If Books has a ForeignKey to Sections, then Django will automatically create a reverse relationship from Sections back to Books, which will be called books_set. This is a Manager, which means you can use .filter(), .get() and .count() on it – and you can use these in your template.
{{ sec.books_set.count }}
(By the way, you should use singular nouns for your model names, not plurals – Book instead of Books. An instance of that model holds information for one book, not many.)