If you want to retrieve Book instances and prefetch the related tags use Book.objects.prefetch_related('tags'). No need to use the reverse relation here.
You can also have a look at the related tests in the Django source code.
Also the Django documentation states that prefetch_related() is supposed to work with GenericForeignKey and GenericRelation:
prefetch_related, on the other hand, does a separate lookup for each relationship, and does the ‘joining’ in Python. This allows it to prefetch many-to-many and many-to-one objects, which cannot be done using select_related, in addition to the foreign key and one-to-one relationships that are supported by select_related. It also supports prefetching ofGenericRelationandGenericForeignKey.
UPDATE: To prefetch the content_object for a TaggedItem you can use TaggedItem.objects.all().prefetch_related('content_object'), if you want to limit the result to only tagged Book objects you could additionally filter for the ContentType (not sure if prefetch_related works with the related_query_name). If you also want to get the Author together with the book you need to use select_related() not prefetch_related() as this is a ForeignKey relationship, you can combine this in a custom prefetch_related() query:
from django.contrib.contenttypes.models import ContentType
from django.db.models import Prefetch
book_ct = ContentType.objects.get_for_model(Book)
TaggedItem.objects.filter(content_type=book_ct).prefetch_related(
Prefetch(
'content_object',
queryset=Book.objects.all().select_related('author')
)
)