Django 1.7 added Prefetch
objects which let you customise the queryset used when prefetching.
In particular, see only()
.
In this case, you’d want something like:
queryset = Contact.objects.all().prefetch_related(
Prefetch('Groups', queryset=Group.objects.all().only('id')))