You can do this by following the backwards relation in the lookup.
>>> qs = Department.objects.filter(
... departmentvolunteer__isnull=True).values_list('name', flat=True)
>>> print(qs.query)
SELECT "app_department"."name"
FROM "app_department" LEFT OUTER JOIN "app_departmentvolunteer"
ON ( "app_department"."id" = "app_departmentvolunteer"."department_id" )
WHERE "app_departmentvolunteer"."id" IS NULL
Here are the docs on queries “Spanning multi-valued relationships”: https://docs.djangoproject.com/en/stable/topics/db/queries/#spanning-multi-valued-relationships