git and vimdiff – close all files at once

OK, I found it.

in .gitconfig:

[difftool]
    # Be able to abort all diffs with `:cq` or `:cquit`
    trustExitCode = true

Then inside vim just enter :cq or :cquit. This will exit vim with error codes and git has been told to rely on the error codes with trustExitCode.