OK, I found it.
in .gitconfig:
[difftool]
# Be able to abort all diffs with `:cq` or `:cquit`
trustExitCode = true
Then inside vim just enter :cq
or :cquit
. This will exit vim with error codes and git has been told to rely on the error codes with trustExitCode
.