You can make vim exit with a non-zero exit code to cancel the rebase by typing :cq. The result of this is:
$ git rebase -i HEAD~4
# enter vim, then type :cq<enter>
Could not execute editor
From the vim help page we see that cq is short for cquit:
*:cq* *:cquit*
:cq[uit][!] Quit Vim with an error code, so that the compiler
will not compile the same file again.
WARNING: All changes in files are lost! Also when the
[!] is not used. It works like ":qall!" |:qall|,
except that Vim returns a non-zero exit code.