This is controlled by the -F (--quit-if-one-screen) option to less.
Git uses the options FRSX for/of less by default, if none are specified by the $LESS or $GIT_PAGER environment variables. To change it, specify the core.pager option and set it to RSX:
git config --global core.pager 'less -+F'
Older versions of Git used to recommend the following in their documentation:
git config --global core.pager 'less -+$LESS -RSX'