Git never gets hold of the GnuPG passphrase. You must rely on GnuPG’s capabilities of caching passphrases, which happens through gpg-agent which are easily set up by editing ~/.gnupg/gpg-agent.conf (hidden somewhere in your AppData folder in Windows).
Set default-cache-ttl to the number of seconds the passphrase is cached after each invocation of GnuPG. maximum-cache-ttl sets the time after the passphrase was initially entered at which the cache is wiped. Make sure ignore-cache-for-signing is not set — otherwise GnuPG will ignore the cache for signing operations.
If you want to sign commits without any user interaction, you can prefill the cache through gpg-preset-passphrase, often hidden somewhere in a location like /usr/lib/gnupg2/gpg-preset-passphrase; or by running an arbitrary decryption or signing operation. You might also configure git to use an option like --passphrase [your passphrase] to be passed to gpg, but read up on the restrictions and security implications of this approach (it involves your passphrase being stored in plaintext somewhere).
Full list of options is here.