The problem was that I made an assumption that wasn’t true. If you look at section 14.2 of the gradle documentation, it says:
You can place a gradle.properties file in the Gradle user home directory (defined by the “GRADLE_USER_HOME” environment variable, which if not set defaults to USER_HOME/.gradle) or in your project directory.
My incorrect assumption was that USER_HOME just defaulted to the standard linux HOME environment variable. This is not true.
As soon as I export USER_HOME=$HOME in my ~/.bashrc everything works