Play does not appear to pick up the jvm.memory settings for dependencies or even test command. One way to force it to use specific JVM settings would be to use _JAVA_OPTIONS.
For example:
export _JAVA_OPTIONS="-Xms800m -Xmx1500m -XX:PermSize=64m -XX:MaxPermSize=256m"
play test
or
play deps
and you should see
~ _ _
~ _ __ | | __ _ _ _| |
~ | '_ \| |/ _' | || |_|
~ | __/|_|\____|\__ (_)
~ |_| |__/
~
~ play! 1.2.3, http://www.playframework.org
~ framework ID is test
~
~ Running in test mode
~ Ctrl+C to stop
~
Picked up _JAVA_OPTIONS: -Xms800m -Xmx1500m -XX:PermSize=64m -XX:MaxPermSize=256m
Listening for transport dt_socket at address: 8000
Note that this would apply those settings to all java programs run on that terminal where _JAVA_OPTIONS is set.