By default, <javac> runs in-process with Ant. It is a general limitation of Java that you can’t adjust a JVM process’ Xms and Xmx once that JVM process has launched. So, the error message that you are seeing is the software rejecting your attempt to violate this principle (using an unhelpful, unfriendly error message.)
If, however, you specify the attribute fork="true" on the <javac> tag you will be able to specify a new Xms and Xms. This is because fork instructs Ant to launch a new JVM subprocess in which to run javac. Because the JVM process is new, it gives Ant an acceptable opportunity to specify Xms and Xmx for it.
You might try something like this:
<project name="project" default="all" basedir="[yourvalue]">
<target name="all">
<javac srcdir="[yourvalue]" destdir="[yourvalue]" fork="true">
<!-- javac requires that -Xmx and -Xms be prefixed with -J -->
<compilerarg line="-J-Xms128m -J-Xmx512m" />
</javac>
</target>
</project>
(Notice I am using compilerarg line="" rather than compilerarg value="". The line attribute lets you specify multiple space-separated arguments. The value attribute is for passing a single argument.)
Ant will wait for the forked <javac> to exit, which happens after the javac process finishes its work (i.e. compiling). Ant then continues running the build script inside its own original JVM process. Ant will check if the forked javac failed or succeeded, and take the usual actions based on this information.
Performance
It’s usually more performant to not fork javac, and instead simply tune the relevant memory settings for the initial Ant JVM overall. This is often (but not always) the best choice because launching a separate JVM is usually slower and takes more memory than simply allowing javac to run in-process.
If you are using the Ant-provided ant.bat or ant.sh to launch Ant, an easy way to tune Ant’s Xms and Xmx is to define the environment variable ANT_OPTS to contain the arguments you want. There many ways to set environment variables, but you could just edit ant.bat:
set ANT_OPTS=-Xms128m -Xmx512m