I would modify your jar task to include multiple filesets; one for the classes and one for the source files.
<jar destfile="${target.dir}/my-app.jar">
<fileset dir="${target.dir}/classes" />
<fileset dir="${src.dir}" includes="**/*.java"/>
</jar>
Packaging should be treated as a separate concern from compiling. This will give you more flexibility. For example, you may want to add other filesets to the jar (e.g. properties files), or you may want to package your sources in a jar file that is separate from your class files.