Try running the signing process as a separate process, and providing custom memory settings for this process.
You can use the <jvmarg> inner tag for the <java> tag to set -Xms and -Xmx attributes.
-Xms takes starting JVM memory
-Xmx takes maximum JVM memory allowed.
See the following link for how to use these tags
http://ant.apache.org/manual/index.html
- Rakesh