<p>The default 64 MB max heap memory can be small for a large Java application. Are there any applet parameter to increment this for a signed applet?</p>

<p>For a Java program this is a simple command line parameter but how this work for an applet in the browser.</p>

Not that I know for certain, it's been a long time since I wrote applets, but I don't think you can set this from the applet.
Apparently, you can set the JVM's heap size for the browser's JVM from the Java plug-in control panel, but that's something the user has to do before starting your applet.
You can always check
http://forums.sun.com/thread.jspa?threadID=523105&messageID=3033288
for more discussion on the topic.