If the project specifies a QSYS file using "set_global_assignment -name QSYS_FILE" then the quartus build will automatically synthesize the .qsys file. With this mechanism quartus is implicitly calling qsys-generate/ip-generate and I cant explicitly specify the "--jvm-max-heap-size" argument, and therefore the build fails. I did find the project assignment option that sets the MAX_QSYS_JVM_MEMORY variable in the quartuis2.ini file in the login directory. However, this apparently doesn't work for quartus 13.1, and furthermore we prefer for our build scripts to work independently of the login context of a particular user.
Is there any mechanism for specifying the java heap size using the qsf file using a TCL set_global_assignment?
Is the build fail due to OutOfMemoryError? I currently do not know how to set java heap size through the qsf file.
Perhaps this KDB could be an alternate solution for your case.
To answer my own question...
This works within bash or also within the GNU make file environment. On windows the set command can be used in a DOS environment.