Also, is there some substitute for the lack of support for MPI-2 process spawning? We produce simulation libs which clients (including GUI-enabled) link against and we need to implement portable distributed processing, so launching via mpiexec isn't a viable option - we need the (MPI-enabled) client app to be able to launch worker MPI processes and communicate with/manage those.
I don't see any likelihood of MPI jobs on Windows and linux communicating while running, but it might be interesting, if you would explain.
You could submit feature requests about additional MPI-2 functionality on your premier.intel.com account.
My recollection from (several years ago) was that this was possible with MPICH to some extent. We were able to run a single large job across a heterogeneous collection of systems, but it required a seperate compile for each platform and a configuration file that specified the correct binary for each node. We succeeded in getting demonstration codes and a couple simple problems distributed across the whole collection. However, we were really trying to leverage our Unix platforms and the data file incompatibilities killed the project.