Best way to build large applications in parallel

[Henry_G_Intel]

Hello everyone,

At the HPC site where I worked before joining Intel, I used a program called pmake to build applications in parallel. Pmake worked on distributed-memory parallel systems. One of the proprietary UNIX systems also had a regularmake program with a -P option for parallel builds.

What is the best way to build large applications in parallel on a Linux cluster? A Google search on 'pmake' turned up a couple of candidates: Parallel Make and Berkeley Pmake. Does anyone have other recommendations?

Thanks,

Henry

[ClayB]

Henry -

The GNU make utility allows for parallel compilation with the "-j" flag.

gmake -j 2 MyApp will spawn off two threads to divide up the building of multiple files and libraries. Unfortunately, this only works within a single node, not across a cluster.

--clay

Message Edited by ClayB on 08-24-2004 10:55 AM

[jolsson]

I'm also interested in this, but on Windows. We are using the cons

build system, which has a pcons flavor http://www.baldmt.com/parallel-cons/.

We are not using pcons yet, and also might need a better cluster file system

than just a share on windows.

There is also the newer http://www.scons.org/build system that supports parallel build on Linux and Windows.

I think there has been cluster versions of either cons or scons, but I cannot

find them now.

cons/scons have the advantage over make that dependencies are automatically generated, including what libs to build before what

execiutables and what source code needs to be generated before

compilation starts. It also handles changes using MD5 checksums

which means that things will rebuild properly even if time stamp

of file doesn't change or goes backwards (like if you undo a version

control checkout).

Jonas

[Intel_C_Intel]

I believe you are looking for:

http://distcc.samba.org/

It is very easy to setup and use.

RPMS for RHEL or CentOS can be found at:

http://dag.wieers.com/packages/distcc/