The answer depends on the budget you are ready to spend on interconnects. 1. Very cheap solution is 1G ethernet. If your application doesn't use a lot of communications that's your choice. 2. A bit more expensive - 10G ethernet. There are number of producers - you can choose. E.g. NetEffect 10G works fine with Intel MPI - you can use iWARP in this case. 3. Any InfiniBand interconnect - there are a lot of different cards. You can use DAPL provider in this case. 4. With Qlogic PSM and Myrinet MX you can use TMI provider which works even faster than IB.
The list of top500 computers may help - take a look at their configurations.
I think that you understand that increasing interconnect speed in 2 times doesn't mean that your application will run 2 times faster.
I'd recommend to consult with a company where you will buy equipment and also find an experienced person who can configure interconnects.
Mass produced motherboards are available with either InfiniBand or more-than-1Gbe adaptors built in, so the differences in cost are mainly in the switch. Many (but by no means all) applications do run twice as fast on IB as on iWARP, e.g. for clusters of more than 64 cores.