As Tim mentions, you're actually running the application over InfiniBand* in both instances. A few releases ago, Intel MPI Library had changed the defaults to use the fastest available network on the cluster at startup (which would be InfiniBand, in your case).
Below, it seems like you specify the RDMA device in your IB run, but don't specify a device in your GigE run (which would default to IB again):
Since the default is the RDSSM device, your two runs are identical. If you'd like to run over GigE, you need to specify the
"sock" device. Alternatively, you can use
"ssm" for GigE communication between the nodes, and shared memory within a node. That means your command line would look like this:
# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE ssm /home/c/OMB-3.1.1/osu_mbw_mrI hope this helps.