The version of the icc compiler differs from node to node. I see at least 3 different versions; there may be more:
on n078: 19.1 build date 20200117
on n142: 18.104.22.16891031
on login-2: 22.214.171.124
This makes it very difficult to get consistent results across the DevCloud. Results would depend on where the code is compiled, which adds an unnecessary complication to benchmarking.
Can you install the same icc compiler across all nodes on the DevCloud?
The icc(by default) in devcloud compute nodes is the one that is made available as part of oneAPI toolkit.The difference in versions of icc is because of the ongoing OneAPI toolkit upgrade on devcloud. The oneAPI toolkit present in devcloud is being upgraded from beta03 to beta04.
beta03 had icc (ICC) 19.1 20191031
beta04 has icc (ICC) 19.1 20200117
This results in 2 different versions of icc compiler across different compute nodes. The login-node does not have oneAPI toolkit installed so will use the icc present in /glob directory.
We will contact the concerned team regarding the conflicting versions.
Meanwhile a quick work around I would suggest for the time being would be to append the path of icc(126.96.36.199) present in /glob directory to your path variable. This can be done by adding the below line to your job script.
As the icc present in /glob is same for all nodes your issue would be resolved
Thanks for the confirmation, the teams are working on completing the installations. Meanwhile as you are able to continue your benchmarking with the workaround provided could we close this case.