Brief summary of the issue:
Some more details:
export OMP_NUM_THREADS=8 ; mpirun -np 2 ./xhpcg_avx2
xhpcg_avx2: src/CheckProblem.cpp:123: void CheckProblem(const SparseMatrix&, Vector*, Vector*, Vector*): Assertion `*currentIndexPointerG++ == curcol' failed
Please, feel free to ask any piece of information you think might be helpful, I'll do my best to provide it.
Thank you in advance for help,
Thank you for your answer.
No, the only version of the Intel MKL I have access to is the one I specified (namely the 2018.3.222).
I am not sure whether or not I could have access to the newest version of the Intel MKL.
Once again, thank you for your answer. And in advance, thanks for the time you will dedicate to help me.
When looking at the requirements on the page you linked, I saw that the Intel MKL library had been tested alongside OpenMPI 1.8.x.
If I am not mistaken, several versions of OpenMPI (not sure which one though) are installed and available on the clusters I use. If I try to compile/link with other versions of OpenMPI, would it make it easier for you to investigate the issue?
yes, i see the same problem with version 2019 also. We will check what's going wrong with this specific case. meantime running the prebuilt avx2 binaries didn't show some problems;
]$ mpirun -np 2 ./xhpcg_avx2
HPCG result is VALID with a GFLOP/s rating of 29.870816
Can you add -DNDEBUG to your CXXFLAG variable and build your binary again? And why do you use gcc instead of Intel compiler to build HPCG, if it is not a secret?
First, thank you for your answer.
I tried building with -DNDEBUG before opening this thread. As far as I remember, same error and no more information output.
I will have access to the test cluster in approximately 16 hours, I will test it again and update this post accordingly.
This is not a secret at all. I am working on a French research-oriented cluster (Grid5000, which Chameleon in the US is "inspired of"). Intel Cluster Studio is not available on this plateform so I cannot compile with the Intel compilation suite.
I tried to download the free version for academic/research purposes, but my institutional e-mail address seems not to be recognised.
And, by the way, any progress on the issue?
We know about this issue and currently we are investigating it. Actually this issue is not correlate to benchmark at all it is more about check the matrix for optimized kernels. I checked mentioned workaround on my side and it's work correctly. Can you double check it on your side? By the way, we publish pre-build exactable files of HPCG with MKL release so why do you need to rebuild them?
Thank you for your answer, you were right. I double-checked the "-DNDEBUG" workaround, and it works like a charm.
The first time I tried it, I must have made a typo (-NDEBUG or -DDEBUG).
Concerning the pre-built binaries of HPCG, I tried to use the version specific to machines with AVX2 support but I could not make it work. Even if I sourced "compilvars.sh" and "mklvars.sh", 3 dynamic libraries related to MPI were still not found by the executable. And the distribution of OpenMPI installed on the nodes seemed not to supply an equivalent of those libraries I could have symlinked to ... I thought it was compiled/linked using Intel MPI, and since I do not have access to it, I tried to compile HPCG from the sources supplied by the MKL. It is not unlikely at all that I missed something.
If you have any clue on how to make the pre-built version work, I would appreciate it (I would rather use it than a custom-compiled version of HPCG).
If not, one more huge "thank you" for the workaround!
First of all, I am very sorry for the huge latency associated with this answer ... I have been quite busy lately.
The issue concerning the missing libraries stemmed from the fact that the Intel MPI libraries were not installed on the cluster I was using (only the Intel MKL libraries were).
After installing the Intel MPI libraries, I was able to use the pre-built version of HPCG.
Thanks again to all the people who helped me through this thread,