syrk mkl armadillo wrong output

[Ramakrishnan_K_]

I have written the following simple program for syrk using armadillo (arma.sourceforge.net).

Environment : Rhea from OLCF. https://www.olcf.ornl.gov/computing-resources/rhea/

MKL : Tried with version 16 and 17. Problem occurs in both.

#define ARMA_DONT_USE_WRAPPER
#define ARMA_USE_BLAS

#include <iostream>
#include <armadillo>
using namespace std;
using namespace arma;
int main() {
  int m = 10000;
  int n = 50; 
  fmat A;
  A.load("H_init.csv");
  cout << "A::" << A.n_rows << "x" << A.n_cols << endl;
  fmat AtA = arma::zeros<fmat>(n, n); 
  AtA = A.t() * A;
  cout << "AtA " << endl;
  cout << max(max(AtA)) << " " << min(min(AtA)) << " " << norm(AtA, "fro") << endl;
  return 0;
}

I have also attached the H_init.csv along with this email. I compile using the following three procedure.

Compilation 1 (gcc with mkl): Based on the article https://software.intel.com/en-us/articles/a-new-linking-model-single-dynamic-library-mkl_rt-since-intel-mkl-103. I compile as "g++ hth.cpp -o hth -O2 -I ~/armadillo-7.800.1/include/ -fopenmp -lmkl_rt"

Compilation 2 (with icc and mkl): icc hth.cpp -o hth -O2 -I ~/armadillo-7.800.1/include/ -fopenmp -mkl

Compilation 3 (gcc with intel linker recommendation): g++ hth.cpp -o hth -O2 -I ~/armadillo-7.800.1/include/ -fopenmp -lmkl_rt  -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl

Compilation based on method compilation 1 is producing wrong output on HtH. Compilation 2 and 3 works fine. Infact in the case of compilation 3, even the ordering of the libraries appears important. If the ordering is change a bit, it is producing wrong output. 

Output of compile 1: (Wrong)

A::10000x50

AtA 

24044.6 3697.91 350951

Output out of compile 2: (Right)

A::10000x50

AtA 

3436.05 2437.46 126222

Output out of compile 3: (Right)

A::10000x50

AtA 

3436.05 2437.46 126222

Am I making any mistake on this? Can't I link w/ gcc using -lmkl_rt?

[Gennady_F_Intel]

>> Can't I link w/ gcc using -lmkl_rt?

this is valid option to link with MKL.  This looks like some problem with armadillio package.  You may try to call mkl's syrk directly and play with different compiling/linking options. 

[Ramakrishnan_K_]

Hello,

I changed my code to directly call syrk. With the new change, compilation 1, intermittently works fine and compilation 3 is always right. Attached is the source code for reference.

Incorrect Output and compile 1:g++ hthexp.cpp -o hthexp -O2 -I ~/armadillo-7.800.1/include/ -fopenmp -lmkl_rt

[seswar@rhea-login1g ~/scratch]$ ./hthexp 

AtA 
6716.41 4864.58

Correct output compile 3 : g++ hthexp.cpp -o hthexp -O2 -I ~/armadillo-7.800.1/include/ -fopenmp -lmkl_rt  -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl

[seswar@rhea-login1g ~/scratch]$ ./hthexp 
AtA 
3381.87 2457.73