Different behavior between ifort 19 and previous versions.

[Matthew_H_6]

I have a large code (10k+lines) that I am trying to debug. When I run with v19 I have no issues. When I switch to v18 of the compiler I see different behavior. I tried to build a small test case to aid in the debugging but I see identical results between the compilers for the small test case. I am wondering if anyone has any tips or tricks on how to track down errors like this.

The same compilation flags are used for both compilers, and I tested version v17, it behaved the sameway as v18.

[andrew_4619]

It might help if you have some specific example of a problem you are experiencing. It is hard to make any helpful suggestions from what you have given. Yes I have experience both both versions and no I do not know of any fundamental differences.

[Matthew_H_6]

I am getting different numerical results. With v19 the code stable, with v18 and prior the code will run for short time but will eventually die. The residual output is different between the two version. It should be nearly identical since the case setup is identical. I have not been able yet to track down why the numerical results are different and which aspect of the code is responsible.

[andrew_4619]

A useful first test if it is numerical differences is to switch of optimisations and see if that changes anything. There are a whole load of options that trade accuracy versus speed. Are you comparing like with like?

[Matthew_H_6]

The same Makefile is used, so the compiler options are identical. I am thinking about generating optimization reports for each version and comparing them to see if differences pop out.

[Ron_Green]

It is not reasonable to expect optimized code to produce identical results between compiler updates, OS updates, or OS changes, or processor changes. I will upload a presentation for you to read, try the options therein, and it has reference articles for more information on conditional numeric reproducibility.

see attached presentation for assistance.

[Matthew_H_6]

Thanks, I will take a look. I am not expecting 100% exact answers between versions, but having one version diverge and generate garbage and the other produce correct result is unexpected. I have never experienced this before and have been using ifort for 10+ years. The only reason I noticed the current issue is because I had to run on a system that only had v18, and started to see problems. I went back to a system with v18 and v19 and confirmed that v19 runs correctly and v18 does not.

[mecej4]

If constructing a small test case causes the issue to cease to exist, one may have to start at the other end, i.e., with the full code, and pare away parts of the code that are either not executed or are not involved in causing the error. This can involve considerable effort, especially in the beginning, and it is helpful to have tools available to measure and document code coverage, effects of changing compiler options, etc.

You have the option to provide the large code, along with any data needed, and instructions to build and run the program to demonstrate the error. You have to contend with the fact that the larger the code, the fewer the count will be of people willing to look into the problem.

[Steve_Lionel]

Here's a 2013 presentation I made on this topic: 

http://sc13.supercomputing.org/sites/default/files/WorkshopsArchive/pdfs/wp129s1.pdf

[JohnNichols]

Not found, error 404

The page you are looking for no longer exists. Perhaps you can return back to the site's homepage and see if you can find what you are looking for. Or, you can try finding it by using the search form below.

The org site throws a bad site error. 

Any chance you could put up the paper I would love to read it. 

[AlHill]

Easily enough found:

https://web.archive.org/web/20150912135546/https://sc13.supercomputing.org/sites/default/files/WorkshopsArchive/pdfs/wp129s1.pdf

Doc

[JohnNichols]

Thanks

[Steve_Lionel]

Gah - I put in https but the site works only with http. Link edited.

[Ron_Green]

unfortunately there is no simple "do this to fix it" for numerical difference issues. Did you try the compiler options in Steve's presentation?

Also, what happens at -O0 between the different compilers? Is this an optimization related difference?

[Matthew_H_6]

I think it may be an optimization issue/difference between the versions. I am attempting to isolate the routine with the difference in the full simulation. It runs on 1000's of cores, so it is a trial and error hunt with write statements. If I uncover anything useful I will post a follow-up message.

[jimdempseyatthecove]

>>I have a large code (10k+lines)

I will assume your code is in multiple source files.

What I suggest to do (instructions in MS VS-speak, you can do the same in Eclipse, make, etc...)

In the solution explorer, Release Build, expand the source folder, multi-select half the files,  right-click on selection, properties, then set the optimization level to none, or /O1 or whatever works).

Build, Run and if you crash or get trash data, the problem is within the unselected files. In this case, select half of the remaining files, set optimization level to non... build and try again.

When you reach a point where you don't crash, then start working to halve the list of un-optimized sources and re-enable optimizations.

What the above procedure is doing is performing a binary search, looking for the source file (or files) having issues with optimization.

Once you've identified the affected file, you can then strategize how to isolate the problem further.

Note, it might be helpful to search this forum for ifort 19 vectorization problems.

Also, should the affected source file(s) when compiled without optimizations, combined with the other files compiled with optimizations, cause little runtime performance issues, Then leave your project with using different optimization levels for different files, and move on.

Jim Dempsey