Automatic bug fixing for intel fortran?

[awa5114]

I have just read this wikipedia article (https://en.wikipedia.org/wiki/Automatic_bug_fixing) about automatic bug fixing in software engineering. It piqued my interest, and so I would like to investigate similar possibilities for Intel Fortran. In our company we have a test suite which compares test file output to existing reference files. When tests fail it is usually required for the testing engineer to go back and revert changes until they find the sub-change which cause a particular test to fail. A decision is then made with respect to how the sub-change needs to be modified to accomodate the test.

As you can imagine this is often a tedious and sometimes impossible process, especially when the testing engineer knows little about the nature of the changes made.

How can I get started on automatic patch generation/automatic bug fixing/fault localization in Intel Fortran? Do any such tools exist? Also our 'bugs' are not Fortran syntax bugs in the true sense of the term, they are really cases where a change (which otherwise allows the code to continue functioning smoothly) causes a deviation in numerical results for a particular test.

[Steve_Lionel]

I have seen numerous tools that leverage change tracking to identify which changed triggered a test failure, but I have also seen these tools fail spectacularly at identifying the actual cause. In one famous case, while I was an Intel, an edit to the copyright year in a header comment was accused of causing a 15% performance drop in a benchmark. Sometimes the real culprit causes some other innocuous change to trigger a fault.

Such tools are useful as a first step in debugging, but I think that calling them "automatic bug fixing" is simply incorrect (or at best, wishful thinking.)

I'll note that you don't need a tool to necessarily be language-specific. All you're doing is looking at what edits were made and what minimal combination of edits triggers the fault.

[mecej4]

The testing procedure described is rather simple and naive.

Consider, for instance, a case where the "reference file" contains erroneous information, but nobody knows of the existence of the errors. The automated "bug-fixer" then serves to make the bug permanent.

Another case that comes to mind is a compiler defect (not in an Intel compiler), which had the effect of decrementing the stack pointer ESP by 4 bytes when a particular RTL routine was called. This bug had gone undetected for over a decade because few programs called that RTL routine more than a dozen times.

Many program bugs are of this nature; that is, the bug is something that affects the running of the program but not the results that you put into your reference file.

As Steve already stated, there is no need to even mention Fortran in this context, let alone Intel Fortran. In fact, if your tool detects a deviation, that could be because the wrong linker or wrong library version was used in building the EXE whose output files you then compare to a reference file.

[jimdempseyatthecove]

There are two techniques which I use and find helpful. Both require that you keep a before copy, complete with data that you can use as reference for both source code and results data as produced by your reference compiler and reference system to produce the results data.

Prior to making any changes:

a) use the newer compiler (if applicable), desired options (if different from original, e.g. FPU vs SSE/AVX), and new system, to produce a new set of reference data (the as-was source on newer compiler, newer system, newer architecture).

b) run difference check on the results data to determine if differences, if any, are significant, if so, determine why, and fix or accept results as new reference data.

Prior to making changed, I strongly suggest buying an excellent difference program. My preference is Beyond Compare. The time it saves you is small change to the minor cost. This program is much better than WinDiff.

Use the method of adding new sequenced base folder names. E.g.

widgit
widgit_20180505
widgit_20180506
...

I use dates, you can use sequence numbers. use leading zeroes and/or yyyymmdd, yyyymmdd.001 format such that the directory names sort properly)

method a) Make periodic backup copies of complete base solution (and project) folders. Make periodic test runs and difference compares of results data.

method b) Same operation as a) however use the conditional compile feature of the compiler and either number the conditional expansion sections such that the same (current) edit sources can be used to produce all versions (or all major feature changes). While the source code looks cluttered, I find it faster to perform verification tests between various revisions (simply change a -Dxxx). You can later strip the un-expanded sections from your final gold code copy of source code.

I agree with Steve and mecej4 that "auto bug fix" just doesn't work. Maybe AI can fix this some time in the next 100 years. And (per mecej4), do not assume that the old reference data is correct. Any differences requires you examine the code to determine the cause of the difference and if the difference is significant.

Jim Dempsey

[Steve_Lionel]

The OP probably mentioned Intel Fortran specifically because the Wikipedia article lists a number of language-specific tools. My opinion is that these make great graduate school projects, but are unlikely to be useful in the real world, no matter how attractive the concept is.