Porting Itanium OpenVMS FORTRAN to Intel x86 FORTRAN - Memory overruns

[JP_SECTOR7_COM]

More in the porting to Intel x86 Fortran series.

S1.FOR
SUBROUTINE DATAOVERRUN(INT4)
INTEGER *4 INT4
INT4 = 324
END SUBROUTINE

P1.FOR
PROGRAM TESTME
INTEGER *2 INT2
CALL DATAOVERRUN(INT2)
END PROGRAM


This will cause memory overruns on x86 with data corruption, but not on Itanium (INT2's "effectively" promoted to INT8 on the stack).

Due to natural word alignment the OpenVMS titanium compiler will align all integer (and real & records) quantities (INT1,INT2,INT4) on 8 byte boundaries. x86 will align INT2 on a 2 byte boundary.

So - this code will run without data corruption on Itanium but will need remediating on x86 (I did not see a compiler option to pad the stack)

Same can happen if the I2 is in a record and:

(1) the record is in PSECT, 8192 page alignment will provide plenty of slack on itanium
(2) the record is on the stack and once again, 8 byte titanium alignment provides a buffer.

Jon Power

Sector7 USA

http://www.sector7.com

[Steve_Lionel]

Sorry, but this is simply invalid code. That it happened to work on one platform doesn't make it right.

The Intel compiler has an option that will help you detect such errors, /warn:interface. For example:

D:\Projects>ifort /c /warn:interface s1.for
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.1 Build 20201112_000000
Copyright (C) 1985-2020 Intel Corporation.  All rights reserved.


D:\Projects>ifort /warn:interface p1.for s1.obj
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.1 Build 20201112_000000
Copyright (C) 1985-2020 Intel Corporation.  All rights reserved.

p1.for(3): warning #6075: The data type of the actual argument does not match the definition.   [INT2]
      CALL DATAOVERRUN(INT2)
-----------------------^

[JP_SECTOR7_COM]

I have to agree with you. However, when you have 1 or 2 million lines of code to,port, you could be looking at 20k or 30k instances. The inevitable question always arises “well, it worked on VMS”.

I highlighted the issue for people wanted to port their code to ifort, 

i hoe I’ve answered the “why” question which is not always obvious.

the most expensive bugs  are the ones found at runtime.

FPT (FORTRAN partner) can detect and fix these systemic global issues and automatically fix the code BEFORE expending huge amounts of  time tracking down data overruns.

In migrating large systems to significantly different platforms we live by 2 maxims:” an ounce of prevention is better than a pound of cure” and “measure twice, cut once”.

When we migrate FORTRAN applications from VMS or HP9000 making manual code changes is out of the question, 1/ human make mistakes 2/ asking the client to freeze their code while 3m LOC is hand remediated is a non starter 

[mecej4]

Jon Power:

Given such Fortran source code, you had better keep an Itanium-VMS-OneTrueCompiler system running for ever, or have a reliable emulator on an X64 host. Otherwise, since the code was written with disdain for the safety belt of the Fortran standard, how will you ever know what output the Itanium gives for every possible input data? 

Even with standard-conforming source code, we know that often the output is "processor-dependent". That is already hard enough to cope with.

If the body of code is large, you may need to design a streamlined procedure to 1) produce a standard-conforming version on X64, 2) run the original and the ported code on the Itanium, 3) Repeat 1) and 2) on the Itanium until the two versions produce identical results for a wide range of inputs, and 4) run the ported code on X64 and make modification until the old code on Itanium, and the ported code on Itanium and X64 all produce the same results.

It may be more efficient to rewrite some subprograms from scratch.

[JP_SECTOR7_COM]

We typically port 1M to 3M LOC of old fortran 77 - rewriting any code by hand is simply not an option.

FPT (Fortran Partner) reads all of the FORTRAN source and creates databases of all the functions and their arguments and down to which arguments are READ ONLY and which also WRITE (all the way don the call tree).

FPT is an expert system with build in knowledge of 12 fortran dialects (mainly ancient machines) and 6 or 7 modern "to" dialects [favorite being ifort].

In the case of data overruns (I*4 overwriting I*2) FPT creates a temporary I*4, passes to the offending function and depending on the functions use of the arguments (read or write - down the the past level of functions calling functions)  it will then assign the temporary I*4 to the original I*2.

 This automatically handles hundreds of cases that would otherwise corrupt data.

Also on the "most wanted list" is IOSTAT = I*2 or even L*1 (detected and handled by the same mechanism).

FPT can convert millions of lines of OpenVMS 'lazy' fortran to x86 intel fortran in a few minutes.

The goal is ALWAYS to have 100% conversion - this allowing the corporation to continue to develop their application on OpenVMS and have FPT handle the syntax and semantic conversion - it is this 100% automated conversion that allows our Continuous Integration Methodology - i.e - no code freeze - cant expect a client to freeze development int he year it can take to convert and test the fortran and also provide all the VMS API's SYS/LIB/STR/OTS/FDV/FORMS$ etc etc.

We thought we'd seen and handled everything, but, with 30yr old code, there are always surprises that FPT handles (or can be made to handle).

(1) STRUCTURE with INITIALIZATION - which OpenVMS allows to be used in a record in a common block but IFORT pukes. FPT clones the STRICTURE and automatically created BLOCK DATA files for the common block initialization.

(2) VMS API's - being called as subroutines STR$TRIM and also as FUNCTIONS. VMS is forgiving - ifort - not so much - FPT has knowledge of all VMS API's and ads temp = XXX$YYY where needed and also adds the declaration for the API. [normally THOUSANDS of these]

(3) VMS allows multiple initialized data segments as long as the initialized data is the same - i.e initialized COMMON blocks being declares twice or more (in an include file). Linux allows 1 initialized segment and all the otters much be uninitialized overlays. FPT collects all the common blocks into a single place and validates the initializations are the same.

(4) FPT is also clever enough to know that the target machines and fortran compiler is iFORT and does not try to make temporaries for I*4 being passed to I*2 (little endian dependent).

(5) PRESENT - FPT inserts %VAL(0) or ,,,,, to pad the caller to the correct number of arguments (as VMS passes the number of arguments on the stack and linux has no concept) this is vital.

(6) IARGCOUNT - due to FPT knowing all of the functions / subroutines declarations through out the entire system and all the source files -and as shown above pads the calling arguments - we can also implement an IARGCOUNT  (effectively using the PRESENT functions by checking for %VAL(0) as the address).

(7) FPT "knows" about all of the FORTRAN language syntax and semantics of 7 FORTRAN dialects and 5 or 6 target FORTRAN compliers - and as such can make all the changes required for 100% remediation with 0% human intervention. The truly excellent part is that IF we come across an bug (like the X8000 compiler bug) we just tell FPT to change I*2 negative PARAMETER to the decimal equivalent - naturally assuming that I*4 X80000000 has the same issue passing to an I*8.

(8) to get 1M lines of old f77 VMS FORTRAN to compile CORRECTLY on intel x86 ifort - FPT will typically make 60,000 remediations .. some just "good practice like SAVE - other vital for the correct operation on x86 (I*2 to I*4)

(9) FPT also converts all FORTRAN I/O to API thus enabling  our linux version of RMS to fill the oenVMS RMS usage (indexed, relative, sequential, RABSs, FAB's XAB;s). [hence my first post about intercepting the FORTRAN I/O - which would have been nice but we get by].

(10) as an example - not that we would every want a client to port to gfortran but - on gfortran a logical is considered true if NON ZERO - obviously ifort tests bit 0. FPT can also make all of the logic changes to handle source syntax needing bit 0 T/F to zero/non zero.

 (11) its been used to convert hundreds of millions of lines of fortran running on ancient slow machines to modern fortran on x86 and others. very popular with the military and weather forecasters.

Anyway - I had in mind to post "differences" and work arounds for users wanting to port OpenVMS fortran to linux x86 intel fortran i.e using mmap() to reproduce $CRMPSC and $MGBLSC functionality / PSECST / $CDEC OpenVMS attributes or whatever OpenVMS FORTRAN to ifort on linux porting challenges we find. 

There doesn't seem to be forum for sharing this knowledge.

ifort (as you know) is very compatible with VMS - but - certainly has enough differences to make a large port "challenging" without something like FPT.

the key to a large operating system port > 1M LOC + operating system API's - is have an environment that means NO code freeze during the port and the ability to convert the source FORTRAN to x86 ansi fortran [intel] is a few minutes.

FPT allows us to take a code drop (1MLOC OpenVMS FORTRAN), 100% automatically remediate for intel fortran, compile, link and deploy  executables in about 12 minutes [4 processor 16GB VM] - end to end.

TO successfully port a large (pre template) fortran application the tool MUST have visibility to ALL of the code and a full understanding.

FTP is a compiler that emits FORTRAN [and oddly FPT is also written in FORTRAN].

it can also generate makefiles automatically [as it knows what subs/functions are called and where they live].

[andrew_4619]

That seems like an very impressive and extensively featured means of applying sticky plasters to buggy non-conforming old code. I would have through putting more emphasis on automatically producing code that was more standard conforming and using that as the base for future work would have been more worthwhile than making bodged versions of old buggy code. Hey ho each to there own. Happy Christmas.

[JP_SECTOR7_COM]

HI Andrew, thank you for your excellent thoughts.

FPT can also rewrite the fortran into any specific "standard". Add modules / templates etc.

The issue is that clients want the minimum number of changes done to their code as possible. 

Obviously when we have finished porting their application to Linux or wherever - they have to maintain the code and their developers are used to their F77 style.

We almost NEVER get asked, to update their fortran to 90 or 2018 standards - yet - its a single button click  (one that they can undo if they hate the new format) and one that they'e ALREADY PAID  for with the FTP license !!!. 

Rarely do we even see F90.

I live my professional life in the 1980's - life was simple then .. fixed format F77 where you can pretty much do anything to anyone - the wild west of computing - none of these modern ideas of having the compiler actually see if you're calling a subroutine with the correct arguments - or even ANY arguments.

Hi Ho Silver .. Away !

/Jon

[mecej4]

I thank you for the detailed explanation, and now I appreciate better the formidable nature of your task, and the need for a complex set of tools and procedures for performing that task.

In the late 1990-s and early 2000-s, I did use a few Itanium servers for some scientific work. Some were Linux servers run by my organization, and others were HP servers provided as a service to users, some running HPUX, some VMS, and others running Linux. 

You may find it interesting to read one thread from two years ago on Usenet comp.lang.fortran. The topic was the venerable Eispack package, which had been regarded as having no known bugs for  almost 50 years. Most of the posts in that thread were about an aliasing error. This error occurs in the code of the test problems, rather than in the Eispack routines themselves. While running some of those tests, I discovered another bug, again in the testing code. An array contains all equal real numbers in one case. Their mean is computed, and the test code expects that the mean can never exceed all the elements of the array. The following example shows that it can happen. I do not know if the bug does occur on a VAX.

program tst
implicit none
double precision x(10), y
integer :: i, n = 10, iy(2), ix(2)
equivalence (iy,y),(ix,x)
!
y=0.0d0
do i=1,n
   x(i) = sqrt(0.5d0)
   y = y+abs(x(i))
end do
y = y/n
print 10,ix(2),ix(1),'item-1',iy(2),iy(1),'mean'
10 format(1x,2Z8.8,2x,A)
end program

In the Eispack test code, a certain DO loop may run to completion, against expectations. The loop index variable I will then have the value n+1, which is used as an array index for a variable with dimension (n).

I am curious to know how your software porting system will handle this case, but it is certainly a toy example. Nevertheless, you can still see the bug in http://netlib.org/eispack/ex/cgtest.f (and other *test*.f files in the same directory), and Cleve Moler's highly readable article on Eispack ends with "As far as I know, Burt never received any serious bug reports or complaints."