Undocumented behavior of fpp with alternative preprocessor

[ivanp]

The Intel Fortran compilers (both ifx and ifort) provide the compile switch -fpp-name to supply an alternative preprocessor instead of the built-in one.

According to the Intel Fortran compiler documentation, the alternative preprocessor needs to support an invocation of the form:

alt_fpp [ [–D<define>]..] [[-I<include directory>]..] inputfile 

In practice I have observed this is not entirely the case (see https://github.com/aradi/fypp/issues/24) and that the alternative preprocessor should be capable of reading the input commands from a response file (a file prefixed with the @ symbol).

Can we rely upon this undocumented behavior long term?

[jimdempseyatthecove]

It may be difficult to use a different preprocessor.

Intel fpp

1) defaults to processing the source, as an example, xyz.f90 to xyz.i90, then compiles the xyz.i90 file as if it were the xyz.f90

2) Furthermore, module files may (will depending on content) generate multiple xxx.i90 file for compilation by ifort/ifx

If you are using MS VS, you may find it productive to use the Project's Configuration Properties | Pre-Build Event to specify a command to pre-process your .f90 (etc...)  and *** output as ....f90.

Have the pre-alt_fpp sources in a folder that is not your build source folder, but rather a pre_source folder. Your alternate fpp command, for example would process

  alt_source\foo.f90 -> source\foo.f90

Then your build would build from the source folder.

IOW you edit the files in the alt_source folder, but build from the source folder (after preprocessing)

*** Note, build your @fileList from files that are newer than the corresponding .obj files (or .exe/.lib file)

Note-2

As you are not using an alt_fpp (because you are using a pre-build event), you could potentially preprocess using your alternat fpp, then supply the option to use ifort's fpp as well. Though it would be a little tricky handling the #'s as to which preprocessor processes what.

I hope I made this clear enough.

Note 3, If you are not using MS VS, then you can do a similar thing using make files.

Jim Dempsey

[ivanp]

Thanks for describing the options. Currently I am using a separate preprocessing step in the spirit of Note 2. This doesn't change the fact the Intel Fortran compiler documentation is incomplete.

The alternative preprocessor I am trying to use - fypp (https://fypp.readthedocs.io/en/stable/) - matches the expected command-line interface and behavior described in the documentation of -fpp-name: "Output from the preprocessor goes to STDOUT and will be captured for any further processing."

In practice however, it doesn't work and fypp exits with the error

error: Failed to open file '@/tmp/ifxargBf8BjW' for read [FyppFatalError]

since it attempts to read '@/tmp/ifxargBf8BjW' as the location of the file to be preprocessed and not a response file for passing the command-line arguments.

Either I interpret the documentation as incomplete, or as your post suggests, avoid the built-in compiler solution and defer my preprocessing step to the IDE or build-system.

---

[Barbara_P_Intel]

Please attach a reproducer and I can investigate the issue.

[jimdempseyatthecove]

>>error: Failed to open file '@/tmp/ifxargBf8BjW' for read [FyppFatalError]

Apparently, ifort/ifx (?IDE) created a temp file and placed it somewhere else (or not).

I suggest you create a same named program (fypp.a?) and place it in path (or CD) before the real fypp.

The content of this file is to print out the  command line args, then pause for user input.

When the program is paused, then search for the temporary file name.

This may provide some insight as to what is going on.

Jim Dempsey

[ivanp]

I've taken up Jim's advice and written a bash script which pauses so the contents of the file can be read. 

A minimal example with a description is in the attached tar file.

*I find it a bit odd the compiler automatically adds include directories for /usr/include twice:

-I/usr/include/
-I/usr/include

[Barbara_P_Intel]

Thank you. I'm investigating.

[jimdempseyatthecove]

>>...pauses so the contents of the file can be read. 

>>>>error: Failed to open file '@/tmp/ifxargBf8BjW' for read [FyppFatalError]

The intention was to figure out why the file could not be found, or opened.

Note, if the case is the file is in the correct location ('@/tmp/ifxargBf8BjW') ...

then the issue is likely that ifort/ifx has written the file...

AND has it open (possibly for reuse)

AND your fypp is attempting to open the file for exclusive use

Therefore, if you have the source to fypp, change the open mode to shared.

Jim Dempsey

[ivanp]

The Fypp preprocessor does not recognize a file name prepended with @ as a response file. Even if it did, whatever the compiler has written into this file (arguments to be passed to the alternative preprocessor) is potentially subject to change and hence can not be relied upon by third-party tools. In other words it's an internal implementation detail of the Intel Fortran compiler.

Either the behaviour of passing arguments through a temporary file prefixed with @ is documented or the -fpp-name=<alt_fpp> flag as it stands currently is useless, misleading, and should be removed from the documentation entirely.

[Barbara_P_Intel · Accepted solution]

I worked with the Fortran compiler team and I submitted a bug report on the documentation. Here's is the new Description for fpp-name:

Description

This option lets you specify an alternate preprocessor to use with Fortran.

The compiler invokes the user-specified Fortran preprocessor by spawning a command with the following signature:

alt_fpp @fpp.arg

where alt_fpp is the name of the Fortran preprocessor you want to use, and fpp.arg is an input file containing the preprocessor's command line arguments. fpp.arg is generated by the Fortran compiler driver, and contains the -D<define>, -I<include directory>, and input file name(s) from the compiler command line.  Compiler predefined -D<define> and -I<include directory> options are also included. Output from the preprocessor goes to STDOUT and will be captured for any further processing.

You can use option Qoption,fpp,… to pass options, other than definitions (-D xxx), include directories (-I xxx), or any other options accepted by the preprocessor.

You can use option Qlocation, fpp,… to specify a directory for supporting tools.

[jimdempseyatthecove]

>>The Fypp preprocessor does not recognize a file name prepended with @ as a response file.

Apparently Fypp does not use response files (see). Nothing mentioned about response files on the command line.

Also see: integrating into build environments

Therefore, you should write a script that you specify to be used as a preprocessor...

which takes the @(generatedFileNameHere) as input, edits it to create a makefile out of the file list within the @(generatedFileNameHere), and then calls make to perform the Fypp-ing of the list of files (and defines, if any enclosed within the @(generatedFileNameHere).

This shouldn't be too hard.

If you want some help on this, please post the contents of the @(generatedFileNameHere).

*** On this forum's compose response window's tool bar, press the ... button, then click on the </> button, then click on the pulldown and select Plain Text. And post the contents in there. This will make it easy for forum readers to select/copy/paste the file.

*** Inserting with normal forum text makes it difficult to copy and paste.

*** Use the same process to insert Fortran, C++, etc file formats (supported by the pull-down). If your desired format is not listed, then use Plain Text.

Jim Dempsey

Jim Dempsey

[ivanp]

Thanks Barbara for the bug report. I hope the option can be put to good use. It can simplify work with build systems considerably.

@jimdempseyatthecove , below I've pasted the contents of the response file. In this particular example only -DHELLO was a user option, everything else is included by the compiler by default for my platform.

-D__INTEL_COMPILER=2021
-D__INTEL_COMPILER_UPDATE=4
-D__unix__
-D__unix
-D__linux__
-D__linux
-D__gnu_linux__
-Dunix
-Dlinux
-D__ELF__
-D__x86_64
-D__x86_64__
-D__amd64
-D__amd64__
-D__INTEL_COMPILER_BUILD_DATE=20210910
-D__INTEL_OFFLOAD
-D__MMX__
-D__SSE__
-D__SSE_MATH__
-D__SSE2__
-D__SSE2_MATH__
-D__pentium4
-D__pentium4__
-D__tune_pentium4__
-DHELLO
-I.
-I/dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/x86_64/intel-oneapi-compilers/2021.4.0-gcc-xrzccgg/compiler/2021.4.0/linux/bin/intel64/../../compiler/include/intel64
-I/dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/x86_64/intel-oneapi-compilers/2021.4.0-gcc-xrzccgg/compiler/2021.4.0/linux/bin/intel64/../../compiler/include/icc
-I/dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/x86_64/intel-oneapi-compilers/2021.4.0-gcc-xrzccgg/compiler/2021.4.0/linux/bin/intel64/../../compiler/include
-I/usr/local/include
-I/usr/lib64/gcc/x86_64-suse-linux/7/include
-I/usr/lib64/gcc/x86_64-suse-linux/7/include-fixed
-I/usr/include/
-I/usr/include
hello.F90

The suggestion to modify the contents of this file into a Makefile which then gets called by a shell script is an interesting one, however I think it may be easier to just start directly from Fypp's Python API (https://fypp.readthedocs.io/en/stable/fypp.html#fypp) and write a Python script. The author of Fypp has also offered his view of how to implement the option parsing here: https://github.com/aradi/fypp/issues/24

[jimdempseyatthecove]

Please report back when you have a solution to integrating Fypp with Intel's Fortran via -fpp-name=...

This may be appreciated by other readers of this forum thread.

Jim Dempsey

[Barbara_P_Intel]

The updated description of fpp-name is in the current Fortran DGR (Developer Guide and Reference).