ifx removes object files

[foxtran]

Hello,

I have tried to compile some application which consists of a large number of files. Accidentally, I noticed that ifx started to remove files:

[ build_ifx_reldeb]$ tree > tree.initial
[ build_ifx_reldeb]$ /opt/intel/oneapi/compiler/2023.2.0/linux/bin/ifx  -O2 -g @CMakeFiles/drpa.rsp -o drpa
[ build_ifx_reldeb]$ tree > tree.after
[ build_ifx_reldeb]$ diff tree.initial tree.after -upN --color
--- tree.initial 2023-11-19 22:30:29.674035720 +0100
+++ tree.after 2023-11-19 22:30:42.785903961 +0100
@@ -1186,7 +1186,6 @@
 |   |   |   |   |-- h_g_tc_uc_rangesep.f-pp.f.d
-|   |   |   |   |-- h_h_tc_rangesep.f.o
 |   |   |   |   |-- h_h_tc_rangesep.f.o.ddi​

and more files were deleted (at this time, only 17 of them)
@CMakeFiles/drpa.rsp contains about 4400 lines (so, about 4400 .o files are linked together)

Fortunately, linking of drpa is successful, but other binaries can not be linked because ifx removed some of necessary files.

My shell supports such large number of arguments:

$ getconf ARG_MAX
76800000

Used ifx version:
$ ifx --version
ifx (IFX) 2023.2.0 20230622
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

[foxtran]

Dear, 

I have the same behaviour with IFX 2024.0.0. It does not matter, do I use ninja nor make build scripts. 

Optimization options also has no effect.

[Igor_V_Intel]

What happens if you split this to 2 or 4 commands? It will be interesting to take a look on the output with -### option. Could you please run the linking with this option and attach a log? Will be nice to check if you actually exceeded 76800000 limit (it depends on how long are your lines, hope not more than 17k). The output from -### option should help to understand what is actually finally executed.

[foxtran]

Dear Igor,

> What happens if you split this to 2 or 4 commands?

I suppose it will work.

Fortunately, I created a script that reproduces observed behaviour:

#!/usr/bin/bash

#L1="a b c d e f g h i j k l m n o p q r s t v x y z"
L1="a b c d" # does not remove files with these directory names
L1="a b c d e"
L2=${L1}
L3=${L1}

OPT="-O2 -g"
LIBS="-L${CMPLR_ROOT}/opt/compiler/lib ${MKLROOT}/lib/libmkl_intel_ilp64.so  ${MKLROOT}/lib/libmkl_intel_thread.so  ${MKLROOT}/lib/libmkl_core.so  ${CMPLR_ROOT}/lib/libiomp5.so  -lpthread  -lm  -ldl  ${MKLROOT}/lib/libmkl_intel_ilp64.so  ${MKLROOT}/lib/libmkl_intel_thread.so ${MKLROOT}/lib/libmkl_core.so  ${CMPLR_ROOT}/lib/libiomp5.so  -lpthread  -lm  -ldl  -lpthread  -lm  -ldl  ${MKLROOT}/lib/libmkl_intel_ilp64.so  ${MKLROOT}/lib/libmkl_intel_thread.so  ${MKLROOT}/lib/libmkl_core.so  ${CMPLR_ROOT}/lib/libiomp5.so  ${MKLROOT}/lib/libmkl_intel_ilp64.so  ${MKLROOT}/lib/libmkl_intel_thread.so  ${MKLROOT}/lib/libmkl_core.so  ${CMPLR_ROOT}/lib/libiomp5.so  -lpthread  -lm  -ldl  -lirng"

mkdir -p tree
mkdir -p tree/mods

touch tree/null.f
echo "program main" > tree/main.f90

FILES_LIST=""

for i in ${L1}
do
  for j in ${L2}
  do
    echo "${i}${j}"
    for k in ${L3}
    do
      mkdir -p tree/${i}/${j}/${k}
      cat >tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90 <<EOF
  subroutine print_${i}${j}${k}()
    print *, "call print_${i}${j}${k}()"
  end subroutine print_${i}${j}${k}
EOF
    ifx tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90 ${OPT} -module tree/mods -c -o tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90.o &
    FILES_LIST="$FILES_LIST $(pwd)/tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90.o"
    cat >>tree/main.f90 <<EOF
    call print_${i}${j}${k}()
EOF
    done
  done
done
wait

echo "end program main" >> tree/main.f90

ifx tree/main.f90 ${OPT} -module tree/mods -c -o tree/main.f90.o
ifx tree/null.f -c -o tree/null.f.o
tree > tree.initial
echo ${FILES_LIST} > tree/files
echo ${LIBS} > tree/libs
echo "ifx ${OPT} @Tree/files $(pwd)/tree/main.f90.o $(pwd)/tree/null.f.o @Tree/libs -o tree/main"
ifx ${OPT} @Tree/files $(pwd)/tree/main.f90.o $(pwd)/tree/null.f.o @Tree/libs -o tree/main
tree > tree.after

diff tree.initial tree.after -upN --color

rm tree.initial tree.after
rm -rf tree

It expects that you have loaded oneAPI, so, MKLROOT, CMPLR_ROOT are defined and ifx is available.
At final after a few seconds, you will see the following log:

--- tree.initial	2023-11-22 19:24:32.198791170 +0100
+++ tree.after	2023-11-22 19:24:32.609787386 +0100
@@ -334,8 +334,7 @@
 |   |   |   |   |-- eab_mod.f90
 |   |   |   |   `-- eab_mod.f90.o
 |   |   |   |-- c
-|   |   |   |   |-- eac_mod.f90
-|   |   |   |   `-- eac_mod.f90.o
+|   |   |   |   `-- eac_mod.f90
 |   |   |   |-- d
 |   |   |   |   |-- ead_mod.f90
 |   |   |   |   `-- ead_mod.f90.o
@@ -406,11 +405,15 @@
 |   |       `-- e
 |   |           |-- eee_mod.f90
 |   |           `-- eee_mod.f90.o
+|   |-- files
+|   |-- libs
+|   |-- main
 |   |-- main.f90
 |   |-- main.f90.o
 |   |-- mods
 |   |-- null.f
 |   `-- null.f.o
+|-- tree.after
 `-- tree.initial

-157 directories, 256 files
+157 directories, 259 files

So, after linking (which was successful: see +main), some files were removed. Probably, the behaviour can be different on different machines, so, you can try to increase a number of created files.

From my experiments, with empty LIBS variable, the compilation is successful up to 26^3 files (I did not test more). As I said earlier, OPT flags does not play any role. Note, also there is an empty null.f. It also may play some role.

Best,
Igor

[Igor_V_Intel]

Thanks a lot for this reproducer and script. I was thinking to do the same but you saved a lot of time. Let me check if I can reproduce it and I will investigate it further.

[foxtran]

@Igor_V_Intel, I just noticed that the script is not correct. Somewhat, after "at" symbol, the "tree" has large symbols while it should be small. Only lines 50 and 51 are affected.

[foxtran]

After some experiments, 
- null.f.o has no effect
- minimal example when the problem is reproducible has the following LIBS variable:

LIBS="-lpthread  -lm  -ldl  -lpthread  -lm -ldl -lpthread  -lm  -ldl"

Then, reducing repeatable libs removes the problem 

[foxtran]

Well... With -###

rm /tmp/ifx1715225509AnpRH2/ifxdashvXrkHm1
rm /tmp/ifx1715225509AnpRH2/ifxdummysjLn0Q.c
rm /tmp/ifx1715225509AnpRH2/ifxclangdashvDEKl9H
rm /tmp/ifx1715225509AnpRH2/ifxsearchdirs5kRXJB
rm /tmp/ifx1715225509AnpRH2/ifxgccdashvPPVbt8
rm /tmp/ifx1715225509AnpRH2/ifxldashvzSYfaD
rm /dev/shm/ger/tree/e/e/d/eed_mod.f90.o

The last deletion is not necessary

[Igor_V_Intel]

I modified the script and use this version now:

#!/usr/bin/bash

L1="a b c d" # does not remove files with these directory names
L1="a b c d e"
L2=${L1}
L3=${L1}

OPT="-O2 -g"
LIBS="-lpthread -lm -ldl"

mkdir -p tree
mkdir -p tree/mods

touch tree/null.f
echo "program main" > tree/main.f90

FILES_LIST=""

for i in ${L1}
do
  for j in ${L2}
  do
    echo "${i}${j}"
    for k in ${L3}
    do
      mkdir -p tree/${i}/${j}/${k}
      cat >tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90 <<EOF
  subroutine print_${i}${j}${k}()
    print *, "call print_${i}${j}${k}()"
  end subroutine print_${i}${j}${k}
EOF
    ifx tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90 ${OPT} -module tree/mods -c -o tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90.o &
    FILES_LIST="$FILES_LIST $(pwd)/tree/${i}/${j}/${k}/${i}${j}${k}_mod.f90.o"
    cat >>tree/main.f90 <<EOF
    call print_${i}${j}${k}()
EOF
    done
  done
done
wait

echo "end program main" >> tree/main.f90

ifx tree/main.f90 ${OPT} -module tree/mods -c -o tree/main.f90.o
ifx tree/null.f -c -o tree/null.f.o
tree > tree.initial
echo ${FILES_LIST} > tree/files
echo ${LIBS} > tree/libs
echo "ifx ${OPT} @Tree/files $(pwd)/tree/main.f90.o $(pwd)/tree/null.f.o @Tree/libs -o tree/main"
ifx ${OPT} @Tree/files $(pwd)/tree/main.f90.o $(pwd)/tree/null.f.o @Tree/libs -o tree/main
tree > tree.after

diff tree.initial tree.after -upN --color

rm tree.initial tree.after
rm -rf tree

This is what I observe on my system:

 ./repro.sh
aa
ab
ac
ad
ae
ba
bb
bc
bd
be
ca
cb
cc
cd
ce
da
db
dc
dd
de
ea
eb
ec
ed
ee
tree/null.f: remark #5133: The input stream is empty
ifx -O2 -g @Tree/files /test/main.f90.o /test/tree/null.f.o @Tree/libs -o tree/main
--- tree.initial        2023-11-27 08:44:59.388925480 -0800
+++ tree.after  2023-11-27 08:44:59.655923016 -0800
@@ -406,11 +406,15 @@
 │   │       └── e
 │   │           ├── eee_mod.f90
 │   │           └── eee_mod.f90.o
+│   ├── files
+│   ├── libs
+│   ├── main
 │   ├── main.f90
 │   ├── main.f90.o
 │   ├── mods
 │   ├── null.f
 │   └── null.f.o
+├── tree.after
 └── tree.initial

-157 directories, 256 files
+157 directories, 260 files

[foxtran]

@Igor_V_Intel,

Right. With your LIBS variable I see the same behaviour.

Please, try the following LIBS:
```
LIBS="-lpthread -lm -ldl -lpthread -lm -ldl -lpthread -lm -ldl"
```

For such case with your script, I see the problem:

ifx -O2 -g @tree/files /dev/shm/ger/tree/main.f90.o /dev/shm/ger/tree/null.f.o @tree/libs -o tree/main
--- tree.initial	2023-11-28 13:14:33.485538397 +0100
+++ tree.after	2023-11-28 13:14:33.646536858 +0100
@@ -422,16 +422,19 @@
 |   |       |   |-- eec_mod.f90
 |   |       |   `-- eec_mod.f90.o
 |   |       |-- d
-|   |       |   |-- eed_mod.f90
-|   |       |   `-- eed_mod.f90.o
+|   |       |   `-- eed_mod.f90
 |   |       `-- e
 |   |           |-- eee_mod.f90
 |   |           `-- eee_mod.f90.o
+|   |-- files
+|   |-- libs
+|   |-- main
 |   |-- main.f90
 |   |-- main.f90.o
 |   |-- mods
 |   |-- null.f
 |   `-- null.f.o
+|-- tree.after
 `-- tree.initial

-157 directories, 277 files
+157 directories, 280 files

I have repeatable link flags due to configuration with CMake. 

[foxtran]

@Igor_V_Intel, was you able to reproduce an issue with repeatable libs in LIBS?

[foxtran]

@Igor_V_Intel, ping

[Igor_V_Intel]

I have reproduced this problem with latest compiler build. Looks like a bug in the compiler driver (ifx). I escalated this to the development team to fix it in the upcoming versions.

Thank you for reporting this problem to us. A workaround may be to remove the duplicated libs entries (if possible).

[foxtran]

Thank you for good news!

> A workaround may be to remove the duplicated libs entries (if possible).

Unfortunately, CMake does not do that

[Igor_V_Intel]

The solution for this bug was found and implemented. Please expect the fix to be available in the upcoming updates.