Fortran “additional relocation overflows” error while reading big data file

[Yeon__Jejoon]

I have following format of input file: 

    4400000
          1       4       4    7056    4203    7190    2856       0       0       0       0       0       0       0   0.942    0.93   0.944   0.925       0       0       0       0       0       0   3.846     0.0   1.627
          2       3       3    8553     481    7734       0       0       0       0       0       0       0       0    0.63   0.923   0.322       0       0       0       0       0       0       0    1.93     2.0  -0.792
          3       3       2    7007    3625       0       0       0       0       0       0       0       0       0   0.359    0.93       0       0       0       0       0       0       0       0   1.889     2.0  -0.728
          4       4       4    1611     854    7901    8306       0       0       0       0       0       0       0   0.923   0.944   0.936   0.915       0       0       0       0       0       0   3.867     0.0   1.488
          5       3       4    7657    1180    6659    4147       0       0       0       0       0       0       0   0.332    0.69   0.346   0.574       0       0       0       0       0       0   1.969     2.0  -0.634
          6       3       2    1978     523       0       0       0       0       0       0       0       0       0   0.931   0.917       0       0       0       0       0       0       0       0   1.849     2.0  -0.805
    ...

This file has total 27 columns. I tried to read this file using following command:  

    do l=1,num
     read (1,42) atindex(l),atype(l),nbonds(l),(conn(l,j),j=1,10),molnr(l),(bos(l,j),j=1,10),abo(l),nlp(l),charge(l)
    enddo

(num = number of lines) and 

    42  format (i7,13i7,11f7.3,f7.1,f7.3)

I compiled this using intel compiler. However, this creates following "additional relocation overflows" error: 

    ....
    /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o): In function `for__signal_handler':
        for_init.c:(.text+0x906): additional relocation overflows omitted from the output

It looks like memory issue due to too much column. That error never happens when I read only upto integer column (column 14) with proper format. However, when I tried to read real numbers, then the error occur. 
  
This is research server cluster, not my laptop or desktop so I wouldn't be possible to modify installed setting of intel compiler. 

How can I escape from this issue? 

[Steve_Lionel]

Please show the complete and actual text of the message. The part you quoted is the least interesting part.

[mecej4]

Sorry, I misdiagnosed the problem and I don't seem able to delete the post.

[Yeon__Jejoon]

Steve Lionel (Ret.) (Blackbelt) wrote:

Please show the complete and actual text of the message. The part you quoted is the least interesting part.

$ ifort test10d.f -o test10d
/tmp/ifortYNdLHJ.o: In function `MAIN__':
test10d.f:(.text+0x59c): relocation truncated to fit: R_X86_64_32S against `.bss'
test10d.f:(.text+0x736): relocation truncated to fit: R_X86_64_32S against `.bss'
/opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o): In function `for__process_start_time':
for_init.c:(.text+0x8): relocation truncated to fit: R_X86_64_PC32 against `for_process_start_time'
/opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o): In function `handle_flt_und':
for_init.c:(.text+0x34): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_fpe_mask' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
for_init.c:(.text+0x356): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_undcnt' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
for_init.c:(.text+0x3a8): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_undcnt' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
for_init.c:(.text+0x3b0): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_undcnt' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
for_init.c:(.text+0x3bc): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_excpt_info' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
for_init.c:(.text+0x3dd): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_excpt_info' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
for_init.c:(.text+0x430): relocation truncated to fit: R_X86_64_PC32 against symbol `for__l_undcnt' defined in .bss section in /opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o)
/opt/shared/intel/2018u4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin/libifcoremt.a(for_init.o): In function `for__signal_handler':

[Yeon__Jejoon]

mecej4 (Blackbelt) wrote:

The READ statement contains 26 elements in the I/O list, whereas the input data has 27 items per record, as does the format specification. Please investigate and correct this before doing anything else.

Hmm but even free format (2,*) creates the same error... I also checked with proper format but still the same error. 

[Steve_Lionel]

See if this helps: https://software.intel.com/en-us/articles/avoiding-relocation-errors-when-building-applications-with-large-global-or-static-data-on-intel64/

[JohnNichols]

Read as a charter array - then take the array apart 

[JohnNichols]

character sorry

[Yeon__Jejoon]

Nichols, John wrote:

Read as a charter array - then take the array apart 

You mean, read all columns as characters, and print them in 2 separate files by 14 columns each, then analyze? 

[John_Campbell]

I looked at the data you posted. It does not agree with the format statement you used. The columns of numbers are more than 7 characters wide. (This may simply be due to re-formating in the forum post.)

You are showing "read(1,42)...". Are you taking the data as terminal input, or are you opening the file on unit 1 ?

An alternative, as John N suggests, would be to try a simple test fortran program and a copy of the data file:

Use a data file and open the file using "open (unit=11, file='data_copy.txt')  "and then

read each line into "character input_line*256". ( trailing blanks are not a problem) using read (11,41)  input_line

you could then read 27 values into a real array(27), using either "read(input_line,42) array" or "read(input_line,*) array" . (read num first)

write out the results you read.

I have not suggested error tests on I/O statements, as you should use the Fortran I/O response.

Start small and expand the test. Hopefully this could identify where the error is not coming from.

There is a lot to be gained in a simple data validation program, which you can enhance as you learn more about the data quirks. eg max, min, avg, median of values, zeros, or bad values is very useful for recorded test results.

Probably not here as no "additional relocation overflows" evident in the approach so far.

Probably the error is in the way you have defined your arrays. Are you using auto allocate arrays in some unusual way?

[Steve_Lionel · Accepted solution]

I am very skeptical that the I/O statement or format is directly related to the relocation error, other than that it might push the data size over the "small" limit. I have not evaluated the correctness of the format, etc., but don't see that as related to the link error.

Short answer - compile with "-mcmodel medium" and see what you get.

[JohnNichols]

Yeon, Jejoon wrote:

Quote:

Nichols, John wrote:

Read as a charter array - then take the array apart 

You mean, read all columns as characters, and print them in 2 separate files by 14 columns each, then analyze? 

Somewhere on this forum is a program by Jim Dempsey for taking apart CSV files which is similar to your problem -- the separator character is different is the only problem.  He provided an elegant solution to my brute force method.  

But I would try the other solution -- but reading files such as yours reading with read(1,*)..... is usually much better

[Yeon__Jejoon]

Steve Lionel (Ret.) (Blackbelt) wrote:

I am very skeptical that the I/O statement or format is directly related to the relocation error, other than that it might push the data size over the "small" limit. I have not evaluated the correctness of the format, etc., but don't see that as related to the link error.

Short answer - compile with "-mcmodel medium" and see what you get.

Thank you so much. This worked like a charm. 

[Yeon__Jejoon]

@John Campbell @Nichols, John Thank you so much for comment.