cmmand line too long for buffer

[Russell_Winans]

/opt/intel/compilerpro-12.0.1.107/bin/ia32/ifort -c -openmp -extend_source 132 -c -O2 -module /media/home2/scripts/cctm/MOD_DIR -I. -DSUBST_GRID_ID='' -DINTERPB=INTERP3 -DSUBST_MODULES=NOOP_MODULES -DSUBST_BARRIER=NOOP_BARRIER -DSUBST_GLOBAL_MAX=NOOP_GLOBAL_MAX -DSUBST_GLOBAL_MIN=NOOP_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=NOOP_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=NOOP_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=NOOP_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=NOOP_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=NOOP_LOOP_INDEX -DSUBST_SUBGRID_INDEX=NOOP_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=NOOP_HI_LO_BND_PE -DSUBST_SUM_CHK=NOOP_SUM_CHK -DSUBST_INIT_ARRAY=NOOP_INIT_ARRAY -DSUBST_COMM=NOOP_COMM -DSUBST_MY_REGION=NOOP_MY_REGION -DSUBST_SLICE=NOOP_SLICE -DSUBST_GATHER=NOOP_GATHER -DSUBST_DATA_COPY=NOOP_DATA_COPY -DSUBST_IN_SYN=NOOP_IN_SYN \\
-DSUBST_PE_COMM=\\"/re/PE_COMM.EXT\\" \\
-DSUBST_CONST=\\"/re/CONST.EXT\\" \\
-DSUBST_FILES_ID=\\"/re/FILES_CTM.EXT\\" \\
-DSUBST_EMPR_VD=\\"/re/EMISPRM.vdif.EXT\\" \\
-DSUBST_EMPR_CH=\\"/re/EMISPRM.chem.EXT\\" \\
-DSUBST_IOPARMS=\\"/media/home2/cmaq/lib/ioapi/ioapi/fixed_src/PARMS3.EXT\\" \\
-DSUBST_IOFDESC=\\"/media/home2/cmaq/lib/ioapi/ioapi/fixed_src/FDESC3.EXT\\" \\
-DSUBST_IODECL=\\"/media/home2/cmaq/lib/ioapi/ioapi/fixed_src/IODECL3.EXT\\" \\
-DSUBST_RXCMMN=\\"/re/cb05cl_ae5_aq/RXCM.EXT\\" \\
-DSUBST_RXDATA=\\"/re/cb05cl_ae5_aq/RXDT.EXT\\" \\
-DSUBST_GC_SPC=\\"/re/cb05cl_ae5_aq/GC_SPC.EXT\\" \\
-DSUBST_GC_EMIS=\\"/re/cb05cl_ae5_aq/GC_EMIS.EXT\\" \\
-DSUBST_GC_ICBC=\\"/re/cb05cl_ae5_aq/GC_ICBC.EXT\\" \\
-DSUBST_GC_DIFF=\\"/re/cb05cl_ae5_aq/GC_DIFF.EXT\\" \\
-DSUBST_GC_DDEP=\\"/re/cb05cl_ae5_aq/GC_DDEP.EXT\\" \\
-DSUBST_GC_DEPV=\\"/re/cb05cl_ae5_aq/GC_DEPV.EXT\\" \\
-DSUBST_GC_ADV=\\"/re/cb05cl_ae5_aq/GC_ADV.EXT\\" \\
-DSUBST_GC_CONC=\\"/re/cb05cl_ae5_aq/GC_CONC.EXT\\" \\
-DSUBST_GC_G2AE=\\"/re/cb05cl_ae5_aq/GC_G2AE.EXT\\" \\
-DSUBST_GC_G2AQ=\\"/re/cb05cl_ae5_aq/GC_G2AQ.EXT\\" \\
-DSUBST_GC_SCAV=\\"/re/cb05cl_ae5_aq/GC_SCAV.EXT\\" \\
-DSUBST_GC_WDEP=\\"/re/cb05cl_ae5_aq/GC_WDEP.EXT\\" \\
-DSUBST_AE_SPC=\\"/re/cb05cl_ae5_aq/AE_SPC.EXT\\" \\
-DSUBST_AE_EMIS=\\"/re/cb05cl_ae5_aq/AE_EMIS.EXT\\" \\
-DSUBST_AE_ICBC=\\"/re/cb05cl_ae5_aq/AE_ICBC.EXT\\" \\
-DSUBST_AE_DIFF=\\"/re/cb05cl_ae5_aq/AE_DIFF.EXT\\" \\
-DSUBST_AE_DDEP=\\"/re/cb05cl_ae5_aq/AE_DDEP.EXT\\" \\
-DSUBST_AE_DEPV=\\"/re/cb05cl_ae5_aq/AE_DEPV.EXT\\" \\
-DSUBST_AE_ADV=\\"/re/cb05cl_ae5_aq/AE_ADV.EXT\\" \\
-DSUBST_AE_CONC=\\"/re/cb05cl_ae5_aq/AE_CONC.EXT\\" \\
-DSUBST_AE_A2AQ=\\"/re/cb05cl_ae5_aq/AE_A2AQ.EXT\\" \\
-DSUBST_AE_SCAV=\\"/re/cb05cl_ae5_aq/AE_SCAV.EXT\\" \\
-DSUBST_AE_WDEP=\\"/re/cb05cl_ae5_aq/AE_WDEP.EXT\\" \\
-DSUBST_NR_SPC=\\"/re/cb05cl_ae5_aq/NR_SPC.EXT\\" \\
-DSUBST_NR_EMIS=\\"/re/cb05cl_ae5_aq/NR_EMIS.EXT\\" \\
-DSUBST_NR_ICBC=\\"/re/cb05cl_ae5_aq/NR_ICBC.EXT\\" \\
-DSUBST_NR_DIFF=\\"/re/cb05cl_ae5_aq/NR_DIFF.EXT\\" \\
-DSUBST_NR_DDEP=\\"/re/cb05cl_ae5_aq/NR_DDEP.EXT\\" \\
-DSUBST_NR_DEPV=\\"/re/cb05cl_ae5_aq/NR_DEPV.EXT\\" \\
-DSUBST_NR_ADV=\\"/re/cb05cl_ae5_aq/NR_ADV.EXT\\" \\
-DSUBST_NR_N2AE=\\"/re/cb05cl_ae5_aq/NR_N2AE.EXT\\" \\
-DSUBST_NR_CONC=\\"/re/cb05cl_ae5_aq/NR_CONC.EXT\\" \\
-DSUBST_NR_N2AQ=\\"/re/cb05cl_ae5_aq/NR_N2AQ.EXT\\" \\
-DSUBST_NR_SCAV=\\"/re/cb05cl_ae5_aq/NR_SCAV.EXT\\" \\
-DSUBST_NR_WDEP=\\"/re/cb05cl_ae5_aq/NR_WDEP.EXT\\" \\
-DSUBST_TR_SPC=\\"/re/trac0/TR_SPC.EXT\\" \\
-DSUBST_TR_EMIS=\\"/re/trac0/TR_EMIS.EXT\\" \\
-DSUBST_TR_ICBC=\\"/re/trac0/TR_ICBC.EXT\\" \\
-DSUBST_TR_DIFF=\\"/re/trac0/TR_DIFF.EXT\\" \\
-DSUBST_TR_DDEP=\\"/re/trac0/TR_DDEP.EXT\\" \\
-DSUBST_TR_DEPV=\\"/re/trac0/TR_DEPV.EXT\\" \\
-DSUBST_TR_ADV=\\"/re/trac0/TR_ADV.EXT\\" \\
-DSUBST_TR_T2AQ=\\"/re/trac0/TR_T2AQ.EXT\\" \\
-DSUBST_TR_SCAV=\\"/re/trac0/TR_SCAV.EXT\\" \\
-DSUBST_TR_WDEP=\\"/re/trac0/TR_WDEP.EXT\\" \\
-DSUBST_PACTL_ID=\\"/re/pa_noop/PA_CTL.EXT\\" \\
-DSUBST_PACMN_ID=\\"/re/pa_noop/PA_CMN.EXT\\" \\
-DSUBST_PADAT_ID=\\"/re/pa_noop/PA_DAT.EXT\\" \\
VGRD_DEFN.F -o VGRD_DEFN.o

fpp: fatal: Command line too long for bufferendif


I am trying to compiler the following program but the buffer runs out of space and I have a few more objects to link. It there a way to increase the buffer size? I don't remember getting this error using previous versions of ifort.

Thank you.

[Ron_Green]

This is a reported bug, the bug ID is DPD200156236. It is new to the 12.0

At least, I think it is. You have a typo above that I hope is just in this report and NOT in your actual command line. You have an unbalanced double quote above: ... -DSUBST_GRID_ID='' -DINTERPB=INTERP3 ...

At least, it looks like a single double quote: could it be double single quotes?

Also, if you don't need FPP, you can disable it with ifort -nofpp

ron

[Russell_Winans]

Those are double single quotations in my script they may have shown up as a double quote when I copy and pasted it to the forums. I need to preprocess as some of the variables get used in other files which are linked in the program. Without it I get errors as below:

...
-DSUBST_PADAT_ID="/re/pa_noop/PA_DAT.EXT" \
VGRD_DEFN.F -o VGRD_DEFN.o

VGRD_DEFN.F(91): error #5082: Syntax error, found IDENTIFIER 'SUBST_IOPARMS' when expecting one of:
INCLUDE SUBST_IOPARMS ! I/O parameters definitions
--------------------^
VGRD_DEFN.F(92): error #5082: Syntax error, found IDENTIFIER 'SUBST_IOFDESC' when expecting one of:
INCLUDE SUBST_IOFDESC ! file header data structure
--------------------^
VGRD_DEFN.F(93): error #5082: Syntax error, found IDENTIFIER 'SUBST_IODECL' when expecting one of:
INCLUDE SUBST_IODECL ! I/O definitions and declarations
--------------------^
VGRD_DEFN.F(93): error #5102: Cannot open include file 'NULL_STRING>'
INCLUDE SUBST_IODECL ! I/O definitions and declarations
--------------------^
compilation aborted for VGRD_DEFN.F (code 1)



I tried to shorten the paths to see if that helped, but still recieve the same error. I am going to try to rename my variables and shorten them. Is there a .h file which contains the command line limits? I thought climits or limits might contain relevant information when it is linked for the size limitations; though not positive about this.

[Ron_Green]

the limit is built into our FPP. I'll ask what that size is from our developers.

ron

[Ron_Green]

The FPP limit is 4K characters BUT remember that ifort is going to invoke FPP and it will add a number of include lines as well, bumping you over the limit ( wc -c shows your ifort line at 3941 characters, add the includes and blammo! )

We can try a few things. First, the -dryrun option will dump the invocation command for ifort without doing anything. You can try that option to see what it's going to try to do.

Next, we can invoke FPP separately, this will save all the ifort include paths and whatnot. Assuming VGRD_DEFN.F is a fixed-form source file, we can do the following. Note that at the end of this we'll add an output file with file extension .i :

fpp -e -fixed -f_com=yes -I. -DSUBST_GRID_ID='' -DINTERPB=INTERP3 -DSUBST_MODULES=NOOP_MODULES ...etc etc etc ...
VGRD_DEFN.F VGRD_DEFN.i

Note that the processed source is now in VGRD_DEFN.i

Now, we can compile with that file:

ifort -c -openmp -extend_source 132 -O2 -module /media/home2/scripts/cctm/MOD_DIR VGRD_DEFN.i


By splitting this up, we reduce the command line being passed to FPP and we SHOULD be able to get it to just under 4K.

ron

[swang]

Hi, Ron:

I tried as you suggested:

pp -e -fixed -f_com=yes -I. -DSUBST_GRID_ID='' -DINTERPB=INTERP3 -DSUBST_MODULES=NOOP_MODULES ...etc etc etc ...
VGRD_DEFN.F VGRD_DEFN.i

I still got:

fpp: fatal: Command line too long for buffer

my command is as follows:

/opt/intel/composerxe-2011.2.137/bin/intel64/fpp -e -fixed -f_com=yes -I. -DSUBST_GRID_ID='' .....

Thank you for your help.

Regards,
zoom

[jimdempseyatthecove]

Place the long list of -Define=somethings into a file as

#define efine=something
#define ...

then in your Fortran program insert

#include YourDefineFile.inc

Jim Dempsey

[kjliao]

Quoting Russell Winans

/opt/intel/compilerpro-12.0.1.107/bin/ia32/ifort -c -openmp -extend_source 132 -c -O2 -module /media/home2/scripts/cctm/MOD_DIR -I. -DSUBST_GRID_ID='' -DINTERPB=INTERP3 -DSUBST_MODULES=NOOP_MODULES -DSUBST_BARRIER=NOOP_BARRIER -DSUBST_GLOBAL_MAX=NOOP_GLOBAL_MAX -DSUBST_GLOBAL_MIN=NOOP_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=NOOP_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=NOOP_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=NOOP_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=NOOP_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=NOOP_LOOP_INDEX -DSUBST_SUBGRID_INDEX=NOOP_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=NOOP_HI_LO_BND_PE -DSUBST_SUM_CHK=NOOP_SUM_CHK -DSUBST_INIT_ARRAY=NOOP_INIT_ARRAY -DSUBST_COMM=NOOP_COMM -DSUBST_MY_REGION=NOOP_MY_REGION -DSUBST_SLICE=NOOP_SLICE -DSUBST_GATHER=NOOP_GATHER -DSUBST_DATA_COPY=NOOP_DATA_COPY -DSUBST_IN_SYN=NOOP_IN_SYN \
-DSUBST_PE_COMM="/re/PE_COMM.EXT" \
-DSUBST_CONST="/re/CONST.EXT" \
-DSUBST_FILES_ID="/re/FILES_CTM.EXT" \
-DSUBST_EMPR_VD="/re/EMISPRM.vdif.EXT" \
-DSUBST_EMPR_CH="/re/EMISPRM.chem.EXT" \
-DSUBST_IOPARMS="/media/home2/cmaq/lib/ioapi/ioapi/fixed_src/PARMS3.EXT" \
-DSUBST_IOFDESC="/media/home2/cmaq/lib/ioapi/ioapi/fixed_src/FDESC3.EXT" \
-DSUBST_IODECL="/media/home2/cmaq/lib/ioapi/ioapi/fixed_src/IODECL3.EXT" \
-DSUBST_RXCMMN="/re/cb05cl_ae5_aq/RXCM.EXT" \
-DSUBST_RXDATA="/re/cb05cl_ae5_aq/RXDT.EXT" \
-DSUBST_GC_SPC="/re/cb05cl_ae5_aq/GC_SPC.EXT" \
-DSUBST_GC_EMIS="/re/cb05cl_ae5_aq/GC_EMIS.EXT" \
-DSUBST_GC_ICBC="/re/cb05cl_ae5_aq/GC_ICBC.EXT" \
-DSUBST_GC_DIFF="/re/cb05cl_ae5_aq/GC_DIFF.EXT" \
-DSUBST_GC_DDEP="/re/cb05cl_ae5_aq/GC_DDEP.EXT" \
-DSUBST_GC_DEPV="/re/cb05cl_ae5_aq/GC_DEPV.EXT" \
-DSUBST_GC_ADV="/re/cb05cl_ae5_aq/GC_ADV.EXT" \
-DSUBST_GC_CONC="/re/cb05cl_ae5_aq/GC_CONC.EXT" \
-DSUBST_GC_G2AE="/re/cb05cl_ae5_aq/GC_G2AE.EXT" \
-DSUBST_GC_G2AQ="/re/cb05cl_ae5_aq/GC_G2AQ.EXT" \
-DSUBST_GC_SCAV="/re/cb05cl_ae5_aq/GC_SCAV.EXT" \
-DSUBST_GC_WDEP="/re/cb05cl_ae5_aq/GC_WDEP.EXT" \
-DSUBST_AE_SPC="/re/cb05cl_ae5_aq/AE_SPC.EXT" \
-DSUBST_AE_EMIS="/re/cb05cl_ae5_aq/AE_EMIS.EXT" \
-DSUBST_AE_ICBC="/re/cb05cl_ae5_aq/AE_ICBC.EXT" \
-DSUBST_AE_DIFF="/re/cb05cl_ae5_aq/AE_DIFF.EXT" \
-DSUBST_AE_DDEP="/re/cb05cl_ae5_aq/AE_DDEP.EXT" \
-DSUBST_AE_DEPV="/re/cb05cl_ae5_aq/AE_DEPV.EXT" \
-DSUBST_AE_ADV="/re/cb05cl_ae5_aq/AE_ADV.EXT" \
-DSUBST_AE_CONC="/re/cb05cl_ae5_aq/AE_CONC.EXT" \
-DSUBST_AE_A2AQ="/re/cb05cl_ae5_aq/AE_A2AQ.EXT" \
-DSUBST_AE_SCAV="/re/cb05cl_ae5_aq/AE_SCAV.EXT" \
-DSUBST_AE_WDEP="/re/cb05cl_ae5_aq/AE_WDEP.EXT" \
-DSUBST_NR_SPC="/re/cb05cl_ae5_aq/NR_SPC.EXT" \
-DSUBST_NR_EMIS="/re/cb05cl_ae5_aq/NR_EMIS.EXT" \
-DSUBST_NR_ICBC="/re/cb05cl_ae5_aq/NR_ICBC.EXT" \
-DSUBST_NR_DIFF="/re/cb05cl_ae5_aq/NR_DIFF.EXT" \
-DSUBST_NR_DDEP="/re/cb05cl_ae5_aq/NR_DDEP.EXT" \
-DSUBST_NR_DEPV="/re/cb05cl_ae5_aq/NR_DEPV.EXT" \
-DSUBST_NR_ADV="/re/cb05cl_ae5_aq/NR_ADV.EXT" \
-DSUBST_NR_N2AE="/re/cb05cl_ae5_aq/NR_N2AE.EXT" \
-DSUBST_NR_CONC="/re/cb05cl_ae5_aq/NR_CONC.EXT" \
-DSUBST_NR_N2AQ="/re/cb05cl_ae5_aq/NR_N2AQ.EXT" \
-DSUBST_NR_SCAV="/re/cb05cl_ae5_aq/NR_SCAV.EXT" \
-DSUBST_NR_WDEP="/re/cb05cl_ae5_aq/NR_WDEP.EXT" \
-DSUBST_TR_SPC="/re/trac0/TR_SPC.EXT" \
-DSUBST_TR_EMIS="/re/trac0/TR_EMIS.EXT" \
-DSUBST_TR_ICBC="/re/trac0/TR_ICBC.EXT" \
-DSUBST_TR_DIFF="/re/trac0/TR_DIFF.EXT" \
-DSUBST_TR_DDEP="/re/trac0/TR_DDEP.EXT" \
-DSUBST_TR_DEPV="/re/trac0/TR_DEPV.EXT" \
-DSUBST_TR_ADV="/re/trac0/TR_ADV.EXT" \
-DSUBST_TR_T2AQ="/re/trac0/TR_T2AQ.EXT" \
-DSUBST_TR_SCAV="/re/trac0/TR_SCAV.EXT" \
-DSUBST_TR_WDEP="/re/trac0/TR_WDEP.EXT" \
-DSUBST_PACTL_ID="/re/pa_noop/PA_CTL.EXT" \
-DSUBST_PACMN_ID="/re/pa_noop/PA_CMN.EXT" \
-DSUBST_PADAT_ID="/re/pa_noop/PA_DAT.EXT" \
VGRD_DEFN.F -o VGRD_DEFN.o

fpp: fatal: Command line too long for bufferendif


I am trying to compiler the following program but the buffer runs out of space and I have a few more objects to link. It there a way to increase the buffer size? I don't remember getting this error using previous versions of ifort.

Thank you.

Hi Russell,

I think that you were trying to compile CMAQ (version?) using ifort on a 32-bit machine. Iam trying to compile CMAQ v4.5using ifort on a 64-bit machine. My compilation stopped at the same place, but theerror message was different from yours. It looked like:

-DSUBST_TR_WDEP="/home/kliao/work/Models_3_CMAQ_4.5/models/include/release/trac0/TR_WDEP.EXT" \

-DSUBST_PACTL_ID="/home/kliao/work/Models_3_CMAQ_4.5/models/include/release/pa_noop/PA_CTL.EXT" \

-DSUBST_PACMN_ID="/home/kliao/work/Models_3_CMAQ_4.5/models/include/release/pa_noop/PA_CMN.EXT" \

-DSUBST_PADAT_ID="/home/kliao/work/Models_3_CMAQ_4.5/models/include/release/pa_noop/PA_DAT.EXT" \

VGRD_DEFN.F -o VGRD_DEFN.o

: catastrophic error: **Internal compiler error: internal command buffer overflowed ** Please report this error along with the circumstances in which it occurred in a Software Problem Report.

: catastrophic error: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error.


I am guessing that your problem and my problem areessentially the samealthough the error messages looked different. Did you find a solution to fix the problem?

Thanks,

[Ron_Green]

Russell,

What you are seeing is a totally different error. What version of the compiler are you using? The above was with 12.0.1.107, aka Composer XE 2011 update 1.

Your error is what happens when the compiler itself crashes. I wonder, since you are on a 32bit system if you've tried

ulimit -s unlimited

prior to compilation. If not, send me your compiler version and I can try this myself.

ron

[Zac_Adelman]

Hi Ron

I'm tracking this ticket as I'm having the same issue with these same codes. I'm definitely on a 32-bit machine and I tried the ulimit command as you suggested. Still same error:

...cut...

-DSUBST_PADAT_ID="/home/training/CMAQ/models/include/release/pa_noop/PA_DAT.EXT" \

VGRD_DEFN.F -o VGRD_DEFN.o

fpp: fatal: Command line too long for bufferendif

-DSUBST_PADAT_ID="/home/training/CMAQ/models/include/release/pa_noop/PA_DAT.EXT" \ VGRD_DEFN.F -o VGRD_DEFN.o
fpp: fatal: Command line too long for bufferendif

[training@jnias cctm]$ uname -a

Linux jnias 2.6.18-194.32.1.el5xen #1 SMP Wed Jan 5 19:32:33 EST 2011 i686 i686 i386 GNU/Linux

[training@jnias cctm]$ ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 58142

max locked memory (kbytes, -l) 32

max memory size (kbytes, -m) unlimited

open files (-n) 1024

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) unlimited

cpu time (seconds, -t) unlimited

max user processes (-u) 58142

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

[training@jnias cctm]$ ifort -v

Version 12.0.2

Any help that you can provide on this will be greatly appreciated.

Zac

[Zac_Adelman]

As an aside, I'm a bit pressed for time with this and I'm wondering if there is a way to downgrade to one of the 11x versions of the Intel Fortran/C. I've been using the 10x and 11x versions of these compilers for quite a while now with no major issues. How can I get the last stable release of the 11x version of the compiler suite?

Zac

[TimP]

Download the version of your choice at https://registrationcenter.intel.com

[Ron_Green]

Zac,

Sorry, I just saw your note above from march 8th. All older versions can be downloaded from IRC:

http://software.intel.com/en-us/articles/older-version-product/

ron

[Ron_Green]

All,

The buffer length was bumped up to 8K in the Update 3 compiler, up from the 4K in Update 2. This should "fix" this bug for now. In a future version there is a more permanent fix to use a variable buffer for FPP.

Ron