Very slow compilation

Guillaume_d_ · ‎10-07-2014

Hello,

Recently, the compilation time of our programs has slow down dramatically. For a very simple source :

@drhpcmss % time ifort -c -fpp hsbib/normR.f90

real    0m12.450s
user    0m0.048s
sys     0m0.084s

We believe that this comes from the license check, but running the compilation with $INTEL_LICENSE_FILE pointing explicitely to a folder containing a license file (which itself point to a server) does not help. Below is the results with $INTEL_LMD_DEBUG=1 (most of the time is between "INTEL_LMD: checkoutfilter: returns ACCEPT" and "INTEL_LMD: checkout: checkout call returns rc 0") .

Any suggestion that could solve the issue ?

Regards,

Guillaume

drhpcmss % time env INTEL_LMD_DEBUG=1 ifort -c -fpp hsbib/normR.f90
    INTEL_LMD: flex_set_license_path: called, path=</opt/intel/composer_xe_2013_sp1.4.211/bin/intel64>
    INTEL_LMD: flex_set_platform: called, plat_name=<i86_r>
    INTEL_LMD: checkout: entered at start_time: 1412689314 (UTC) Tue Oct 7 13:41:54 2014
    INTEL_LMD: checkout: build options:
    INTEL_LMD: checkout:     build date:                2014.0806
    INTEL_LMD: checkout:     build platform:            Intel(R) 64 architecture; Linux*
    INTEL_LMD: checkout:     checkout platform:         i86_r (IA-32 architecture; Linux*)
    INTEL_LMD: checkout:         *Other brands and names are the property of their respective owners.
    INTEL_LMD: checkout:     FLEXlm version:            11.12
    INTEL_LMD: checkout:     source version:            1.132 $ Intel(R) Software Licensing release 2.3 | FlexNet Publisher* version 11.12.0 $
    INTEL_LMD: checkout:     FLEX_MAX_LICENSE_TRIES:    0
    INTEL_LMD: checkout:     FLEX_USE_NOWAIT_AND_SPIN: enabled
    INTEL_LMD: checkout:     FLEX_SPIN_SLEEP_SECONDS:   30
    INTEL_LMD: checkout:     heartbeats:                60 seconds (automatic)
    INTEL_LMD: checkout:     server retry:              60 seconds
    INTEL_LMD: checkout:     FLEX_CHECKOUT_FILTER:      enabled
    INTEL_LMD: checkout:     FLEX_CLEAN_PATHS:          disabled
    INTEL_LMD: checkout:     FLEX_CLOCK_ROLLBACK_CHECK: disabled
    INTEL_LMD: checkout:     FLEX_APP_SPECIFIC:         disabled
    INTEL_LMD: checkout: user options:
    INTEL_LMD: checkout:     feature name:              FCompL
    INTEL_LMD: checkout:     opts:                      { FLEX_OPT_QUIET FLEX_OPT_QUIET_EXPIRE_EVAL }
    INTEL_LMD: checkout:     try_count:                 0
    INTEL_LMD: checkout:     retry_interval:            30
    INTEL_LMD: checkout:     linger_time:               0
    INTEL_LMD: checkout: environment:
    INTEL_LMD: checkout:     INTEL_LICENSE_FILE         </opt/intel/composer_xe_2013_sp1.4.211/licenses:/opt/intel/licenses:/homes/gde-hauteclocque/intel/licenses:/opt/intel/composer_xe_2013_sp1.4.211/Licenses:/homes/gde-hauteclocque/intel/licenses:/opt/intel/licenses:/Users/Shared/Library/Application Support/Intel/Licenses:/opt/intel/composer_xe_2013_sp1.4.211/bin/intel64>
    INTEL_LMD: checkout:     LM_LICENSE_FILE            <NULL>
    INTEL_LMD: checkout:     INTEL_LMD_DEBUG            <1>
    INTEL_LMD: checkout: license file(s) used were (in this order):
    INTEL_LMD: ** 1. /opt/intel/composer_xe_2013_sp1.4.211/licenses
    INTEL_LMD: ** 2. /opt/intel/licenses/USE_SERVER.lic
    INTEL_LMD: ** 3. /homes/gde-hauteclocque/intel/licenses
    INTEL_LMD: ** 4. /Users/Shared/Library/Application Support/Intel/Licenses
    INTEL_LMD: ** 5. /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/*.lic
    INTEL_LMD: checkout: linger_time set to 0 by default
    INTEL_LMD: checkout: begin attempt(s) to checkout
    INTEL_LMD: checkout: about to do checkout call
    INTEL_LMD: checkoutfilter: called for feature: FCompL
    INTEL_LMD: checkoutfilter: VENDOR_STRING: <SUPPORT=COM https://registrationcenter.intel.com>;
    INTEL_LMD: checkoutfilter: next VENDOR_STRING keyword: SUPPORT=<COM>
    INTEL_LMD: checkoutfilter:     ignore: keyword SUPPORT is disabled for filtering
    INTEL_LMD: checkoutfilter: next VENDOR_STRING keyword: https://registrationcenter.intel.com
    INTEL_LMD: checkoutfilter:     ignore: keyword https://registrationcenter.intel.com is disabled for filtering
    INTEL_LMD: checkoutfilter: returns ACCEPT
    INTEL_LMD: checkout: checkout call returns rc 0
    INTEL_LMD: checkout: license now granted
    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: checkout: contents of particular license actually checked out:
    INTEL_LMD: checkout:     feature name:                  FCompL (INCREMENT line)
    INTEL_LMD: checkout:     serial number:                 SMSADNV27DB5
    INTEL_LMD: checkout:     license expires:               perpetual license (does not expire)
    INTEL_LMD: checkout:     license maintenance expires:   2015.0926
    INTEL_LMD: checkout:     type of license:               counted, 2 users maximum
    INTEL_LMD: checkout:     allowed platforms:
    INTEL_LMD: checkout:             i86_r          (IA-32 architecture; Linux*)
    INTEL_LMD: checkout:             i86_re         (IA-32 architecture; Linux*)
    INTEL_LMD: checkout:             it64_lr        (IA-64 architecture; Linux*)
    INTEL_LMD: checkout:             it64_re        (IA-64 architecture; Linux*)
    INTEL_LMD: checkout:             amd64_re       (Intel(R) 64 architecture; Linux*)
    INTEL_LMD: checkout:         *Other brands and names are the property of their respective owners.
    INTEL_LMD: checkout:     VENDOR_STRING= string contents:
    INTEL_LMD: checkout:             SUPPORT=<COM>
    INTEL_LMD: checkout:             https://registrationcenter.intel.com
    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: flex_expire_days: called
    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: flex_expire_days: returns 3650000
    INTEL_LMD: checkout: returns GRANTED
    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_return_license: called
    INTEL_LMD: flex_return_license: considering feature name: FCompL
    INTEL_LMD: flex_return_license: feature name needs checkin: FCompL
    INTEL_LMD: flex_return_license: checkin succeeded for feature: FCompL
    INTEL_LMD: flex_free_job: called
    INTEL_LMD: flex_free_job: lc_new_job was called
    INTEL_LMD: flex_return_license: done

Guillaume_d_ · ‎11-04-2014

Hello,

The problem persists and make the compilation so slow (more than 4hours for the whole program) that the compiler is barely usable. A few hint that could help in helping us :

1- We do not observe the problem on windows (the license server for Linux and Windows is the same, and linux based)

2- On our linux system, with a few exception, the computers run with AMD processors. The few computer with intel processor does not show similar issue. Might that be relevant ?

Regards,

Guillaume

Steven_L_Intel1 · ‎11-04-2014

I am not aware that the CPU brand has any significance here. We do have some similar complaints about license checkout times on Linux. I'll let the people investigating this know of the problem you are seeing.

Steven_L_Intel1 · ‎11-05-2014

The developers have asked that you try an experiment and replace the host name in the local license file (the one with USE SERVER) with the IP address of the license server. See if that makes a difference.

littlejohn75 · ‎11-06-2014

NO, I don't think it is a licensing configuration problem.

The intel ifort compiler is installed on a DL365G7 HP Proliant server (hostname drhpcmpp ) :

fp2x@drhpcmpp:/tmp> df -kT /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
Sys. de fichiers Type 1K-blocs Util. Dispo. Uti% Monté sur
/dev/mapper/opt_vg-opt_lv ext3 61927420 58473500 2825648 96% /opt

fp2x@drhpcmpp:/tmp> time /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
ifort: command line error: no files specified; for help type "ifort -help"

real    0m12.315s
user    0m0.004s
sys     0m0.012s

Please note that there is no licensing error message.

fp2x@drhpcmpp:/tmp> cat /opt/intel/licenses/USE_SERVER.lic
SERVER drsvn.ho.bureauveritas.com ANY 27000
USE_SERVER

Now I try the same ifort invocation from another linux 64 system with /opt/intel directory automounted :

fp2x@drbuildbot:~$ time /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
ifort: command line error: no files specified; for help type "ifort -help"

real    0m0.550s
user    0m0.000s
sys     0m0.012s

fp2x@drbuildbot:~$ df -kT /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
Sys. fich. Type 1K-blocks Util. Disponible Uti% Monté sur
drhpcmpp:/opt/intel nfs4 61927424 58473472 2826240 96% /opt/intel

Depending on the time to mount the /opt/intel nfs4 file system the time of invocation varies, it can be as low as 0.125 second

fp2x@drbuildbot:~$ time /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
ifort: command line error: no files specified; for help type "ifort -help"

real    0m0.125s
user    0m0.004s
sys     0m0.008s

But on the very machine which holds the file system the delay is consistently always above 12.125 seconds

My explaination :

On the machine with a > 12 sec timeout delay

fp2x@drhpcmpp:/tmp> fgrep vendor_id /proc/cpuinfo
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD
vendor_id       : AuthenticAMD

On the no delay machine

fp2x@drbuildbot:~$ fgrep vendor_id /proc/cpuinfo
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel

Now, let's try with an openSUSE 11.4 (x86_64) system (named linux02 ) :

fp2x@linux02:~> time /optl/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
ifort: command line error: no files specified; for help type "ifort -help"

real    0m0.102s
user    0m0.001s
sys     0m0.007s
fp2x@linux02:~> df -kT /opt/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort
Sys. fich.    Type    1K-blocs   Utilisé    Dispo. Uti% Monté sur
drhpcmpp:/opt/intel/
              nfs4    61927424 58473472   2826240 96% /optl/intel
fp2x@linux02:~> fgrep vendor_id /proc/cpuinfo
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel
vendor_id       : GenuineIntel

The automount is made on /optl as /opt is used by the system

I remove the sybolic link /opt/intel -> /optl/intel :

fp2x@linux02:~> sudo rm /opt/intel
fp2x@linux02:~> time /optl/intel/composer_xe_2013_sp1.4.211/bin/intel64/ifort

Error: A license for FCompL is not available (-76,61026,2).

License file(s) used were (in this order):
1. Trusted Storage
** 2. /optl/intel/composer_xe_2013_sp1.4.211/Licenses/USE_SERVER.lic
** 3. /home/fp2x/intel/licenses
** 4. /opt/intel/licenses
** 5. /Users/Shared/Library/Application Support/Intel/Licenses
** 6. /optl/intel/composer_xe_2013_sp1.4.211/bin/intel64/*.lic

Please visit http://software.intel.com/sites/support/ if you require technical assistance.

ifort: error #10052: could not checkout FLEXlm license

real    0m1.051s
user    0m0.010s
sys     0m0.020s

The 1 second duration to check the paths of licences seems reasonable

The systematic 12 seconds delay when the compiling system has AMD processors is unacceptable.

As the compilation of most source files is very quick, there is a two orders of magnitude performance lost in case of AMD64 CPUs.

The prejudice for our Research Department is considerable as we bought 6 DL165G7 ( 2x12 cores ), one DL365G7 ( 2x8 cores ) and a 1x8 cores DL365G7 (drhpcmpp) machines, 4 years ago.

It is the same Debian binaries on the DL165G7 machines and the drbuildbot system ( GenuineIntel CPU ). No compatibility problem of any sort, except with Intel Software products

Cordialement,
Regards,
Mit freundlichen Grüssen,
ﻢﻋ ﺖﺤﻳﺎﺘﻳ ﺎﻠﺧﺎﻠﺻﺓ
---
F. Petitjean
01 55 24 75 05

Steven_L_Intel1 · ‎11-06-2014

I have to think that there is something else at work other than the processor brand. We do not even look at this in our code and we use a third-party licensing library for the actual license checks. Thanks for the detailed information and I will forward that to the developers.

littlejohn75 · ‎12-12-2014

Trying with the last version

fp2x@drbuildbot:~$ time /opt/intel/composer_xe_2015.1.133/bin/intel64/ifort
ifort: command line error: no files specified; for help type "ifort -help"

real    0m0.120s
user    0m0.000s
sys     0m0.008s

fp2x@drbuildbot:~$ time /opt/intel/composer_xe_2015.1.133/bin/intel64/icc
icc: command line error: no files specified; for help type "icc -help"

real    0m0.120s
user    0m0.004s
sys     0m0.004s

very easy to reproduce

fp2x@drbuildbot:~$ fgrep vendor_id /proc/cpuinfo | uniq
vendor_id : GenuineIntel

___________

fp2x@drhpcmpp:~> time /opt/intel/composer_xe_2015.1.133/bin/intel64/ifort
ifort: command line error: no files specified; for help type "ifort -help"

real    0m12.133s
user    0m0.012s
sys     0m0.008s

fp2x@drhpcmpp:~> time /opt/intel/composer_xe_2015.1.133/bin/intel64/icc
icc: command line error: no files specified; for help type "icc -help"

real    0m12.133s
user    0m0.000s
sys     0m0.020s

fp2x@drhpcmpp:~> fgrep vendor_id /proc/cpuinfo | uniq
vendor_id : AuthenticAMD
fp2x@drhpcmpp:~>

Please, why a penalty of twelve seconds ?

The AuthenticAMD AMD64 eight cores CPU has the Magny-Cours architecture

---

Cordialement,
Regards,
Mit freundlichen Grüssen,
ﻢﻋ ﺖﺤﻳﺎﺘﻳ ﺎﻠﺧﺎﻠﺻﺓ
---
F. Petitjean
01 55 24 75 05

Intel société américaine qui à la suite de pratiques anticoncurrentielles systématiques s'est vue infliger une amende record de 1060 millions € par la Commission Européenne.

Steven_L_Intel1 · ‎12-12-2014

No answers yet - sorry.

Steven_L_Intel1 · ‎08-12-2015

I know it's been a long time, but I finally got an answer. The developers are reasonably certain that the issue is slow response from the DNS server used by the system. Even putting an IP address in the license file for the server location doesn't skip DNS lookups. As I indicated before, the CPU type is not relevant.

The recommendation I have received is to check your "nsswitch.conf" file and make sure it checks /etc/hosts first. Another thing to look at is resolv.conf and perhaps change the order of DNS servers to have a faster one come first.

Aboulghasem_K_ · ‎09-20-2015

This comment has been moved to its own thread

Yi_C_ · ‎11-05-2015

We were suffering the same problem. I used the same machine for both license server and client. The DNS setup is absolutely correct. But the checkout is very slow. The moment we disabled the DNS in /etc/resolve.cnf the checkout became fast.

Johannes_Rieke · ‎11-06-2015

Hi,

could it be possible that these two topics are related to each other?

'ifort.exe is slow on windows': https://software.intel.com/en-us/forums/intel-visual-fortran-compiler-for-windows/topic/598158

It's Windows and not Linux but it's the same license management bib (flexlm) used by Intel compilers and the description sounds similar.

Regards, Johannes

Kevin_D_Intel · ‎11-06-2015

I think separate issues are getting confused.

@Yi C - The OP was about slowness between two systems with different CPUs. It sounds like your issue relates to having DNS enabled. What happens if you reordered nssswitch.conf as Steve described? What release of the Parallel Studio XE are you using where you see the slow checkout, 2015 or 2016?

@Johannes - The slowness reported on Windows in the thread you cited is reportedly only occurring with the latest PSXE 2016 (16.0 compiler) and that it does not occur with the previous PSXE 2015 (15.0) compiler so that makes me believe these issues (the non-Intel CPU and DNS configuration) are not connected; however, we won't dismiss the possibility and will share Yi C's observation with our Developers.

samuel_p_ · ‎12-28-2015

This comment has been moved to its own thread

Thomas_H_3 · ‎02-02-2016

I had a similar issue, I realized that the DNS query that was sometimes slow was an IPV6 query (AAAA) on the short hostname of the building host. I resolved it by putting a fake IPV6 entry in /etc/hosts, is there a way to avoid intel compiler issuing this name query ? And why is it issued in the first place, I don't get its purpose ?

Steven_L_Intel1 · ‎02-02-2016

It's the licensing check that does the DNS lookup - I'm not sure exactly what it's doing here as it's part of the licensing solution we use. I have heard of other issues relating to DNS lookups slowing the compiler from starting.

DCarv5 · ‎05-20-2016

I install CentOS 7.2 and had the same problem with the Intel compiler license delay. CentOS 7.2 enables ipv6 by default.

login1.hikari(1)$ time icc -V

Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 16.0.1.150 Build 20151021

real 0m25.096s

user 0m0.014s

sys 0m0.009s

I disabled ipv6 and rebooted

login1.hikari(1)# cat /etc/default/grub

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL="serial console"

GRUB_SERIAL_COMMAND="serial --unit=1 --speed=115200"

GRUB_DISABLE_RECOVERY="true"

GRUB_CMDLINE_LINUX="crashkernel=auto biosdevname=0 net.ifnames=0 ipv6.disable=1 selinux=0 console=tty0 console=ttyS1,115200n8"

login1.hikari(2)# grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

login1.hikari(3)# reboot

Now the delay for Intel license checkout is reasonable.

login1.hikari(1)$ time icc -V

Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 16.0.1.150 Build 20151021

real 0m0.055s

user 0m0.015s

sys 0m0.006s