Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

Slow Builds on Mojave

Thomas_F_1
Beginner
522 Views

My employer has forced me to upgrade from High Sierra to Mojave this weekend and now my build times have tripled in length from 3-4 minutes to 9-14 minutes. This affects ifort 2015 and 2020.

What information I could find online relates to license server (DNS) issues, but I am using an uncounted local license. I have confirmed that this is related to ifort, because our Makefile effectively defines FPP as "ifort -E". If I change it to "fpp" (to avoid running ifort for generating dependencies), the build time is cut significantly, but it's still very slow.

On an old (2011) iMac running High Sierra, "ifort -v" takes about 0.095 seconds, but on my MacBook Pro running Mojave, it takes about 0.8 seconds.

If I set "export INTEL_LMD_DEBUG=1", the license is acquired very quickly, but there is a delay later in the process:

    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: flex_expire_days: called
    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: flex_expire_days: returns 3650000
    INTEL_LMD: checkout: returns GRANTED
    INTEL_LMD: flex_config: called
    INTEL_LMD: flex_config: at least one license has been granted
    INTEL_LMD: flex_config: the most recently checked out feature was checked out successfully
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called

(delay occurs here)

    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
    INTEL_LMD: flex_pre_fork: called
    INTEL_LMD: flex_post_fork: called
ifort version 15.0.7
    INTEL_LMD: flex_return_license: called
    INTEL_LMD: flex_return_license: considering feature name: FCompM
    INTEL_LMD: flex_return_license: feature name needs checkin: FCompM
    INTEL_LMD: flex_return_license: checkin succeeded for feature: FCompM
    INTEL_LMD: flex_free_job: called
    INTEL_LMD: flex_free_job: lc_new_job was called
    INTEL_LMD: flex_return_license: done

Has anyone run into this problem before or have suggestions?

Thanks!

0 Kudos
7 Replies
Steve_Lionel
Honored Contributor III
522 Views

Do you have the environment variable INTEL_LICENSE_FILE defined?  If I recall correctly, the license checking code looks at all the available license sources, even if it finds a sufficient license locally, and if this causes it to query the network it can be slow.

Are you really running version 15?

0 Kudos
Thomas_F_1
Beginner
522 Views

Hi Steve,

I have INTEL_LICENSE_FILE defined (and exported) to point directly to the license file I want to use, but the debug messages show that it still appends other search locations. I have eliminated all unnecessary license files, and do not have any that reach out to a network server (that I know of).

The problem we are experiencing happens with both 2015 and 2020. The 0.8s-delay adds about 6-8 minutes to our overall build time given the number of files that need to be compiled.

$ time ifort -v
ifort version 19.1.0.166

real	0m0.866s
user	0m0.435s
sys	0m0.265s

$ time ifort -v
ifort version 15.0.7

real	0m0.681s
user	0m0.433s
sys	0m0.227s

I have also tried using dtruss to track underlying system calls, but nothing overt stands out. Most of the time ifort is sitting in "psynch_cvwait".

As I noted, if I use `fpp` instead of `ifort -E` for generating dependencies, we can reduce the build time, so this points at ifort (or the license libraries) as the source of our problems.

0 Kudos
Steve_Lionel
Honored Contributor III
522 Views

Any other xxx_LICENSE_FILE variables? Are there other .lic files in the directory pointed to by INTEL_LICENSE_FILE with SERVER lines that point to other servers?

Licensing issues are difficult to diagnose in the forum - you would be well served to file an issue at the Intel Online Service Center to get the attention of licensing experts. You didn't (and shouldn't) post the entire debug output here, but if you provide that info to the support team they can probably help you. I don't have a Mac but when I do a test compile on Windows I don't see flex_pre_fork and flex_post_fork lines - I don't know what those signify.

0 Kudos
Thomas_F_1
Beginner
522 Views

Thanks Steve. I will follow up with support.

I started in the forum with the hope that someone else has already encountered (and solved) the problem. I dug through as many relevant threads as possible and knew that most cases were the result of DNS issues. This is a mobile Mac so it's always had it's own license. I still checked, and didn't find anything with "USE SERVER".

Regarding environment variables, nothing else with "LICENSE" or "LM" exists. It appears that what is defined is the default set up by the "compilervars.sh" script.

Thanks again.

0 Kudos
Ron_Green
Moderator
522 Views

But wait, the ONLY thing you changed was the OS?  You did not touch the compiler versions or get a new/renewed license?  If you did nothing but an OS upgrade how could this be a license library problem or a compiler problem of any kind?  You upgraded the OS and the compilation slowed - so why isn't this a problem with the upgrade?
Isn't it more likely that the system call to psych_cvwait is slower on the new OS?   That's what I'm taking away from this discussion.

Did you check ls /Users/Shared/Library/Application\ Support/Intel/Licenses to make sure only 1 license is in there?  Remember, the license check is brain dead and opens each and every file and scans top to bottom EVEN IF IT FINDS A VALID LICENSE on the first file!  Keep that directory clean.

As for the 2015 compiler, the most current Mac OS supported was OS X 10.10.  I would not be surprised if the upgrade to macOS 10.14 broke something.  I hope it's working, but it's not supported any longer.  

And the compilers are installed locally in /opt/intel and not network mounted file systems.  And the SOURCE files and Makefile(s) are local also, correct?  You are NOT using remote filesystems at all for anything?

 

 

 

0 Kudos
JohnNichols
Valued Contributor III
521 Views

Remember, the license check is brain dead and opens each and every file and scans top to bottom EVEN IF IT FINDS A VALID LICENSE on the first file!  Keep that directory clean.

Excellent description -- sounds like my bank 

0 Kudos
Thomas_F_1
Beginner
522 Views

Ronald W Green (Blackbelt) wrote:

But wait, the ONLY thing you changed was the OS?  

Yep. Only upgraded from High Sierra to Mojave. All files local. Only one license file. 'fpp' is not affected (no license check), but 'ifort' is (along with 'ifort -E'). This also affects the 2020 installation (which I re-installed to make sure). 'psych_cvwait' is simply a system wait call, so it's only "slow" because it's waiting for some condition. I just don't know what that condition is.

For clarification, I am not blaming Intel or ifort. I am just seeking help. I've Googled, Googled, and Googled some more. I've dug through the forums. I've rebuild my system's dynamic library cache, I've rebooted, I've checked the console logs, etc...

0 Kudos
Reply