Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
569 Views

What are the optimal Fortran XE2020 update 4 compiler settings for AMD Threadripper 3990x on Win10?

What are the optimal Fortran Parallel Studio Pro XE2020 update 4 compiler settings for AMD Threadripper 3990x on Win10?

0 Kudos
43 Replies
Highlighted
194 Views

Run VTune on your program to see what&where the hangups are.

Note, for any timing section of code, enclose it within a loop of 2 or a few iterations. It initial pass may exhibit additional overhead than the 2nd and later passes.

a) Tread pool creation
b) "first touch" of allocated arrays
c) cache preloading
d) ...

As an experiment:

Turn off (disable) OpenMP and MKL affinity settings (we will work on enabling these later)
Set OMP_NUM_THREADS=32
Set MKL_NUM_THREADS=32

IOW let both thread pools float, and each pool at half the number of available hardware threads.

Make your test run in VTune (if the runtime is longer than 30 second or so, use the Stop button to collect the results). The past the Bottom-Up screenshot (Alt-PrintScreen).

You may need multiple screenshots to get the threads listing at the bottom.

Jim Dempsey

Jim Dempsey

0 Kudos
Highlighted
Beginner
179 Views

In my current project from within Visual Studio I clicked on "Open Vtune Profiler" and in Performance Snapshot I get two error messages stating that Vtune can not recognize the microprocessor. Does anyone in the Vtune development group have an AMD Threadripper 39nnX to test this on, with possibly a lot fewer cores if cost is an issue?

0 Kudos
Highlighted
Black Belt Retired Employee
175 Views

My understanding is that VTune is limited when using non-Intel processors, as it makes use of internal processor registers. But it can be used - it might be best to ask in the Vtune forum.

0 Kudos
Highlighted
Beginner
168 Views

I posted this subject in the Analyzers section: Vtune in Fortran XE2020 update 4 using Visual Studio is not working with AMD Threadripper 3990X

0 Kudos
Highlighted
160 Views

VTune has two sampling modes:

Hardware Sampling (you won't be able to use this on a non-Intel CPU)
Software (interrupt timer) sampling. No reason this shouldn't work on AMD CPU other than Intel messing with you.

The VTune forum would be the place to go. (there may be an option for doing what you need)

Were you able to run with the reduced thread counts and non-affinity pinned threads?

Jim Dempsey

0 Kudos
Highlighted
Beginner
154 Views

Both hardware sampling and software sampling are thwarted by error messages when running Vtune on Win10 from either Visual Studio or as a Standalone program using Admin rights with AMD 39nnX. Have you tried it?

0 Kudos
Highlighted
150 Views

Sorry to say, I do not have an AMD system. Well... I do have a 16 year old Dual AMD Opteron server gathering dust in a back corner, but that will be of little help.

You might explore using the MS VS 2019 profiling tool:

https://docs.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019

Or AMD uProf: https://developer.amd.com/amd-uprof/

Jim Dempsey

0 Kudos
Highlighted
107 Views

RE: no all logical processors being used.

I recently installed Windows 10 on my KNL system (formerly CentOS). This is a 64 core, 256 HW thread system.

I was curious to see if the OpenMP number of threads were restricted to one processor group (64-bits).

The simple Fortran test program, 64-bit, used 256 threads. I decided to test Intel C++ too. In my haste, I configured for Win32 instead of x64. The OpenMP test used 32 threads. Switching configuration to x64, the test used 256 threads. Going back to Fortran, setting to Win32 configuration, 32 threads.

FWIW apparently, Win32 builds have a (?default?) upper limit for number of threads.

Jim Dempsey

0 Kudos
Highlighted
Black Belt Retired Employee
102 Views

I tend to think that it's a restriction in the 32-bit API. Please build and run the attached source for both 32-bit and 64-bit and show the results.

0 Kudos
Highlighted
82 Views

RE: ProcessorInfo.f90

IMHO this program is incomplete (on Windows) when run on a system with more than 64 logical processors (or possibly more than one socket). Without further investigation, I presume it is enumerating the logical processors contained within the current processor group (that in which the process was launched).

Win32 output:

Trying buffer length of 480
Required buffer length is 1392
Reallocated buffer length to 1392
GetLogicalProcessorInformation results:
Number of physical processor packages: 1
Number of NUMA nodes: 1
Number of processor cores: 16
Number of logical processors: 64
Number of processor L1/L2/L3 caches: 32/8/0

x64 Output:

Trying buffer length of 640
Required buffer length is 1856
Reallocated buffer length to 1856
GetLogicalProcessorInformation results:
Number of physical processor packages: 1
Number of NUMA nodes: 1
Number of processor cores: 16
Number of logical processors: 64
Number of processor L1/L2/L3 caches: 32/8/0

x64 OpenMP run with KMP_AFFINITY=compact,verbose:

OMP: Info #155: KMP_AFFINITY: Initial OS proc set not respected: 0-63
OMP: Info #214: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #156: KMP_AFFINITY: 256 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #285: KMP_AFFINITY: topology layer "LL cache" is equivalent to "core".
OMP: Info #285: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #191: KMP_AFFINITY: 1 socket x 64 cores/socket x 4 threads/core (64 total cores)
OMP: Info #216: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to socket 0 core 0 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to socket 0 core 0 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to socket 0 core 0 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to socket 0 core 0 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to socket 0 core 1 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to socket 0 core 1 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to socket 0 core 1 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to socket 0 core 1 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 64 maps to socket 0 core 2 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 65 maps to socket 0 core 2 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 66 maps to socket 0 core 2 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 67 maps to socket 0 core 2 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 68 maps to socket 0 core 3 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 69 maps to socket 0 core 3 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 70 maps to socket 0 core 3 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 71 maps to socket 0 core 3 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 128 maps to socket 0 core 4 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 129 maps to socket 0 core 4 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 130 maps to socket 0 core 4 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 131 maps to socket 0 core 4 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 132 maps to socket 0 core 5 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 133 maps to socket 0 core 5 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 134 maps to socket 0 core 5 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 135 maps to socket 0 core 5 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 192 maps to socket 0 core 6 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 193 maps to socket 0 core 6 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 194 maps to socket 0 core 6 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 195 maps to socket 0 core 6 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 196 maps to socket 0 core 7 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 197 maps to socket 0 core 7 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 198 maps to socket 0 core 7 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 199 maps to socket 0 core 7 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to socket 0 core 8 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to socket 0 core 8 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to socket 0 core 8 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to socket 0 core 8 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to socket 0 core 9 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to socket 0 core 9 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to socket 0 core 9 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to socket 0 core 9 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 72 maps to socket 0 core 10 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 73 maps to socket 0 core 10 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 74 maps to socket 0 core 10 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 75 maps to socket 0 core 10 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 76 maps to socket 0 core 11 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 77 maps to socket 0 core 11 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 78 maps to socket 0 core 11 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 79 maps to socket 0 core 11 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 136 maps to socket 0 core 12 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 137 maps to socket 0 core 12 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 138 maps to socket 0 core 12 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 139 maps to socket 0 core 12 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 140 maps to socket 0 core 13 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 141 maps to socket 0 core 13 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 142 maps to socket 0 core 13 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 143 maps to socket 0 core 13 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 200 maps to socket 0 core 14 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 201 maps to socket 0 core 14 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 202 maps to socket 0 core 14 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 203 maps to socket 0 core 14 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 204 maps to socket 0 core 15 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 205 maps to socket 0 core 15 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 206 maps to socket 0 core 15 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 207 maps to socket 0 core 15 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 80 maps to socket 0 core 18 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 81 maps to socket 0 core 18 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 82 maps to socket 0 core 18 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 83 maps to socket 0 core 18 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 84 maps to socket 0 core 19 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 85 maps to socket 0 core 19 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 86 maps to socket 0 core 19 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 87 maps to socket 0 core 19 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 144 maps to socket 0 core 20 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 145 maps to socket 0 core 20 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 146 maps to socket 0 core 20 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 147 maps to socket 0 core 20 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 148 maps to socket 0 core 21 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 149 maps to socket 0 core 21 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 150 maps to socket 0 core 21 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 151 maps to socket 0 core 21 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 208 maps to socket 0 core 22 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 209 maps to socket 0 core 22 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 210 maps to socket 0 core 22 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 211 maps to socket 0 core 22 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 212 maps to socket 0 core 23 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 213 maps to socket 0 core 23 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 214 maps to socket 0 core 23 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 215 maps to socket 0 core 23 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to socket 0 core 24 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to socket 0 core 24 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to socket 0 core 24 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to socket 0 core 24 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to socket 0 core 25 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to socket 0 core 25 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to socket 0 core 25 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to socket 0 core 25 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 88 maps to socket 0 core 26 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 89 maps to socket 0 core 26 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 90 maps to socket 0 core 26 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 91 maps to socket 0 core 26 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 92 maps to socket 0 core 27 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 93 maps to socket 0 core 27 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 94 maps to socket 0 core 27 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 95 maps to socket 0 core 27 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 152 maps to socket 0 core 28 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 153 maps to socket 0 core 28 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 154 maps to socket 0 core 28 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 155 maps to socket 0 core 28 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 156 maps to socket 0 core 29 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 157 maps to socket 0 core 29 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 158 maps to socket 0 core 29 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 159 maps to socket 0 core 29 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 216 maps to socket 0 core 30 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 217 maps to socket 0 core 30 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 218 maps to socket 0 core 30 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 219 maps to socket 0 core 30 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 220 maps to socket 0 core 31 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 221 maps to socket 0 core 31 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 222 maps to socket 0 core 31 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 223 maps to socket 0 core 31 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 24 maps to socket 0 core 32 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 25 maps to socket 0 core 32 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 26 maps to socket 0 core 32 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 27 maps to socket 0 core 32 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 28 maps to socket 0 core 33 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 29 maps to socket 0 core 33 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 30 maps to socket 0 core 33 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 31 maps to socket 0 core 33 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 96 maps to socket 0 core 34 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 97 maps to socket 0 core 34 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 98 maps to socket 0 core 34 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 99 maps to socket 0 core 34 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 100 maps to socket 0 core 35 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 101 maps to socket 0 core 35 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 102 maps to socket 0 core 35 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 103 maps to socket 0 core 35 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 160 maps to socket 0 core 36 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 161 maps to socket 0 core 36 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 162 maps to socket 0 core 36 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 163 maps to socket 0 core 36 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 164 maps to socket 0 core 37 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 165 maps to socket 0 core 37 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 166 maps to socket 0 core 37 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 167 maps to socket 0 core 37 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 224 maps to socket 0 core 38 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 225 maps to socket 0 core 38 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 226 maps to socket 0 core 38 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 227 maps to socket 0 core 38 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 228 maps to socket 0 core 39 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 229 maps to socket 0 core 39 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 230 maps to socket 0 core 39 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 231 maps to socket 0 core 39 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 32 maps to socket 0 core 40 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 33 maps to socket 0 core 40 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 34 maps to socket 0 core 40 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 35 maps to socket 0 core 40 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 36 maps to socket 0 core 41 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 37 maps to socket 0 core 41 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 38 maps to socket 0 core 41 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 39 maps to socket 0 core 41 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 104 maps to socket 0 core 42 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 105 maps to socket 0 core 42 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 106 maps to socket 0 core 42 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 107 maps to socket 0 core 42 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 108 maps to socket 0 core 43 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 109 maps to socket 0 core 43 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 110 maps to socket 0 core 43 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 111 maps to socket 0 core 43 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 232 maps to socket 0 core 46 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 233 maps to socket 0 core 46 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 234 maps to socket 0 core 46 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 235 maps to socket 0 core 46 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 236 maps to socket 0 core 47 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 237 maps to socket 0 core 47 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 238 maps to socket 0 core 47 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 239 maps to socket 0 core 47 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 40 maps to socket 0 core 48 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 41 maps to socket 0 core 48 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 42 maps to socket 0 core 48 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 43 maps to socket 0 core 48 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 44 maps to socket 0 core 49 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 45 maps to socket 0 core 49 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 46 maps to socket 0 core 49 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 47 maps to socket 0 core 49 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 112 maps to socket 0 core 50 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 113 maps to socket 0 core 50 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 114 maps to socket 0 core 50 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 115 maps to socket 0 core 50 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 116 maps to socket 0 core 51 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 117 maps to socket 0 core 51 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 118 maps to socket 0 core 51 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 119 maps to socket 0 core 51 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 168 maps to socket 0 core 52 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 169 maps to socket 0 core 52 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 170 maps to socket 0 core 52 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 171 maps to socket 0 core 52 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 172 maps to socket 0 core 53 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 173 maps to socket 0 core 53 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 174 maps to socket 0 core 53 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 175 maps to socket 0 core 53 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 48 maps to socket 0 core 56 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 49 maps to socket 0 core 56 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 50 maps to socket 0 core 56 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 51 maps to socket 0 core 56 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 52 maps to socket 0 core 57 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 53 maps to socket 0 core 57 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 54 maps to socket 0 core 57 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 55 maps to socket 0 core 57 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 120 maps to socket 0 core 58 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 121 maps to socket 0 core 58 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 122 maps to socket 0 core 58 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 123 maps to socket 0 core 58 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 124 maps to socket 0 core 59 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 125 maps to socket 0 core 59 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 126 maps to socket 0 core 59 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 127 maps to socket 0 core 59 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 176 maps to socket 0 core 60 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 177 maps to socket 0 core 60 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 178 maps to socket 0 core 60 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 179 maps to socket 0 core 60 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 180 maps to socket 0 core 61 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 181 maps to socket 0 core 61 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 182 maps to socket 0 core 61 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 183 maps to socket 0 core 61 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 240 maps to socket 0 core 62 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 241 maps to socket 0 core 62 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 242 maps to socket 0 core 62 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 243 maps to socket 0 core 62 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 244 maps to socket 0 core 63 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 245 maps to socket 0 core 63 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 246 maps to socket 0 core 63 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 247 maps to socket 0 core 63 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 184 maps to socket 0 core 68 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 185 maps to socket 0 core 68 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 186 maps to socket 0 core 68 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 187 maps to socket 0 core 68 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 188 maps to socket 0 core 69 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 189 maps to socket 0 core 69 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 190 maps to socket 0 core 69 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 191 maps to socket 0 core 69 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 248 maps to socket 0 core 70 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 249 maps to socket 0 core 70 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 250 maps to socket 0 core 70 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 251 maps to socket 0 core 70 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 252 maps to socket 0 core 71 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 253 maps to socket 0 core 71 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 254 maps to socket 0 core 71 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 255 maps to socket 0 core 71 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 56 maps to socket 0 core 72 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 57 maps to socket 0 core 72 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 58 maps to socket 0 core 72 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 59 maps to socket 0 core 72 thread 3
OMP: Info #171: KMP_AFFINITY: OS proc 60 maps to socket 0 core 73 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 61 maps to socket 0 core 73 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 62 maps to socket 0 core 73 thread 2
OMP: Info #171: KMP_AFFINITY: OS proc 63 maps to socket 0 core 73 thread 3
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2972 thread 0 bound to OS proc set 0
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23080 thread 1 bound to OS proc set 1
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23444 thread 2 bound to OS proc set 2
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 18784 thread 12 bound to OS proc set 68
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26184 thread 4 bound to OS proc set 4
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20876 thread 5 bound to OS proc set 5
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28512 thread 6 bound to OS proc set 6
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28196 thread 7 bound to OS proc set 7
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15892 thread 8 bound to OS proc set 64
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15016 thread 9 bound to OS proc set 65
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 7112 thread 10 bound to OS proc set 66
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 7636 thread 11 bound to OS proc set 67
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23072 thread 193 bound to OS proc set 169
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6620 thread 204 bound to OS proc set 52
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 17264 thread 215 bound to OS proc set 127
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21020 thread 15 bound to OS proc set 71
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14520 thread 16 bound to OS proc set 128
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25840 thread 17 bound to OS proc set 129
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21124 thread 18 bound to OS proc set 130
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28340 thread 19 bound to OS proc set 131
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26336 thread 20 bound to OS proc set 132
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20212 thread 21 bound to OS proc set 133
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15460 thread 22 bound to OS proc set 134
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22592 thread 23 bound to OS proc set 135
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22736 thread 24 bound to OS proc set 192
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2976 thread 25 bound to OS proc set 193
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20816 thread 26 bound to OS proc set 194
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19116 thread 27 bound to OS proc set 195
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26864 thread 28 bound to OS proc set 196
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27140 thread 29 bound to OS proc set 197
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25008 thread 30 bound to OS proc set 198
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 18340 thread 31 bound to OS proc set 199
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25304 thread 32 bound to OS proc set 8
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23684 thread 33 bound to OS proc set 9
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28284 thread 34 bound to OS proc set 10
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22612 thread 35 bound to OS proc set 11
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26780 thread 36 bound to OS proc set 12
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6224 thread 37 bound to OS proc set 13
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26148 thread 38 bound to OS proc set 14
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26272 thread 39 bound to OS proc set 15
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 9076 thread 40 bound to OS proc set 72
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23612 thread 41 bound to OS proc set 73
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28072 thread 42 bound to OS proc set 74
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19688 thread 43 bound to OS proc set 75
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 9276 thread 44 bound to OS proc set 76
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22904 thread 45 bound to OS proc set 77
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 10380 thread 46 bound to OS proc set 78
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26192 thread 47 bound to OS proc set 79
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23320 thread 48 bound to OS proc set 136
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27100 thread 49 bound to OS proc set 137
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 4680 thread 50 bound to OS proc set 138
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2928 thread 51 bound to OS proc set 139
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23764 thread 52 bound to OS proc set 140
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24168 thread 53 bound to OS proc set 141
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26796 thread 54 bound to OS proc set 142
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20012 thread 55 bound to OS proc set 143
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15668 thread 56 bound to OS proc set 200
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23656 thread 57 bound to OS proc set 201
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26388 thread 58 bound to OS proc set 202
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23560 thread 59 bound to OS proc set 203
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26848 thread 60 bound to OS proc set 204
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27768 thread 61 bound to OS proc set 205
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 4612 thread 62 bound to OS proc set 206
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 10284 thread 63 bound to OS proc set 207
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 7936 thread 64 bound to OS proc set 80
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24108 thread 65 bound to OS proc set 81
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25396 thread 66 bound to OS proc set 82
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22208 thread 67 bound to OS proc set 83
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23952 thread 68 bound to OS proc set 84
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 8960 thread 69 bound to OS proc set 85
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19408 thread 70 bound to OS proc set 86
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2604 thread 71 bound to OS proc set 87
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6312 thread 72 bound to OS proc set 144
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28648 thread 73 bound to OS proc set 145
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26144 thread 74 bound to OS proc set 146
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 9080 thread 75 bound to OS proc set 147
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25972 thread 76 bound to OS proc set 148
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23784 thread 77 bound to OS proc set 149
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 5004 thread 78 bound to OS proc set 150
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25184 thread 79 bound to OS proc set 151
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28136 thread 80 bound to OS proc set 208
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19912 thread 81 bound to OS proc set 209
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24112 thread 82 bound to OS proc set 210
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 5576 thread 83 bound to OS proc set 211
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19396 thread 84 bound to OS proc set 212
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 3644 thread 85 bound to OS proc set 213
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19556 thread 86 bound to OS proc set 214
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 5316 thread 87 bound to OS proc set 215
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6640 thread 88 bound to OS proc set 16
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25656 thread 89 bound to OS proc set 17
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28492 thread 90 bound to OS proc set 18
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28472 thread 91 bound to OS proc set 19
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 3460 thread 92 bound to OS proc set 20
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19612 thread 93 bound to OS proc set 21
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6112 thread 94 bound to OS proc set 22
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23884 thread 95 bound to OS proc set 23
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2376 thread 96 bound to OS proc set 88
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26296 thread 97 bound to OS proc set 89
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14056 thread 98 bound to OS proc set 90
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 17384 thread 99 bound to OS proc set 91
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14600 thread 100 bound to OS proc set 92
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6176 thread 101 bound to OS proc set 93
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 4864 thread 102 bound to OS proc set 94
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21128 thread 103 bound to OS proc set 95
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22584 thread 104 bound to OS proc set 152
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23680 thread 105 bound to OS proc set 153
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28640 thread 106 bound to OS proc set 154
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 10600 thread 107 bound to OS proc set 155
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21860 thread 108 bound to OS proc set 156
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26200 thread 109 bound to OS proc set 157
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26556 thread 110 bound to OS proc set 158
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27312 thread 111 bound to OS proc set 159
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 8636 thread 112 bound to OS proc set 216
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26300 thread 113 bound to OS proc set 217
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6416 thread 114 bound to OS proc set 218
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24512 thread 115 bound to OS proc set 219
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15264 thread 116 bound to OS proc set 220
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24088 thread 117 bound to OS proc set 221
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24508 thread 118 bound to OS proc set 222
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27512 thread 119 bound to OS proc set 223
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24272 thread 120 bound to OS proc set 24
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25000 thread 121 bound to OS proc set 25
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21072 thread 122 bound to OS proc set 26
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15636 thread 123 bound to OS proc set 27
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27908 thread 124 bound to OS proc set 28
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20260 thread 125 bound to OS proc set 29
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2352 thread 126 bound to OS proc set 30
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19680 thread 127 bound to OS proc set 31
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27484 thread 128 bound to OS proc set 96
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 8968 thread 129 bound to OS proc set 97
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26092 thread 130 bound to OS proc set 98
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 17120 thread 131 bound to OS proc set 99
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 3200 thread 132 bound to OS proc set 100
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20808 thread 133 bound to OS proc set 101
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23076 thread 134 bound to OS proc set 102
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26568 thread 135 bound to OS proc set 103
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14392 thread 136 bound to OS proc set 160
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22524 thread 137 bound to OS proc set 161
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14780 thread 138 bound to OS proc set 162
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 5556 thread 139 bound to OS proc set 163
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26140 thread 140 bound to OS proc set 164
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 18388 thread 141 bound to OS proc set 165
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2392 thread 142 bound to OS proc set 166
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19852 thread 143 bound to OS proc set 167
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 1388 thread 144 bound to OS proc set 224
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22084 thread 145 bound to OS proc set 225
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 5432 thread 146 bound to OS proc set 226
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26056 thread 147 bound to OS proc set 227
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20472 thread 148 bound to OS proc set 228
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25308 thread 149 bound to OS proc set 229
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 18828 thread 150 bound to OS proc set 230
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15752 thread 151 bound to OS proc set 231
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25688 thread 152 bound to OS proc set 32
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14872 thread 153 bound to OS proc set 33
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23776 thread 154 bound to OS proc set 34
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14540 thread 155 bound to OS proc set 35
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15268 thread 156 bound to OS proc set 36
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22184 thread 157 bound to OS proc set 37
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26544 thread 158 bound to OS proc set 38
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24880 thread 159 bound to OS proc set 39
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 7844 thread 160 bound to OS proc set 104
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19204 thread 161 bound to OS proc set 105
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19008 thread 162 bound to OS proc set 106
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28488 thread 163 bound to OS proc set 107
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26724 thread 164 bound to OS proc set 108
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25740 thread 165 bound to OS proc set 109
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 16276 thread 166 bound to OS proc set 110
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6568 thread 167 bound to OS proc set 111
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25580 thread 168 bound to OS proc set 232
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21492 thread 169 bound to OS proc set 233
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14688 thread 170 bound to OS proc set 234
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23228 thread 171 bound to OS proc set 235
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24744 thread 172 bound to OS proc set 236
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19568 thread 173 bound to OS proc set 237
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22392 thread 174 bound to OS proc set 238
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22896 thread 175 bound to OS proc set 239
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14424 thread 176 bound to OS proc set 40
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2388 thread 177 bound to OS proc set 41
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21384 thread 178 bound to OS proc set 42
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 3336 thread 179 bound to OS proc set 43
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20704 thread 180 bound to OS proc set 44
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 4824 thread 181 bound to OS proc set 45
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27044 thread 182 bound to OS proc set 46
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 7312 thread 183 bound to OS proc set 47
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2612 thread 184 bound to OS proc set 112
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25572 thread 185 bound to OS proc set 113
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23584 thread 186 bound to OS proc set 114
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26940 thread 187 bound to OS proc set 115
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22648 thread 188 bound to OS proc set 116
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 15512 thread 189 bound to OS proc set 117
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21448 thread 190 bound to OS proc set 118
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24556 thread 191 bound to OS proc set 119
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 8948 thread 192 bound to OS proc set 168
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23692 thread 3 bound to OS proc set 3
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24028 thread 194 bound to OS proc set 170
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19216 thread 195 bound to OS proc set 171
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22536 thread 196 bound to OS proc set 172
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 8596 thread 197 bound to OS proc set 173
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22344 thread 198 bound to OS proc set 174
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14052 thread 199 bound to OS proc set 175
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24288 thread 200 bound to OS proc set 48
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2868 thread 201 bound to OS proc set 49
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 1560 thread 202 bound to OS proc set 50
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23600 thread 203 bound to OS proc set 51
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 1516 thread 13 bound to OS proc set 69
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26712 thread 205 bound to OS proc set 53
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19044 thread 206 bound to OS proc set 54
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21824 thread 207 bound to OS proc set 55
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 19380 thread 208 bound to OS proc set 120
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 5392 thread 209 bound to OS proc set 121
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 28476 thread 210 bound to OS proc set 122
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26536 thread 211 bound to OS proc set 123
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 9608 thread 212 bound to OS proc set 124
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24540 thread 213 bound to OS proc set 125
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25388 thread 214 bound to OS proc set 126
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 4804 thread 14 bound to OS proc set 70
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 20300 thread 216 bound to OS proc set 176
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27628 thread 217 bound to OS proc set 177
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23236 thread 218 bound to OS proc set 178
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22876 thread 219 bound to OS proc set 179
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6292 thread 220 bound to OS proc set 180
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25504 thread 221 bound to OS proc set 181
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25700 thread 222 bound to OS proc set 182
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25896 thread 223 bound to OS proc set 183
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26976 thread 224 bound to OS proc set 240
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23872 thread 225 bound to OS proc set 241
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26816 thread 226 bound to OS proc set 242
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 8820 thread 227 bound to OS proc set 243
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 9332 thread 228 bound to OS proc set 244
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25792 thread 229 bound to OS proc set 245
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25604 thread 230 bound to OS proc set 246
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 18324 thread 231 bound to OS proc set 247
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 26552 thread 232 bound to OS proc set 184
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27172 thread 233 bound to OS proc set 185
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 16352 thread 234 bound to OS proc set 186
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24336 thread 235 bound to OS proc set 187
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25140 thread 236 bound to OS proc set 188
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2552 thread 237 bound to OS proc set 189
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 9012 thread 238 bound to OS proc set 190
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24340 thread 239 bound to OS proc set 191
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24208 thread 240 bound to OS proc set 248
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 21276 thread 241 bound to OS proc set 249
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27820 thread 242 bound to OS proc set 250
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 22232 thread 243 bound to OS proc set 251
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23312 thread 244 bound to OS proc set 252
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27280 thread 245 bound to OS proc set 253
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 27432 thread 246 bound to OS proc set 254
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 3012 thread 247 bound to OS proc set 255
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 2940 thread 248 bound to OS proc set 56
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 25976 thread 249 bound to OS proc set 57
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24816 thread 250 bound to OS proc set 58
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 18084 thread 251 bound to OS proc set 59
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 24116 thread 252 bound to OS proc set 60
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 6884 thread 253 bound to OS proc set 61
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 23176 thread 254 bound to OS proc set 62
OMP: Info #252: KMP_AFFINITY: pid 23804 tid 14132 thread 255 bound to OS proc set 63

OpenMP does use all 4 Processor Groups on my KNL system

Jim Dempsey

 

0 Kudos
Highlighted
Black Belt Retired Employee
75 Views

True - the use of GetLogicalProcessorInformation gets info on the current processor group only. There was a later addition of GetLogicalProcessorInformationEx that can deal with processor groups. Let me see if I can rework the example to do this, though I don't have access to a system with multiple groups.

0 Kudos
Highlighted
Beginner
66 Views

ProcessorInfo:

There are compiler warnings and the code stops before generating any meaningful results.

Build Log
 	 	

 Build started: Project: ProcessorInfo, Configuration: Release|x64 

Output
 	 	

Deleting intermediate files and output files for project 'ProcessorInfo', configuration 'Release|x64'.
Compiling with Intel(R) Visual Fortran Compiler 19.1.3.311 [Intel(R) 64]...
ifort /nologo /MP /O3 /heap-arrays1000 /arch:AVX /Qopenmp /fpscomp:general /stand:f18 /Qdiag-disable:5268,10182,5462 /warn:all /fpe:0 /fpconstant /module:"x64\Release/" /object:"x64\Release/" /Fd"x64\Release\vc160.pdb" /traceback /check:none /libs:qwin /Qmkl:sequential /c /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX64\x64" /Qm64 "D:\ProcessorInfo\ProcessorInfo.f90"
D:\ProcessorInfo\ProcessorInfo.f90(36): warning #6463: This is not a derived type name.   [T_SYSTEM_LOGICAL_PROCESSOR_INFORMATION]
    type(T_SYSTEM_LOGICAL_PROCESSOR_INFORMATION), allocatable, dimension(:) :: buffer
---------^
D:\ProcessorInfo\ProcessorInfo.f90(37): warning #6463: This is not a derived type name.   [T_SYSTEM_LOGICAL_PROCESSOR_INFORMATION]
    type(T_SYSTEM_LOGICAL_PROCESSOR_INFORMATION) :: buffer_element ! For size only
---------^
D:\ProcessorInfo\ProcessorInfo.f90(106): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
        select case (buffer(i)%Relationship)
---------------------^
D:\ProcessorInfo\ProcessorInfo.f90(112): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
            if (buffer(i)%Flags == 1) then
----------------^
D:\ProcessorInfo\ProcessorInfo.f90(116): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
                processorCoreCount = processorCoreCount + popcnt(buffer(i)%processorMask)
-----------------------------------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(120): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
            logicalProcessorCount = logicalProcessorCount + popcnt(buffer(i)%processorMask)
-------------------------------------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(124): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
            if (buffer(i)%Cache%Level > 0 .and. buffer(i)%Cache%Level <= 3) then
----------------^
D:\ProcessorInfo\ProcessorInfo.f90(124): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
            if (buffer(i)%Cache%Level > 0 .and. buffer(i)%Cache%Level <= 3) then
------------------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(125): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
                processorCacheCount(buffer(i)%Cache%Level) = processorCacheCount(buffer(i)%Cache%Level) + 1
------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(125): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
                processorCacheCount(buffer(i)%Cache%Level) = processorCacheCount(buffer(i)%Cache%Level) + 1
---------------------------------------------------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(127): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
                print *, "Invalid processor cache level ", buffer(i)%Cache%Level
-----------------------------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(135): warning #6135: DEC Fortran field is being referenced as if it were a Fortran 2018 derived type; use '.' and not '%' as the separator.   [BUFFER]
            print *, "Unrecognized relationship code ", buffer(i)%Relationship
--------------------------------------------------------^
D:\ProcessorInfo\ProcessorInfo.f90(45): remark #7712: This variable has not been used.   [NLPI]
    integer :: nlpi, lpi_element_length, i
---------------^
D:\ProcessorInfo\ProcessorInfo.f90(46): remark #7712: This variable has not been used.   [DUMPBUF]
    integer(1), pointer, dimension(:) :: dumpbuf
-----------------------------------------^

Linking...
Link /OUT:"x64\Release/ProcessorInfo.exe" /INCREMENTAL:NO /MANIFEST /MANIFESTFILE:"x64\Release\ProcessorInfo.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /SUBSYSTEM:WINDOWS /HEAP:100000000,10000000 /STACK:100000000,10000000 /IMPLIB:"D:\ProcessorInfo\x64\Release\ProcessorInfo.lib" -qm64 /qoffload-ldopts="-mkl=sequential" "x64\Release/ProcessorInfo.obj"
Link: executing 'link'
Microsoft (R) Incremental Linker Version 14.27.29111.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/OUT:x64\Release/ProcessorInfo.exe 
/INCREMENTAL:NO 
/MANIFEST 
/MANIFESTFILE:x64\Release\ProcessorInfo.exe.intermediate.manifest 
"/MANIFESTUAC:level='asInvoker' uiAccess='false'" 
/SUBSYSTEM:WINDOWS 
/HEAP:100000000,10000000 
/STACK:100000000,10000000 
/IMPLIB:D:\ProcessorInfo\x64\Release\ProcessorInfo.lib 
x64\Release/ProcessorInfo.obj

Embedding manifest...
mt.exe /nologo /outputresource:"D:\ProcessorInfo\x64\Release\ProcessorInfo.exe;#1" /manifest "x64\Release\ProcessorInfo.exe.intermediate.manifest"

ProcessorInfo - 0 error(s), 12 warning(s)

 

0 Kudos
Highlighted
Black Belt Retired Employee
62 Views

Don't compile with /warn.

0 Kudos
Highlighted
Beginner
56 Views

Warnings don't affect the EXE. Results are attached.

0 Kudos
Highlighted
Black Belt Retired Employee
52 Views

Interesting - I'll take a look.

I was looking at the Microsoft documentation of GetLogicalProcessorInformationEx and it included this bit which might explain the 32-bit results you saw

If a 32-bit process running under WOW64 calls this function on a system with more than 64 processors, some of the processor affinity masks returned by the function may be incorrect. This is because the high-order DWORD of the 64-bit affinity masks for processors 32 through 63 are incorrectly represented as duplicates of the masks for processors 0 through 31. In addition, the affinity masks for processors 32 through 63 are incorrectly represented as duplicates of the masks for processors 0 through 31. In addition, the sum of all per-group ActiveProcessorCount and MaximumProcessorCount values reported in PROCESSOR_GROUP_INFO structures may exclude some active logical processors.

When this function is called with a relationship type of RelationProcessorCore, it returns a PROCESSOR_RELATIONSHIP structure for every active processor core in every processor group in the system. This is by design, because an unaffinitized 32-bit thread can run on any logical processor in a given group, including processors 32 through 63. A 32-bit caller can use the total count of PROCESSOR_RELATIONSHIP structures to determine the actual number of active processor cores on the system. However, the affinity of a 32-bit thread cannot be explicitly set to logical processor 32 through 63 of any processor group.

0 Kudos
Highlighted
Beginner
46 Views

I stopped creating 32-bit programs over a decade ago and stopped installing the 32-bit IVF compilers at that time. The results are 64-bit EXE.

0 Kudos
Highlighted
32 Views

T_SYSTEM_LOGICAL_PROCESSOR_INFORMATION is defined in ifwinty.f90 (Windows only systems).

Is your copy of the program missing "use kernel32" (which has use ifwinty)?

Steve,

In looking at kernel32.f90, at the top

!DEC$ IF .NOT. DEFINED (KERNEL32_ )
!DEC$ DEFINE xKERNEL32_

Note the "x" on the DEFINE, but missing on the DEFINED??

 

Jim Dempsey

0 Kudos
Highlighted
30 Views

Steve,

ifwinty.f90 defines T_SYSTEM_LOGICAL_PROCESSOR_INFORMATION as a STRUCTURE, not TYPE "because of the union". I've used UNION in TYPE's before. Has something changed??

Jim Dempsey

0 Kudos
Highlighted
Beginner
23 Views

I sent the build log yesterday which has all the compiler and linker settings for you to debug/modify and test your code before I build it and run it and generate results for you.

0 Kudos
Highlighted
18 Views

The Capture.png may be indicative of a difference in the CPUID implementation (leaf) difference between an AMD CPU and Intel CPU.

It may be that the kernel32.dll expects/uses the Intel CPUID specification as opposed to the AMD.

https://www.amd.com/system/files/TechDocs/25481.pdf

https://software.intel.com/content/dam/develop/external/us/en/documents-tps/325462-sdm-vol-1-2abcd-3... page 3-198

Note, due to different leaf (table) layouts, when querying an invalid leaf, then 0 is returned as all four results.

Jim Dempsey

0 Kudos