reaching max speed in xeon gold 6138

[Valjean__Jean]

Dear enthusiasts,

I have a supermicro x11dpi-n with two xeon gold 6128 which runs at a terrible speed 2GHz in comparison with the 3.7Ghz advertised.
Thinking that supermicro is too conservative (specialized on servers) I bought an asus ws c621 sage (advertised as the mother and father of overclocking).
With asus a small improvement came up, 2.3Ghz and intel_pstae/turbo_pct shows 61 (makes sense 3.7Ghz x 0.61 = 2.3GHz).
Playing with cpupower/acpi/intel_pstate I've managed another small step up - 3 core (out of 40 by rotation) to work at 2.7GHz.
So xeon gold is a big/fat fake (it is not running at 3.7GHz how it is advertised) and also asus a fake with a too less 61% from nominal speed, not even thinking at overclocking.
After spending consistent amount of money/time and reading articles on this subject, my questions are:
1 - how can I reach over 80% from the 3.7GHz advertised ? maybe modifying turbo_pct or other OS can go over, I've tried only Fedora 32 server.
(it is such a mess on acpi vs intel_pstate, bios on asus vs supermicro, everyone with it's own name and functions overlapiung)
(if xeon 6138 runs over 3GHz I can forget all the nerves/time/money spent and still bet on intel)
2 - to search for a crack in bios ?
I saw some comments saying that newer cpus have inside management of frecs and volts and cannot be modified from outside.
This is not true because speed on supermicro < speed on asus, so it is depending motherboard/bios.
Also I found for xeon E5-V3 there are cracks/hacks in bios (especially on chinese motherboards) to go at nominal speed, even overclocking.

temp is in asus below 35°C and in supermicro below 42°C

I have the same question for xeon E5-2680V4 in a gaming motherboard (but I didn't spend so much time/nerves/hopes in trying to reach the max 3.3GHz)

thank you

[Emeth_O_Intel]

Hello Valjean__Jean,

Thank you for contacting Intel® Xeon® Community Support.

I was checking the information you provided us, and I would like to request you the following commands:

"$ cat /proc/cpuinfo"

"$ lscpu"

"$ sudo lshw -C CPU"

Additionally, there is a utility call "Inxi tool", you can install the utility using the following command:

($ sudo dnf install inxi- #Fedora 22+ )

Then, in order to run the utility use the following command: "$ inxi -C".

Also, I would like to ask you the following questions:

1. Did the processor provide you a normal speed before?
2. Is it a brand-new processor or and old processor?
3. Did you buy the processor or it were included into a server system?
4. Which type utility are you using in order to test the processor speed?
5. What type of server applications are you trying to run in order to test the productivity of the processor?
6. Have you noticed any degradation in performance using the processor?

As soon as you have the outputs, and share with us the details in order to proceed with the analysis of the processor information in order to provide you the most accurate solution about it.

Regards,

Emeth O.

Intel® Server Specialist.

[Valjean__Jean]

Hello Emeth,

thank you for your supporting me, here are the answers.

1 - I bought the cpus two weeks ago, I cannot tell you how they behave before, but I think everything was fine.
At me, they just work at 2.3GHz or 2.0GHz depending on the motherboard (asus/supermicro) I used.
2 - they are second-hand cpus, I bought them from a company which are selling server parts in Ebay
3 - I bought the cpus as standalone/spare parts, no server system
4 - As test's application I use exactelly the application what I'll use the server for.
A C-code application compiled in gcc with all optimizations turned on, with a pool of mpi tasks in a small network, omp threads, intrinsics dispached by architecture (that means for xeon 6138 the application dispache AVX512 intrinsics).
The application fullfill the whoile capacity of the few servers in the network for few hours.
As speed/temp/load monitors I use linux Fedora commands >> sensors << and
something like >> watch -n.1 "cat /proc/cpuinfo | grep \"^[c]pu MHz\"" <<
5 - No degradation

The requested commands are attached.
You will see mitigation=off in grub/kernel, because I squeeze everthing from the processor.
I don't know if in the newer cpus is so important, but in old cpus I've managed 32% improvement on xeon E5-2450 and 47% improvement on xeon X5670.

Regards,
Jean

[Valjean__Jean]

Hello Emeth,

thank you for supporting me, here are asnwers and commands:

1 - I bought the cpus three weeks ago, I don't know how they behave before
2 - I bought the cpus from a company which are selling server parts in Ebay, so second-hand
3 - no server, cpus were stand_alone/spare_parts parts
4 - As a speed/temp monitors I use linux commands from lm_sensors and watch "cat /proc/cpuinfo | grep \"^[c]pu MHz\""
5 - I use as test exectelly the application what I will use the servor for because other tests are not relevant on this particular case
A C-code application compiled in gcc with all optimisatios turned on, with pool mpi tasks for a small network, omp threads and intrinsics dispachted by architecture, that means for xeon 6138 the application dispacht AVX512 intrinsics
The application/algorithm is carefully tunned for computation bound (I passed all the stages L2 bound, L1 bound, it points the roof line on the horizontal area ...)
The application fully fullfill the capacity of few servers for quite a good time.
6 - no degradation noticed

attached are the results of the commands, you will see a mitigation=off in the grub/kernel (isolated network)
I don't know if for newer cpus is so important, I know that in older cpu are very important (32% improvement in xeon E5-2450 and 47% in xeon X5670)
the results are how the server starts (no tune with cpupower)

Regards,

Jean

[Valjean__Jean]

it is strange ... excuse me for the delay, I've sent the answer in the morning, I received the confirmation on the private email, but now I see no answer, ... so I write it again.

Hello Emeth,

thank you for supporting me, here are asnwers and commands:

1 - I bought the cpus three weeks ago, I don't know how they behave before
2 - I bought the cpus from a company which are selling server parts in Ebay, so second-hand
3 - no server, stand_alone/spare_parts parts
4 - As a speed/temp monitors I use linux commands from lm_sensors and watch "cat /proc/cpuinfo | grep \"^[c]pu MHz\""
5 - I use as test exactly the application what I will use the servor for because other tests are not relevant on this particular case
A C-code application compiled in gcc, with pool mpi tasks for a small network, omp threads and intrinsics dispatched by architecture, that means for xeon 6138 the application dispatch AVX512 intrinsics
The application/algorithm is carefully tuned for computation bound (I passed all the stages L2 bound, L1 bound, it points the roof line on the horizontal area ...)
The application fully fullfill the capacity of few servers for quit a good time.
6 - no degradation noticed

attached are the results of the commands, you will see a mitigation=off in the grub/kernel.
I don't know if for newer cpus is so important, I know that in older cpu are very important - 32% improvement in xeon E5-2450 and 47% in xeon X5670.
the results are how the server starts (no tune with cpupower)

Regards,

Jean

[Emeth_O_Intel · Accepted solution]

Hello Valjean__Jean, 

Thank you for the information provided. 

I was reviewing the outputs provided and I did not see anything unusual. 

I would like to explain you some details about the processor base frequency of your Intel® Xeon® Gold 6138 Processor because it is for 2.00GHz and the Max Turbo Frequency of 3.70GHz. (Max turbo frequency is the maximum single core frequency at which the processor is capable of operating using Intel® Turbo Boost Technology and, if present, Intel® Thermal Velocity Boost. Frequency is typically measured in gigahertz (GHz), or billion cycles per second.)

So, The operation is dependent on headroom available in one or more cores. The amount of time the system spends in turbo boost mode depends on workload, environment, and platform design. 

Also, The Intel Turbo Boost Technology is enabled by default on supported processors. You can disable or enable the technology with a switch in the BIOS. No other user-controllable settings to change Intel Turbo Boost Technology operation are available. Once enabled, Intel Turbo Boost Technology works automatically under operating system control.

The board vendors normally enable Intel Turbo Boost Technology by default. You can typically enable or disable through a switch in the server board BIOS. Refer to your motherboard vendor for BIOS options to enable and disable Intel Turbo Boost Technology.

I was reviewing some posts about how to verify that the feature is enable in the BIOS of a SuperMicro Board and I found the following link: 

https://www.supermicro.com/support/faqs/faq.cfm?faq=21555#:~:text=Then%20Advanced%20%3E%3E%20CPU%20Configuration%20%3E,the%20setting%20are%20as%20follows.&text=5.,the%20Max%20Turbo%20mode%20speed.

My best recommendation will be to check the server BIOS configuration in order to confirm you have the Intel® Turbo Boost Technology enable, additionally if possible to contact SuperMicro Support in order to confirm with them if the Server Board Supports this feature. 

Have a wonderful day. 

Regards, 

Emeth O. 

Intel Server Specialist.

[Valjean__Jean]

thank you, now I have a direction where to look for a solution (the bios is the reason of lower freqs)

it is a bit strange that all cores are working at 2.3GHz but according to wikichip they must work on different speed

https://en.wikichip.org/wiki/intel/xeon_gold/6138

I will check also lower intrinsics (AVX2, AVX, SSE) on which speed they are working on xeon gold 6138

or to make another condition in the code, if I have data only for an AVX2 vector, to go on AVX2/SSE intrinsisc, maybe I can speed up the CPU on small data

Best Regards

Jean

[Emeth_O_Intel]

Hello Valjean__Jean, 

Alright, I hope with the information provided you will figure out the most accurate solution for this scenario.

If you have more question do not hesitate and contact us back and we will be more than happy to assist you. 

Have a wonderful day. 

Regards, 

Emeth O. 

Intel Server Specialist.