High CPU usage for 1st 1 or 2 hours on RHEL 7.6/7.7/7.8

[Soumit]

Hello All,

  We have multicore/multithread application media-intensive software developed with IPP and we are using intel compiler with AVX optimization enabled. 

We also have a JDK ( 1.8.0_181-b13) installed and a Java code to handle the signaling and control operations of media applications running on C/C++. This is done through JNI running on other cores. 

On Rhel 6 our application runs with constant CPU usage at a fixed load testing. But on Rhel 7 we are seeing for 1st one hour or so (sometimes it takes 2 hours) the CPU usage is high. And post that without any change the CPU usage is coming down by 5-10%. 

Initially, we thought it is caused by "TUNED" but even uninstalling tuned.  and we are using the default rhel kernel (3.10.0-957.el7.x86_64). 

• We are using  Intel Xeon CPU[Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz ] with 2 Sockets and multiple CPUS[ ProLiant DL380 Gen10 ].
• Our product/application is running on a bare-metal server. Not in any VM or KVM.
• The OS is Vanilla HOE RHEL 7.X . [Can be omitted]
• We are using numcatl cores and the product is pinned on 3 cores in NUMA 1.
• The cpupower.service is not running in our server.
• The Intel P_state and C_State drivers are not installed.
• Turbo Boost is disabled.
• The nohz_full and CPU isolation is disabled in our setup
• The irqbalance service is running in default mode.
• The /proc/sys/vm/ data is restored to default OS values after the uninstallation of TUNED.
• Spectre Meltdown patches are applied.

Any clue or help will be highly appreciated. 

[Emeth_O_Intel · Accepted solution]

Hello Soumit,

I was reviewing your case and in order to provide you the most accurate information about this matter, my best recommendation will be to create a thread in the following forum from Intel called: Software Tuning, Performance Optimization & Platform Monitoring (https://community.intel.com/t5/Software-Tuning-Performance/bd-p/software-tuning-perf-optimization.)

They will be more than happy to assist you on this specific matter.

Best regards,

Emeth O.

Intel Server Specialist.

[Soumit]

Thanks,

 Posted under 

• Software Tuning, Performance Optimization & Platform Monitoring