Problem with reduced CPU clock after reboot on Intel C2758

[AMyhr]

I'm having a performance issue with our systems using the Intel Atom C2758:

The core clocks seems to be reduced after a reboot, and is then permanently stuck at this low clock speed until I do a power cycle.

Running "cpupower monitor" reports that the "Freq" is 2200 MHz after a power cycle and 1200 MHz after a reboot.

I have also confirmed this by using a small "for-loop" program, where the execution time approximately doubles after a reboot.

Looking at the MSR registers, the IA32_PERF_STS register is different between reboot and power cycle.

After a power cycle this register is set to 0xa6000000164a and after reboot it is set to 0xa60000000c1d.

Looking at the "BIOS Writers Guide (BWG), Volume 1 of 2 Rev 1.8" table 2-34, these bits is VID_SEL and BUS_RATIO_SEL, which is used to set the clock frequency for each core.

I suspected this could be something with the power control of the CPU and ran the Intel PCM tools (https://github.com/opcm/pcm, 202005 release).

This gave the following output when having high load on one core (same for power-cycle and reboot):

 Core   C-state distribution 11111111116666666666666666666666666666666666666666666666666666666666666666666666

 Package C-state distribution 00000000000000000000000000000000000000000000000000000000000000000000000000000000

I would expect the core to go into C0 state and not C1 state. Could this maybe be the cause of the problems?

I'm running Red Hat 7.4 on the system, but I see the same problems on Debian 9.

The systems is using coreboot v4.9 as BIOS, where the configuration is based on Intel Mohonpeak board.

Thanks in advance

Best Regards,

Andreas Myhre

[CarlosAM_INTEL]

Hello, @AMyhr​:

Thank you for contacting Intel Embedded Community.

We suggest verify if the situation persists using any of the Operating Systems listed on page 2 of the Intel(R) Atom(TM) Processor C2000 Product Family for Communications Infrastructure: Platform Brief document # 329404 that can be found in the following website:

https://cdrdv2.intel.com/v1/dl/getContent/329404

We are waiting for your confirmation.

Best regards,

@Mæcenas_INTEL​.

[AMyhr]

Hi Mæcenas_INTEL

Thanks for replying.

I downloaded the yocto mohonpeak-1.0-daisy-1.6.1 32 bit image and booted it from a USB drive.

Since the Yocto image is quite limited, I couldn't run the "for-loop" program used earlier.

But I read out "1200000" from "/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq", which indicates that the CPU cores is running at 1.2GHz

I also compressed 1GB file containing random data with the "tar" command and measured the time it took to finish.

On Red Hat 7, it took3.7 seconds and on the Yocto image used 6.6 seconds.

So it seems to have the same problem.

I also tried to test with Red Hat 8 instead of Red Hat 7, and I got a slightly different behaviour:

1. After a power cycle, the core clocks are set to 1200MHz (from "cpupower monitor").

  The MSR register IA32_PERF_STS is set to 0xa60000000c1d

2. Writing VID and BUS_RATIO values to IA32_PERF_CTL changes the frequency.

  For example: If I write 0x1854 to IA32_PERF_CTL, the frequency is set to 2400MHz. (IA32_PERF_STS changes to 0xa60000001853)

3. I can then lower the frequency back to 1200MHz by writing "0xc1d" to the IA32_PERF_CTL MSR register.(IA32_PERF_STS changes to 0xa60000000c1d)

4. But after lowering the frequency, I can't change the frequency at all.

  For example writing 0x1853 to IA32_PERF_CTL doesn't do anything (IA32_PERF_STS is 0xa60000000c1d and IA32_PERF_CTL is 0x1853).

It seems that the CPU is ignoring the values written to the IA32_PERF_CTL MSR register after I have changed the frequency twice.

Do you have any idea of why this is happening?

Best Regards,

Andreas

[CarlosAM_INTEL]

Hello, @AMyhr​:

Thanks for your reply.

In order to better understand your request, we want to address the following questions:

Could you please give top side markings pictures of the processors associate to this situation?

Could you please clarify if this situation happens on design made by you or by a third- party company? 

In case that it is a third-party device, could you please inform the name of the manufacturer, its model, the part number, and where its documentation is stated?

On the other hand, could you please let us know how many units of the project related to this circumstance have been manufactured? How many are affected? Could you please give the failure rate? Also, could you please list the sources that you have used to design it and if it has been verified by Intel?

Could you please let us know how and where you obtain the coreboot implementation associated to this condition?

We are waiting for your reply to these questions.

Best regards,

@Mæcenas_INTEL​.

[AMyhr]

Hi @Mæcenas_INTEL​ 

Here is a picture of the CPU used on our system.

This is a system designed by us.

Over 100 units have been produced, and the problem occurs on all the systems (100% failure rate). I therefore think it's probably some configuration/setup error happening (perhaps in the BIOS).

The design haven't had any verification by Intel, since it's based on the Mohon Peak Reference Board.

We're using coreboot 4.9 as BIOS, cloned from "https://review.coreboot.org/coreboot.git" git repository.

It's based on the mohonpeak peak board configuration, but with some changes related to our hardware.

I have also tested this on two evaluation systems: Supermicro with a C2358 (using AMI bios) and AVNet board with the C2338 (using coreboot),

and I can't reproduce the problem on these boards.

One significant difference is that C2358 and C2338 supports Intel Turbo Boost technology, but C2758 (the CPU used in our system) doesn't have any turbo support.

Could this have anything to do with the problem?

Best Regards,

Andreas

[CarlosAM_INTEL]

Hello, @AMyhr​:

Thanks for your update.

We suggest you check your Coreboot implementation through the channel mentioned as a reference in the following website:

https://ticket.coreboot.org/

By the way, you can also verify your schematics and layout implementation following the procedure stated at:

https://edc.intel.com/Tools/Design-Review/Default.aspx

Best regards,

@Mæcenas_INTEL​.