std::condition_wait sometimes waits forever (and doesn't time out)

Mark_L_1 · ‎06-10-2015

I've got code that works fine on our Xeon host system, but when it runs on our MICs, it hangs forever. I'm basically calling the C++ standard library "condition_variable::wait_for" with a timeout of 1000 milliseconds. The condition never becomes true, so this function should wait for 1000 ms and then return false. It works fine for a whileand prints out "timeout" every second, but after about 10 seconds, it just hangs entirely. However, if I "disturb" the MIC by either a key stroke (into a ssh session) or ping the MIC from the host, then it immediately "wakes up" and starts printing "timeout" again every second, but after another 10 seconds it hangs again.

     template< class Rep, class Period>
     bool wait(std::chrono::duration<Rep,Period> timeout)
     {
         std::unique_lock<std::mutex> lock(mutex_);

         if (condition_.wait_for(lock, timeout) == std::cv_status::timeout)
                return false;

         return true;
     }

Here's the calling loop

      
    while( wait(std::chrono::milliseconds(1000)))
    {
         cout << "timeout" << endl;
    }

It appears that the process is going to sleep. How do I prevent it from going to sleep? Do I need to change power management parameters? Is there a way of polling like this without hanging forever and using the default power management settings ?

Frances_R_Intel · ‎06-16-2015

As you say, you can disable power states to prevent the coprocessor from going to sleep. It may be sufficient to disable pc6, which requires a signal from the host before the coprocessor wakes up. However, it seems to me that the coprocessor should have set the C3WakeUpTimer and not gone into the pc6 state. I couldn't find any known problems, but if you will give me your host operating system version and the MPSS version you are running, I will try looking around some more.

Mark_L_1 · ‎06-16-2015

pa-poca$ micinfo
MicInfo Utility Log
Created Tue Jun 16 16:17:21 2015

System Info
HOST OS : Linux
OS Version : 3.10.0-123.el7.x86_64
Driver Version : 3.5-1
MPSS Version : 3.5

Host Physical Memory : 263941 MB

Device No: 0, Device Name: mic0

Version
Flash Version : 2.1.02.0391
SMC Firmware Version : 1.17.6900
SMC Boot Loader Version : 1.8.4326
uOS Version : 2.6.38.8+mpss3.5
Device Serial Number : ADKC32603169

Board
Vendor ID : 0x8086
Device ID : 0x225c
Subsystem ID : 0x7d95
Coprocessor Stepping ID : 2
PCIe Width : Insufficient Privileges
PCIe Speed : Insufficient Privileges
PCIe Max payload size : Insufficient Privileges
PCIe Max read req size : Insufficient Privileges
Coprocessor Model : 0x01
Coprocessor Model Ext : 0x00
Coprocessor Type : 0x00
Coprocessor Family : 0x0b
Coprocessor Family Ext : 0x00
Coprocessor Stepping : C0
Board SKU : C0PRQ-7120 P/A/X/D
ECC Mode : Enabled
SMC HW Revision : Product 300W Passive CS

Cores
Total No of Active Cores : 61
Voltage : 1007000 uV
Frequency : 1238095 kHz

Thermal
Fan Speed Control : N/A
Fan RPM : N/A
Fan PWM : N/A
Die Temp : 56 C

GDDR
GDDR Vendor : Samsung
GDDR Version : 0x6
GDDR Density : 4096 Mb
GDDR Size : 15872 MB
GDDR Technology : GDDR5
GDDR Speed : 5.500000 GT/s
GDDR Frequency : 2750000 kHz
GDDR Voltage : 1501000 uV

Device No: 1, Device Name: mic1

Version
Flash Version : 2.1.02.0391
SMC Firmware Version : 1.17.6900
SMC Boot Loader Version : 1.8.4326
uOS Version : 2.6.38.8+mpss3.5
Device Serial Number : ADKC32603342

Board
Vendor ID : 0x8086
Device ID : 0x225c
Subsystem ID : 0x7d95
Coprocessor Stepping ID : 2
PCIe Width : Insufficient Privileges
PCIe Speed : Insufficient Privileges
PCIe Max payload size : Insufficient Privileges
PCIe Max read req size : Insufficient Privileges
Coprocessor Model : 0x01
Coprocessor Model Ext : 0x00
Coprocessor Type : 0x00
Coprocessor Family : 0x0b
Coprocessor Family Ext : 0x00
Coprocessor Stepping : C0
Board SKU : C0PRQ-7120 P/A/X/D
ECC Mode : Enabled
SMC HW Revision : Product 300W Passive CS

Cores
Total No of Active Cores : 61
Voltage : 998000 uV
Frequency : 1238095 kHz

Thermal
Fan Speed Control : N/A
Fan RPM : N/A
Fan PWM : N/A
Die Temp : 56 C

GDDR
GDDR Vendor : Samsung
GDDR Version : 0x6
GDDR Density : 4096 Mb
GDDR Size : 15872 MB
GDDR Technology : GDDR5
GDDR Speed : 5.500000 GT/s
GDDR Frequency : 2750000 kHz
GDDR Voltage : 1501000 uV

Device No: 2, Device Name: mic2

Version
Flash Version : 2.1.02.0391
SMC Firmware Version : 1.17.6900
SMC Boot Loader Version : 1.8.4326
uOS Version : 2.6.38.8+mpss3.5
Device Serial Number : ADKC32603243

Board
Vendor ID : 0x8086
Device ID : 0x225c
Subsystem ID : 0x7d95
Coprocessor Stepping ID : 2
PCIe Width : Insufficient Privileges
PCIe Speed : Insufficient Privileges
PCIe Max payload size : Insufficient Privileges
PCIe Max read req size : Insufficient Privileges
Coprocessor Model : 0x01
Coprocessor Model Ext : 0x00
Coprocessor Type : 0x00
Coprocessor Family : 0x0b
Coprocessor Family Ext : 0x00
Coprocessor Stepping : C0
Board SKU : C0PRQ-7120 P/A/X/D
ECC Mode : Enabled
SMC HW Revision : Product 300W Passive CS

Cores
Total No of Active Cores : 61
Voltage : 998000 uV
Frequency : 1238095 kHz

Thermal
Fan Speed Control : N/A
Fan RPM : N/A
Fan PWM : N/A
Die Temp : 55 C

GDDR
GDDR Vendor : Samsung
GDDR Version : 0x6
GDDR Density : 4096 Mb
GDDR Size : 15872 MB
GDDR Technology : GDDR5
GDDR Speed : 5.500000 GT/s
GDDR Frequency : 2750000 kHz
GDDR Voltage : 1501000 uV
pa-poca$