Process in offload mode affects other MIC cards

[kadir]

I have a system with 4 MIC cards.

When I start a process in offload mode on mic0, one core of other mic cards is occupied with coi_daemon process. Why?

Unfortunately, I get high variances in timing when other mic cards are used by other users.

mic0:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                            
114977 micuser   20   0 17.7g 1.1g 1.1g S 9999 15.0 617:24.73 /tmp/coi_procs/1/114977/Spmm-O1?iccoutvPt2a7
  4787 micuser   20   0  185m  12m 1576 S   73  0.2   1728:18 /usr/bin/coi_daemon --coiuser=micuser

mic1:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                            
  4787 micuser   20   0  185m  11m 1576 S   70  0.2   1723:23 /usr/bin/coi_daemon --coiuser=micuser
 93060 micuser   20   0  355m 7152 4744 S    0  0.1   0:00.13 /tmp/coi_procs/2/93060/Spmm-O1?iccoutvPt2a7

mic3:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                            
  4787 micuser   20   0  185m  11m 1576 S   64  0.2   1713:04 /usr/bin/coi_daemon --coiuser=micuser
105339 micuser   20   0  355m 7152 4744 S    0  0.1   0:00.13 /tmp/coi_procs/4/105339/Spmm-O1?iccoutvPt2a7

[Ravi_N_Intel · Accepted solution]

The default behavior is to start offload process on all cards on encountering the 1st offload.
Use export OFFLOAD_INIT=on_offload
See below for different setting.
Ravi

OFFLOAD_INIT

The environment variable specifies a hint for offload runtime when it should do initialization of MIC devices.

Supported values:

on_start:                    All available devices are initialized before entering main.

on_offload:               Device initialization is performed right before the first offload to it. Initialization is done only on the MIC device which handles offload.

on_offload_all:       All available MIC devices are initialized right before the first offload in a program.

Default is on_offload_all (for backward compatibility).