How to Reduce CAL (Function Call Interrupts ) on x86_64 architectures in /proc/interrupts

[Kumar__Satish]

Hi all,

I am trying to run DPDK application on x86_64 Machine, On this process trying to do core isolation and assign one application to isolated Core.

While doing Core Isolation, achieved "Interrupt count increment stop on isolated core"  in all interrupts except CAL (Function Call Interrupts)

Searched in google, but did not get enough data on CAL

While checking the CAL in the Books: Understanding Linux Kernel & Professional Linux Kernel Archiecture , did not get any info for CAL Interrupts generated.

Can you guys please explain reason to generate CAL and to stop/control  for the same in x86_64 (IA64 or AMD64 ISA).

Thanks & Regards

Satish.G

[Cui__Shujie]

Hi Kumar,

I have the same question.

Moreover, in my case, the Local timer interrupts are still increasing on isolated cores.

How do you stop that?

Thanks & Regards,

Shujie

[Richard_Nutman]

Even with NOHZ Full mode, you will still get timer interrupts every second.

There is a patch I believe that removed this, but I can't find it atm.  Sorry this is not more help.

[Cui__Shujie]

It seems in Kernel 4.19 and 4.20, the residual 1HZ tick can be removed in NOHZ Full mode when running a task on the CPU.

I tried that, but the 1 HZ tick is still there.  I am thinking maybe I missed some steps.

Here are what I did:

 1. Set  CONFIG_NOHZ_FULL=y,   CONFIG_RCU_FAST_NO_HZ=y, CONFIG_RCU_NOCB_CPU=y

 2. Set boot parameters: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=nohz,domain,1-3 nohz=on nohz_full=1-3  rcu_nocbs=1-3 intel_pstate=disable irqaffinity=0 nmi_watchdog=0 nosoftlockup

3. Modify /sys/devices/virtual/workqueue/cpumask to 1

4. systemctl stop irqbalance

5. Turn off turbo mode, P state, and hythreading in BIOS

Has anybody ever tried this? What did I miss?

[Richard_Nutman]

That's similar to what we did on 3.10 kernel, but just left the 1Hz timer in there. We left turbo mode on, hyperthreading on, and Pstate to 1 to avoid the idle spin.

In Centos (RedHat) we also had to change the realtime timer scheduling to 100%, and set thread scheduling to FIFO with priority 99.

/proc/sys/kernel/sched_rt_runtime_us to -1

Have you tried executing irqbalance with a mask excluding your NOHZ cores after you stop the service ?

[Cui__Shujie]

how did you change the realtime timer scheduling to 100%?

[Richard_Nutman]

Set 

/proc/sys/kernel/sched_rt_runtime_us to -1

and in your code, change your thread scheduling like this;

int policy;
struct sched_param param;

pthread_getschedparam(pthread_self(), &policy, &param);
param.__sched_priority = 99;
pthread_setschedparam(pthread_self(), SCHED_FIFO, &param);

[Cui__Shujie1]

Hi Kumar,

Have you found the solution to reduce the CAL interrupts? I have the same issue now.

Best,

Shujie

[Ivanov__Stanislav]

This topic needs to be raised :) - the first line of the google search for "Function Call Interrupts" and there is no solution.

I have the same problem - some jitter caused by CAL and I  can't find manual how to tune it.

[ADIBACCO1]

Anyone has an updated on preventing CAL interrupts to be fired onto isolated cores ?

Even using a realtime kernel (Ubuntu for example) the CAL interrupts are still called on all cores causing a big (7-10 usec) interruption on every task that is executing on a given core.