I have an Intel Arria 10 SoC FPGA system with 5.4.104-lts Linux built with Yocto 3.3.1 and Poky. The installed FPGA image is doing nothing more than making interrupts to an UIO device, 50 times a sec.
The UIO device is defined in the device tree like this:
tx_trig_irq {
/* /dev/uio4 */
compatible = "test_irq", "generic-uio";
interrupts = < 0 0x16 IRQ_TYPE_EDGE_RISING >;
interrupt-parent = <&intc>;
}; The interrupts arrive in the OS correctly.
Here is the simple program which handles the interrupts, there is no other software running.
#include <iostream> #include <cmath> #include <fcntl.h> #include <unistd.h> typedef struct { int txTrigIrqFd; } TX_IRQ_HANDLE_S; bool wait_for_irq (TX_IRQ_HANDLE_S* pHandle) { if (!pHandle || pHandle->txTrigIrqFd < 0) { std::cout << "ERROR: handle"; return false; } uint32_t info = 1; ssize_t nb = write(pHandle->txTrigIrqFd, &info, sizeof(info)); if (nb != (ssize_t)sizeof(info)) { std::cout << "ERROR: writing"; return false; } nb = read(pHandle->txTrigIrqFd, &info, sizeof(info)); if (nb == (ssize_t)sizeof(info)) { return true; } return false; } int main(int argc, char* argv[]) { //Init IRQ TX_IRQ_HANDLE_S* tx_irq_handle = NULL; tx_irq_handle = (TX_IRQ_HANDLE_S*) ( malloc (sizeof (TX_IRQ_HANDLE_S)) ); if (!tx_irq_handle) { std::cout << "irq init failed"; return 1; } tx_irq_handle->txTrigIrqFd = open ("/dev/uio4", (O_RDWR)); if (tx_irq_handle->txTrigIrqFd < 0) { free (tx_irq_handle); std::cout << "irq init failed"; return 1; } if (!tx_irq_handle) { std::cout << "irq init failed"; return 1; } // Do IRQ while (true) { auto status = wait_for_irq(tx_irq_handle); } std::cout << "Stopped.\n"; return 0; }
It handles the interrupts correctly, I also instrumented the code and see the interrupt handling's timing with Tracy, everything works as intended.
However there is a strange CPU anomaly when starting this simple irq handling program:
- There are ~5 sec long spikes in CPU usage every minute, periodically
- When I reduce the number of interrupts coming from the FPGA to 25, the 1 minute period doubles to 2 minutes.
- When stopping the FPGA to send interrupts, and waiting some time (for eg. 1 minute), then resuming again, the priodicity of the spikes continues from the last state when the FPGA generated interrupts. So when pausing the interrupts in the midde of a spike for some minutes (the cpu usage goes to ~0%), after the resume, the CPU usage continues from the middle of the spike.
- The characteristics of the CPU usage graph can be tuned by doing work between the waiting for the interrupts, so the spikes can transform into a wave:
// Do IRQ while (true) { //Doing some work volatile double x = 0.0001; for (int i = 0; i < 20000; ++i) { x += std::sin(x) * std::cos(x); } auto status = wait_for_irq(tx_irq_handle); }
Here are some pictures:
Doing no additional work:
Doing some work:
Doing more work:
- I've already instrumented the UIO kernel driver and watched with dmesg how the interrupt handling goes in the kernel, but I found nothing suspicious.
- This extra, periodical CPU usage is not assigned to any process (watching with top or htop), the cpu usage of my test program is constant, however the total cpu usage shows this periodicity.
I can't imagine what is happening, I ran out of ideas. Do you have any suggestions what could cause this periodic cpu usage?
Thank you!
Link Copied
