I am curious to know if DMA writes to coherent memory (allocated by dma_alloc_coherent) by PCIe device are re-ordered somehow e.g. PCH reorders memory stores for optimizations OR cache infrastructure flushes the cacheline to memory not in the order in which DMA writes are issued etc.
Setup:
1. Supermicro X11DPi-N (chipset – C621, CPU- Intel Xeon Gold 5118)
2. Coherent buffers are allocated in kernel using dma_alloc_cohernet and mapped to userspace using mmap(). Buffer is slotted in to 256 Bytes chunks.
3. Our proprietary device issues 256 Bytes single DMA write to these chunks over PCIe bus.
4. Software detects availability of new data by polling on first 2 bytes and last 8 bytes of single chunk.
5. After detecting the arrival of fresh data, CPU reads the 256 bytes chunk.
6. Sfence and lfence is used before reading the chunk.
Observation:
Reading 256 bytes coherent memory by CPU gives the unexpected (stale) data at random location within chunk (between of first and last Qword of chunk), however CPU can read the expected data after some time.
Question:
Can someone explain the Observed behaviour? And Suggest solution to take care of this.
