Intel QuickData DMA Engine Data Transfers via PCIe NTB Port and Request for Detailed Documentation

[mpcukur]

We are experiencing issues with data transfers between two Intel XeonD-1500 processors using the Intel QuickData DMA engine in a system running VxWorks. Although the DMA engine reports successful completion of transfers, the data does not appear at the destination processor as expected. Additionally, we are seeking more detailed documentation on the DMA engine configuration.

Detailed Issue Description:

1. System Configuration:

   • Processors: Two Intel XeonD-1500 Series SoC
   • OS: VxWorks 7.0 SR640
   • PCIe Configuration: Data transfers are executed over the PCIe bus, facilitated through a PCIe switch using a Non-Transparent Bridge (NTB) port.

2. Observations:

   • Transfers between local memory regions on the same CPU using the DMA engine are successful.
   • Transfers from a CPU to an external endpoint (e.g., FPGA) over TB ports are successful.
   • Direct CPU writes (bypassing the DMA engine) to the destination CPU via the PCIe switch and NTB port are successful.

3. Problematic Scenario:

   • When using the Intel QuickData DMA engine to transfer data from one CPU to another over the NTB port, the DMA engine reports that it has successfully processed the descriptor list and completed the transfer. However, the expected data is not present at the destination CPU.

4. Specific Issue with BAR Configuration:

   • Our PCIe switch setup includes multiple Base Address Registers (BARs) used as message windows. Writing directly to BAR2 (using CPU instructions) successfully initiates and completes the data transfer to the second CPU. However, attempts to perform this operation via the DMA engine do not result in visible data at the destination, despite successful completion reports from the DMA engine.

5. Documentation Concern:

   • The available documentation for setting up and configuring the Intel QuickData DMA engine lacks specific details on the process sequence and register settings necessary for proper data transfer setup. This includes vague guidelines on the descriptor list format and a lack of clear instructions for the sequence of register writings for specific operational purposes using the DMA engine.

Request: We seek your guidance on resolving these discrepancies in DMA engine behavior, particularly concerning the NTB port and BAR configurations for cross-CPU data transfers. Additionally, we request more detailed and explicit documentation or guidance that clearly explains the required processes and register configurations for the Intel QuickData DMA engine, to better understand and utilize this technology in our systems.

[Diego_INTEL]

Hello @mpcukur,

Thank you for contacting Intel Embedded Community.

The Intel® Xeon® Processor D-1500 Series are not supported anymore, you can check this link:

https://www.intel.com/content/www/us/en/support/articles/000022396/processors.html

Regarding your issue, I looked into this and you can check the following:

-You can check that you are using the latest MRC.

-You can try checking the following White Paper:

https://www.intel.com/content/dam/doc/white-paper/quickdata-technology-software-guide-for-linux-paper.pdf

Also, you can check these documents in RDC that are still available:

#570950

#544040

#544041

As you are using VxWorks, you may try contacting Wind River support, maybe they can help further in this issue.

Best regards,

@Diego_INTEL