Direct GPU-FPGA DMA

Altera_Forum · ‎01-18-2015

Hello,

I am trying to implement direct DMA communication between a Nvidia Quadro and a Stratix V board over PCIe. Nvidia supports this with its GPUDirect RDMA technology. I have read through the code of Altera's PCIe driver and as far as I can tell I only need to get the physical address of the GPU memory and send it to the DMA controller, is that correct?

I already succeeded to transfer data from the FPGA to the GPU that way, however when I try the other direction (by swapping read and write addresses), the whole system freezes.

My kernel module code looks something like this (simplified & error checking omitted):


//pin gpu memory and get the physical address
nvidia_p2p_get_pages(0, 0, virtual_gpu_address, size, &page_table, free_callback, NULL);
unsigned long physical_gpu_address=page_table->pages->physical_address;
//set up a dma descriptor
unsigned int att_row=0;
size_t pcietxs_addr = (size_t) ACL_PCIE_TX_PORT 
                     | (att_row * ACL_PCIE_DMA_MAX_ATT_PAGE_SIZE) 
                     | (physical_gpu_address & (ACL_PCIE_DMA_MAX_ATT_PAGE_SIZE-1));
struct DMA_DESCRIPTOR dmadesc;
//direction from FPGA to GPU (works)
//dmadesc.read_address=    LO32(fpga_address);
//dmadesc.read_address_hi= HI32(fpga_address);
//dmadesc.write_address=   LO32(pcietxs_addr);
//dmadesc.write_address_hi=HI32(pcietxs_addr);
//direction from GPU to FPGA (crashes)
dmadesc.read_address=    LO32(pcietxs_addr);
dmadesc.read_address_hi= HI32(pcietxs_addr);
dmadesc.write_address=   LO32(fpga_address);
dmadesc.write_address_hi=HI32(fpga_address);
dmadesc.bytes = size;
dmadesc.burst = 0;
dmadesc.stride = 0x00010001;
dmadesc.control = (unsigned int)( ACL_PCIE_GET_BIT(DMA_DC_GO) 
                |  ACL_PCIE_GET_BIT(DMA_DC_EARLY_DONE_ENABLE)
                |  ACL_PCIE_GET_BIT(DMA_DC_TRANSFER_COMPLETE_IRQ_MASK)
                   );
//send data to the DMA controller
//functions from aclpci_dma.c
set_att_entry(aclpci, physical_gpu_address, att_row);
dma_desc_write (aclpci, 0, &dmadesc);

Does anyone have an idea what could be wrong?

Thanks,

Alexander

He4Forum · ‎01-25-2022

Hi Alexander,

I's working on direct DMA communication between GPU and FPGA recently. Is your program developed on Intel Quartus Prime? And how did you get the driver code? Could you please share the link to me?

Thanks,

Heron