Intel® Quartus® Prime Software
Intel® Quartus® Prime Design Software, Design Entry, Synthesis, Simulation, Verification, Timing Analysis, System Design (Platform Designer, formerly Qsys)

Direct GPU-FPGA DMA

Altera_Forum
Honored Contributor II
3,244 Views

Hello, 

 

I am trying to implement direct DMA communication between a Nvidia Quadro and a Stratix V board over PCIe. Nvidia supports this with its GPUDirect RDMA technology. I have read through the code of Altera's PCIe driver and as far as I can tell I only need to get the physical address of the GPU memory and send it to the DMA controller, is that correct? 

 

I already succeeded to transfer data from the FPGA to the GPU that way, however when I try the other direction (by swapping read and write addresses), the whole system freezes. 

 

My kernel module code looks something like this (simplified & error checking omitted): 

//pin gpu memory and get the physical address nvidia_p2p_get_pages(0, 0, virtual_gpu_address, size, &page_table, free_callback, NULL); unsigned long physical_gpu_address=page_table->pages->physical_address; //set up a dma descriptor unsigned int att_row=0; size_t pcietxs_addr = (size_t) ACL_PCIE_TX_PORT | (att_row * ACL_PCIE_DMA_MAX_ATT_PAGE_SIZE) | (physical_gpu_address & (ACL_PCIE_DMA_MAX_ATT_PAGE_SIZE-1)); struct DMA_DESCRIPTOR dmadesc; //direction from FPGA to GPU (works) //dmadesc.read_address= LO32(fpga_address); //dmadesc.read_address_hi= HI32(fpga_address); //dmadesc.write_address= LO32(pcietxs_addr); //dmadesc.write_address_hi=HI32(pcietxs_addr); //direction from GPU to FPGA (crashes) dmadesc.read_address= LO32(pcietxs_addr); dmadesc.read_address_hi= HI32(pcietxs_addr); dmadesc.write_address= LO32(fpga_address); dmadesc.write_address_hi=HI32(fpga_address); dmadesc.bytes = size; dmadesc.burst = 0; dmadesc.stride = 0x00010001; dmadesc.control = (unsigned int)( ACL_PCIE_GET_BIT(DMA_DC_GO) | ACL_PCIE_GET_BIT(DMA_DC_EARLY_DONE_ENABLE) | ACL_PCIE_GET_BIT(DMA_DC_TRANSFER_COMPLETE_IRQ_MASK) ); //send data to the DMA controller //functions from aclpci_dma.c set_att_entry(aclpci, physical_gpu_address, att_row); dma_desc_write (aclpci, 0, &dmadesc);  

 

Does anyone have an idea what could be wrong? 

 

Thanks, 

Alexander
0 Kudos
1 Reply
He4Forum
Employee
2,203 Views

Hi Alexander,

 

I's working on direct DMA communication between GPU and FPGA recently. Is your program developed on Intel Quartus Prime? And how did you get the driver code? Could you please share the link to me?

 

Thanks,

Heron

0 Kudos
Reply