pcie DMA memory read & write TLP

[EastGun-A]

Hello, while studying the pcie protocol, I was confused about DMA and PIO, so I asked a question. Thank you so much for always responding kindly.

PIO seems to work like the existing pcie spec, but DMA is confusing

When using DMA as an EP, if the host sends a memory write request, DMA reads data from the host memory through a memory read request. Is this correct?

Then, when the host sends a memory read request, it waits for completion with data. In DMA, data is stored in the host memory through a memory write request. In that case, how does the host receive the completion with data?

I'm not sure about the relationship between pcie TLP and DMA, so I asked a question.
How does TLP work when using DMA engine?

umm, i mean, I'm wondering if you're not passing the Memory Write TLP from the host to the endpoint, but which one.
As far as I know, it seems that PIO is delivered in the form of Memory Write TLP (fmt+type: 0h60).

[wchiah]

Hi,

In PCIe, DMA (Direct Memory Access) and PIO (Programmed Input/Output) are two different methods used for data transfer between devices. Let's clarify how they work and their relationship with PCIe TLPs (Transaction Layer Packets).

1. DMA (Direct Memory Access):
2. When using DMA, the host device initiates the data transfer by sending a PCIe Memory Read Request TLP to the endpoint (EP). This request includes the address in the host memory from where the data should be read. The DMA engine in the EP receives this request and performs a memory read operation to fetch the data from the host memory. Once the data is read, the DMA engine can store it locally in the EP or perform other processing tasks.

When the host wants to receive data from the EP through DMA, it sends a PCIe Memory Write Request TLP to the EP, specifying the address in host memory where the EP should write the data. The EP's DMA engine reads the data from its local memory and writes it to the specified address in the host memory.

The completion of the DMA transfer is typically handled using PCIe Completion TLPs. When the EP completes the requested operation (read or write), it generates a Completion TLP to indicate the status and transfer the completion data (if applicable) back to the host. The Completion TLP contains the necessary information for the host to identify the completion and retrieve the transferred data, if any.

1. PIO (Programmed Input/Output):
2. PIO, on the other hand, involves the host or EP directly reading or writing data to/from the PCIe configuration or I/O space registers of the target device. This method is typically used for control and configuration purposes rather than large data transfers. PIO operations are not explicitly associated with PCIe Memory Read or Write Request TLPs.

Regarding your specific question about the relationship between PCIe TLPs and DMA engine, the TLPs (such as Memory Read/Write Request TLPs and Completion TLPs) are used to initiate and acknowledge the DMA operations. They carry the necessary information for the host and EP to coordinate and perform the data transfer. The DMA engine in the EP is responsible for processing the TLPs and performing the actual data movements between the host memory and local EP memory.

Please note that the exact implementation details of DMA and PIO can vary depending on the specific PCIe device and its configuration, but this general explanation should help you understand the concept behind these mechanisms.

Let me know if further clarification is needed.

Regards,

Wincent_Intel

[EastGun-A]

Thank you very much for your reply.

1. It is said that the Memory Read TLP (Fmt+ type = 0010000) sends a request pointing to the host memory address through the PCI express link from the host to the DMA. Doesn't the address field of the Memory Rd TLP point to the address of the destination (EP)? (PCI express 4.0 spec, P-tile)

I am confused whether the form of the packet transmitted to the DMA EP through the PCI express link is different

2. Then, I understand that it is specified in the form shown in the figure. Does the request to the host memory also have the same TLP form and the address of the host memory represents the address field of Memory Rd?
Is there a separate address? (Does the host memory also have an independent EP?)

3. In the case of the Mwr in the figure, is the DMA EP receiving the Memory Write TLP, reinterpreting it, generating the Memory Write TLP again from the DMA to the host memory address, and sending it?

Sorry for the difficult question

The TLP that meets the Pcie 4.0 spec is transmitted through the PCI express link, but I ask a question because I do not understand how DMA works according to the TLP.

In addition, if you do a PIO test and look at the dumping signal with a Stratix 10 FPGA, it seems to be proceeding with MWR and MRD while following the Pcie 4.0 spec. I wonder what it means that TLP and PIO are not explicitly associated.

 

[EastGun-A]

Thank you very much for your reply.

1. It is said that the Memory Read TLP (Fmt+ type = 0010000) sends a request pointing to the host memory address through the PCI express link from the host to the DMA. Doesn't the address field of the Memory Rd TLP point to the address of the destination (EP)? (PCI express 4.0 spec, P-tile)

I am confused whether the form of the packet transmitted to the DMA EP through the PCI express link is different

2. Then, I understand that it is specified in the form shown in the figure. Does the request to the host memory also have the same TLP form and the address of the host memory represents the address field of Memory Rd?
Is there a separate address? (Does the host memory also have an independent EP?)

3. In the case of the Mwr in the figure, is the DMA EP receiving the Memory Write TLP, reinterpreting it, generating the Memory Write TLP again from the DMA to the host memory address, and sending it?

Sorry for the difficult question

The TLP that meets the Pcie 4.0 spec is transmitted through the PCI express link, but I ask a question because I do not understand how DMA works according to the TLP.

In addition, if you do a PIO test and look at the dumping signal with a Stratix 10 FPGA, it seems to be proceeding with MWR and MRD while following the Pcie 4.0 spec. I wonder what it means that TLP and PIO are not explicitly associated.

[FvM]

When doing DMA PCIe access controlled by the host, your host DMA driver is communicating with the device DMA descriptor controller, providing read and write transaction descriptors. You typically get completion info by an interrupt message. 

Review e.g. https://www.intel.com/content/www/us/en/docs/programmable/683425/18-0.html for details.

[EastGun-A]

Thank you for answer.

One thing I'm curious about is that MWr TLP accompanies data, but isn't that data not used in DMA? Then, is the data of MWr TLP unnecessary?
I'm curious about the behavior in terms of packet.

[wchiah · Accepted solution]

Hi

In the context of PCIe (Peripheral Component Interconnect Express), a Memory Write (MWr) Transaction Layer Packet (TLP) is used to transfer data from the initiator (the device initiating the transaction) to the target (the device receiving the transaction).

During a Memory Write transaction, the data is indeed used in the DMA (Direct Memory Access) process. DMA allows devices to transfer data directly to and from system memory without involving the CPU. The data sent in the MWr TLP is typically stored in a specific memory location or register within the target device.

So, the data carried by the MWr TLP is not unnecessary. It serves the purpose of updating or modifying data in the target device's memory or registers, and DMA facilitates this data transfer without CPU intervention.

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

[wchiah]

Hi,

Thanks for accepting the solution.

Therefore following our support policy, I have to put this case in close status.

Hence, This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If your support experience falls below a 9 out of 10, I kindly request the opportunity to rectify it before concluding our interaction. If the issue cannot be resolved, please inform me of the cause so that I can learn from it and strive to enhance the quality of future service experiences. 

Regards,

Wincent_Intel

[Rajat_Dongre_79]

1. I am getting page fault while accessing another PCI NTB switch config space.

1. The steps that I followed are below:

1. I am using two PCIe NTB switch.

• Microchip pcie ntb switch          
• Microchip Pcie ntb switch

Structure:

   

2.both switch is having two partition one is upstream and NT and second is NT only and    cross link between both switch.

1. I have added translated base address to 0x4000000 to BAR2 LUT table at index 0. and added this address in another switch partition 1 BAR 0 (translated base address)

1. In switch 1: I have added base address to LUT table at 0th Index Translated Base address : 0x04000000 Destinition partion : 1

1. I am trying to fetch address of it to access config space of another switch (SW2 NTB1 BAR0) so the address for this is BAR2 + LUT0.

1. BAR2's physical address is 0xaf310004

1. To access LUT0 entry. = BAR2 + LUT0 = af310004

1. but when i am trying access Device ID of another PCIe switch it is giving page fault.

1. please let me know where I am wrong

10 I checked physical crosslink using its chiplink tool In switch 2 I have added BAR0 base address to 0x4000000 using static enumeration.

Your help is really appreatiated

  `/*short code */

  PHYS_ADDR *sw2Ntb1regAddr;

  UINT32 sizeMapReg;

  UINT32 data;

  idt_ntb_peer_mw_set_trans(ndev, 0, 0, 0x04000000, 0);

  /* lukup table idx 0 */

    ret = idt_ntb_peer_mw_get_addr(ndev, 0, &sw2Ntb1regAddr, &sizeMapReg);

    if(ret != Success)

    {

    console_printf("NP: %s @ line %d, ret = %d\n", __FUNCTION__, __LINE__, ret);

    }

 /* Trying to access Device ID : register address : 0x2000 from SW2 PCIe NTB config space * /

/*sw2NtbregAddr = 0xaf310004*/

 memcpy(&data, sw2NtbregAddr + 0x2000, 4);  /* page fault error */

 printf("Device ID %X", data); `