How to stop Avalon DMA?

[MehmetFide]

Hello,

We have an NXP SoM module (with Linux) connected to an Intel Arria 10 FPGA through a PCIe bus.

We use Arria 10 Avalon-MM DMA Interface to transfer data from FPGA to our Linux processor. The DMA is master on the bus and fed by a FIFO in the FPGA by controlling waitrequest pin on the Avalon bus. Because of waitrequest, whenever a particular data is available in the FIFO of the FPGA, it will be pushed automatically to Linux Memory via this DMA. And the DMA generates an MSI interrupt once the size reaches to the one specified in the CONTROL register. Whole mechanism explained above works fine for us.

Now I need a command to terminate an active DMA operation previously programmed, before it reaches to a size specified in the CONTROL register.

I went through the solution manual below which describes the registers (CONTROL, WR_RC_LOW_SRC_ADDR, WR_CTLR_LOW_DEST_ADDR etc..) but couldn't find any useful information about how to terminate an active DMA transaction.  The documentation somehow didn't touch to this topic.

Intel Arria 10 Avalon-MM DMA Interface for PCIe pdf is attached.

Please suggest a way to terminate active DMA operation in the FPGA. (it may be through control registers, not sure exactly)

Your helps are highly appreciated. 

Thank you.

[VenTingT]

Hi @MehmetFide,

Thanks for reaching out to Intel Community Forum.

It seems like there's a related forum post discussing how to stop/cancel the AVMM DMA operation after it starts. 

Could you please check out the solution in the post?

https://community.intel.com/t5/FPGA-Intellectual-Property/How-to-stop-or-cancel-an-Read-Write-AVMM-DMA-operation-after-it/m-p/1288312

The Intel Arria 10 or Intel Cyclone 10 GX FPGA for PCI Express is compliant with PCI Express Base Specification 3.0.

Thanks.

Best Regards,

VenTing_Intel

[MehmetFide]

Hi @VenTingT,

Thank you for your response.  I went through the link you provided and followed the way suggested.

In the Linux kernel code, I have found following api which reset the secondary bus according to the PCIe standard and then wait for secondary bus to be ready again:

void pci_reset_secondary_bus(struct pci_dev *dev)
{
	u16 ctrl;

	pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl);
	ctrl |= PCI_BRIDGE_CTL_BUS_RESET;
	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl);

	/*
	 * PCI spec v3.0 7.6.4.2 requires minimum Trst of 1ms.  Double
	 * this to 2ms to ensure that we meet the minimum requirement.
	 */
	msleep(2);

	ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET;
	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl);
}

/**
 * pci_bridge_wait_for_secondary_bus - Wait for secondary bus to be accessible
 * @dev: PCI bridge
 * @reset_type: reset type in human-readable form
 *
 * Handle necessary delays before access to the devices on the secondary
 * side of the bridge are permitted after D3cold to D0 transition
 * or Conventional Reset.
 *
 * For PCIe this means the delays in PCIe 5.0 section 6.6.1. For
 * conventional PCI it means Tpvrh + Trhfa specified in PCI 3.0 section
 * 4.3.2.
 *
 * Return 0 on success or -ENOTTY if the first device on the secondary bus
 * failed to become accessible.
 */
int pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, char *reset_type)
{
	struct pci_dev *child;
	int delay;

	if (pci_dev_is_disconnected(dev))
		return 0;

	if (!pci_is_bridge(dev))
		return 0;

	down_read(&pci_bus_sem);

	/*
	 * We only deal with devices that are present currently on the bus.
	 * For any hot-added devices the access delay is handled in pciehp
	 * board_added(). In case of ACPI hotplug the firmware is expected
	 * to configure the devices before OS is notified.
	 */
	if (!dev->subordinate || list_empty(&dev->subordinate->devices)) {
		up_read(&pci_bus_sem);
		return 0;
	}

	/* Take d3cold_delay requirements into account */
	delay = pci_bus_max_d3cold_delay(dev->subordinate);
	if (!delay) {
		up_read(&pci_bus_sem);
		return 0;
	}

	child = list_first_entry(&dev->subordinate->devices, struct pci_dev,
				 bus_list);
	up_read(&pci_bus_sem);

	/*
	 * Conventional PCI and PCI-X we need to wait Tpvrh + Trhfa before
	 * accessing the device after reset (that is 1000 ms + 100 ms).
	 */
	if (!pci_is_pcie(dev)) {
		pci_dbg(dev, "waiting %d ms for secondary bus\n", 1000 + delay);
		msleep(1000 + delay);
		return 0;
	}

	/*
	 * For PCIe downstream and root ports that do not support speeds
	 * greater than 5 GT/s need to wait minimum 100 ms. For higher
	 * speeds (gen3) we need to wait first for the data link layer to
	 * become active.
	 *
	 * However, 100 ms is the minimum and the PCIe spec says the
	 * software must allow at least 1s before it can determine that the
	 * device that did not respond is a broken device. Also device can
	 * take longer than that to respond if it indicates so through Request
	 * Retry Status completions.
	 *
	 * Therefore we wait for 100 ms and check for the device presence
	 * until the timeout expires.
	 */
	if (!pcie_downstream_port(dev))
		return 0;

	if (pcie_get_speed_cap(dev) <= PCIE_SPEED_5_0GT) {
		u16 status;

		pci_dbg(dev, "waiting %d ms for downstream link\n", delay);
		msleep(delay);

		if (!pci_dev_wait(child, reset_type, PCI_RESET_WAIT - delay))
			return 0;

		/*
		 * If the port supports active link reporting we now check
		 * whether the link is active and if not bail out early with
		 * the assumption that the device is not present anymore.
		 */
		if (!dev->link_active_reporting)
			return -ENOTTY;

		pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &status);
		if (!(status & PCI_EXP_LNKSTA_DLLLA))
			return -ENOTTY;

		return pci_dev_wait(child, reset_type,
				    PCIE_RESET_READY_POLL_MS - PCI_RESET_WAIT);
	}

	pci_dbg(dev, "waiting %d ms for downstream link, after activation\n",
		delay);
	if (!pcie_wait_for_link_delay(dev, true, delay)) {
		/* Did not train, no need to wait any further */
		pci_info(dev, "Data Link Layer Link Active not set in 1000 msec\n");
		return -ENOTTY;
	}

	return pci_dev_wait(child, reset_type,
			    PCIE_RESET_READY_POLL_MS - delay);
}

But whole those mechanisms require about 1.5sec to be completed which is not acceptable for our system.

Isn't there a command/way on the  FPGA side just to terminate active Avalon DMA transaction rather then resetting whole PCIe subsystem?

Thank you.

[VenTingT]

Hi @MehmetFide,

Thanks for your update.

The users cannot stop active DMA transfer from the FPGA side because, in the reference design, the DMA transfer is set up using drivers from the host.

Thanks.

Best Regards,

VenTing_Intel

[MehmetFide]

Hi @VenTingT ,

Is there a way to stop it from the driver side by controlling DMA registers in the FPGA (like CONTROL, WR_RC_LOW_SRC_ADDR, WR_CTLR_LOW_DEST_ADDR etc..)?

Thank you.

[VenTingT]

Hi @MehmetFide,

These registers are specified in the write descriptor table. Each of them is dedicated to carrying the specified information correspondingly, and the user cannot stop the DMA by controlling these registers.

Thanks.

Best Regards,

VenTing_Intel

[MehmetFide]

Hi @VenTingT ,

What I understand from you is that there is no practical way that the user can stop the DMA in action then. Is this correct?

Is it possible to add a RESET functionality to the DMA in the FPGA and map this pin to one of the PCIe BARs registers so that user can reset it from the driver side? Is this DMA hard IP or soft one? (our PFGA is Arria 10 GX -> 10AX016E4F27I3SG)

I'm not an FPGA engineer but the engineer in our company uses QSYS and ready IPs in Quartus provided by Intel.  I wonder if it is possible for him to alter existing DMA IP and add such a reset functionality to it?

Thank you.

[VenTingT]

Hi @MehmetFide,

Yes, as there is no method that mentioned in the User Guide to stop the ongoing DMA.

The PCIe AVMM with DMA is a hard IP. From my understanding, it's not workable to add the Reset functionality to the PCIe BAR Registers as the BAR Registers (Base Address Register) are designated for memory addressing. 

User Guide of BAR in Intel Arria 10 AVMM DMA interface for PCIe:

https://www.intel.com/content/www/us/en/docs/programmable/683425/18-0/base-address-register-bar-and-expansion.html

May I know what is your objective to terminate an active DMA operation?

Thanks.

Best Regards,

VenTing_Intel

[MehmetFide]

Hi @VenTingT,

We use this DMA to push data in FIFO in FPGA to Linux and at the and of operation we get MSI interrupt. But the FIFO in the FPGA does not always have the amount of the data desired by DMA. Therefore the FIFO throttle the DMA operation using WaitRequest pin until FIFO has enough data requested by the DMA. FIFO may have enough data in 1ms or 3sec or never depends on user action in our final product.

The main problem of the DMA is that once we program it to transmit some data (1024 dword for example) from FPGA to Linux through PCIe, we can not reuse this DMA for other purposes (like read direction instead of write direction) until it completes its existing operations. Since the existing operation completion time is unknown due to user interactions, we want to cancel existing DMA operation and reprogram the DMA for some other tasks.

Thanks.

[VenTingT]

Hi @MehmetFide,

Thanks for your reply.

We cannot intervene when the DMA operation is in the process.  

To shorten the time to wait for a DMA operation, you may try to reduce the number of dword for one DMA operation, so that for each DMA operation it takes less time to complete the transfer. And, in between each DMA operation, add some code at the user application level to exit the loop before the next DMA operation starts.

Thanks.

Best Regards,

VenTing_Intel

[MehmetFide]

Hi @VenTingT ,

Even If I set the number of the dword size of the DMA to be transferred to 1, the DMA will wait forever until that dword became available in the FIFO because of WaitRequest control between FIFO and DMA. The FIFO may not have amount of data requested by the DMA when we initiate the DMA at the first place. Therefore we need to be able to cancel active DMA operation to make the DMA usable again.

Thanks.

[VenTingT]

Hi @MehmetFide,

Thanks for your reply.

I'll file the feature request internally.

You may also reach out to your nearest distributor for any new feature requests. 

Partner Directory (intel.com)

Thanks.

Best Regards,

VenTing_Intel

[VenTingT]

Hi @MehmetFide,

We have not received any response from you on the previous solution that we provided. Please login to ‘https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you.

After 15 days, this thread will be transitioned to community support. The community users will be able to help you with your follow-up questions.

If you feel your support experience was less than a 4 or 5, please allow me to correct it before closing, or please let me know the cause so that I may improve your future support experience.

Thanks.

Best Regards,

VenTing_Intel

p/s: If any answers from the community or Intel support are helpful, please feel free to mark them as solutions, give them kudos, and rate the survey 4/5.