Programmable Devices
CPLDs, FPGAs, SoC FPGAs, Configuration, and Transceivers
20638 Discussions

Alternate driver for Avalon-MM DMA for PCIe

AlexGo
Novice
1,965 Views

Hi All,

 

If anyone is interested in a Linux device driver for Avalon-MM DMA for PCIe? The existing reference design code is quite difficult to maintain, while the alternate solution basically boils down to these APIs:

 

typedef void (*avalon_dma_xfer_callback)(void *dma_async_param);   int avalon_dma_submit_xfer( struct avalon_dma *avalon_dma, enum dma_data_direction direction, dma_addr_t dev_addr, dma_addr_t host_addr, unsigned int size, avalon_dma_xfer_callback callback, void *callback_param);   int avalon_dma_submit_xfer_sg(struct avalon_dma *avalon_dma, enum dma_data_direction direction, dma_addr_t dev_addr, struct sg_table *sg_table, avalon_dma_xfer_callback callback, void *callback_param);   int avalon_dma_issue_pending(struct avalon_dma *avalon_dma);

 

 

The callbacks are called from the tasklet context - much like standard Linux dmaengine framework does.

 

If the idea gathers enough interest I can think of posting it upstream or coming up with implementation that conforms to the Kernel dmaengine model.

 

Please, let me know if the use of suggested API is not clear and needs clarification.

 

Thanks!

12 Replies
BoonT_Intel
Moderator
1,559 Views

@AlexGo​ Thank you very much for the sharing.

0 Kudos
JET60200
New Contributor I
1,559 Views

Hello @AlexGo,

 

Where can I found that you mentioned " The existing reference design Linux device driver code for Avalon-MM DMA for PCIe ", is it ok for A10 SoC design ?

 

Also <where can we find your updated "Alternate driver for PCie" ? Thanks

0 Kudos
AlexGo
Novice
1,559 Views

Hello!

 

I remember downloading the reference design from wiki, but the link does not appear available anymore. This one looks the same:

https://www.intel.com/content/www/us/en/programmable/support/support-resources/software/download/refdesigns/ip/interface/dnl-pciexpress-hp-avalonmm.html

 

You could find the recent version of "avalon-dma" and "avalon-drv" drivers here:

git@github.com:a-gordeev/linux.git avalon-dma-engine

git@github.com:a-gordeev/avalon-drv-tool.git master

 

Here is some explanation:

avalon-dma - DMA engine driver that provides generic interface to the Avalon DMA engine;

avalon-drv - sample driver on top of a custom A10 design (see Alaric board link below) that uses avalon-dma engine;

avalon-drv-tool - user-level sample utility that uses avalon-drv interface to execute DMAs;

 

I used Arria® 10 SoC FMC IDK for development:

https://www.reflexces.com/intel-fpga/arria-10-intel-fpga/alaric-instant-devkit-arria-10-soc-fmc-idk

 

It would be interesting if you give it a go.

 

Thanks!

0 Kudos
JET60200
New Contributor I
1,559 Views

Thanks Alex.Go for your head up,

I saw your github commit are :

https://github.com/a-gordeev/linux/commits/avalon-dma-engine

  1. Commits on Sep 11, 2019 staging: avalon-drv: Avalon DMA driver

a-gordeev committed on Jul 17

Signed-off-by: Alexander Gordeev <a.gordeev.box@gmail.com>

IS THIS YOUR CODE CHANGES FOR "Avalon-MM DMA for PCIe DRiver" ? Please confirm, and what's the linux kernel version, as well ? thanjks

0 Kudos
AlexGo
Novice
1,559 Views

I am not quite sure what is your question, but here is some explanation (I updated my previous message as well):

 

avalon-dma - DMA engine driver that provides generic interface to the Avalon DMA engine;

avalon-drv - sample driver that uses avalon-dma engine interface;

avalon-drv-tool - sample user-level utility that uses avalon-drv interface to execute DMAs;

 

The Linux version used is upstream v5.3-rc8.

It is not big deal to backport it to any earlier version.

 

HTH

 

Thanks!

0 Kudos
MehmetFide
Novice
1,180 Views

Hi @AlexGo ,

 

 

Thank you for the drivers. I could clone your repo and integrated it into our Linux 5.4.193 for iMX8M-Plus processor.

Whole image contains avalon-dma and avalon-drv has been build successfully using yocto.

 

I could successfully tested DMA and PCIe bus connected to our iMX8M-Plus platform using AN690 provided by Intel.

But I'm not happy with the way they implemented the driver therefore I want to try yours as well.

 

I'm going to test it on Arria 10 SX Development board

But before that, could you give me some hints for the avalon-tool and required structure for the FPGA side? 

Do you have a FPGA Quartus project which can work with default driver parameters directly?

 

image (2).png

 

Thank you.

0 Kudos
MehmetFide
Novice
1,178 Views

Hi @AlexGo ,

 

Also noted that avalon-tool.c requires avalon-ioctl.h file which is missing. 

If you meant to include avalon-drv-ioctl.h instead, there are still many missing definitions required by avalon-tool.c to be compiled.

for example "struct avalon_dma_info" or "IOCTL_AVALON_DMA_GET_INFO"

 

Could you suggest?

 

0 Kudos
MehmetFide
Novice
1,133 Views

By the way I use avalon-dma-engine branch by hoping that it is the latest.

0 Kudos
MehmetFide
Novice
1,112 Views

Hello @AlexGo 

 

I started testing drivers.

 

I had problems to receive MSI interrupts by the driver. It basically hangs at xfer_rw() function if MSI interrupts are not received.

To make it work, I added init_arria_msi() function and called it in your init_interrupts() function as suggested by @OlegT  here

 

static void init_arria_msi(struct pci_dev *pci_dev)
{
	u32 conf_reg;
	
	// set interrupt disable for legacy
	pci_read_config_dword(pci_dev, 0x04, &conf_reg);
	dev_info(&pci_dev->dev, "previous conf_reg = %08llx", (u64)conf_reg);
	conf_reg |= (1 << 10);	
	pci_write_config_dword(pci_dev, 0x04, conf_reg);

	//Set bit[1] (Memory space) and bit[2]
	conf_reg |= 0x06;
	pci_write_config_dword(pci_dev, 0x04, conf_reg);
	dev_info(&pci_dev->dev, "after conf_reg = %08llx", (u64)conf_reg);

	// set msi Ena
	pci_read_config_dword(pci_dev, 0x50, &conf_reg);
	dev_info(&pci_dev->dev, "previous msi Ena = %08llx", (u64)conf_reg);
	conf_reg |= (1 << 16);
	pci_write_config_dword(pci_dev, 0x50, conf_reg);	
	dev_info(&pci_dev->dev, "after msi Ena = %08llx", (u64)conf_reg);
}

static int init_interrupts(struct pci_dev *pci_dev)
{
	int ret;
	
	init_arria_msi(pci_dev);

	ret = pci_alloc_irq_vectors(pci_dev,
				    PCI_MSI_COUNT, PCI_MSI_COUNT,
				    PCI_IRQ_MSI);
	if (ret < 0) {
		goto msi_err;
	} else if (ret != PCI_MSI_COUNT) {
		ret = -ENOSPC;
		goto nr_msi_err;
	}

	ret = pci_irq_vector(pci_dev, PCI_MSI_VECTOR);
	if (ret < 0)
		goto vec_err;

	return ret;

vec_err:
nr_msi_err:
	pci_disable_msi(pci_dev);

msi_err:
	return ret;
}

 

 

After the modification, it seems I can complete dma transfers for writing and reading successfully. I will do couple of performance tests and let you know.

 

Thank you.

0 Kudos
MehmetFide
Novice
1,085 Views

My tests show that reading speed from FPGA memory to user-space memory is only 28MB/sec with IOCTL_AVALON_DMA_READ command. Other commands like IOCTL_*_SG and IOCTL_*_SG_SMP crashed for me, therefore I couldn't test them.

 

On our device (iMX8M-Plus and Arria 10 GX), the PCIe bus has 1 lane and the link trained at Gen3 speed. Previously I have tested the speed using AN690 mentioned above and it was about 300MB/sec on the same hardware. But in that case the Altera's application test software just keeps the data in the kernel and never transfers it to user space.

 

I have just realized that you have more updated code in avalon-dmaengine-v5.7-rc1 branch instead of avalon-dma-engine branch. I will test that as well to see if there are any improvements at reliability and speed.

0 Kudos
MehmetFide
Novice
1,078 Views

Well, I jumped to the branch avalon-dmaengine-v5.7-rc1, and after couple of adjustment I measured the speed as 60MB/sec with IOCTL_AVALON_DMA_READ cmd.

 

total: 10737418240 bytes, duration: 169 secs, speed = 60.59 MB/sec

 

Our target is to reach 100MB/sec in the user-space domain. Therefore I will continue digging. 

0 Kudos
MehmetFide
Novice
1,069 Views

With the above branch it seems that IOCTL_AVALON_DMA_READ_SG command is also stable. This command and mmap() approach yielded  66.07MB/sec.

 

Could you explain what are *_SG_SMP commands and how they differ comparing to IOCTL_AVALON_DMA_*_SG?

 

Thank you.

 

0 Kudos
Reply