Arria 10 PCIe Avalon-MM DMA Reference Design: Read/Write Timeout

[Altera_Forum]

Hi guys, 

 I tried searching through the forum for this and found some posts on it but no solution to the issue. I am currently in possession of a Nallatech 385a PCIe card with an Arria 10 on it. I am running this on CentOS 7 and using Quartus Prime version 17.1. What I've done with the Avalon-MM DMA reference design ( https://www.altera.com/en_us/pdfs/literature/an/an690.pdf ) is change the device and the pin assignments. The reference design archive package I am using is "hip_a10gx_g3_x8_avmm_dma256_1602_PS.qar". I am able to generate the HDL, fully compile, and program the .jic file into flash and restart the PC so the design saves and re-enumerates the PCIe bus properly. I am also able to make the test throughput code, install the driver, and run the program. However, the read/write test is timing out. I am able to see the device in lspci and output its details. Attached are screenshots of Quartus and the terminal output is below. Any and all help is greatly appreciated!   $ cd Downloads/ $ cd avmm_dma_linux/ $ su Password: # make make -C /lib/modules/3.10.0-862.2.3.el7.x86_64/build M=/home/arclab/Downloads/avmm_dma_linux make: Entering directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64' Building modules, stage 2. MODPOST 1 modules make: Leaving directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64' # ./install rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions *.symvers *.order make -C /lib/modules/3.10.0-862.2.3.el7.x86_64/build M=/home/arclab/Downloads/avmm_dma_linux make: Entering directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64' LD /home/arclab/Downloads/avmm_dma_linux/built-in.o CC /home/arclab/Downloads/avmm_dma_linux/altera_dma.o /home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘init_rp_mem’: /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:272:9: warning: unused variable ‘increment_value’ u32 increment_value = 0; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘rp_ep_compare’: /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:287:9: warning: unused variable ‘count’ u32 count = 1; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:286:12: warning: unused variable ‘rp_tmp’ u32 j, rp_tmp, ep_tmp; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘dma_test’: /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:362:13: warning: unused variable ‘ep_tmp’ u32 rp_tmp, ep_tmp; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:362:5: warning: unused variable ‘rp_tmp’ u32 rp_tmp, ep_tmp; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:361:12: warning: unused variable ‘j’ int i, j; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:63: warning: unused variable ‘simul_write_count’ int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:45: warning: unused variable ‘simul_read_count’ int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:25: warning: unused variable ‘num_loop_count’ int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count; ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:9: warning: unused variable ‘loop_count’ int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count; ^ In file included from /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:15:0: /home/arclab/Downloads/avmm_dma_linux/altera_dma.c: At top level: /home/arclab/Downloads/avmm_dma_linux/altera_dma.h:171:12: warning: ‘rp_compare’ declared ‘static’ but never defined static int rp_compare(u8 *virt_addr1, u8 *virt_addr2, u32 num_dwords); ^ /home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘rp_ep_compare.isra.5’: /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:313:14: warning: ‘ep_tmp’ may be used uninitialized in this function printk(KERN_DEBUG "ep_tmp = %08x\n", ep_tmp); ^ Building modules, stage 2. MODPOST 1 modules CC /home/arclab/Downloads/avmm_dma_linux/altera_dma.mod.o LD /home/arclab/Downloads/avmm_dma_linux/altera_dma.ko make: Leaving directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64' # ********************************************** ** ALTERA AVMM DMA driver ** ** version 2.02 ** ** 1) start DMA ** ** 2) enable/disable read dma ** ** 3) enable/disable write dma ** ** 4) enable/disable simul dma ** ** 5) set num dwords (256 - 2048) ** ** 6) set num descriptors (1 - 128) ** ** 8) loop dma ** ** 10) exit ** ********************************************** Run Read ? 1 Run Write ? 1 Run Simultaneous ? 1 Read Passed ? 0 Write Passed ? 0 Simultaneous Passed ? 0 Read EPLast timeout ? 1 Write EPLast timeout ? 1 Number of Dwords/Desc : 512 Number of Descriptors : 128 Length of transfer : 256 KB Rootport address offset : 0 Read Time : 0 s and 249767 us Read Throughput : 0.000978 GB/S Write Time : 0 s and 216415 us Write Throughput : 0.001128 GB/S Simultaneous Time : 0 s and 270368 us Simultaneous Throughput : 0.001807 GB/S# $ sudo lspci -s 01:00.0 -vvv password for arclab: 01:00.0 Non-VGA unclassified device: Altera Corporation Device e003 (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 33 Region 0: Memory at f0120000 (64-bit, prefetchable) Region 4: Memory at f0100000 (64-bit, prefetchable) Capabilities: MSI: Enable+ Count=1/4 Maskable- 64bit+ Address: 00000000fee003d8 Data: 0000 Capabilities: Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port# 1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: Vendor Specific Information: ID=1172 Rev=0 Len=044 <?> Capabilities: # 19 Capabilities: Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: Altera DMA $ $ ls /dev/ -l total 0 crw-rw-rw-. 1 root root 243, 0 May 16 19:16 altera_dma

[Altera_Forum]

Some additional output from dmesg: 

  altera_dma: loading out-of-tree module taints kernel. altera_dma: module verification failed: signature and/or required key missing - tainting kernel Altera DMA: altera_dma_init(), May 18 2018 17:29:59 Altera DMA 0000:01:00.0: pci_enable_device() successful Altera DMA 0000:01:00.0: irq 33 for MSI/MSI-X Altera DMA 0000:01:00.0: pci_enable_msi() successful Altera DMA 0000:01:00.0: using a 64-bit irq mask Altera DMA 0000:01:00.0: irq pin: 1 Altera DMA 0000:01:00.0: irq line: 11 Altera DMA 0000:01:00.0: irq: 33 Altera DMA 0000:01:00.0: request irq: 11 Altera DMA 0000:01:00.0: BAR 0xf0110000-0xf01101ff flags 0x0014220c, length 512 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0 Altera DMA 0000:01:00.0: BAR 0xf0100000-0xf010ffff flags 0x0014220c, length 65536 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0 Altera DMA 0000:01:00.0: BAR mapped to 0xffffabf100e46000, length 512 Altera DMA 0000:01:00.0: BAR mapped to 0xffffabf100ee0000, length 65536 Read DMA times out DWORD = 00000200 Desc = 00000080 Write DMA times out DWORD = 00000200 Desc = 00000080 Simultaneous DMA times out DWORD = 00000200 Desc = 00000080

[Altera_Forum]

Still struggling to understand what's going on here. Same error on both Windows 10 and CentOS. I have a dual boot setup on the same PC where the card is installed. Extension cable is not used here. I've also attempted to change the memory maps for RD/WR master and the device ID, still the same result. I never have an issue with seeing the device nor is the BARs disabled, I just can't read/write to the on chip RAM. I'm at a loss here as to what the issue would be.

[Altera_Forum]

I have successed run PCIE reference design on my a10 board. 

 I change the device and the pin assignments, then compiled.  I think you should check your pin assignment.

[arod412]

So after months of trying to figure out what happened I finally have a solution for this. The npor and perstn signals must both be set to the reset line. Or at least that's how I got it to work. In my setup, I left the npor signal "open" and only connected perstn. I did this because I had found posts on this about which one to connect and it only mentioned perstn. The documentation also states that npor is an OR output of perstn and some other signal. After connecting both perstn and npor to the same reset the card started functioning with no problem. This makes sense since on SignalTap I only had the clocks working but the card seemed dead as in no other signals would change.

For people searching and finding this topic, this was my environment:

CentOS 6.9

Nallatech N385a (Arria 10 [10AX115N3F40E2SG])

Quartus Prime Pro 18.0.1

I was able to verify the qsys design, which was the Arria 10 Reference Design, using Dave's program found here:

http://www.alteraforum.com/forum/showthread.php?t=35678

[Jose_G_Intel3]

Hello arod412,

So in order to fix this issue did you connected npor and perstn signals to reset instead of configuring them as conduit?

Raul

[guna1]

so connecting npor and persetn to which reset signal?

did it work? timeout error is not there?

[homeexpert12]

In the case of the "Arria 10 PCIe Avalon-MM DMA Reference Design: Read/Write Timeout" issue, it's essential to check for potential configuration errors or data transfer bottlenecks. This troubleshooting process resembles the attention to detail required in the UAE embassy attestation. I am sure it's right way.