Arria 10 PCIe Gen3x8 DMA(AN 829 ) - Linux Driver Error " Couldn't open the device"

[sachin_blr]

Hi all,

We were trying to install the DMA test driver and run the Linux DMA software as mentioned in "AN 829: PCI Express* Avalon®-MM DMA Reference Design" under "1.2.7. Installing the DMA Test Driver and Running the Linux DMA Software", and are facing the below issue: 

% sudo./install arria10

% sudo ./run

Got the below:

% Couldn't open the device.

On running dmesg command, getting the following:

[ 466.649278] tegradc 15200000.nvdisplay: unblank
[ 466.649291] tegradc 15210000.nvdisplay: blank - powerdown
[ 466.649298] tegradc 15220000.nvdisplay: blank - powerdown
[ 1047.293720] altera_dma: loading out-of-tree module taints kernel.
[ 1047.294647] Altera DMA: altera_dma_init(), Dec 11 2020 17:14:57
[ 1047.294959] Altera DMA 0005:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x000001ff 64bit pref] not claimed
[ 1047.295159] Altera DMA 0005:01:00.0: pci_enable_device() failed
[ 1047.295263] Altera DMA 0005:01:00.0: goto err_enable
[ 1047.295351] Altera DMA 0005:01:00.0: goto err_initchrdev
[ 1047.295449] Altera DMA 0005:01:00.0: goto err_bk_alloc
[ 1047.295558] Altera DMA: probe of 0005:01:00.0 failed with error -22
grl-jetson@grljetson-desktop:~$

On running "lspci -vvk", getting the following:

0005:01:00.0 Non-VGA unclassified device: Altera Corporation Device e093 (rev 01)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 38
Region 0: Memory at <unassigned> (64-bit, prefetchable) [disabled] [size=512]
Region 2: Memory at <unassigned> (64-bit, prefetchable) [disabled] [size=512M]
Capabilities: <access denied>

P.S: Earlier we tried by having Device Id as e003 in both altera_dma driver and Arria 10 reference design, but that also gave the same errors as above

Could someone please give any input(s) with regards to addressing the above issue.

Thanks in advance.

- Sachin

[SengKok_L_Intel]

Hi,

Did you use the root (or su)  to “make” and “install” the driver?

After programming the SOF, did you restart the host to ensure the PCIe link is enumerated?

Regards -SK

[sachin_blr]

Hi SK,

Thank you for your response.

I had used "sudo" when running install.

Further, the host was restarted after programming the SOF.

We are still facing the issues mentioned above. 

Regards,

Sachin

[SengKok_L_Intel]

After reboot, do you able to see the FPGA PCIe is enumerated?

What is the output if you run the commands below?

lspci -nn | grep 1172

lspci -s 5:01.0 -vvv (assuming the BDF is 5.01.0).

Besides, you may try to type "make" compile the driver and then run it again.

[sachin_blr]

Thank you for the response, SK

Following is the output of running the commands that you mentioned.

root@grljetson-desktop:/home/grl-jetson/FPGA_PCIe_driver/AN829_driver# lspci -nn | grep 1172
0005:01:00.0 Non-VGA unclassified device [0000]: Altera Corporation Device [1172:e093] (rev 01) 

root@grljetson-desktop:/home/grl-jetson/FPGA_PCIe_driver/AN829_driver# lspci -s 5:01:0 -vvv
0005:01:00.0 Non-VGA unclassified device: Altera Corporation Device e093 (rev 01)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Region 0: Memory at <unassigned> (64-bit, prefetchable) [disabled] [size=512]
Region 2: Memory at <unassigned> (64-bit, prefetchable) [disabled] [size=512M]
Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
Capabilities: [300 v1] #19
Capabilities: [800 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

Also, I compiled the driver again(using "make") and ran it. Still getting the same output of "Couldn't open the device." as before

Following are the commands that were used to compile and run (as root user):

- make EXTRA_CFLAGS=-DARRIA10

-  ./altera_dma_load

- ./run

Regards,

Sachin

[SengKok_L_Intel]

The PCIe link is linked up correctly. This is uncertain why the driver is not running, perhaps there might have some dependency on this host/OS. If possible, please try it out on another server or machine and see if you have encountered the same problem.

Regards -SK

[SengKok_L_Intel]

If further support is needed in this thread, please post a response within 15 days. After 15 days, this thread will be transitioned to community support. The community users will be able to help you with your follow-up questions. 

[sachin_blr · Accepted solution]

Thank for your following up on this issue, SK.

Our issue got resolved, on setting the base class code (upper byte at offset 0Bh in the PCIe configuration space) to 0xFF (Device does not fit any defined class). We did not have to change our host device or OS. 

We were able to successfully run the Linux DMA software as mentioned in "AN 829: PCI Express Avalon -MM DMA Reference Design" document.

Regards,

Sachin