Configuration via Protocol (CvP) fail after loading the periphery image

[JonGoh]

Hi,

May I know if this issue has been resolved in newer versions of the CvP driver? I am still facing CvP intermittent failing problems and was wondering whether there are other workarounds other than the need for a power recycle.

"Due to a CvP upstream driver issue, CvP may intermittently fail after successfully loading the periphery image into all Intel Agilex® devices with package code R31C / R31B.".

Here is the link where I found the article:

https://www.intel.com/content/www/us/en/support/programmable/articles/000089044.html

Thanks,

Jon

[JohnT_Intel]

Hi,

The issue has been fixed in Quartus 22.2. May I know which version are you using that you are still observing the issue?

[JonGoh]

Hi @JohnT_Intel ,

I am using Quartus 24.2 and Linux kernel version 5.15.30, and I am following the Agilex 7 Device Configuration via Protocol (CvP) Implementation User Guide that is available. What are some areas that I can look into to try resolve the issue?

Thanks,

Jon

[JohnT_Intel]

Hi,

Please use git checkout socfpga-6.6.37-lts version of the opensource driver https://github.com/altera-opensource/linux-socfpga/tree/socfpga-6.6.37-lts which is already have the fixed implemented.

[JonGoh]

Hi @JohnT_Intel ,

I have changed to 6.6.37-lts version. However,  it seems like the driver is not loaded properly. After entering lspci -vvv -d1172:, I do not see the output "Kernel driver in use: altera-cvp", and I can't upload my core image either. Is there a difference in the procedure for version 6.6.37-lts? And where else can I look to troubleshoot this issue?

Thanks,

Jon

[JohnT_Intel]

Hi,

After you getting the new source code, have you rebuild the code?

Below is the guide to build the driver.

https://www.intel.com/content/www/us/en/docs/programmable/683763/23-1/installing-the-upstream-open-source.html

Please help to run "uname -mrs" to see which kernel you are using.

[JonGoh]

Hi @JohnT_Intel ,

Yes I have done git checkout 6.6.37-lts, built the kernel and verified that it is "Linux 6.6.37+ x86_64". I have also ensured that these are selected:

• CONFIG_FPGA=y
• CONFIG_FPGA_MGR_DEBUG_FS=y
• CONFIG_FPGA_MGR_ALTERA_CVP=y

I also noticed that fpga_manager directory does not exist at "/sys/kernel/debug". Do you have any idea what could be the problem?

Thanks,

Jon

[JohnT_Intel]

Hi,

Can you share with the the log file so that we can further look into the issue?

[JonGoh]

Hi @JohnT_Intel ,

How do I generate the log file? Or where can I locate the log file?

Thanks,

Jon

[JohnT_Intel]

Hi,

Please copy the "lspci -vv" and "uname -mrs" printout of the FPGA board. Please also provide the step you use to update the kernel.

Thanks.

[JonGoh]

Hi @JohnT_Intel ,

01:00.0 Unassigned class [ff00]: Altera Corporation Device 09c4 (rev 01)

Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Region 0: Memory at f0000000 (64-bit, prefetchable) [disabled] [size=4M]

Region 4: Memory at f0400000 (64-bit, prefetchable) [disabled] [size=256K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)

Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [70] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W

DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-

LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported

ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 8GT/s (ok), Width x8 (ok)

TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+

10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1

EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-

FRS- TPHComp+ ExtTPHComp-

AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,

AtomicOpsCtl: ReqEn-

LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer+ 2Retimers+ DRS-

LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+

EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-

Retimer- 2Retimers- CrosslinkRes: Upstream Port

Capabilities: [b0] MSI-X: Enable- Count=16 Masked-

Vector table: BAR=0 offset=00100000

PBA: BAR=0 offset=00180000

Capabilities: [100 v2] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-

CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+

AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-

MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-

HeaderLog: 00000000 00000000 00000000 00000000

Capabilities: [148 v1] Virtual Channel

Caps: LPEVC=0 RefClk=100ns PATEntryBits=1

Arb: Fixed- WRR32- WRR64- WRR128-

Ctrl: ArbSelect=Fixed

Status: InProgress-

VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

Status: NegoPending- InProgress-

Capabilities: [174 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 1

ARICtl: MFVC- ACS-, Function Group: 0

Capabilities: [184 v1] Secondary PCI Express

LnkCtl3: LnkEquIntrruptEn- PerformEqu-

LaneErrStat: 0

Capabilities: [46c v1] Data Link Feature <?>

Capabilities: [d00 v1] Vendor Specific Information: ID=1172 Rev=0 Len=05c <?>

apd@apd-pc1:~$ uname -mrs
Linux 6.6.37+ x86_64

Steps that I took to update the kernel (following the guide that you mentioned above):

git clone https://github.com/altera-opensource/linux-socfpga

cd linux-socfpga

git checkout socfpga-6.6.37-lts

cd /usr/src/linux-socfpga

cp -v /boot/config-$(uname -r) .config

make menuconfig

- Set Altera CvP FPGA Manager and FPGA Manager DebugFS to *

Make -j 12

export INSTALL_MOD_PATH=/usr/src/linux-socfpga

MODPATH="INSTALL_MOD_PATH=/usr/src/linux-socfpga"

ARGS="$MODPATH"

make $ARGS modules

sudo make modules_install

sudo make install

grub-mkconfig

cat /boot/grub/grub.cfg

sudo cp /etc/default/grub /etc/default/grub.bak

cat /etc/default/grub

gedit/etc/default/grub

sudo update-grub

Hope this gives more insight to the problem.

Thanks,

Jon

[JohnT_Intel]

Hi,

From the log file, you are running in PCIe Gen3x8. Is this the design loaded into the FPGA?

[JohnT_Intel]

HI,

Please also provide the OS and initial kernel version used.

[JohnT_Intel]

Hi,

Can you try Kernel 5.1*.50 and above? I have check internally and it is updated in 5.1*.50 and later kernel version. Maybe you may try 5.15.100-lts.

[JonGoh]

Hi @JohnT_Intel ,

Yes the design of PCIe Gen3x8 is loaded into the FPGA. I am using Ubuntu 22.04, and used kernel version of 5.15.30-lts at first, and also  tried 6.6.37-lts and encountered the problem mentioned above. I also attempted to compile 5.15.100-lts, but there seems to be an error in compiling, here is the error.

fs/ksmbd/smb2pdu.c: In function ‘smb2_write’:
fs/ksmbd/smb2pdu.c:6478:13: error: ‘is_rdma_channel’ undeclared (first use in this function)
6478 | if (is_rdma_channel == false) {
| ^~~~~~~~~~~~~~~
fs/ksmbd/smb2pdu.c:6478:13: note: each undeclared identifier is reported only once for each function it appears in
make[2]: *** [scripts/Makefile.build:289: fs/ksmbd/smb2pdu.o] Error 1
make[1]: *** [scripts/Makefile.build:552: fs/ksmbd] Error 2
make: *** [Makefile:1910: fs] Error 2

Meanwhile, I'll try other versions from what you mentioned above,

Thanks,

Jon

[JohnT_Intel]

Hi,

Can you try modify the source code of 5.15.3-lts as below?

drivers/fpga/altera-cvp.c file.

Change "#define V2_CREDIT_TIMEOUT_US 20000" to "#define V2_CREDIT_TIMEOUT_US 40000"

Thanks.

[JonGoh]

Hi @JohnT_Intel ,

The original source code for 5.15.30-lts is already set as 40000. For kernel version 6.6.37-lts, may I know if the problem of /sys/kernel/debug/fpga_manager directory not being created is replicable?

Thanks

Jon

[JohnT_Intel]

Hi,

Please try out 5.10.110-lts version as that is verified kernel that is working.

Thanks.

John Tio

[JohnT_Intel]

Hi,

If you are still facing intermittent failure can you provide the log file from this command "dmesg | tail -4"? This will provide more detail on what is happening when you are performing CvP programming

[JonGoh]

Hi @JohnT_Intel ,

Yes I am still facing intermittent failures. Here are the log files.

[12201.065931] altera-cvp 0000:01:00.0: Timeout waiting for credit
[12201.065934] altera-cvp 0000:01:00.0: Wait Credit ERR: 0xffffff92
[12201.065936] fpga_manager fpga0: Error while writing image data to FPGA
[12201.066005] fpga_manager fpga0: fpga_mgr_load returned with value -110

Thanks,

Jon

[JohnT_Intel]

Hi,

While I am working with the engineering team on this, can you help try to increase the timeout value to see if it can resolve the issue?

drivers/fpga/altera-cvp.c file.

Change "#define V2_CREDIT_TIMEOUT_US 40000" to "#define V2_CREDIT_TIMEOUT_US 60000"

Thanks.