Programmable Devices
CPLDs, FPGAs, SoC FPGAs, Configuration, and Transceivers
20750 Discussions

PCIe Gen 4 16 Lanes Core in Agilex - ready out signal behave

Avi_V_888
Beginner
1,819 Views

Hi, 

I am using PCIe gen 4 (16 lanes) (512bit/500MHz clk , payload size:512B) CORE, in Agilex device: AGFB014R24A2E2VR0, for transfer DMA writings from the FPGA to the PC’s memory.

 

Since the rate of the ready out signal (coming from the PCIe core) toggling is not constant, a FIFO must be placed between the data source (Sensor) and the DMA, to absorb the volatility.

 

The problem is the ready signal stays in off position for a long time (instead of toggling more homogeneously) and causes the FIFO to overflow (I'm talking about FIFO in the size of 64MB).

 

Are you familiar with this issue  ?

What do you think can be done to make the ready signal not to create such long breaks, but to disperse them over time for a number of short breaks?

 

Thanks

Avi

0 Kudos
17 Replies
Avi_V_888
Beginner
1,786 Views

????

0 Kudos
wchiah
Employee
1,774 Views

Hi Avi,

 

As a start, Programs that become unresponsive can be due to a myriad of reasons and might be difficult to pinpoint exactly.
However, I've laid out some suggestions to help you narrow this down further. As a normal data transfer condition, FIFO buffer overflow normally does not occurs, but it can result if there is a programming error. Example transmit PBL =4, TX watermark =1, if the FIFO buffer has only one location empty, the DMA attempts to read four words from memory even though there is only one word of storage available. This results in a FIFO Buffer Overflow interrupt.

The driver must ensure that the number of bytes to be transferred, as indicated in the descriptor, is a multiple of four bytes. For example, if the bytcnt register = 13, the number of bytes indicated in the descriptor must be rounded up to 16 because the length field must always be a multiple of four bytes.

 

Please let me know if it is helpful.

Regards,

WeiChuan_C_Intel

0 Kudos
wchiah
Employee
1,747 Views

Hi,

We do not receive any response from you to the previous answer that I have provided. This thread will be transitioned to community support. If you have a new question, feel free to open a new thread to get support from Intel experts. Otherwise, the community users will continue to help you on this thread. Thank you

 

Regards,

WeiChuan_C_Intel

0 Kudos
Avi_V_888
Beginner
1,732 Views

First of all sorry for the delay in my response , I switched to another urgent short mission.

 

I checked the FIFO management and found that everything is fine in terms of the reason you mentioned.

 

Can you tell me the size of the FIFO that other developers have had to place in order to absorb the volatility of the ready signal coming from the PCIe4 core?


If there is not yet enough data for gen 4 , I would happy to get an answer even for gen 3.

 

Thanks

Avi

0 Kudos
wchiah
Employee
1,721 Views

Hi Avi,

 

Thanks for your reply. To accommodate PCIe gen3 protocol requirements and to compensate for clock frequency differences of up to ±300 ppm between source and termination equipment, receiver channels have a rate match FIFO. The rate match FIFO adds or deletes four SKP characters (32 bits) to keep the FIFO from becoming empty or full. It monitors the block synchronizer for a skip_found signal. If the rate match FIFO is almost full, the FIFO deletes four SKP characters. If the rate match FIFO is nearly empty, the FIFO inserts an SKP character at the start of the next available SKP ordered set.

 

Hope this answers your question, let me know if it is helpful.

Regards,

WeiChuan_C_Intel

0 Kudos
Avi_V_888
Beginner
1,713 Views

Hi,

 

The FIFO is to absorb the writing to the PCIe core due to the delay resulting from the processing of the ready signal coming from the PC, and not due to clock frequency differences.

 

The PC seems to be responding to the change in the rate of the data I am transferring to it.

Initially a change from a state of non-data transfer to a state of high rate transfer.
So, it seems that the PC does not allow ready at the desired rate, until it "realizes" that there is a high demand of data transferring as a result of which the ready signal becomes more and more valid.

Indeed, at first the FIFO overflows, and then only a small part of it is occupied.

 

Have you encountered a similar case?
And what can be done to cause constant readiness for a high rate of data transfer from the PC and as a results of it from the PCIe core?

 

Thanks

Avi

0 Kudos
wchiah
Employee
1,702 Views

Hi,

 

Below is some analysis and thoughts that I have.
If the PCI Express bus is unable to accept data at a rate at least as fast as the incoming rate from the DUT,
the onboard memory will eventually fill up and the FIFO overflow error is generated. 
When the device is full, it doesn’t have a way to tell the device that it is full. New image data keep coming, but there is nowhere to store it so the board returns the FIFO overflow error to notify the user. This is different from the user experience when other devices run out of memory. For example, on an Ethernet device, the user doesn’t even know that the device filled its memory. Like an image acquisition board the Ethernet TCP/IP protocol checks for missing data. However, when missing data is found on Ethernet, a request is sent to the sending device to retransmit the data. Usually, by the time that the data is re-transmitted, there is space to hold it and the application continues with only a minor delay.

 

Check and ensure the memory installed in your system is paired correctly. Most modern chipsets can simultaneously access two memory modules if they are installed in the correct memory slots. Also, you can try to check your motherboard documentation to verify that your memory is installed in the correct slots. Otherwise, your memory throughput will be halved. Besides that, you can try to Upgrade to the latest BIOS for your motherboard. 

0 Kudos
wchiah
Employee
1,674 Views

Hi,

 

I wish to follow up with you 

So, do you still have further inquiries on this matter?

 

Regards, 
WeiChuan_C_Intel

0 Kudos
Avi_V_888
Beginner
1,667 Views

Hi,

 

Thank you for your help,
I checked your suggestions but unfortunately none of them cause the ready signal not to be available for such long periods of time at the beginning of the burst transfer

 

Avi

0 Kudos
wchiah
Employee
1,647 Views

Hi,

 

1. Can I get your device info from "lspci -vvv -s" ?

2. What is the design example that you currently using ? 
custom design or design provided in Intel FPGA design store ?

 

Looking forward to hear back from you.

Regards,

WeiChuan_C_Intel

0 Kudos
Avi_V_888
Beginner
1,629 Views

Hi,

 

1.

31:00.0 Unassigned class [ff00]: Device 1bba:7204 (rev 01)
Subsystem: Device 1bba:7204
Physical Slot: 4
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 18
NUMA node: 0
Region 0: Memory at 202fffe00000 (64-bit, prefetchable) [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #1, Speed 16GT/s, Width x16, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Capabilities: [178 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [188 v1] #19
Capabilities: [1b8 v1] #26
Capabilities: [1e8 v1] #27
Capabilities: [470 v1] #25
Capabilities: [d00 v1] Vendor Specific Information: ID=1172 Rev=0 Len=05c <?>
Kernel driver in use: nxtgen
Kernel modules: mvcnxtgen

 

2.

This is not an example design' it's my design with my DMA writing to Agilex's PCIe 4 core

 

Thanks 

Avi

 

0 Kudos
wchiah
Employee
1,605 Views

Hi Avi,

 

  1. Do you enable the burst transfer function in your design?
  2. Since the signal cannot accumulate the FIFO, do you check the host PC credit limit? 
  3. Do you check why the rate of the ready-out signal is not constant?

Regards,

WeiChuan_C_Intel

0 Kudos
Avi_V_888
Beginner
1,593 Views

I'm think there's a power saving mode on the PC that causes the PC to drop priority when it feels like data has not been sent for a certain amount of time.

0 Kudos
wchiah
Employee
1,587 Views

Hi Avi,

 

Thanks for sharing, did you manage to get the desired result after turn OFF the power saving mode ?

Regards,

WeiChuan_C_Intel

0 Kudos
Avi_V_888
Beginner
1,557 Views

I am a FPGA designer, I do not understand the computer's operating system.

I do not know if there is any power saving mode in the PC (as there is in the PCIe core) and if there is, how to turn it off

 

Avi

0 Kudos
wchiah
Employee
1,548 Views

Hi Avi,


I wish to help out, unfortunately the PC issue is out of our scope of support.
Feel free to let me know if you have any other questions.

Regards,

WeiChuan_C_Intel

0 Kudos
wchiah
Employee
1,517 Views

Hi,


I saw in the system, your question has been addressed in IPS.

I will set this as close, if you have any other questions in the future feel free to open a new case.


Regards,

WeiChuan_C_Intel


0 Kudos
Reply