PCIe freezing

[Altera_Forum]

Hi, 

 I have strange problem with PCIe IP. I have a custom board with ArriaGX and PCIe instantiated using MegaCore flow with AvalonST interface. The PCIe works (there are some DMAs running, target access, everything works fine), but occasionally the PC freezes when accessing my device. The design is quite complicated, so I started to minimize it and I ended with simple PCIe interface to on-chip memory and SW application that only writes to the device as fast as is possible. In this situation the system freezes after few seconds.  Using SignalTap I see folowing: - there are coming writes before the freeze - only rx_xxx, nothing else. - no interrupt is signalled - rx_ready, tx_ready are continuously asserted - tx_credits are nonzero - there are just no transactions coming out of the IP when the system freezes even though my application is ready to receive.  So I configured the IP to show me the test_out (512 bit) interface to see more and I found this: - in the time, when the packets stop coming there is bit 313 and 314 set (received erroneous TLP, packet with wrong sequence number) and they are coming repeatedly after that moment - also in the same times there is bit 85 set (that is Data link layer error - TLP error) - in the reaction to bad TLP it seems to me, that the IP sends a NAK to the root port (just few tick after the first erroneous TLP is received) - some ticks after the NAK is sent there is stop in receiving PLP ACKs for some time (I suppose that the IP tries to reset the link)  and that's all - I do not see anything more coming.  At first I thought that It could be some weird problem with the TLP sequencing, but I discovered, that time from time there is exactly the same situation (bad TLP sequence number, NAK, re-training of the link) but IP recovers from that error without any problem - the packets start coming again.  Do you have any ideas, where should I look, what could cause this problem? Maybe I did not find the real moment where the IP freezes. How to track/debug it?  Martin

[Altera_Forum]

Hello, noone has any idea? Any clues? 

 Martin

[Altera_Forum]

If it’s a lower-level issue, it should be visible without traffic as well. Maybe you can implement a detector, triggered by the condition on test_out that acts without PCIe traffic. Maybe connect it with a LED to see the rate of such incidents. 

 In any case you have to make sure that you correctly respond to any non-posted requests sent to you, otherwise the sender will not regain his credits and run out of them quickly. Then his completion timeout mechanism will start to act, and depending on hardware and software configuration, this leads to soft or hard errors, lockups and maybe an immediate reboot.  I am not familiar with the soft IP and have never used test_out, so bear with me.

[Altera_Forum]

Hi Matthias, 

 I can do a detector, so I can see rate of the "problem", but will that show anything to me? What I don't know is if it is a problem that this happens or it's a normal condition?  As a second - I do not issue any non-posted requests, the software only writes to my device, so there should not be any problem with credits, am I right?  Martin

[Altera_Forum]

If the detector shows trouble even in absence of the host/software doing requests, then you know there is something seriously wrong on the lower level. If these errors are just related to the requests or the associated responses to them, it’s hard to tell who is to blame: The PCIe soft IP or the application behind AST. 

 When writing to your device, there could be problems with credits only if your model behind AST does not fetch the data. Once you fetch data from the buffer (rx_ready/rx_valid asserted), the credit is automatically returned to the Requester by the PCIe IP. If the IP does not return credit for whatever reason, you can get into trouble with write-only requests as well.  Remaining issues are: Buggy Soft IP, some synthesis defects – maybe constraints, clocking, clock domain crossing – hardware defects on the FPGA board or the motherboard.

[Altera_Forum]

Hi Matthias, 

 when the PC freezes and I have a SignalTap connected, I do not see any problem with rx_ready- it's continuously asserted. The function sends me rx_valids, sops and eops and my logic should handle them. I am issuing rx_st_mask for one tick after each packet starts, but this should not be an issue, am I right?  And for the detector - I still don't know if the situation it detects is a problem or if it is some standard situation on the line.  Martin

[Altera_Forum]

Refuse from asserting rx_st_mask at that time. The IP has to tamper with credits when you use it, so for debugging, nail it to idle. You are not handling non-posted requests anyway, so you should throttle posted requests only using rx_ready. 

 Put the detector idea aside for now. Let’s try to be exact and easy on the AST interface first.

[Altera_Forum]

Okay, I removed the rx_st_mask logic and tied it to ground. The problem remains with no change... Do you have any other idea?

[Altera_Forum]

Do you know any way how to get the credit status of the PCIe root?

[Altera_Forum]

Hi Martin, 

 it looks as if test_out[199:144] contains the flow control information for the receiver. You cannot easily find out how the root is counting credit. Check whether that changes along the process of accessing your device. If credit gets lower over time, there is something wrong with the soft IP or the application interface. If it stays high there might still be something wrong with the soft IP but on a lower level, and it could be another hardware issue together with the root side of the PCIe link.  Still, I think it might be an issue with posted requests sent to your IP that get dropped somewhere, most probably in your application. Try to make sure that you don’t just drop such packets but respond accordingly, i.e. with a Completer Abort TLP. Otherwise, the root port’s receive credit will be eaten up, not allowing it to issue new requests until the requests time out. I don’t know precisely how a root port deviates in acting on a timeout event, but it might lead to lockup or reboot.  I’m on Arria II GX using hard IP AST (125 MHz, 64 bit), and there is little trouble with handling unsupported requests, though.

[Altera_Forum]

Hi Matthias, 

 the test_out at this bits is: PH (7-0) : 50 PD (19-8) : 360 NHP (27-20) : 54 NPD (35-28) : 0 CPLH (43-36) : 0 CPLD (55-44) : 0  the PH and PD credits are changing (+-1) when the PC is not frozen and I send some "writes" to my devices, other stays intact. When the PC freezes, the values are the same as I have written above - so no change.  Couldn't be a problem that the NPD, CPLH and CPLD are zeroes? But they are zeroes also when the PC is working...  Martin

[Altera_Forum]

PH/PD changing while receiving write requests is expected, and they do recover. NPH looks good as well. NPD=0 looks bad but might be correct as it’s useless to allocate memory for non-posted data when a native Endpoint does not support non-posted traffic with data. At least I think it doesn’t. 

 About CPLH/CPLD: Amazing they are zero. Maybe they get positive once non-posted request are issued by the IP.

[Altera_Forum]

Hi Matthias, 

 In altera datasheet to the IP core is written that "PCI Express endpoints must offer an infinite number of completion credits", I don't know if it is also rule for root devices, but there is also mentioned, that "When infinite credits are available, the corresponding credit field is all 1's". So all zeroes does really mean zero credits available. Could this be a problem? When I do not issue writes too much fast, the device works with a problem - there are reads/writes and dma read/writes running without any problem - so posted, nonposted transactions and completions are issued on both ends. Just when I issue big bunch of "target writes" from the PC, it freezes...  Martin

[Altera_Forum]

I think it does offer infinite completion credits, but they are not allocated on the RX memory and that’s what this debug output should show – otherwise the debug output would be useless. You could watch out for an increase of CPLH/CPLD values when you issued a non-posted DMA read from the application to verify this behavior. 

 Generally, I don’t know why large and/or frequent writes should be an issue for PCIe, the IP core or your application. They are the simplest form of transaction, and the system should not be harmed by your application improperly generating TLPs, like violation of max_read_request_size, max_payload_size, byte enables or read completion boundary (RCB).  Sorry to repeat the question: Are you handling read requests properly, i.e. either responding with valid data or sending CA? Those requests, once sent to your application, will not show up on the RX buffer usage but affects the root complex’ buffer handling and error reporting.

[Altera_Forum]

Hi Matthias, 

 currently I have disabled lot of my application functionality for debugging of this problem, so I do not handle read requests anyhow - now they are just getting timeouted. As a part of debugging I disconnected the whole AST transmitter side from my application (I tied it to ground) to be sure, that some invalid packet from my application is not a cause of this problem. When I issue a read on the PC side, I receive all FFs. When I track changes in credits with the Signal Tap, the NPH stays the same, they are just decremented by one and then incremented back again. I possibly can send CA - I'll only have to find out how to do it as I did not implemented it before, I handled all request from the PCIe.

[Altera_Forum]

Hi Martin, 

 If I understand you correctly, the problem appears more frequent if you remove your TX part with all functionality required for read completion and DMA transmission. This makes me think your – missing/wrong – completions migth be the cause for the problems.  Sending CA as a Cpl is typically faster than sending a CplD. You always have the problem of managing the reception of non-posted requests while you are still handling the last Cpl/CplD, probably slowed down even more by de-assertion of tx_ready by the Hard/Soft IP. Altera’s approach to this problem by adding rx_st_mask is nice but rather useless as you have to buffer a whoppin’ 14 more read requests in 64-bit AST mode, and you have to do it inside the application which contradicts the idea of a wrapper IP.  What you actually could do is play with the max_payload_size and RX buffer space allocation performance parameters, actually try to make the PCIe buffer parameters as default as possible.

[Altera_Forum]

Hi Matthias, 

 the problem started to appear on fully functional design. I found, that PC with my device freezes time from time. It was running correctly for hours or days until that freeze - there were running loads of read/write access, DMA read/writes without any problem. After some searching I found that in some situation the control software writes lots of data to my device (reconfiguration of control registers) and that in some cases freezes the PC. So I tried to remove as much as possible from my design to find which part could be responsible for this problem, but until now I did not find any part of my design responsible for freezing of the system - I've ended with design that does not transmit anything and the software only writes to my device...  Martin

[Altera_Forum]

Martin, 

 even if you don’t have to transmit anything I would suggest that you at least design a detector to see whether any unsupported request is handed over on the rx port. Those requiring a completion should be properly managed by your design, even in the cut-down minimum debugging version.  I hope your configuration registers are in a non-prefetchable BAR.  Another issue could be the PC driver. Are you sure the hangups are not a driver’s fault?

[Altera_Forum]

Hi Matthias, 

 all my BARs are 32-bit non-prefetchable and I doubt that the driver is fauly as I also simplified the driver so it is only simple "mapper" of the BARs to user mode memory and the software only writes bunch of data to that memory. There is no special logic left in the driver now.  As for the detector - at least there is some vendor message from the chipset (I suppose) that appears on RX periodically, I think that there is nothing other, but I will create some detector for it. Maybe a dumb question - how can I find that the request requires completion? Just now I have only a list of types of request (from PCIe specification) and I return a completion for every "read" - is there some general bit in the header that tells me that the request is non-posted and needs a completion?  Martin

[Altera_Forum]

Hi Martin, 

 “Read Requests and Non-Posted Write Requests require Completion.[…]”, from PCIe Section 2.2.9. »Completion Rules«. In fact, all non-posted requests require completion, and they are given, for example, in the »Ordering Rules Summary Table«, Cols 3 & 4: Read Requests, I/O (Read and Write), Configuration Requests (Read and Write) or Trusted Configuration Write Requests.  Maybe you can cut down the PC driver+app thing as well? How about writing a driver that just writes some amount of bytes to the device’s configuration register space over and over (e.g. timer-based)? You could write a bottom-half function that does one write cycle and re-schedules itself immediately.

[Altera_Forum]

Hi Matthias, 

 right now I have simplified everything that is possible - the FPGA design now consist only of PCIe IP and some virtual pins - rx_ready is '1', rx_mask is '0', there is some reset logic, interrupt and msi are '0', and the tx side is grounded also.  PC detects my device and I am able to write to it. The driver is now just a "skeleton driver" that only connects to my device and then writes to to it, nothing else. After the write sequence is started, the PC freezes - sometimes it does it immediatelly, sometimes after a few seconds.  I will add some detection logic for detecting non-posted requests, if there is a need to answer to some request, also I will create a detector for the error condition from the very beginning of this post. But after that I am stuck.  This is already the simplest design that I can make - there is nothing more left to put away and it still freezes... Do you have any simple design that has only the Soft PCIe and some driver that communicates with it that you could share with me? Or do you know about any that I could use? Maybe there could be some problem with the hardware...  Martin