How Credit Works In Stratix V PCIe G3x1 Reference Design

[#1]

How Credit Works In Stratix V PCIe G3x1 Reference Design

Overview

The article covers the concepts of Flow Control mechanism, what is Flow Control credit and how the credits are consumed. And the article also illustrates how the credit works in Altera Stratix V Hard IP for PCI Express Gen3 ×1 reference design. The goal of the article is to briefly guide the user the general idea of the Flow Control mechanism and how it works in the reference design (<install_dir>/ip/altera/altera_pcie/altera_pcie_hip_ast_ed/example_design/sv) through waveforms from simulation.

Flow Control Concepts

Flow Control (FC) mechanism is used by the Requester to track the queue/buffer space available in the Receiver in order to prevent overflow of Receiver buffers and to enable compliance with the ordering rules. The PCI Express protocol requires each device to implement credit-based link flow control for each virtual channel on each port. Flow Control guarantees that transmitters will never send Transaction Layer Packets (TLPs) that the receiver can’t accept. This prevents receive buffer over-runs and eliminates the need for inefficient disconnects, retries, and wait-states on the link. Flow Control also helps enable compliance with PCI Express ordering rules by maintaining separate virtual channel Flow Control buffers for three types of transactions: Posted (P), Non-Posted (NP) and Completions (Cpl).

Each Virtual Channel maintains an independent Flow Control credit pool. The FC information is conveyed between two sides of the link using DLLPs. The VC ID field of the DLLP is used to carry the Virtual Channel Identification that is required for proper flow-control credit accounting.

Flow Control mechanisms is handled by the Transaction Layer in cooperation with the Data Link Layer. The Transaction Layer performs Flow Control accounting functions for Received TLPs and “gates” TLP Transmissions based on available credits for transmission. 

Flow Control Credits

Buffer space is reported by the receiver in units called Flow Control credits. The unit value of Flow Control credits (FCCs) may differ between header and data as listed below:

    • Header FCCs – 4 DWs for completions, 5 DWs for requests

    • Data FCCs – 4 DWs

Based on the three different types of transactions mentioned above, the Flow Control credits can be distinguished as followings:

    • Posted Header (PH)

    • Posted Data (PD)

    • Non-Posted Header (NPH)

    • Non-Posted Data (NPD)

    • Completion Header (CPLH)

    • Completion Data (CPLD)

Flow Control Credits Consumption

According to the type of the specific transaction, the rule for Flow Control credits consumption is as followings:

TLP	Credit Consumed
Memory, I/O, Configuration Read Request	1 NPH unit
Memory Write Request	1 PH unit + n PD units
I/O, Configuration Write Request	1 NPH unit + 1 NPD credit unit Note: size of data written is never more than 1 (aligned) DW
AtomicOp Request	1 NPH unit + n NPD units
Message Requests without data	1 PH unit
Message Requests with data	1 PH unit + n PD units
Memory Read Completion	1 CPLH unit + n CPLD units
I/O, Configuration Read Completions	1 CPLH unit + 1 CPLD unit
I/O, Configuration Write Completions	1 CPLH unit
AtomicOp Completion	1 CPLH unit + 1 CPLD unit Note: size of data returned is never more than 4 (aligned) DWs

Signals Associated With Flow Control In Avalon-ST TX Datapath

The Flow Control credit related signals in Avalon-ST TX datapath of Stratix V reference design are as listed below:  

tx_cred_datafccp	Data credit limit for the completion data payload (associated with corresponding NP Request). The unit of data credit is 4DW.
tx_cred_datafcnp	Data credit limit for the non-posted request data payload (all Reads, I/O Writes, Configuration Writes, and AtomicOps). The unit of data credit is 4DW.
tx_cred_datafcp	Data credit limit for the posted request data payload (Message and Memory Writes). The unit of data credit is 4DW.
tx_cred_hdrfccp	Header credit limit for the completion headers (associated with corresponding NP Request). The unit of header credit is one maximum-size header.
tx_cred_hdrfcnp	Header credit limit for the non-posted request headers (all Reads, I/O Writes, Configuration Writes, and AtomicOps). The unit of header credit is one maximum-size header.
tx_cred_hdrfcp	Header credit limit for the posted request headers (Message and Memory Writes). The unit of header credit is one maximum-size header.
tx_cred_fchipcons	Asserted for 1 cycle each time the Hard IP consumes a credit. The 6-bit of this vector correspond to the following 6 types of credit types: [5]: posted headers [4]: posted data [3]: non-posted header [2]: non-posted data [1]: completion header [0]: completion data During a single cycle, the IP core can consume either a single header credit or both a header and a data credit.
tx_cred_fc_inifinite	When asserted, indicates that the corresponding credit type has infinite credits available and does not need to calculate credit limits. The 6-bit of this vector correspond to the following 5 types of credit types: [5]: posted headers [4]: posted data [3]: non-posted header [2]: non-posted data [1]: completion header [0]: completion data

Examples From Reference Design

Since the transactions in the transceiver side (/pcie_de_gen3_x1_ast64_tb/pcie_de_gen3_x1_ast64_inst/dut/tx) of SV PCIe G3x1 reference design contain only Memory Read transactions and Memory Write transactions, we can only oberve the increment of siganl hdrfcnp, hdrfcp and datafcp. Specifically, signal hdrfcnp which stands for the header credit limit for non-posted request is going to change associated with the Memory Read transactions and the datafcp (data credit for posted request) and hdrfcp (header credit for posted request) is going to change associated with Memory Write transactions. Some examples are given to illustrate how credit increases in Avalon-ST interface.

Initial Value

b/b2/Ini_value.PNG ( Ini value.PNG - click here to view image )

The above figure is observed at the very beginning of the simulation before any transactions, we can see the initial value of those associated signals as follows (all showed in hex):

• Non-posted: header = 38, data = 000;
• Posted: header = 32, data = 166;
• Completion: header = 00, data = 000.

Single Memory Read transaction

The following figure shows a Transaction Layer Packet (TLP) in reference design. 

d/da/Single_MRd_pkt_c.png ( Single MRd pkt c.png - click here to view image )

The content in the red rectangle is the header of the TLP which contains the ID information of the package. We can see it is a Memory Read transaction containing 16DW data after decoding the package. In this case, only one header non-posted credit increases since it is a non-posted transaction. The snapshot indicates the change of the credit is as below.

1/11/Single_MRd_full.PNG ( Single MRd full.PNG - click here to view image )

We can see that the header non-posted credit increases to 39 by 1 and the data non-posted credit has no change.

Multiple Memory Read transactions

The flow control credits increase consecutively when multiple memory read transactions occur.

1/1f/Multiple_MRd_full.PNG ( Multiple MRd full.PNG - click here to view image )

Memory Write transactions

Different with Memory Read transactions, Memory Write transactions consume both header flow control credits and data flow control credits based on the length of the data.