This article explains how to implement PCIe MSI-X interrupt in Altera FPGA devices. The article is separated into two parts. The first part explains what is MSI-X and how to use it in a PCIe system. The second part explains how to implement it in the FPGA.
MSI-X allows a larger number of interrupts compare to MSI interrupts, it gives each one a separate target address and data word.
MSI-X defines a separate extension to basic MSI functionality. Compared to MSI, MSI-X supports a larger maximum number of vectors per function, the ability for the software to control each vector, and the ability for each vector to use an independent set of address and data values. All this is implemented through a table which resides in the device’s memory space.
MSI-X is an optional feature that provides a method of in-band signaling of an interrupt. This feature enables a device function to request service by writing a system-specified data value to a system-specified address. System software initializes the message address and message data during device configuration.
Figure 1 shows the flow diagram for MSI interrupt (MSI-X is similar to the flow diagram as below too).
Figure 1 - MSI Generation and Handling Flow
A function is permitted to implement both MSI and MSI-X, but only one is allowed to be enabled at a time. If the system software enables both at the same time, the result is undefined.
For the sake of software backward compatibility, MSI and MSI-X use separate and independent capability structures.
For comparison purpose, the MSI-X capability structure is illustrated in Figure 2 below and the MSI capability structure is shown in Figure 3 as a reference. Each function can only have one MSI-X capability structure.
Figure 2 - MSI-X Capability Structure
Figure 3 - MSI Capability Structure
Compared to the MSI capability structure, the MSI-X capability structure does not contain all controls and status information. The MSI-X capability structure instead points to a MSI-X Table structure and a MSI-X Pending Bit Array (PBA) structure, both of them reside in the device’s memory space. Each structure is mapped to the device’s memory space by a PCIe BAR which is located beginning at offset 10h in the Configuration Space.
The Table BIR (BAR Indicator Reigster) and PBA BIR in the MSI-X capability structure indicates which PCIe BAR the structure is mapped to. For example, if the value in BIR is 0, it means the structure is mapped through PCIe Bar0 at offset 10h in the Configuration Space. If the value in BIR is 1, it means the structure is mapped through PCIe Bar1 and so on. The BAR is permitted to be either 32-bit or 64-bit, but the BAR must be memory mapped, not mapped to IO space. The two structures can be mapped to the same BAR or two different BARs.
The Offset is QWORD-aligned and indicates where the structure begins relative to the base address associated with the BAR.
The MSI-X Table Structure contains multiple entries and eachentry represents one interrupt vector. Each entry has 4 QWORDs and consists of a32-bit lower Message Address, 32-bit upper Message Address, 32-bit data, and asingle Mask bit in the Vector Control field as shown in Figure 4 below. When the device wants to transmit a MSI-X interrupt message, it does
Figure 4 - MSI-X Table Structure
The MSI-X PBA Structure is shown Figure 5 below. Each pending bit is associated with one entry in the MSI-X Table Structure.
Figure 5 - MSI-X PBA Structure
The number of interrupts a function can generate depend on the number of entries in the Table Structure. This is defined when configuring the PCIe HIP and this value is reflected in the Table Size portion of the Message Control register. This read only register is part of the MSI-X Capability Structure. The system software will read this field to determine the size of the table and then configure the MSI-X Table Structure. For example, if the software reads back the value of 3, it means the table has 4 entries, from entry 0 to entry 3
Altera PCIe HIP contains the MSI-X Capability Structure, but not the Table and PBA structures. When the user configures the HIP in Qsys, the parameters of the MSI-X Capability Structure need to be filled in, so Quartus II can generate the IP as requested. Select the MSI-X tab in PCI Express/PCI Capabilities section. Enable the Implement MSI-X option, fill in the parameters, such as the Table Size, Table Offset, Table BAR Indicator (Table BIR), Pending bit array (PBA) Offset, and PBA BAR Indicator (PBA BIR) as shown in Figure 6.
Altera PCIe Hard IP core supports up to 2048 MSI-X Table size. Legal range is 0–2047.
5/51/Msix_figure6.png ( Msix figure6.png - click her to view image )
Figure 6 - MSI-X Tab when Configuring PCIe HIP in Qsys
Table size states the size of the MSI-X Table. The offset tells the memory map offset of the structure in the PCIe BAR. The BAR indicator decides which PCIe BAR the structure is mapped to.
For the Table Structure and PBA Structure, both of them are implemented in user application logic. They can be implemented using on-chip memory or registers in the FPGA. The Figure 7 shows a block diagram where the two structures use two separated BARs and both of them have zero offset. The MSI-X Table structure uses BAR0 in the PCIe configuration space and the PBA Structure uses BAR1.
7/7f/Msix_figure7.png ( Msix figure7.png - click here to view image )
Figure 7 - Example Implementation of MSI-X Table and PBA through two BARs
Two MSI-X structures can be mapped to one BAR, but the two structures should have two different offsets and must not overlap. Figure 8 shows the block diagram of this implementation. In the example, both structures are mapped to BAR2 in the PCIe configuration space. The MSI-X PBA structure has offset 0 and MSI-X Table structure has offset 1000h.
d/df/Msix_figure8.png ( Msix figure8.png - click here to view image )
Figure 8 - Example Implementation of MSI-X Table and PBA through one BAR
For Avalon-ST designs, user logic can determine whether the BAR is hit by monitoring rx_st_bardec1 or rx_st_bardec1 and rx_st_bardec2 depending on the PCIe HIP configuration. For Avalon-MM designs, each bar has its own Avalon-MM bus. The two structures can connect to the corresponding Avalon-MM bus. The user logic also needs to check the lower few bits in the address field to determine if the offset is correct. For example, assume the offset of the Table Structure is 1000_0000h. In this case, the user logic should monitor the lower 29 bits. The packet targets the structure when the address is between 1000_0000h to 1000_0000h + table size * 4.
The following is the flow about how the MSI-X is configured and is used.
The PCIe IP core’s exported below interrupt signals when you turn on MSI-X under PCI Express/PCI Capabilities setting.
- MsixIntfc_o[15:0]: Provides for system software control of MSI-X as defined in Section 18.104.22.168 Message Control for MSI-X in the PCI Local Bus Specification, Rev.3.0. The following fields are defined:
MsixIntfc_o[10:0]: Table size
This example design shows the MSI-X interrupts simulation. It is based on Cyclone V device. MSI-X Design Example for Cyclone V
Follow the steps below to generate the testbench system to simulate the MSI-X design
1. Unzip the folder.
2. Open Quartus.
3. Open msix_sys.qsys.
4. Generate > Generate Testbench System: select "Standard" and the simulation model language.
5. Generate the testbench.
6. Replace ../msix_avmm_ref_design/msix_sys/testbench/msix_sys_tb/simulation/submodules/altpcietb_bfm_driver_avmm.v with ../msix_avmm_ref_design/altpcietb_bfm_driver_avmm.v.
7. Simulate using scripts.
a) Open ModelSim
b) cd <your project directory>/msix_avmm_ref_design/msix_sys/testbench/mentor
c) do msim_setup.tcl
e) do wave.do
e) run -all
8/83/Figure_1-2.png ( Figure 1-2.png - click here to view image )
e/e2/Figure_3.png ( Figure 3.png - click here to view image )
Note: Avalon-MM-to-PCI Express supports both 32- and 64-bit addresses.
If you select 64-bit addressing the bridge does not perform address translation.
However, when you specify 32-bit addresses, the Avalon-MM address of a received request on the TX Avalon-MM slave port is translated to the PCI Express address before the request packet is sent to the Transaction Layer.
The Avalon-MM-to PCIe Address (A2P) translation table is access through CRA interface.
Assume a design has 2 pages each with 1MB size (20 bits):
Bit[19:0] are passed through and become PCI Express address bits [19:0]. (due to the 1MByte page size -20 bits)
Bit select the address translation table entry (0=select page#0, 1=select page#1)
Bit[31:21] select the TX slave module port from among other slaves connected to the same master by the system interconnect fabric. The decode is based on the base addresses assigned in Qsys. For setting up the translation table page 1:
At 0x1000 bit[1:0] set to 0x0 for 32 bit addressing.
At 0x1000 bit[31:21] is the host physical address.
At 0x1004 bit[31:0] is set to 0x00000000 (Writing all 0 to this address for 32 bits PCIe system.)
The steps to set up second address page is almost similar to page 1. (also applicable if there are more than 2 address pages)
To setup second page, use the PCIe CRA (Control Register Access) at offset 0x1008 - 0x100C.
At 0x1008 bit[1:0] set to 0x0 for 32 bit addressing.
At 0x1008 bit[31:21] is the host physical address.
At 0x100C bit[31:0] is set to 0x00000000 (Writing all 0 to this address for 32 bits PCIe system)