PCIe core MSI-X support

[Altera_Forum]

I have added MSI-X support to an implementation of the Altera PCIe core we're already testing in the lab. My understanding, is that with MSI-X enabled in the megawizard and my vector table offset set appropriately, I have created the vector table in my own register set at that offset. I then, as I understand it, trigger an MSI-X interrupt simply by issueing a single DW PCIe Memory Write to the appropriate MSI-X address with the appropriate data, as specified in the vector table. As I understand it, the specific MSI I/O to the core isn't used for MSI-X, and you just do a normal memory write. 

 However, my write doesn't seem to appear in shared memory in the simulation. I'm not sure what I'm doing wrong. Is my above understanding correct? I have other data moving back and forth over the link without trouble.  I notice that in the shared memory the code 0x00207811 appears, in the first interrupt address, even with no interrupt happening, image attached. Also attached is a wave view of my MSI-X single DW write.

[Altera_Forum]

--- Quote Start ---  MSI-X interrupt simply by issueing a single DW PCIe Memory Write to the appropriate MSI-X address with the appropriate data, as specified in the vector table.  --- Quote End ---   Memory Write address is 32 bit or 64 bit?

[Altera_Forum]

I don’t think your understanding of the location of the MSI-X vector table is correct. 

 The vector table is not placed inside the PCIe IP (hard or soft). Instead, the application has to present it behind the application interface (probably Avalon ST), and the values you enter in the MegaWizard for the address offsets and BARs denote your application’s expectations of those accesses.  Your application must be aware of the data written by the OS/driver to the given address ranges anyway, as it needs it for sending out the proper MSI-X messages.  The only job that the MegaWizard will do is implement the MSI-X Capability Structure including the control registers, so that your device will be recognized by the OS/driver as MSI-X capable and controllable. Before you are allowed to issue MSI-X writes, you have to make sure you are allowed to do so by polling cfg_msixcsr. This is done by the OS only when all entries of the MSI-X table have been written, i.e. are known to the application so that successive MSI-X write accesses can hit the right target (interrupt number).

[Altera_Forum]

--- Quote Start ---  The vector table is not placed inside the PCIe IP (hard or soft). Instead, the application has to present it behind the application interface (probably Avalon ST), and the values you enter in the MegaWizard for the address offsets and BARs denote your application’s expectations of those accesses.  --- Quote End ---   Yes. I'm get one megabyte in PCI memory and set MSI table to 0x80000. I'm make four 32bit registers for data and address ({32_bit_adress_low,32_bit_data}). But this is only low 32 bit of address. MSI-X required full 64 bit address. For generate IRQ I need generate memory write with 64bit address {32_bit_adress_high,32_bit_adress_low} with data in "32_bit_data".

[Altera_Forum]

Look at the other thread, I gave you a link to the correct MSI-X description (here it is again (http://www.pcisig.com/specifications/conventional/msi-x_ecn.pdf)). Your description is most probably from the PCI 3.0 pre-release which incorporates an early version of msi-x. I have that pre-release, too, so I understand your confusion :)

[Altera_Forum]

Thank for all.  

Now my MSI-X table for two entry looks line:  msix0_addr = msix1_addr = FFFFFFFFFFF0F00F // I test it on 64 bit Linux.   msix0_ctrl = msix1_ctrl = FFFFFFFE // LSB is 0, all ok.   msix0_data = FFFFFFA1  msix1_data = FFFFFFA9  looks good too.      I'm get msixcsr data from tl_cfg bus.   cfg_msixcsr[15] - MSI-X Enable – good.   cfg_msixcsr[14] - 0, “each vector’s Mask bit determines whether the  vector is masked or not. and msi_ctrl[0] i have 0.” Yes. I have.      I'm make memory write.      TLP_FMT_4DW_W, TLP_TYPE_WRITE, Last BE = 4'b0000, First BE = 4'b1111.  Adress is FFFFFFFFFFF0F00F.      But no interrupt.   looks very strange LSB in FFFFFFFFFFF0F00F. It's must be 2'b00.

[Altera_Forum]

The address looks strange indeed. Can you trap the write accesses to these registers and look what address Linux actually writes, on the AST level (as write data)? Is your MSI-X Table handler prepared for all kinds of write accesses, especially if they are aggregated? 

 Remember: The MSI-X Table layout specifies for Message Address bits 01:00: «For proper DWORD alignment, software must always write zeroes to these two bits; otherwise the result is undefined. The state of these bits after reset must be 0. These bits are permitted to be read only or  read/write.»

[Altera_Forum]

--- Quote Start ---  The address looks strange indeed. Can you trap the write accesses to these registers and look what address Linux actually writes, on the AST level (as write data)? Is your MSI-X Table handler prepared for all kinds of write accesses, especially if they are aggregated?   --- Quote End ---   I'm support no more than 64bit burst. Because I'm read x86 CPU can't generate burst bigger. I will check. Other's registers works ok.      --- Quote Start ---   Remember: The MSI-X Table layout specifies for Message Address bits 01:00: «For proper DWORD alignment, software must always write zeroes to these two bits; otherwise the result is undefined. The state of these bits after reset must be 0. These bits are permitted to be read only or  read/write.»  --- Quote End ---   Yes.

[Altera_Forum]

btw, I don’t know why msix*_ctrl has so many '1's in it. Those bits should be cleared at reset, and the OS should not write anything beside zero into it, anyway.

[Altera_Forum]

You might be curious to read intel’s apic spec regarding msi (http://www.intel.com/assets/pdf/manual/253668.pdf) (see Chapter 10.11) to get a feeling about valid values of the MSI/MSI-X message address and data.

[Altera_Forum]

--- Quote Start ---  You might be curious to read intel’s apic spec regarding msi (http://www.intel.com/assets/pdf/manual/253668.pdf) (see Chapter 10.11) to get a feeling about valid values of the MSI/MSI-X message address and data.  --- Quote End ---   Thank! It's help.   Now MSI-X is working.

[Altera_Forum]

Hello! It's seems that I'm made everything ok and interrupts came to CPU. But problem appear - MSI-X is working only on 1 PC (I have tried on other PC and on 2 servers). I saw on Signal TAP that MSI-X write-packet is formed exactly like on 1st PC... 

Some additional information: 1. I'm using Arria II GX and PCIe Hard IP.  2. MSI interrupt are ok on all computers 3. All other write packets (not MSI-X) are also ok on all 4. Format of the MSI-X packet assign Sig_Packet_Type_field = 8'h60; assign Sig_Packet_Length_field = 10'b00_0000_0001; Clk1: PCIe_data_out [63:0]<= {Requester_ID, 16'h00_0f, Sig_Packet_Type_field, 14'b0, Sig_Packet_Length_field}; Clk2: PCIe_data_out [63:0]<= {msix_Msg_Lower_Addr[msix_Number],msix_Msg_Upper_Addr[msix_Number]} Clk3: PCIe_data_out [63:0]<= {Sig_Packet_Data_field, 32'b0};

[Altera_Forum]

You know that you must use a 3DW header if msix_Msg_Upper_Addr[msix_Number] is all zeroes?

[Altera_Forum]

Do You mean 3-DWord Header TLP and memory write request 32bit addr? 

So how should packet look like if msix_Msg_Upper_Addr[msix_Number] is all zeroes? Like this - only two clk: assign Sig_Packet_Type_field = 8'h40; // 32bit addr assign Sig_Packet_Length_field = 10'b00_0000_0001; // 1 DWORD // LastBE = 4'h0, FirstBE = 4'hF Clk1: PCIe_data_out [63:0]<= {Requester_ID, 16'h00_0f, Sig_Packet_Type_field, 14'b0, Sig_Packet_Length_field};  Clk2: PCIe_data_out [63:0]<= {msix_Msg_Lower_Addr[msix_Number], Sig_Packet_Data_field [31:0]}  Or like this – 3 clk: assign Sig_Packet_Type_field = 8'h40; // 32bit addr assign Sig_Packet_Length_field = 10'b00_0000_0001; // 1 DWORD // LastBE = 4'h0, FirstBE = 4'hF Clk1: PCIe_data_out [63:0]<= {Requester_ID, 16'h00_0f, Sig_Packet_Type_field, 14'b0, Sig_Packet_Length_field}; Clk2: PCIe_data_out [63:0]<= {msix_Msg_Lower_Addr[msix_Number],32'b0} Clk3: PCIe_data_out [63:0]<= {Sig_Packet_Data_field, 32'b0};  It seems to me that I tried already both variants but they don't work :( HELP! :) P.S. I checked msix_Msg_Upper_Addr is all 0! On 1st PC where MSI-X are working, I send 4DW header and msix_Msg_Upper_Addr still is all 0...

[Altera_Forum]

--- Quote Start ---  Do You mean 3-DWord Header TLP and memory write request 32bit addr? […] Or like this – 3 clk: […]   --- Quote End ---    Yes, either the first one (if addr mod 8 == 4) or the second one (if addr mod 8 == 0).    --- Quote Start ---   It seems to me that I tried already both variants but they don't work :(    --- Quote End ---    So it’s probably something else, additionally. Does the system properly enable MSI-X at all? There are systems that do support MSI but don’t support MSI-X.

[Altera_Forum]

--- Quote Start ---  Yes, either the first one (if addr mod 8 == 4) or the second one (if addr mod 8 == 0).  --- Quote End ---   I made in this way... but problem was that in 2clk packet I send Clk2: PCIe_data_out [63:0]<= {msix_Msg_Lower_Addr[msix_Number], Sig_Packet_Data_field [31:0]} but should be Clk2: PCIe_data_out [63:0]<= {Sig_Packet_Data_field [31:0],msix_Msg_Lower_Addr[msix_Number]}  Thank You for help!!!! :) Now everything is ok. P.S. 1st PC understand write packet with 64 bit addr even if upper addr is all 0... but other don't understand... (they understand just 32bit addr packets)

[Altera_Forum]

Congratulations! ;)