FPGA Intellectual Property
PCI Express*, Networking and Connectivity, Memory Interfaces, DSP IP, and Video IP
6343 Discussions

Problem with HIP pcie gen3x16 two functions

AJ
Beginner
2,604 Views

Hi everyone,

 

  Operating system: CentOS Linux release 7.4.1708 

     Base board: ASUS P8Z77-V LK

        FPGA: 1SG280LN2F43E2VG

 

  I have problem with pcie gen3x16 x2 (two functions). FPGA card inserted in slot PCIEX16_1 (blue), slot PCIEX16_2 is available (not used).

Base board provides maximum speed and width (gen3x16), all functions are displayed in the status (lspci):

  01:00.0 Unassigned class [ff00]: Altera Corporation Device e10a (rev 01)

  Subsystem: Device a106:2804

  Flags: bus master, fast devsel, latency 0, IRQ 16

  Memory at f0000000 (32-bit, non-prefetchable) [size=32M]

  Memory at f2000000 (32-bit, non-prefetchable) [size=2M]

  Memory at f22c0000 (32-bit, non-prefetchable) [size=64K]

  Memory at f2280000 (32-bit, non-prefetchable) [size=256K]

  Capabilities: [40] Power Management version 3

  Capabilities: [50] MSI: Enable- Count=1/16 Maskable+ 64bit-

  Capabilities: [70] Express Endpoint, MSI 00

  Capabilities: [100] Advanced Error Reporting

  Capabilities: [148] Virtual Channel

  Capabilities: [178] Alternative Routing-ID Interpretation (ARI)

  Capabilities: [188] #19

  Capabilities: [b80] Vendor Specific Information: ID=1172 Rev=0 Len=05c <?>

  Kernel driver in use: dd_board

 

  01:00.1 Unassigned class [ff00]: Adobe Systems, Inc Device e10b (rev 01)

  Subsystem: Device a106:2804

  Flags: bus master, fast devsel, latency 0, IRQ 16

  Memory at f2240000 (32-bit, non-prefetchable) [size=256K]

  Memory at f2200000 (32-bit, non-prefetchable) [size=256K]

  Capabilities: [40] Power Management version 3

  Capabilities: [50] MSI: Enable- Count=1/16 Maskable+ 64bit-

  Capabilities: [70] Express Endpoint, MSI 00

  Capabilities: [100] Advanced Error Reporting

  Capabilities: [178] Alternative Routing-ID Interpretation (ARI)

  Kernel driver in use: dd_board2

 

  No errors in drivers. 

  The error occurs when addressing the second function - function 1 is always ignored. The problem is in the decoding signal from HIP (rx_st_func_num_o), the signal value is always 0.

In the SignalTap, I can see that the packages for the first function are coming correct: header and data. The headers contain the bar addresses for the first function but rx_st_func_num_o = 0.

Because of this, all requests go to function 0. The project was built in two different software versions (18.0pro, 18.1pro).

  The following pcie configurations work without errors (bar settings, drivers (and test soft) and slot (PCIEX16_1) are the same): gen3x16 with one function, gen3x8 with two functions or one function, 

gen2x8 with two functions or one function.

  Has anyone encountered such a problem?

 

  Intel pcie HIP documentation:

    https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_s10_pcie_avst.pdf

  Base board documentation (page 33):

    https://dlcdnets.asus.com/pub/ASUS/mb/LGA1155/P8Z77-V_LK/E8534_P8Z77-V_LK.pdf?_ga=2.83170837.2056190884.1548158451-500877768.1548158451

Best regards,

Alex

0 Kudos
8 Replies
SengKok_L_Intel
Moderator
710 Views
Hi Sir, To better understand the problem here, are you facing the problem when using the SRIOV where the function 1 can't access? Difference virtual functions should have a difference BDF. Regards -SK
0 Kudos
AJ
Beginner
710 Views

Hi SK Lim,

 

   It's true. I have two physical functions and tlp requests always come in zero function. There are two drivers for working with the device:

dd_board and dd_board2. Driver dd_board only works with ph0 (01:00.0 - e10a), dd_board2 with ph1 (01:00.1 - e10b). In my tests, I only change

the speed and width. BAR settings, drivers (and test soft) are the same. I wrote about this in detail.

 

Best regards,

Alex

0 Kudos
AJ
Beginner
710 Views

HIP settings for gen3x16 two functions

0 Kudos
AJ
Beginner
710 Views

   I did new tests on the new PC (host1). First PC - host0.

   Host0:

       Operating system: CentOS Linux release 7.4.1708

                    CPU: i7-3770K (host0_cpu file)

             Base board: ASUS P8Z77-V LK

                   FPGA: 1SG280LN2F43E2VG

   Host1 (new):

       Operating system: CentOS Linux release 7.6.1810 (Core)

                    CPU: i7 950 (host1_cpu file)

             Base board: ASUS P6X58D-E

                   FPGA: 1SG280LN2F43E2VG

 

   The problems are the same. Test for host1 (slot PCIEX16_1(blue)): Gen3x16 (two functions) - failure. Gen2x16 (two functions), Gen3x8 (two functions) - OK.

New test for host0: Card in PCIEX16_1: Gen2x16 (two functions) - OK, card in PCIEX16_2: Gen3x16 (two functions) - failure, Gen2x16 (two functions) - OK.

   I also attach the full output of pcie statuses for host0 and host1.

 

   Host1 Base board documentation (page 40):

       https://www.asus.com/Motherboards/P6X58DE/HelpDesk_Manual/

0 Kudos
SengKok_L_Intel
Moderator
710 Views

​HI Sir,

 

What Quartus version you are using? Could you please try to use the latest version v18.1.1? Are the design can pass PCIe Gen3x8 with two Physical Function?

 

Regards -SK

0 Kudos
AJ
Beginner
710 Views

Hi SK Lim,

 

  The PCIE Gen3x8 with two functions passed the test (18.0). In the previous messages listed the tested settings. 

  Explanations for my messages: 18.0 - 18.0.0 Build 219, 18.1 - 18.1.0 Build 222. These versions I use.

  Only one configuration does not pass tests - gen3x16 two functions (Avalon-ST 512). Project was compiled in two versions: 18.0, 18.1.

  Are you looked the hip settings? It's are correct settings?

  I will try to compile in the new version (v18.1.1).

Best regards,

Alex

0 Kudos
AJ
Beginner
710 Views

Hi SK Lim,

 

  I found the cause of the error. Parameter enable_sriov_hwtcl must be equal to 1 (hip settings in hip_1811 file). I compiled the project with the corrected parameter in the latest version (v18.1.1).

Gen3 x16 configuration passed tests.

  The reason is in the adapter.

  Path: .../hip/altera_pcie_s10_gen3x16_adapter_inst/u_rx_st_if 

  Module: u_rx_st_if (altera_pcie_s10_gen3x16_rx_st_if)

 

    module altera_pcie_s10_gen3x16_rx_st_if

     #(

      parameter pld_rx_parity_ena = "enable",

      parameter enable_sriov_hwtcl = 0

     )( 

      ...

       

      //avst rx interface

      ...

      input logic       rx_st_func_num_i,

      ...  

     

      output logic [1:0]    rx_st_func_num_o,

      ...

      );

   

      localparam  par_sriov  = (pld_rx_parity_ena == "enable" && enable_sriov_hwtcl)? 1 : 0;

      localparam  only_sriov = (pld_rx_parity_ena != "enable" && enable_sriov_hwtcl)? 1 : 0;

      localparam  only_par  = (pld_rx_parity_ena == "enable" && !enable_sriov_hwtcl)? 1 : 0;

     

      ...

     

      //async FIFO read interface

      always @ (posedge clk250)..

       begin

         ...

         if (rst_clk250)

          begin

            ...

            rx_st_func_num_o <= '0;

            ...        

          end

         else

          begin

            ...

            rx_st_func_num_o  <= (par_sriov) ? (rx_fifo_rdreq_q? {rx_fifo_rddata_hi[304], rx_fifo_rddata_lo[304]} : '0) : (only_sriov) ? (rx_fifo_rdreq_q? {rx_fifo_rddata_hi[272], rx_fifo_rddata_lo[272]} : '0) : '0 ;

            ...

          end

       end // always @ (posedge clk250)

 

      ...

    endmodule

 

    If enable_sriov_hwtcl = 0 then par_sriov = 0, only_sriov = 0 and rx_st_func_num_o is always zero.

 

  I have two questions.

  1. Is this a wrong description in the documentation?

  From documentation:

    4.2. Multifunction and SR-IOV System Settings

    Table 13. Multifunction and SR-IOV System Settings

    ...

    Enable SR-IOV Support  On/Off  When On, the variant supports multiple VFs. When Off, supports PFs only. SR-IOV is only available in H-Tile devices.

    ...

 

  My device is L-Tile. For non-Gen3x16 configurations, this parameter can be omitted (tested in 17.1.0, 18.0.0, 18.1.0, 18.1.1). For Gen3x16 physical function 1 is not available (enable_sriov_hwtcl = 0).

  My system is built using composition scripts. Adding an HIP instance with 2 functions:

 

  Before (non-Gen3x16 configurations):

 

    add_instance  HIP  altera_pcie_s10_hip_ast

    ...

    set_instance_parameter_value  HIP  enable_multi_func_hwtcl  1

    set_instance_parameter_value  HIP  total_pf_count_hwtcl   2

    ...

 

  After:

 

    add_instance  HIP  altera_pcie_s10_hip_ast

    ...

    set_instance_parameter_value  HIP  enable_multi_func_hwtcl  1

    set_instance_parameter_value  HIP  enable_sriov_hwtcl    1 <-- FIX Gen3x16

    set_instance_parameter_value  HIP  total_pf_count_hwtcl   2

    ...

 

  2. I use the parameter enable_multi_func_hwtcl, its was in 17.1:

    4 Parameters

    ...

    Table 10. System Settings

    ...

    Enable multiple physical functions  On/Off  When On, you can enable multiple physical functions (PFs) on the Multifunction and SR-IOV System Settings tab. You can also also enable TLP Processing

                            Hints (TPH) and Address Translation Services (ATS) on the TPH/ATS Capabilities tab

 

    ...

    4.2 Multifunction and SR-IOV System Settings

    ...

    Enable SR-IOV Support   On/Off  When On, the variant supports multiple VFs. When Off, supports PFs only.

    ...

 

  This parameter is missing in the latest documentation, but I can assign it in the scripts. Do I need to use it? 

  The description of SR-IOV is also different. In the latest edition there is a clarification about the tile. Is that the correct description?

Best regards,

Alex

0 Kudos
SengKok_L_Intel
Moderator
710 Views

​Hi Sir,

 

Thank for this update. You are actually using L tile device, and the "multiple physical functions" is actually not available for the L tile device. If you open PCIe GUI for H-tile, you will see there is an option for "enable multiple physical functions". Since this feature is not officially available for L-tile,  you probably can hack the lower level file to achieve the same feature, but this is not guarantee that the functional behavior is work as expected.

 

Regards -SK 

0 Kudos
Reply