FPGA passive serial configuration -- detect as PCIe device in Linux after configuration

[Steve-Mowbray-ENL]

Hi

In passive serial configuration the Linux platform powers on and is booted with FPGA in unconfigured state -- our RBF file is then configured via passive serial and working but is not detected as a PCIe connected device in Linux -- the command "echo 1 > /sys/bus/pci/rescan" has no effect...

Is there a user guide or instructions anywhere as to how to establish the PCIe connection in Linux after passive  serial configuration

Thanks

Steve

[Wincent_Altera]

Hi,

May I know which Device that you are using for this ?

Also, are you using any PCIe design example ?

Regards,

Wincent

[Steve-Mowbray-ENL]

Hi Wincent

Arria 10 (10AX066H4F34E3SG) and not using a design example but system is working fine in active serial mode but we are planning on using passive serial mode in product deployment

Thanks

Steve

[Wincent_Altera]

Hi Steve,

For Passive serial configuration overall, you can refer to below link for the guide

https://www.intel.com/content/www/us/en/support/programmable/support-resources/configuration/cfg-ps.html

Hope that able to help you to move forward, let me know if there is any further clarification is needed.

Regards,

Wincent

[Steve-Mowbray-ENL]

Hi Wincent

I dont see any Linux guidance related to the above link -- specifically it is the Linux steps required to get the PCIe endpoint correctly detected in the Linux PCIe master device after passive serial configuration that we need

Regards

Steve

[Wincent_Altera]

Hi Steve,

I am supporting PCIe IP area, based on your reply.
I believe you try to convert the Programming Files RBF into passive serial configuration in order to run the design , am I get this right ?
This is out of my support area actually, however I can try my best to assist

In order to successful run the passive serial file please check below guide
https://cdrdv2-public.intel.com/705024/ug-qpp-programmer-19-3-683039-705024.pdf

1. Can you try to use JTAG mode and see if it can be configure successfully?

2. If in JTAG mode it can be configure successfully, it can be something is wrong with the connection on board or flash. Do you follow the pin connection guidelines? https://www.intel.com/content/www/us/en/docs/programmable/683814/current/intel-arria-10-gx-gt-and-sx-device-family.html 

3. How is the MSEL setting?

Hope my suggestion able to help you to move a step forward, Please accept my apology that this is out of my support area.
IF your issue is still unable to solve, I suggest to file a new forum thread with title example "Arria 10: Passive Serial Configuration-programming failure" . So that our configuration team support specialist is able to look at this.

Let me know if you have different thoughts.

Regards,
Wincent

[FvM]

Hi,
not sure if the observed problem is failed configuration or only missing PCIe device enumeration by PC Bios. In the latter case, the problem may be simply too late configuration. Depending on Bios behaviour, rescan doesn't necessarily help, a reboot may be necessary.

[Wincent_Altera]

Thanks @FvM , for your help.

Hi Steve, 

Do you try to reboot the system and see if the PCIe is able to enumerate ?

Regards,

Wincent

[Steve-Mowbray-ENL]

Hi Wincent -- reboot doesn't appear to force Linux to recognise the FPGA PCIe endpoint but there is the suspicion that the reboot causes the GPIO line tied to NCONFIG to go low...

Regards

Steve

[Steve-Mowbray-ENL]

Hi Wincent -- passive serial programming seems to be working fine as indicated by other non PCIe behaviour is fully functional -- problem is specifically the FPGA endpoint is not recognised in the Linux system after programming and a PCIe rescan doesnt work

Regards

Steve

[Wincent_Altera]

Hi Steve,

Since the Intel FPGA stores its configuration data in SRAM memory, and the PCIe HIP is part of the configuration that needs to happen, this gets configured each time the system boots up, or if a full reconfiguration happens. The Arria 10 handbook (link) briefly describes the configuration sequence and how to trigger a reconfiguration, in specific the configuration step is the one in which the data is written to the FPGA, however it does not explicitly mention what happen to the PCIe HIP block. The Configuring Altera FPGAs  doc (link) describes in more detail the configuration and the reconfiguration process. 

• Can you please check and ensure that those is been followed ?

As far as I know,  At the time the FPGA will be configured via PS, but that needs about 12 secs. To speed up, you may want to change the initialization to CvP. Still the minimal configuration to start the CvP will be done via PS and afterwards just to load the application via PCIe.

• Could you give some time more than 12 secs and see if the system is able to enumerate due to late configuration of FPGA via PS ?

IF those still unable to solve your problem, I suspect to look it can be something is wrong with the connection on board or flash. Do you follow the pin connection guidelines? https://www.intel.com/content/www/us/en/docs/programmable/683814/current/intel-arria-10-gx-gt-and-sx-device-family.html . IF the design didn’t have the PCIe component that it is expected you are not seeing the FPGA as end point.

• Did you see the conf_done asserted high in the fail scenario ?
  

Looking forward to hear back from you.

Regards,

Wincent

[Steve-Mowbray-ENL]

Hi Wincent

• "Can you please check and ensure that those is been followed ?"
  • Yes we have followed that document and seems to be working as evidenced by FPGA operating as expected after passive configuration eg LED's flashing and serial port connectivity etc except Linux doesn't enumerate the PCIe bus
• "Could you give some time more than 12 secs and see if the system is able to enumerate due to late configuration of FPGA via PS ?"
  • I dont understand this request -- we can see FPGA in correct operation by observing LEDs and serial port connection but at this stage rescanning the PCIe bus doesnt work
• "Did you see the conf_done asserted high in the fail scenario ?"
  
  • Yes conf_done is asserted high after passive serial configuration

I am confident that the FPGA side of the configuration is working as expected -- the issue we have is getting the Linux PCI subsystem to connect to the FPGA after passive serial configuration -- if we do an active serial configuration FPGA build and change MSEL pins the FPGA gets PCIe enumerated correctly but we assume this is because the FPGA is ready in time for BIOS level enumeration. For passive serial configuration we are using the Linux shell commands:

echo 1 >/sys/bus/pci/devices/0000:00:00.0/remove
#do FPGA passive serial programming here...
echo 1 > /sys/bus/pci/rescan

 Regards

Steve

[Wincent_Altera]

Hi Steve,

I wondering , do you have any backup host to try this out ?
I am trying to narrow this issue that is not specific host/CPU/BIOS related.

Regards,
Wincent

[Steve-Mowbray-ENL]

Hi Wincent -- I am 100% confident this is not FPGA side rather it is Linux side -- we need guidance on the situation where FPGA is in unconfigured state Linux host boots up with the FPGA PCIe endpoint not present as to what the procedure in Linux is to rescan the PCIe bus to establish PCIe connectivity with the FPGA after Linux host completes passive serial configuration of the FPGA -- so far the possible avenues of investigation we have discovered are:

• Linux kernel boot options (pci=realloc etc)
• Linux native PCI hotplug 
• Linux ACPI PCI hotplug
• Linux custom PCI register bashing (setpci etc and sysfs exposed functions...)

This must be a pretty standard scenario when using any FPGA with Linux host that is based on the passive serial configuration scheme so I am hopeful there is some pre-existing guidance or experience with this but so far we have not been able to find anything detailed enough to replicate on our platform

Regards

Steve

[Wincent_Altera]

Hi Steve,

I am 100% confident this is not FPGA side rather it is Linux side
>> thanks for confirming that, I have program the .sof via normal configuration in either CentOS 7 or Ubuntu 22.04 LTS
>> I never seeing such scenario before

I am not sure if you can try to 

• Perform BIOS update
• Update your Linux version into latest 

And see if you still monitor the same behavior or not.
I trying to check any existing guide, but only option that I am able to find is
https://www.intel.com/content/www/us/en/docs/programmable/683461/current/passive-serial-configuration.html

But I think those are not so targeted to Linux specific use case. Perhaps those can provide you some hint to move forward.

If the issue still not solved, and you confirm that the issue is from Linux sides, I suggest to try post the same question on Linux related forum.

Please accept my apology for cannot help too much on this. 
Let me know if there is anything else on the PCIe IP that you feel I could provide my best assist.

Regards,
Wincent

[Steve-Mowbray-ENL]

Hi Wincent

Thanks for your input thus far -- I will update this thread with the solution when we eventually discover one

Regards

Steve

[Wincent_Altera]

Hi Steve,

Do you think you need any of my support at this moment ?
If not , do I have your permission to close this thread ticket from my end ?
Don't worry it just our internal procedure, this thread is still accessible by community support.

If you discover a new finding related to FPGA , you may file a new thread in future.
Our specialist will be there to support you.

Regards,
Wincent

[Steve-Mowbray-ENL]

Hi Wincent

Actually I have maybe a few questions about your previous response:

>> thanks for confirming that, I have program the .sof via normal configuration in either CentOS 7 or Ubuntu 22.04 LTS
>> I never seeing such scenario before

1. When you power on your Linux system is the FPGA configured (by an active serial image) or completely unconfigured?
2. If unconfigured can you see any trace of the FPGA PCIe endpoint using "lspci" command?
3. What does "program the .sof via normal configuration" mean? Do you program the FPGA via JTAG perhaps?
4. Is there any Linux scripts to run after configuration that are required to reset/rescan the FPGA PCIe endpoint?
5. What Linux kernel boot options on your system?
6. Is PCI hotplug enabled in your system BIOS?

Regards

Steve

[Wincent_Altera]

Hi Steve,

1. When you power on your Linux system is the FPGA configured (by an active serial image) or completely unconfigured?
   >> I just use the default setting without any additional configuration.
   >> But in my cases it is "Passive Serial"
   >> 
2. If unconfigured can you see any trace of the FPGA PCIe endpoint using "lspci" command?
   >> Yes, the PCIe endpoint is able to enumerate after performing host reboot
   >> 

   >> Signaltap shows link up in L0 stage as well
   >>

   
3. What does "program the .sof via normal configuration" mean? Do you program the FPGA via JTAG perhaps?
   >> I use the default settings and JTAG cable , perhaps you can try USB blaster to program it.
4. Is there any Linux scripts to run after configuration that are required to reset/rescan the FPGA PCIe endpoint?
   >> I dont use any linux scripts, I just perform reboot , using $ sudo reboot
5. What Linux kernel boot options on your system?
   >> I am not sure this is helpful for you or not
   >> 
6. Is PCI hotplug enabled in your system BIOS?
   >> I believe Hotplug Flag is enabled by default in linux
   

I share my success design .qar for you as a reference since that it is an "passive serial", perhaps it can give you some clue to resolve the problem that you facing now. The project is compile in Quartus v24.2.
One things I realize that will causing your configuration fail would be, do you check your device "dip switch"
https://cdrdv2-public.intel.com/670687/ug_a10-fpga-prod-devkit-683526-670687.pdf 
I am not sure what is your setting requirement, but you may try to set it as default settings if possible, I ever experience programming fail due to those switch toggling randomly in my board.

Regards,

Wincent

[Steve-Mowbray-ENL]

Hi Wincent

I think you misunderstood my power on Q1 and Q2 questions above -- without any manual FPGA programming steps does your system show the FPGA as a PCI device indicating that there is a working PCIe endpoint running on the FPGA after power on Linux boot? No working PCIe endpoint on the FPGA is the starting state for our system on power on since MSEL pins are set to passive serial.  

Thank you for confirming that $sudo reboot is used to initiate the PCI rescan -- we are hopeful we can use the same providing we identify the missing Linux kernel configurations

You can list kernel boot options with the following:

$cat /proc/cmdline

 Anything PCI would be of interest (eg perhaps pci=realloc)

Regards

Steve

[Wincent_Altera]

Hi Steve,

without any manual FPGA programming steps does your system show the FPGA as a PCI device indicating that there is a working PCIe endpoint running on the FPGA after power on Linux boot?
>> For step to programing the .sof file , you may refer to our official document
>> https://www.intel.com/content/www/us/en/docs/programmable/784468/current/programming-hardware-sof-file.html
>> if you need more clear picture about the how to check the PCIe link up status, programming the .sof , capturing .stp for the ltssm status, you may refer to 
>> https://community.intel.com/t5/FPGA-Intellectual-Property/How-to-run-PCIe-Gen-5-Design-Example-using-Altera-FPGA-device/m-p/1640988 
>> Honestly I am authors of that document on link above, although is targeted Agilex 7 Gen 5 , but the fundamental and step would be 90 % above the same.

You can list kernel boot options with the following:

$cat /proc/cmdline

>>  Here you go, hope that able to help you

If unconfigured can you see any trace of the FPGA PCIe endpoint using "lspci" command?
>> First, I compile the design, then I program the .sof file.
>> then perform sudo reboot. then lspci, detail step you may refer to the guide no2 mentioned above.
>> If you found anything unclear , appreciate if you can provide your valuable feedback to me

Regards,

Wincent