Updating FIM and BMC for Arria 10 PAC

[vincenzo_m]

Hi, I am running on Ubuntu 18.04, kernel 4.15.0-201-generic, and I need to update the firmware for an Arria 10 PAC to the latest version.

I cannot load any GBS on the board, e.g.,:

$ sudo fpgaconf /inteldevstack/a10_gx_pac_ias_1_2_1_pv/hw/samples/hello_afu/bin/hello_afu_unsigned.gbs -vvvv
Reading bitstream
Looking for slot
No suitable slots found.
Expected Interface id:  d4a76277-07da-528d-b623-8b9301feaffe
Actual Interface id:    38d782e3-b612-5343-b934-2433e348ac4c

This is the current output of fpgainfo:

$ sudo fpgainfo fme
Board Management Controller, microcontroller FW version unavailable
Last Power Down Cause: unavailable
Last Reset Cause: unavailable (can't open)
//****** FME ******//
Object Id : 0xEF00000
PCIe s:b:d:f : 0000:01:00:0
Device Id : 0x09C4
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x123456789ABCDEF
Bitstream Version : 1.2.3
Pr Interface Id : d4a76277-07da-528d-b623-8b9301feaffe
Boot Page : user

From https://www.intel.com/content/www/us/en/docs/programmable/683415/1-0-production/1-0-reference-table.html , I can make out that a flash failover occurs and the factory image is loaded.

I installed the Acceleration Stack v1.2.1 with both updates from https://www.intel.com/content/www/us/en/software-kit/665840/intel-pac-with-intel-arria-10-gx-fpga-acceleration-stack-version-1-2-1.html . This are the resulting installed modules:

$ lsmod | grep fpga
intel_fpga_pac_iopll 16384 0
intel_fpga_pac_hssi 24576 0
intel_fpga_fme 94208 0
intel_fpga_afu 36864 0
intel_fpga_pci 28672 2 intel_fpga_fme,intel_fpga_afu
fpga_mgr_mod 16384 1 intel_fpga_fme
ifpga_sec_mgr 16384 2 intel_fpga_fme,intel_max10

I tried to perform an OTSU update, as suggested in https://www.intel.com/content/www/us/en/docs/programmable/683633/1-2-1/updating-fpga-flash-and-bmc-firmware.html . It seems that some drivers are either missing or corrupt:

$ sudo fpgaotsu --log-level=debug /usr/share/opae/a10-gx-pac/fpgaotsu/base/otsu-09C4.json
[2023-01-13 19:46:12,823] [DEBUG ] [MainThread] found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2023-01-13 19:46:12,823] [DEBUG ] [MainThread] found device at 0000:01:00.0 -tree is
[pci_address(0000:00:01.0), pci_id(0x8086, 0x1901)]
[pci_address(0000:01:00.0), pci_id(0x8086, 0x09c4)]

[2023-01-13 19:46:12,823] [DEBUG ] [MainThread] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/spi*/spi_master/spi*/spi*"
[2023-01-13 19:46:12,824] [DEBUG ] [MainThread] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/ifpga_sec*"
[2023-01-13 19:46:12,824] [INFO ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:01:00.0 is not secure.
[2023-01-13 19:46:12,824] [DEBUG ] [MainThread] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/avmmi-bmc.*.auto"
[2023-01-13 19:46:12,824] [WARNING ] [MainThread] "requires" expression bmc_aux_fw_rev >= 26889 failed
[2023-01-13 19:46:12,824] [WARNING ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:01:00.0 one or more prerequisite checks failed. Skipping this device.
[2023-01-13 19:46:12,824] [WARNING ] [MainThread] Update starting. Please do not interrupt.
[2023-01-13 19:46:12,824] [INFO ] [MainThread] Total time: 0:00:00.000003
[2023-01-13 19:46:12,824] [ERROR ] [MainThread] One-Time Secure Update failed

I also tried a SUPER-RSU update, getting similar errors:

$ sudo super-rsu --log-level=debug /usr/share/opae/a10-gx-pac/super-rsu/base/rsu-09c4.json
[2023-01-13 19:52:57,877] [DEBUG ] [MainThread ] - found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2023-01-13 19:52:57,878] [DEBUG ] [MainThread ] - found device at 0000:01:00.0 -tree is
[pci_address(0000:00:01.0), pci_id(0x8086, 0x1901)]
[pci_address(0000:01:00.0), pci_id(0x8086, 0x09c4)]

[2023-01-13 19:52:57,878] [DEBUG ] [MainThread ] - could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/ifpga_sec*"
[2023-01-13 19:52:57,878] [WARNING ] [MainThread ] - [0000:01:00.0] does not support secure update
[2023-01-13 19:52:57,878] [ERROR ] [MainThread ] - missing one or more items required by rsu config
[2023-01-13 19:52:57,878] [INFO ] [MainThread ] - super-rsu exiting with code '78'

The same errors happen if I manually try to update a specific piece of firmware, .e.g.,:

$ sudo fpgasupdate /usr/share/opae/a10-gx-pac/super-rsu/base/a10sa4-26895-fw_Release.bin --log-level=state
[2023-01-13 20:29:58,766] [DEBUG   ] fw file: a10sa4-26895-fw_Release.bin
[2023-01-13 20:29:58,766] [DEBUG   ] addr: ssss:bb:dd.f
[2023-01-13 20:29:58,767] [DEBUG   ] hash256: d2ec8a2dc11299da8e4eace776596ba52a685d44e81987f1abb85c846079af6b
[2023-01-13 20:29:58,767] [DEBUG   ] hash384: ec235c4a07ce7a0a2b43bfe9f2fcf2fc18d4a5410abf0fba31bdff5f9e47285d75bd45658ade878988fc736997827045
[2023-01-13 20:29:58,767] [DEBUG   ] file type: BMC Image (Update)
[2023-01-13 20:29:58,767] [DEBUG   ] found fpga objects: ['/sys/class/fpga/intel-fpga-dev.0']
[2023-01-13 20:29:58,767] [DEBUG   ] found device at 0000:01:00.0 -tree is
 [pci_address(0000:00:01.0), pci_id(0x8086, 0x1901)]
    [pci_address(0000:01:00.0), pci_id(0x8086, 0x09c4)]

[2023-01-13 20:29:58,767] [WARNING ] Update starting. Please do not interrupt.
[2023-01-13 20:29:58,768] [DEBUG   ] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/spi*/spi_master/spi*/spi*"
[2023-01-13 20:29:58,768] [DEBUG   ] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/ifpga_sec*"
[2023-01-13 20:29:58,768] [ERROR   ] Failed to find secure device for PAC 0000:01:00.0

I downgraded the Acceleration Stack to v1.1 and tried to flash the stock bitstream, getting a timeout error:

~/intelrtestack/a10_gx_pac_ias_1_1_pv/hw/blue_bits$ sudo fpgaflash user dcp_1_1.rpd
2023-01-13 20:04:23.053942 reversing bits
2023-01-13 20:04:33.040600 erasing 0x04000000 bytes starting at 0x04000000
Traceback (most recent call last):
File "/usr/bin/fpgaflash", line 11, in <module>
load_entry_point('opae.admin==1.0.2', 'console_scripts', 'fpgaflash')()
File "/usr/lib/python2.7/dist-packages/opae/admin/tools/fpgaflash.py", line 1283, in main
ret = rc_fpga_update(args.file, args.type, bdf, args.no_verify)
File "/usr/lib/python2.7/dist-packages/opae/admin/tools/fpgaflash.py", line 872, in rc_fpga_update
no_verify)
File "/usr/lib/python2.7/dist-packages/opae/admin/tools/fpgaflash.py", line 416, in fpga_update
no_verify)
File "/usr/lib/python2.7/dist-packages/opae/admin/tools/fpgaflash.py", line 368, in update_flash
flash_erase(mtd_dev, target_offset, erase_len)
File "/usr/lib/python2.7/dist-packages/opae/admin/tools/fpgaflash.py", line 156, in flash_erase
fcntl.ioctl(file_.fileno(), MEMERASE, ioctl_data)
IOError: [Errno 110] Connection timed out

I also tried to downgrade the Acceleration Stack to v1.0, for both RTE and DEV environments, but they do not support Ubuntu and the installation hangs.

W.r.t. v1.2, the download links, available at https://www.intel.com/content/www/us/en/software-kit/665840/intel-pac-with-intel-arria-10-gx-fpga-acceleration-stack-version-1-2-1.html , do not work.

Furthermore, I do not have JTAG access to the board.

Is it possible to install the latest firmware version from my situation?

Thanks for the help,
Vincenzo.

[JohnT_Intel]

Hi,

We have fixed the download link for 1.2 Acceleration Stack. Please download it and update youur board to 1.2 before moving towards 1.2.1 version

[JohnT_Intel]

Hi,

May I know if you are able to move forward after using DCP 1.2?

[vincenzo_m]

Hi John, 

sorry for the delay, I was still working it out.

I am trying to follow the instructions at https://www.intel.com/content/www/us/en/docs/programmable/683633/1-2-1/selecting-the-correct-update-method.html , but they are outdated:

1. there is no "Archived Versions" in the Acceleration Stack download page anymore;
2. the same goes for "Get Resources" button.

Nevertheless, I succesfully downloaded and installed  v1.2, but the script "setup_gim_and_bmc.h" requires access to Bittware tools and JTAG, but I have none of the two.

I also moved to CentOS 7.4 in order to restart the installation process from scratch from v1.0. Both for v1.0 or v1.1,  after running the installer with no errors or warning, OPAE drivers are not installed in the kernel.

Furthermore, the output of "fpgainfo fme" now shows a different Pr Interface Id, which I can't find reference of in any user guide or anywhere the internet.

The output of all the commands for v1.2 and v1.2.1 is still the same. I think the probles is that the FME driver can't allocate all its subdevices, e.g., from the output of "sudo fpgaotsu --log-level=debug /usr/share/opae/a10-gx-pac/fpgaotsu/base/otsu-09C4.json" (same as above):

[...]
[2023-01-13 19:46:12,823] [DEBUG ] [MainThread] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/spi*/spi_master/spi*/spi*"
[2023-01-13 19:46:12,824] [DEBUG ] [MainThread] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/ifpga_sec_mgr/ifpga_sec*"
[2023-01-13 19:46:12,824] [INFO ] [MainThread] Intel PAC with Intel Arria 10 GX FPGA 0000:01:00.0 is not secure.
[2023-01-13 19:46:12,824] [DEBUG ] [MainThread] could not find: "/sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/avmmi-bmc.*.auto"
[...]

As a matter of fact, these devices are not available on sysfs, not even if I manually instert modules with modprobe, e.g., avmmi-bmc for the flash.

[JohnT_Intel]

Hi,

Can you provide the print information of "fpgainfo fme" and "lspci -s 01:00.0 -vv"?

Thanks

[vincenzo_m]

Of course, here is the output of "fpgainfo fme", sorry I forgot to attach it in the previous post:

$ fpgainfo fme
Board Management Controller, microcontroller FW version unavailable
Last Power Down Cause: unavailable
Last Reset Cause: unavailable (can't open)
//****** FME ******//
Object Id                     : 0xF100001
PCIe s:b:d:f                  : 0000:05:00:0
Device Id                     : 0x09C4
Socket Id                     : 0x00
Ports Num                     : 01
Bitstream Id                  : 0x123456789ABCDEF
Bitstream Version             : 1.2.3
Pr Interface Id               : 4c11003e-c024-51ae-a88b-dc3153ac3ae7
Boot Page                     : user
Board Management Controller, microcontroller FW version 26889
Last Power Down Cause: POK_CORE
Last Reset Cause: None
//****** FME ******//
Object Id                     : 0xF100000
PCIe s:b:d:f                  : 0000:01:00:0
Device Id                     : 0x09C4
Socket Id                     : 0x00
Ports Num                     : 01
Bitstream Id                  : 0x124000200000367
Bitstream Version             : 1.2.4
Pr Interface Id               : 38d782e3-b612-5343-b934-2433e348ac4c
Boot Page                     : user

From this, you can see how I could successfully installed another board, which was previously updated at version v1.2.1, on the system. On the PCIe bus, now I have:

• A new, updated and working board in the slot 01:00.0;
• The old non-working board in the slot 05:00.0, discussed in the previous posts.

Here is the output of lspci for the working board:

$ lspci -s 01:00.0 -vv
01:00.0 Processing accelerators: Intel Corporation Device 09c4
        Subsystem: Intel Corporation Device 0000
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 90100000 (64-bit, prefetchable) [size=512K]
        Region 2: Memory at 90000000 (64-bit, prefetchable) [size=1M]
        Capabilities: <access denied>
        Kernel driver in use: intel-fpga-pci
        Kernel modules: intel_fpga_pci

Here is the lspci output for the non-working board:

$ lspci -s 05:00.0 -vv
05:00.0 Processing accelerators: Intel Corporation Device 09c4
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 90280000 (64-bit, prefetchable) [size=512K]
        Region 2: Memory at 90200000 (64-bit, prefetchable) [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: intel-fpga-pci
        Kernel modules: intel_fpga_pci

My issue, now, is to update the firmware (FIM and BMC) of the  non-working board to v1.2.1 without JTAG access.

[JohnT_Intel]

Hi,

May I know how do you get the board from 05:00.0? It is having non official bitstream.

Thanks.

[vincenzo_m]

The board from 05:00.0 is the board that was previously plugged in the 01:00.0 slot. 

This was is original "fpgainfo" before I attempted any update:

$ sudo fpgainfo fme
Board Management Controller, microcontroller FW version unavailable
Last Power Down Cause: unavailable
Last Reset Cause: unavailable (can't open)
//****** FME ******//
Object Id : 0xEF00000
PCIe s:b:d:f : 0000:01:00:0
Device Id : 0x09C4
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x123456789ABCDEF
Bitstream Version : 1.2.3
Pr Interface Id : d4a76277-07da-528d-b623-8b9301feaffe
Boot Page : user

After trying some updates with fpgaotsu and super-rsu, with versions v1.1 and v1.2, the signatures changed:

Furthermore, the output of "fpgainfo fme" now shows a different Pr Interface Id, which I can't find reference of in any user guide or anywhere the internet.

[JohnT_Intel]

Hi,

Since the bitstream is currently incorrect then the only way is to use USB Blaster to prograrm the FPGA. May I know if you have the blaster to program the FPGA?

[vincenzo_m]

Thanks for the feedback.

Unfortunately, I do not have an USB Blaster at my location. I have to send the board back to the main lab for that.

One last question: with the USB Blaster, can the bitstream and BMC from version 1.2.1 be programmed directly? Or must the v1.0 be flashed first, and then the various updates applied subsequently?

[JohnT_Intel · Accepted solution]

Hi,

Bitstream with 1.2.1 version will have AES which you will not be able to directly update it. Older version will work without any issue as it wil not have the security feature enable.

[vincenzo_m]

Does this mean v1.2 could be programmed directly via the USB Blaster II?

This way I could update directly to v1.2.1 using the instructions in the User Guide, i.e., with fpgaotsu and super-rsu, without passing through all the old versions.

[JohnT_Intel]

Hi,

It is possible.

[vincenzo_m]

Great.

Thank you very much for the support.

[JohnT_Intel]

Hi,

May I know if you have any other queries?

[JohnT_Intel]

We do not receive any response from you to the previous question/reply/answer that I have provided. This thread will be transitioned to community support. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread. Thank you.