Creating an SD Card for the Intel Agilex® 7 SoC

symmt_Intel · ‎02-28-2024

This time, I'm writing about creating an SD card for the Intel Agilex® 7 SoC. The catalyst for this was the announcement of Intel® Agilex™ FPGAs (code named Sundance Mesa) Delivering Industry-Leading Power-Efficient Performance. I believe this device could serve as a compelling reason to transition from the Cyclone® V SoC in the future.

However, as the device is not yet available, I aim to tackle the creation of an SD card for the Intel Agilex® 7 SoC using the already mass-produced Intel Agilex® 7 F-Series Transceiver SoC Development Kit.

The goal is to migrate the SD card image used in the DE10-Nano/FPGA Cloud Connectivity Kit (Cyclone® V SoC) to enable container operation. This article is intended for those who have experience with Cyclone® V SoC but are new to development with Intel® Stratix® 10 SoC/Intel Agilex® 7 SoC.

Author's Environment

Build SD Card Image
- FPGA Image: CentOS 7.4 with Intel® Quartus® Prime Software 22.3
- Other Image: CentOS 7.4 using Docker (Base Image: Debian 11.5/bullseye)
Write and Customize SD Image: Ubuntu 20.04
Device: Intel Agilex® 7 FPGA F-Series Transceiver-SoC Development Kit (Engineering Sample/ES, Device OPN: AGFB014R24A2E3VR0)
Intel® Quartus® Prime Software Version: v22.3 Pro Edition
GCC Arm Cross Compiler: arm-11.2-2022.02-x86_64

Note:
Regarding the host OS, the version is a bit old since it is the environment I usually use. Articles on RocketBoards.org recommend Ubuntu 20.04. I had to use Docker for building because the versions I used did not work directly.
For this project, I used the ES version of the device. This will only affect the part of creating the FPGA image.

Introduction

It's important to note that most of the commands used in this article are a combination of the following two documents:

The first document allows for building everything at once using Yocto, an open-source-based build project. The second document describes building each component separately.

While building with Yocto provides the advantage of automatically building and using internal tools, it can be challenging to understand what exactly is being executed within the project. Building each component separately requires executing more commands, but it makes it easier to understand what is being built, which might be more familiar to those with Cyclone® V SoC development experience. Furthermore, it might be easier when it comes to customizing options later in the article.

Given this background, this article will be based on Building Bootloader for Stratix 10 and Agilex. (I also aim to cover Yocto customization, as I believe I have a decent understanding of it, time permitting.)

The document mostly outlines the steps, paying little attention to the technical background. This article will cover those steps in practice, including the underlying knowledge that supports them.

Note: As the article became quite extensive, for readability reasons, some parts overlapping with commands found on RocketBoards.org have not been included here. Please make sure to refer to the links on RocketBoards.org for comprehensive guidance.

About the SD Image Configuration

As a recap, let's look at the SD card image used in the FPGA Cloud Connectivity Kit tutorial. Note that this is a direct URL to the SD card image, so please be cautious.

For Cyclone® V SoC, it looked like this:

root@de10nano:~# ls -l /dev/mmcblk0*
brw-rw---- 1 root disk 179, 0 Oct 20 22:19 /dev/mmcblk0
brw-rw---- 1 root disk 179, 1 Oct 20 22:19 /dev/mmcblk0p1 #U-Boot, zImage, DTB
brw-rw---- 1 root disk 179, 2 Oct 20 22:19 /dev/mmcblk0p2 #Linux Filesystem
brw-rw---- 1 root disk 179, 3 Oct 20 22:19 /dev/mmcblk0p3 #Preloader

Three partitions were created: Partition 1 contained U-Boot, zImage, DTB (Device Tree Blob), and essential components for Linux boot; Partition 2 housed the Linux Filesystem (Ubuntu18.04); and Partition 3, which cannot be directly mounted, contained the Preloader.

Now, looking at the Intel Agilex® 7 SoC image:

root@agilex:~# ls -l /dev/mmcblk0*
brw-rw---- 1 root disk 179, 0 Apr 21 21:54 /dev/mmcblk0
brw-rw---- 1 root disk 179, 1 Apr 21 21:54 /dev/mmcblk0p1 #U-Boot, Image, DTB
brw-rw---- 1 root disk 179, 2 Apr 21 21:54 /dev/mmcblk0p2 #Linux Filesystem

The number of partitions has been reduced to two. This is because the Boot Up procedure has changed.

Differences in Boot Up Procedures

The Boot Up procedure for the Intel Agilex® 7 SoC (precisely from Intel® Stratix® 10 SoC onwards) may seem to be refined compared to the Cyclone® V. This is influenced by the SDM (Secure Device Manager).

Cyclone V SoC GSRD

GSRD for Agilex 7 F-Series Transceiver-SoC DevKit (P-Tile and E-Tile)

For a rough understanding, the end of the Preloader stage for Cyclone® V and the end of the ATF (Arm Trusted Firmware) stage for Intel Agilex® 7 SoC present a similar scenario. The SDM stage offers more functionality than the Boot ROM of Cyclone® V, so thinking of SDM as a part of the Boot ROM + Preloader stage might be helpful.

The Presence of SDM

From Intel® Stratix® 10 onwards, the writing of FPGA images has been managed by something known as the Secure Device Manager (SDM). The existence of SDM allows the FPGA to always verify a secure image before configuration, among other benefits.

SDM, in addition to the Boot ROM, possesses a processor and requires initialization at boot time. For the Intel Agilex® 7 SoC Development Kit, the boot image cannot be read from an SD card connected to the HPS (since the SD card is optional), hence it is read from the onboard QSPI Flash.

Note: With the introduction of SDM, there are two types of configuration methods: HPS First and FPGA First. It can be a bit confusing, but the Intel Agilex® 7 SoC FPGA Boot User Guide: Boot Flow Overview details different stages of boot. The method described on RocketBoards.org primarily follows the HPS First boot. The document's FSBL (First Stage Bootloader) includes what was previously described as U-Boot SPL and ATF.

ARM Trusted Firmware-A (TF-A/ATF)

Starting with the Intel® Stratix® 10 SoC, using Arm within the HPS requires the use of Trusted Firmware. This can be understood as the protocol for using Arm A-series processors. Customization situations for this firmware are not commonly needed. For reference, here are some URLs:

This firmware is later stored inside a file named uboot.itb.

U-Boot SPL (Secondary Program Loader)

The Secondary Program Loader (SPL) of U-Boot is not the second loader of U-Boot, but rather the second in the overall boot sequence. The SPL stage involves setting up the minimum necessary I/O and SDRAM configurations. There are various resources on U-Boot SPL that can be referred to for more detailed information.

Develop U-Boot: Generic SPL framework: U-Boot Phases

Summary of SD Card Configuration

Let's recap the necessary files for booting the Intel Agilex® 7 SoC Development Kit. The specific file names mentioned are visible when the SD card is mounted.

Outside the SD Card (Stored in QSPI Flash)

Phase 1 FPGA Image (.jic)
- Phase 1 FPGA Image
- U-Boot SPL Hex/Binary (New)

SD Card: Partition 1

Linux Kernel Image (Image)
Phase 2 FPGA Image (ghrd.core.rbf)
Device Tree Blob (socfpga_agilex_socdk.dtb)
u-boot.itb
- U-Boot Image
- Trusted Firmware-A (New! Described as ATF in articles on RocketBoards.org)
- Device Tree File

SD Card: Partition 2

Rootfs Image

The commands to execute this document are essentially the same as those in the Building Bootloader for Stratix 10 and Agilex document, so section names are directly quoted from there.

Note: In the case of a Yocto build, the Linux Kernel Image, Phase 2 FPGA Image, and Device Tree Blob are all combined into a single file named kernel.itb.

B. Build Hardware Design (Differences Exist Between ES and Production)

In this section, we create configuration files (.sof) containing both Phase 1 and Phase 2 from the GHRD (Golden Hardware Reference Design). Note that some functionalities are limited on the ES board, and the device's OPN differs. Reference: Intel Agilex® 7 FPGA F-Series Transceiver-SoC Development Kit User Guide.

cd $TOP_FOLDER
rm -rf ghrd-socfpga-QPDS-22.3pro-21.1std QPDS-22.3pro-21.1std.zip agilex_soc_devkit_ghrd
wget https://github.com/altera-opensource/ghrd-socfpga/archive/refs/tags/QPDS-22.3pro-21.1std.zip
unzip QPDS-22.3pro-21.1std.zip
mv ghrd-socfpga-QPDS-22.3pro-21.1std/agilex_soc_devkit_ghrd .
rm -rf ghrd-socfpga-QPDS-22.3pro-21.1std QPDS-22.3pro-21.1std.zip
cd agilex_soc_devkit_ghrd
# disable sgmii and partial reconfiguration - to decrease build time
export HPS_ENABLE_SGMII=0
export ENABLE_PARTIAL_RECONFIGURATION=0

export QUARTUS_DEVICE=AGFB014R24A3E3VR0 #ES Board Only
export ENABLE_HPS_EMIF_ECC=0 #ES Board Only

~/intelFPGA_pro/22.3/nios2eds/nios2_command_shell.sh make scrub_clean_all
~/intelFPGA_pro/22.3/nios2eds/nios2_command_shell.sh make generate_from_tcl
# change the board id to be 4 - the one used when booting from SD card

#Production Board
#sed -i 's/set_global_assignment -name STRATIX_JTAG_USER_CODE .*/\
#set_global_assignment -name STRATIX_JTAG_USER_CODE 4/g' ghrd_agfb014r24b2e2v.qsf

# ES Board
sed -i 's/set_global_assignment -name STRATIX_JTAG_USER_CODE .*/\
set_global_assignment -name STRATIX_JTAG_USER_CODE 4/g' ghrd_agfb014r24a3e3vr0.qsf

~/intelFPGA_pro/22.3/nios2eds/nios2_command_shell.sh make all
cd ..

This results in the creation of either `ghrd_agfb014r24b2e2v.sof` (for Production) or `ghrd_agfb014r24a3e3vr0.sof` (for ES). The division into Phase 1 and Phase 2 is done after creating the bootloader (U-boot SPL) binary.

C. Build Arm Trusted Firmware / D. Build U-Boot

This section does not have any changes, so please refer to the documentation. For building the Arm Trusted Firmware, create bl31.bin, and for U-Boot, alongside U-Boot SPL u-boot-spl-dtb.hex, create U-boot's image u-boot.itb.

Building Bootloader for Stratix 10 and Agilex: Agilex - Boot from SD Card

E. Prepare QSPI Image (! ES/Production difference exists !)

For Production boards, follow the documentation on RocketBoards.org exactly. For ES boards, you need to change the QSPI Flash and FPGA's OPN.

~/intelFPGA_pro/22.3/nios2eds/nios2_command_shell.sh \
quartus_pfg -c agilex_soc_devkit_ghrd/output_files/ghrd_agfb014r24a3e3vr0.sof \
ghrd.jic \
-o hps_path=u-boot-socfpga/spl/u-boot-spl-dtb.hex \
-o device=MT25QU02G \
-o flash_loader=AGFB014R24A3E3VR0 \
-o mode=ASX4 \
-o hps=1

Executing this will generate the image ghrd.hps.jic (Phase 1 FPGA Image with U-boot SPL) and ghrd.core.rbf (Phase 2 FPGA Image) to be stored in QSPI Flash.

Linux Kernel Build

The RocketBoards.org documentation moves on to creating the SD Card image next, but since we're still missing some components, we'll adjust the order slightly. Jump to the section Appendix - Building Linux Binaries in the documentation and follow the instructions up to "Building Linux Kernel".

Rootfs

The documentation then uses Yocto's Poky, an open-source build project for creating an embedded Linux distribution. While Poky is lightweight and good for initial trials, using a more general distribution like Ubuntu offers more packages and conveniences. This time, we aim to switch to Ubuntu 20.04.

First, access Ubuntu Base 20.04.5 LTS (Focal Fossa). Here, you can download just the base rootfs. Download the tar.gz file with -arm64 in its name. As of writing (December 2022), version 20.04.5 was the latest, so download that.

cd $LINUX_BIN/a53
wget https://cdimage.ubuntu.com/ubuntu-base/releases/20.04/release/ubuntu-base-20.04.5-base-arm64.tar.gz

The downloaded file is equivalent to the core-image-minimal-agilex.tar.gz built with Yocto.

Note: The downloaded file is minimal, so simply creating and writing the SD card with it will not boot. After writing, you need to edit the SD card image (more on this later).

G. Prepare SD Card Image

Now that all materials are gathered:

(Reiterated)

Outside SD Card (stored in QSPI Flash)
- Phase 1 FPGA Image (ghrd.hps.jic) ([Section E](#e-prepare-qspi-image--esproduction-difference-exists))
  - Phase 1 FPGA Image
  - U-Boot SPL Hex/Binary (u-boot-spl-dtb.hex) ([Section C](#c-build-arm-trusted-firmware--d-build-u-boot))
SD Card: Partition 1
- Linux Kernel Image (Image) ([Linux Kernel Build](#linux-kernel-build))
- Phase 2 FPGA Image (ghrd.core.rbf) ([Section E](#e-prepare-qspi-image--esproduction-difference-exists))
- Device Tree Blob (socfpga_agilex_socdk.dtb) ([Section D](#c-build-arm-trusted-firmware--d-build-u-boot))
- u-boot.itb ([Section D](#c-build-arm-trusted-firmware--d-build-u-boot))
  - U-Boot Image
  - Trusted Firmware-A ([Section C](#c-build-arm-trusted-firmware--d-build-u-boot))
  - Device Tree File
SD Card: Partition 2
- Rootfs Image (ubuntu-base-20.04.5-base-arm64.tar.gz) ([Rootfs](#rootfs))

Combine all these to create one image. For Rootfs, the document differs, so I suggest creating a symlink to easily switch between Yocto and Ubuntu. If using Ubuntu, it's recommended to allocate a larger partition due to the need for downloading packages later.

Also, the document does not mention it, but remember to copy over the kernel modules.

Navigate to the main directory, remove any existing SD card directory, then create a new SD card directory:

cd $TOP_FOLDER
sudo rm -rf sd_card && mkdir sd_card && cd sd_card
wget https://releases.rocketboards.org/release/2020.11/gsrd/tools/make_sdimage_p3.py
chmod +x make_sdimage_p3.py
mkdir sdfs && cd sdfs
cp $TOP_FOLDER/u-boot-socfpga/u-boot.itb .
cp $LINUX_BIN/a53/Image .
cp $LINUX_BIN/a53/socfpga_agilex_socdk.dtb .
cp $TOP_FOLDER/ghrd.core.rbf .
cd ..

A clever trick:

mkdir rootfs-yocto && cd rootfs-yocto # Not needed if not building with Yocto
sudo tar xf $LINUX_BIN/a53/core-image-minimal-agilex.tar.gz
sudo rm -rf lib/modules/*
sudo cp -r $LINUX_BIN/a53/modules/* lib/modules/
cd ..

mkdir rootfs-ubuntu && cd rootfs-ubuntu 
sudo tar xf $LINUX_BIN/a53/ubuntu-base-20.04.5-base-arm64.tar.gz
sudo rm -rf lib/modules/*
sudo cp -r $LINUX_BIN/a53/modules/* lib/modules/
cd ..
ln -s rootfs-ubuntu rootfs

To adjust the partition size, modify the size parameter in lines 2, 3, and 4 (in the following example, partition 1 is 56MB and partition 2 is 800MB, totaling 856MB):

sudo python3 make_sdimage_p3.py -f \
-P sdfs/*,num=1,format=fat32,size=56M \
-P rootfs/*,num=2,format=ext3,size=800M \
-s 856M \
-n sdcard.img
cd ..

Note: Increasing the size will enlarge the image size, thereby increasing the writing time to the SD card.

Writing the JIC File to QSPI Flash

Execute the command as directed in the G. Boot section. The text mentions setting MSEL to JTAG, which is the state "SW1: ON-ON-ON-ON". Conversely, to read from QSPI, return to the state "SW1: ON-OFF-OFF-ON".

cd $TOP_FOLDER/flash_image/
~/intelFPGA_pro/22.3/nios2eds/nios2_command_shell.sh \
quartus_pgm -m jtag -o "pvi;./ghrd.hps.jic"

Writing to the SD Card

Write the completed SD card image using your preferred application or tool. I used dd or balenaEtcher.

$ sudo dd if=sdcard.img of=/dev/<your device path> status=progress

As mentioned in the previous note, inserting the SD card as is into the Intel Agilex® 7 SoC DK will not boot. U-Boot cannot find the Linux init process and stops. Therefore, after writing, you need to customize various settings.

Customizing the SD Card

Re-inserting the SD card should show a new disk added, with two partitions visible.

$ ls /dev/sd* # Before inserting the SD card
sda sda1 sda2 sda3 
$ ls /dev/sd* # After inserting the SD card
sda sda1 sda2 sda3 sdb sdb1 sdb2
$ mkdir mnt # Create directory for mounting
$ sudo mount /dev/sdb2 mnt

In the above example, sdb is the SD card disk. Since sdb2 corresponds to the filesystem, we'll configure this part. There are various ways to configure, and using `mmdebstrap` as mentioned in the Cyclone® V SoC Linux Construction Procedure: 7. Building the Root File System (rootfs) is safe, but let's try a more straightforward push.

First, install QEMU since the architecture is different.

sudo apt install qemu-user-static binfmt-support

After installation, copy the necessary files and mount the subdirectories that are handy when chrooting.

sudo cp -b /etc/resolv.conf mnt/etc/resolv.conf
sudo cp /usr/bin/qemu-aarch64-static mnt/usr/bin
sudo mount -t proc /proc mnt/proc
sudo mount -t sysfs /sys mnt/sys
sudo mount -o bind /dev mnt/dev
sudo mount -o bind /dev/pts mnt/dev/pts

This alone makes it possible to chroot.

sudo chroot mnt

Once you've chrooted, start by installing packages with apt. The author notes that attempting to use apt directly led to errors, so chmod was adjusted on /tmp first. (Reference: sudo apt-get update couldn't create temporary file) It's recommended to install packages for partition size adjustment (`parted`), network-related applications, the `systemd` init process, and your favorite editor.

chmod 1777 /tmp
apt clean && apt update
apt install -y vim git parted iproute2 netplan.io iputils-ping network-manager isc-dhcp-client kmod sudo openssh-server systemd

Also, configure users and other settings.

passwd
adduser agilex
usermod -aG sudo agilex

It's handy to write network settings as well if necessary.

vim /etc/ssh/sshd_config # uncomment PasswordAuthentication yes
cat << EOF > /etc/netplan/00-default.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: yes
EOF

Once finished, exit the session and unmount each directory. Be aware, if subdirectories are not unmounted first, you may encounter a "Target is busy" refusal.

exit # Exit chroot
sudo umount mnt/proc
sudo umount mnt/sys
sudo umount mnt/dev/pts
sudo umount mnt/dev
sudo umount mnt

Now, it's time to boot!

Boot Up and Disappointment

Up to the kernel boot, you should see logs similar to those in the Building Bootloader for Agilex 7 documentation. However, once the Root Filesystem is loaded, it transitions to Ubuntu. Ultimately, if you reach the screen showing:

Ubuntu 20.04.5 LTS localhost.localdomain ttyS0

localhost login: %

This indicates completion. Try logging in with the user ‘agilex’ created earlier. If you've set up the network settings correctly, connecting an Ethernet cable to the same board as the SD card should automatically fetch an IP address, allowing internet access. This marks the successful creation and booting of the SD card image using the latest Intel® Quartus® Prime Software v22.3.

However, unlike the image from FPGA Cloud Connectivity Kit, containers do not operate. Attempting to start services within the container will result in errors during installation:

sudo apt install -y lxc
# Installation error occurs

Refer to: Linux Containers.

In the next part, we will address this issue by introducing how to customize Linux kernel options to enable container operation. Stay tuned.

Tips: Using Docker Containers to Build SD Card Components

Despite having a high-speed machine for building, it's hosted on CentOS 7.4, which is slightly outdated. Due to various circumstances, it's not feasible to simply switch to Ubuntu 20.04. Attempts to work directly with CentOS 7.4 were abandoned due to unresolved package dependencies. However, utilizing Docker containers proved to be a convenient solution for building.

As a base, the following page within RocketBoards.org is available: Docker Yocto Build. Yet, the Dockerfile provided there is designed for Yocto and lacks the packages necessary for direct kernel building. A bit of customization allows for direct building.

The task is straightforward: add the Ubuntu 20.04 packages listed in Building Bootloader for Agilex 7 to the appropriate sections of the Dockerfile. Specifically, execute apt-get as root around line 17, where useradd is located. For execution methods, refer to the original Dockerfile: OE Build Container: Dockerfile.

Note that ‘swig’ and ‘kmod’ were additionally required during the build process. For reference, here's a comparison before and after customization:

$ diff Dockerfile Dockerfile.custom
17a18,26
>
> ARG DEBIAN_FRONTEND=noninteractive
> RUN apt-get update && apt-get install -yq gawk wget git-core diffstat unzip texinfo \
>     gcc-multilib build-essential chrpath socat cpio python python3 \
>     python3-pip python3-pexpect xz-utils debianutils iputils-ping \
>     python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint3 xterm \
>     libncurses-dev gawk flex bison openssl libssl-dev swig kmod && \
>     rm -rf /var/lib/apt-lists/*

After completing the Docker build, run it. Ensure to mount directories containing the Intel® Quartus® Prime Software and sources along with GCC toolchains for convenience. Once logged into the container, proceed with the build as usual:

$ docker build -t buildSA --build-arg username=$(whoami) --build-arg build_dir=$PWD .
$ docker run -v ~/intelFPGA_pro/22.3:~/intelFPGA_pro/22.3 -v $PWD:$PWD -it --rm --user=$UID --name builder buildSA

References

Code names are used by Intel to identify products, technologies, or services that are in development and not publicly available. These are not "commercial" names and not intended to function as trademarks.

Statements in this document that refer to future plans or expectations are forward-looking statements. These statements are based on current expectations and involve many risks and uncertainties that could cause actual results to differ materially from those expressed or implied in such statements. For more information on the factors that could cause actual results to differ materially, see our most recent earnings release and SEC filings at www.intc.com.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or visiting www.intel.com/design/literature.htm.

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

symmt_Intel · ‎02-28-2024

Note: This article is an English translation of this Japanese article. Please refer to the original article for updates.