AGFB014R24B2E3V Agilex hanging during Linux boot

[letmein]

Hello,

We brought up a board using AGFB014R24A2E2VR0 Agilex parts, but now that we received new boards with the AGFB014R24B2E3V Agilex, the new boards hang in Linux.

Our board has 4GB of RAM. If I reduce the amount of memory available to Linux (passing in up to 'mem=3584M') then the new boards boot as expected.

If I don't reduce the amount available, during the Linux boot process:

 - Issues interpreting the devicetree properties(?)

 - Bad page state in the process swapper

I've attached the boot log for the failed boot on the new board with the new Agilex variant.

The linux version used is 5.4.34-lts from the linux-socfpga repo, and the u-boot version used is 2020.07 from the uboot-socfpga repo. The .jic for the board was built using Quartus 21.2 tools.

Any idea what could be causing this? Let me know if you need any other information.

Thanks!

[BoonBengT_Intel]

Hi @letmein,

Thank you for posting in Intel community forum and hope all is well.
Due to some unforeseen system glitches, which is causing some delayed in the response.

However request has reached our attention and we are actively working on it.
Will get back to you as soon as possible.
Thanks you for your patients.

Best Wishes
BB

[EBERLAZARE_I_Intel]

Hi,

From the boot logs, it can see that your Uboot booting fine, can you share you Uboot device tree and your Linux device tree?

[EBERLAZARE_I_Intel]

Hi,

Also, what is the DDR device part number? And also could you share your Quartus project? I want to see the HPS DDR EMIF IP settings.

[letmein]

Hi,

The DDR device part number is: Micron MT40A512M16LY-062EAT:E

I've attached the Linux and U-boot device tree files along with the Quartus project Qsys and top level files. If you need more from the Quartus project, I can provide it privately.

Thanks!

[EBERLAZARE_I_Intel]

Hi,

I will contact you via PM/email to request the project file. I am still checking the files you have sent.

Also, I am drafting some pre-check list from my side to check the board differences/logs etc., I will sent it to you shortly.

[EBERLAZARE_I_Intel]

Hi,

I am still checking the files. This may take some times around 1-2 weeks I will get back to you soon, also I need to further check your project's Platform Designer file and device tree.

[EBERLAZARE_I_Intel]

Hi,

Can you also provide your Uboot/configs file?

Regarding the Uboot device tree, did you rename the device tree files' name? If yes, have you try using the default names without renaming the device tree files name? Will this also see the same issue previously?

[letmein]

Hi,

I have attached a zipped copy of the uboot defconfig as well as the patch to allow for our filenames.

The same BSP image works on the older revision Agilex parts, so it doesn't seem like the device tree file names are the cause. I can try changing them if you think it's worth a shot.

Thanks!

[EBERLAZARE_I_Intel]

Hi,

Yes that is odd, but there are issues when changing the uboot files name as they are called and included in many files. Did changing the files name helped in booting the kernel?

May I know what are the changes or major changes made from the default device tree and configs files? Did you enable anything additional?

Also, did you try to use arm debugger to debug?

I will check on the error in your logs and try to find any clues, I will work on it this week.

[letmein]

Hi,

Changing the names did not have any effect.

For the device tree files, I used some different peripherals to match our board:
 - GMAC1 instead of GMAC 0
 - USB1 instead of USB0
 - UART1 instead of UART0
 - Enabled both I2C0 and I2C1

I changed some settings too:
 - UART1 clock frequency
 - MMC bus width set to 8 instead of 4
 - Model name changed from Agilex SoCDK
 - Updated memory reg property to allocate 4GB instead of 8GB
 - Watchdog status left as disabled
 - Enabled GPIO0 as well

For the defconfig differences:
 - I don't use the rsu commands in the CONFIG_BOOTCOMMAND.
 - My CONFIG_BOOTCOMMAND has fpgaprivhack, which just toggles a register to allow userspace FPGA access.
 - I target a different board and defaults, but this is supported by the patches
 - CONFIG_SPL_SPI_LOAD is set, and the CONFIG_SYS_MEMTEST_START/END is not set.
 - I don't have CONFIG_FPGA_INTEL_PR enabled at U-boot
 - I don't have CONFIG_PANIC_HANG
 - I don't have any of the watchdog options

I have not used the debugger yet.

Thanks!

[EBERLAZARE_I_Intel]

Hi,

May I know how did you change for "Updated memory reg property to allocate 4GB instead of 8GB" ?

[letmein]

Hi,

In the u-boot-board.dtsi file (the file is available in requested_files.zip from an earlier post), I have the following settings for the 4GB memory:
  memory {
    #address-cells = <2>;
    #size-cells = <2>;
    reg = <0 0x00000000 0 0x80000000>, /* 2GB */
    <2 0x80000000 0 0x80000000>; /* 2GB */
    u-boot,dm-pre-reloc;
  };

Thanks!

[EBERLAZARE_I_Intel]

Hi,

You could try using this:

memory {
    #address-cells = <2>;
    #size-cells = <2>;
    reg = <0 0x00000000 0 0x80000000>, /* 2GB */
    <1 0x80000000 0 0x80000000>; /* 2GB */
    u-boot,dm-pre-reloc;
  };

[letmein]

Hi,

I made the changes recommended, and tested it out. The issue was still present. Comparing the differences in the boot logs, nothing stood out to me, but I've attached the file just in case.

Thanks!

[EBERLAZARE_I_Intel]

Hi,

Can you share how did you compile your U-boot and Kernel ?

Also, Is there any different from the steps from the Building Bootloader? If so, please state what is the different steps made:

https://rocketboards.org/foswiki/Documentation/BuildingBootloader#Agilex_45_Boot_from_SD_Card

Additionally, how actually did you reduce the memory and got it to boot up?, (from where, in what environment etc.) when you mentioned "If I reduce the amount of memory available to Linux (passing in up to 'mem=3584M') then the new boards boot as expected."?

Lastly, can you share the success boot logs? 

Is there any different when you boot the "AGFB014R24A2E2VR0 Agilex parts" vs a success booting the "new boards with the AGFB014R24B2E3V" when reduce the memory available to Linux?

Or if possible, you can share both success logs here.

[letmein]

Hi,

Yes, there are differences in the steps.

 - The ATF steps are skipped

 - Buildroot is used to build everything, so toolchain, u-boot, linux, etc

 - the steps for building the SD card image are also different. Our image is created using the genimage tool

The memory was reduced by adding the 'mem=3584M' to the u-boot bootargs that get passed to Linux.

I will add the boot log for the AGFB014R24A2E2VR0 board at a later date as I don't have that on hand.

I've attached the boot log for the AGFB014R24B2E3V board booting successfully with reduced memory.

Thanks!

[letmein]

Hi,

I've attached the boot log for the AGFB014R24A2E2VR0 board booting successfully with reduced memory.

The build dates are different, since I had to regenerate the .jic for the QSPI, and the eMMC image had been rebuilt with some unrelated file changes in the rootFS. The U-boot and kernel settings were unchanged.

Thanks!

[letmein]

Hi,

Are there any updates? Could you please reach out to me via PM/Email.

Thanks!

[EBERLAZARE_I_Intel]

Hi,

I will reach out to you via email, thanks.

[letmein]

Hi,

Did you already try to reach out? I didn't get an email.

Thanks!