Cyclone 5 first boot linux crashes, howto debug?

[qwitza]

Hello,

We have a design with a Cyclone V FPGA. Some boards only boot after a reset following power-on.

Problem:

The Debian Linux with U-Boot boots from the SD card, but after approximately 10 seconds, the system crashes and becomes unresponsive. The login prompt appears (on UART debug), but I cannot enter anything, and the network interface does not work.

How should I start troubleshooting?

There is a JTAG on the board, but I have never used it until now.

Any hints or links would be greatly appreciated.

Sorry, but I have no idea how to resolve this, or where to start with the troubleshooting.

[qwitza]

Hello,

after some search, it seems that the linux system crashes after writing the fpga firmware.

[JingyangTeh_Altera]

Hi

Are you able to still boot into the uboot?

Regards

Jingyang, Teh

[qwitza]

Hello,

uboot and debian are booted every time.

but after writing the fpga firmware debian crashes, no input or other possible hard crash

9.697986] socfpga-dwmac ff702000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 9.706611] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Debian GNU/Linux 10 ttyS0
login: [ 37.347805] fpga_manager fpga0: writing socfpga.rbf to Altera SOCFPGA FPGA Manager
login: [ 1477.077961] socfpga-dwmac ff702000.ethernet eth0: Link is Down

if i disable the fpga firmware flash, it seems that the system bootup every time. the fpga firmware is written via devicetree update.

thanks for the help!

[JingyangTeh_Altera]

Hi

Is it possible to share how you flash the FPGA firmware?

In normal case we do not suggest to flash the FPGA firmware after we boot into the linux console.

Regards

Jingyang, Teh

[qwitza]

Hello,

here is the source code, how we "upload" the fpga firmware.

if(is_no_error())
{
	std::ofstream wpd("/sys/kernel/config/device-tree/overlays/device/path");
	wpd << "socfpga_overlay.dtb"";
	wpd.close();
	if(wpd.fail())
		return ERROR_WRITE_FAIL;
}

[JingyangTeh_Altera]

Hi

After checking, it seems this is not officially tested for the Cyclone5.

However we could try to get it to work on your end.

The problem could be you not setting the Cyclone5 to have partial reconfiguration?

https://www.intel.com/content/www/us/en/docs/programmable/683694/current/dynamic-reconfiguration-25100.html

The configuration during the linux should only be on the FPGA fabric and not reconfigure the HPS.

You could not reconfigure the HPS when running in linux.

Regards

Jingyang, Teh

[JingyangTeh_Altera]

Hi

Do you have any follow up question from the previous comment?

Regards

Jingyang, Teh

[qwitza]

Hello,
we do not write the rpd partially it is a full flash.
The kernel also offers this option.

[JingyangTeh_Altera]

Hi

Could you change the rpd to partial?

It would need to be partial as the full configuration could cause the HPS to reconfigured too.

When the HPS is configured the linux running will alsoo crashes, I believe what is happening currently in your case.

Regards

Jingyang, Teh

[qwitza]

Hello,

i think we can not change to partial, because we have only the free version.

i am right?

greetings

[JingyangTeh_Altera]

Hi

Any update on this case?

Did you try out the suggestion in the previous comment?

Regards

Jingyang, Teh

[qwitza]

Hello,

i think we can not change to partial, because we have only the free version.

i am right?

greetings

[JingyangTeh_Altera]

Hi

Yes. The partial configuration for CycloneV requires the Quartus Pro licensing.

The crash that you are observing is due to the full configuration.

The full configuration is resetting the HPS which the linux is running on.

Regards

Jingyang, Teh

[JingyangTeh_Altera]

Hi

Do you have any follow up question on this issue?

Regards

Jingyang, Teh

[JingyangTeh_Altera]

Hi

As we do not receive any response from you on the previous question/reply/answer that we have provided. Please login to ‘https://supporttickets.intel.com/s/?language=en_US’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

Regards

Jingyang, Teh

[qwitza]

Hello,
where can I find the documentation that prohibits writing the full FPGA image at runtime?
Why is it allowed to write the entire image in Linux kernel?

[JingyangTeh_Altera]

HI

Writing the full image will include the reconfiguration of the HPS in which the linux is running on.

This will cause the current running linux on the HPS to crash.

Regards

JIngyang, Teh

[qwitza]

Hello,

did you mean the HPS bridge, or the HPS system/processor?

Because in most cases, and on most boards, it works as expected.

Best regards,

[qwitza]

Hello,

I'm looking for a link or documentation to support the claim, but so far I've only found discussion threads suggesting that it's possible.

Also, why would the kernel allow a full reconfiguration if it's not actually permitted?

I'd really appreciate any useful documents or official links that back up this claim.

Thanks in advance!

[qwitza]

Hello,

here is more information

the device tree:

#include "socfpga_cyclone5.dtsi"
/ {
    model = "Model";
    compatible = "altr,socfpga-cyclone5", "altr,socfpga";
    chosen {
        bootargs = "earlyprintk";
        stdout-path = "serial0:115200n8";
    };
    memory@0{
        name = "memory";
        device_type = "memory";
        reg = <0x0 0x40000000>; /* 1GB */
    };
    aliases {
        /* this allow the ethaddr uboot environmnet variable contents
         * to be added to the gmac1 device tree blob.
         */
        ethernet0 = &gmac1;
	rtc0 = &mcp_rtc;
    };
    regulator_3_3v: vcc3p3-regulator {
        compatible = "regulator-fixed";
        regulator-name = "VCC3P3";
        regulator-min-microvolt = <3300000>;
        regulator-max-microvolt = <3300000>;
    };
    fpga_bridge3: fpga-bridge@ffc25080 {
        compatible = "altr,socfpga-fpga2sdram-bridge";
	reg = <0xffc25080 0x4>;
    };
    msgdma_wr: msgdma_wr@0xff204000 {
	compatible = "msgdma_wr";
	reg = <0xff204000 0x40>;
	interrupt-parent = <&intc>;
	interrupts = <0 40 4>;
    };
	msgdma_rd: msgdma_rd@0xff204040 {
	compatible = "msgdma_rd";
	reg = <0xff204040 0x40>;
	interrupt-parent = <&intc>;
	interrupts = <0 41 4>;
    };
};
&gmac1 {
	status = "okay";
	phy-mode = "rgmii";
  /* 0 bis 900*/
	rxd0-skew-ps = <420>;
	rxd1-skew-ps = <420>;
	rxd2-skew-ps = <420>;
	rxd3-skew-ps = <420>;
	rxdv-skew-ps = <420>;

	txd0-skew-ps = <360>;
	txd1-skew-ps = <360>;
	txd2-skew-ps = <360>;
	txd3-skew-ps = <360>;
	txen-skew-ps = <360>;
/* 0 bis 1860*/
	rxc-skew-ps = <1680>;
	txc-skew-ps = <1740>;

	max-frame-size = <3800>;
};

&i2c0 {
	status = "okay";
	speed-mode = <0>;

	i2c-switch@70 {
		compatible = "nxp,pca9548";
		#address-cells = <1>;
		#size-cells = <0>;
		reg = <0x70>;

		i2c@1{
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <1>;

			mcp_rtc: rtc@6f{
         			compatible = "microchip,mcp7941x";
				reg = <0x6F>;
			};
		};
	};
};

&i2c1 {
    status = "okay";
    speed-mode = <0>;
    gpioexp@20 {
        compatible = "ti,tca6416";
        reg = <0x20>;
    };
    e2prom@50 {
        compatible = "at,24c01";
        reg = <0x50>;
    };
    tempsens@48 {
        compatible = "ti,tmp108";
        reg = <0x48>;
	status = "okay";
    };
};

&mmc0 {
    vmmc-supply = <&regulator_3_3v>;
    vqmmc-supply = <&regulator_3_3v>;
    status = "okay";
};
&usb1 {
    status = "okay";
};
&gpio0 {
    status = "okay";
};
&gpio1 {
    status = "okay";
};
&gpio2 {
    status = "okay";
};

the overlay file:

/dts-v1/;
/plugin/;
/ {
	fragment@0 {
		target-path = "/soc/base-fpga-region";
		__overlay__ {
			firmware-name = "socfpga.rbf";
			fpga-bridges = <&fpga_bridge0>, <&fpga_bridge1>, <&fpga_bridge3>;
		};
	};
};

the firmware is written at runtime.

I modified some code for publication purposes (not tested for compilation):

std::string fpga::write_firmware(void)
{
	std::string msg;
	char *dir = nullptr;
	char *dir_ch = nullptr;

	m_firmware_ok = false;

	dir_ch = strdup("/sys/kernel/config/device-tree/overlays/m3/path");
	if(dir_ch == nullptr)
		msg ="Can not malloc memory for /sys/kernel/config/device-tree/overlays/m3/path";

	if(msg.empty() == true)
	{
		dir = dirname(dir_ch);
		if(dir == nullptr)
			msg = "Can not malloc memory for dirname";
	}

	if(msg.empty() == true)
	{
		if(set_enable(false).is_error())
			msg = "Enable not switched off";
	}

	if(msg.empty() == true)
	{
		if(chdir(dir) == 0)
		{
			if(rmdir(dir) != 0)
				msg = "Could not delete sysfs";
		}
	}

	if(msg.empty() == true)
	{
		if(mkdir(dir, S_IRWXU) != 0)
			msg = "Could not create sysfs";
	}

	if(msg.empty() == true)
	{
		std::ofstream wpd("/sys/kernel/config/device-tree/overlays/m3/path");
		wpd << socfpga_overlay.dtb";
		wpd.close();
		if(wpd.fail())
		{
			msg = "Could not open fpga fw errno: " + std::to_string(errno);
		}
	}

	if(msg.is_no_error())
	{
		WORD timeout = 0;

		bool finished_bridge = false;
		bool finished_manager = false;

		do
		{
			if(!(finished_bridge && finished_manager))
				ica::chrono::sleep_for(std::chrono::milliseconds(3000));

			finished_bridge = get_bridge_state();
			finished_manager = get_fpga_manager_state();

			timeout++;
		}while(timeout < ica::fpga::timeout_bridge_restart && !(finished_bridge && finished_manager));

		if(timeout == 10)
			msg = "FPGA bridge or manager not okay";
	}

	if(msg.empty() == true)	
	{
		if(set_enable(true).is_error())
			msg = "FPGA enable not on";
	}

	if(msg.empty() == true)
		m_firmware_ok = true;

	if(dir_ch != nullptr)
		free(dir_ch);

	return msg;
}