Cache coherency on Agilex 5 when booting secondary cores without ATF

[SarahTr]

Hello,

My goal is to boot the secondary cores on an Agilex 5 by directly writing to the cpurstrelease register of the reset manager instead of using the PSCI CPU_ON call.

I used the ARM Trusted Firmware implementation as a reference and have successfully brought up a secondary core. However, the caches of the primary and secondary core are not coherent.

I compared the contents of the CCU registers but did not observe any differences in the configuration. When following the ATF execution of CPU_ON, I also could not find any specific CCU or MMU setup performed by the secondary core so far.

Does the CCU require specific initialization for the caches to be coherent? Or are there other steps that I am missing?

Thanks,

Sarah

[ProFromDover_Altera]

I have used the the PSCI_CPU_ON infrastructure provided by the ATF bl31 environment quite a bit to start secondary cores and I find that they are always coherent within the CPU cluster with no additional effort. I typically boot though the u-boot-spl bootloader into u-boot proper through the ATF flow so bl31 is left behind in DRAM as the secure monitor which handles the PSCI_CPU_ON requests. If you could follow a flow like this, I believe this would be much simpler flow to get working the way that you wish.

If you must do this is a purely bare metal fashion, I suspect that the configuration you are missing is not in the CCU section of the HPS but more likely in the DSU and the snoop filter that manages choerency within the quad core cluster and L3 cache system. Unfortunately I don't have any examples that demonstrate that raw flow, and I don't know if the configurations that you would be missing are in the u-boot-spl or bl31 parts of the boot loader flow that I typically use.

[SarahTr]

Thanks for your response.

The reason I was trying to bring up the secondary cores without using PSCI_CPU_ON was to allow execution at Exception level 3. We depend on being able to configure interrupts that are always signaled as FIQ. As someone new to this, my current understanding is that this requires the interrupts to be Group 0 in the GIC and configured from EL 3.

Is it possible to use PSCI_CPU_ON and still configure a Group 0 interrupt?

I’d be happy to do the necessary configurations in the ATF, but I haven’t yet figured out how to set up the vector table so these interrupts are not handled by ATF itself. I’d really appreciate any help or suggestions.

[ProFromDover_Altera]

I would suggest that you boot through the default bootloaders into your application so you can leverage all of the default configurations that they apply to the HPS for you, rather than trying to accurately replicate that in your own custom boot sequence. The FSBL which can be the ATF bl2 code but is typically the u-boot-spl code, will configure the HPS in general, ensuring that the HPS handoff data is correctly applied to the HPS, which gets the clocking and HPS IO and HPS EMIF all configured as the HPS was configured to operate in the Quartus project, and then it configures all the secure registers in the firewalls and things like that to get the HPS in the operational state that you expect. Then it passes control to the ATF bl31 code which finishes configuring the DSU and CPU cluster for proper default behavior. You would need to replicate all of that functionality in your own custom boot sequence to bring things up to a similar state.

If you need to use group 0 GIC interrupts, then you'll need to configure those from a secure state. Your realistic options are secure EL3 or secure EL1. To do this in S-EL3 state with the ATF code present, you'd be better served to apply your own customizations into the ATF code base and interact with your NS-EL2 application. To do this in S-EL1 state with the ATF code present, you'd need to start your secondary core and it's application to enter S-EL1 state and then operate from there.