Cache coherency on Agilex 5 when booting secondary cores without ATF

[SarahTr]

Hello,

My goal is to boot the secondary cores on an Agilex 5 by directly writing to the cpurstrelease register of the reset manager instead of using the PSCI CPU_ON call.

I used the ARM Trusted Firmware implementation as a reference and have successfully brought up a secondary core. However, the caches of the primary and secondary core are not coherent.

I compared the contents of the CCU registers but did not observe any differences in the configuration. When following the ATF execution of CPU_ON, I also could not find any specific CCU or MMU setup performed by the secondary core so far.

Does the CCU require specific initialization for the caches to be coherent? Or are there other steps that I am missing?

Thanks,

Sarah

[ProFromDover_Altera]

I have used the the PSCI_CPU_ON infrastructure provided by the ATF bl31 environment quite a bit to start secondary cores and I find that they are always coherent within the CPU cluster with no additional effort. I typically boot though the u-boot-spl bootloader into u-boot proper through the ATF flow so bl31 is left behind in DRAM as the secure monitor which handles the PSCI_CPU_ON requests. If you could follow a flow like this, I believe this would be much simpler flow to get working the way that you wish.

If you must do this is a purely bare metal fashion, I suspect that the configuration you are missing is not in the CCU section of the HPS but more likely in the DSU and the snoop filter that manages choerency within the quad core cluster and L3 cache system. Unfortunately I don't have any examples that demonstrate that raw flow, and I don't know if the configurations that you would be missing are in the u-boot-spl or bl31 parts of the boot loader flow that I typically use.

[SarahTr]

Thanks for your response.

The reason I was trying to bring up the secondary cores without using PSCI_CPU_ON was to allow execution at Exception level 3. We depend on being able to configure interrupts that are always signaled as FIQ. As someone new to this, my current understanding is that this requires the interrupts to be Group 0 in the GIC and configured from EL 3.

Is it possible to use PSCI_CPU_ON and still configure a Group 0 interrupt?

I’d be happy to do the necessary configurations in the ATF, but I haven’t yet figured out how to set up the vector table so these interrupts are not handled by ATF itself. I’d really appreciate any help or suggestions.

[ProFromDover_Altera · Accepted solution]

I would suggest that you boot through the default bootloaders into your application so you can leverage all of the default configurations that they apply to the HPS for you, rather than trying to accurately replicate that in your own custom boot sequence. The FSBL which can be the ATF bl2 code but is typically the u-boot-spl code, will configure the HPS in general, ensuring that the HPS handoff data is correctly applied to the HPS, which gets the clocking and HPS IO and HPS EMIF all configured as the HPS was configured to operate in the Quartus project, and then it configures all the secure registers in the firewalls and things like that to get the HPS in the operational state that you expect. Then it passes control to the ATF bl31 code which finishes configuring the DSU and CPU cluster for proper default behavior. You would need to replicate all of that functionality in your own custom boot sequence to bring things up to a similar state.

If you need to use group 0 GIC interrupts, then you'll need to configure those from a secure state. Your realistic options are secure EL3 or secure EL1. To do this in S-EL3 state with the ATF code present, you'd be better served to apply your own customizations into the ATF code base and interact with your NS-EL2 application. To do this in S-EL1 state with the ATF code present, you'd need to start your secondary core and it's application to enter S-EL1 state and then operate from there.

[SarahTr]

Thank you for the explanation.

For my purpose, I believe getting a core into S-EL1 would be the best solution. What would you recommend as the best way to achieve this?

I have found two possible options: running a secure OS entirely in S-EL1, or extending the BL32 stage by using the Trusted Secure Payload (TSP) as a starting point.

If it is supported, booting our custom OS directly into S-EL1 would be my preferred approach. However, I haven’t yet worked out how to set this up. Are there any guides or resources you could point me to?

I am also experimenting with extending the TSP to run on the Agilex 5 HPS so that I can add my code there. Is this a reasonable alternative, or is there a more straightforward way to run code at S-EL1?

[ProFromDover_Altera]

I am not aware of any guides or resources that you could reference to do something like this.  I suspect that this type of use case is documented in the generic ATF documents and an AI chatbot may be able to suggest patterns that could be deployed for something like this.

[JingyangTeh_Altera]

Hi

Do you have any follow up question for this issue?

Regards

Jingyang, Teh

[JingyangTeh_Altera]

Hi

As we do not receive any response from you on the previous reply that we have provided. Please login to ‘https://supporttickets.intel.com/s/?language=en_US’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

Regards

Jingyang,Teh

[SarahTr]

Apologies for the delayed response.

It took some time to work through the technical details, but I’ve now used the Trusted Secure Payload (TSP) as an entry point into Secure EL1. From there, I was able to configure the Group 0 interrupts as needed for our use case.

Thanks again for the guidance. It was very helpful.

[SarahTr]

Small correction, I meant the Test Secure Payload.

[SueC_Altera]

I'm glad you got it working, SarahTr! Thanks for letting us know.

Sue