Issue with Compiling OFS Agilex PCIe Attach on F-Series Development Kit

[Md_Raizz]

Hello Team,

I am looking to compile the OFS Agilex PCIe Attach on the F-series development kit. I was following the compilation flow using the link below:

https://ofs.github.io/ofs-2023.3/hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/#1311-walkthrough-install-quartus-prime-pro-software

After exporting with the respective Quartus and build root, I am facing an issue where the terminal hangs. and the compilation doesn't go any further for hours. 
I have compiled this on both Red Hat Enterprise Linux 8.6 and Ubuntu 22.04 having same observation.
I am using a system with an Intel i7-14700 and 64GB of RAM, so system performance shouldn't be an issue.
I have also attached logs for your reference. Kindly provide a solution or let me know if any changes need to be made.

Thanks & Regards,
Raizz

[khtan]

Hi Raizz,

Thanks for using Intel Forums, I'm Kian and will be looking into this case. 

I saw the log that you provided, looks like the compilation just stalled at that location and seems to be truncated/incomplete (eg sudden stalled) without error

Info: qsys-generate /home/admin/SmartNIC/Intel/Intel-agilex/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_fseries-dk_flat/ofs-common/src/fpga_family/agilex/hps/hps_ss.qsys --synthesis=VERILOG --output-directory=/home/admin/SmartNIC/Intel/Intel-agilex/OFS_BUILD_ROOT/ofs-agx7-pcie-attach/work_fseri

Could you help clarify some basic questions first to help me understand better on the case

1) Just to confirm, does the cpu I7-14700 is the non K or the K version (eg. I7-14700k unlocked)

2) Does the issue always stalled at the same place (eg qsys-generate HDL design files for synthesis) or random? If you have other logs, please do provide them as well.

3) After the issue happened, like leaving the build for sometime, and then if you try to terminate the build (eg ctrl+c) , does the system still responding to commands etc. (just to make sure the system is still working)

4) Is there any specific OFS version that you need? eg 2023.3 or could you use the latest OFS (https://ofs.github.io/ofs-2024.2-1/hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/) , assuming you have the relevant Quartus version as well. 

Meanwhile,  I've an ubuntu 22.04 system with me, and will try out on the guide as well , do bear with me as it will take some time to compile the FIM and I will try to duplicate the issue on my end using the same version.

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

Thanks for your reply.

1) The CPU I7-14700 is the non-K version.

2) Yes, the issue always occurs in the same place during every compilation.

3) When the compilation is going on the CPU and RAM Utilization are at peak, causing the PC to hang. When the issue arises after some time, the CPU starts to idle and memory usage return to normal, After that I am able to terminate the process (Ctl+C).

4) No need of any specific OFS version, I went ahead with the 23.3 version because I had the Quartus 23.3 version in my System and also I had back in February of 2024 compiled the OFS version 23.3 once. If this is an issue with the OFS version I can install the latest Quartus 24 and follow the OFS version and try it out once. Will install the same and get back to you on this.

Thanks & Regards
Raizz

[khtan]

Hi Raizz,

Sorry for the wait, was setting up the server on ubuntu with the relevant Quartus and OFS version. I've tested 3 times and unfortunately I don't see the "hang" part though, the part where it "hangs" passes without issue on my side although on the particular FIM version 2023.3 do have compilation error at later stage (seems mentioned in release notes)

From your log

On my side

but eventually compilation will fail with error (after around 1hour + into the compilation, average it will take around 7-8 hours on my system on Xeon processor)

might be due to this

I will check with OFS team on this but I think the main focus is getting the compilation to work ,so probably will focus on 'freeze/hang' at the qsys-generate. 

When you mentioned stuck for hours, may i know like the exact/estimate time on how long like how many hours? I'm wondering whether to leave it much longer and see whether the qsys-generate will complete (it is depending on processor type on how long the duration)

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

Thank you for your reply.

Based on your previous message, I moved to the Intel OFS 2024.2-1 version and installed Quartus 24.1. I followed the guide here
" https://ofs.github.io/ofs-2024.2-1/hw/ftile_devkit/dev_guides/fim_dev/ug_ofs_ftile_dk_fim_dev/ "

Initially, I built the default F-Series DK design, but I encountered some compilation errors. I then tried the Flat FIM and faced different issues, so I moved on to the HE_Null FIM.

However, I am still encountering issues during compilation. I've attached the logs for your reference:

F-Series DK : build_fim_work_fseries-dk.log
Flat FIM : build_fim_work_fseries-dk_flat.log
HE_NULL : build_fim_work_fseries-HE-NULL-dk_flat.log

Thanks and regards,
Raizz

[khtan]

Hi Raizz,

Thanks for the info, let me go through the logs and feedback to you later.

Thanks

Regards

Kian

[khtan]

Hi Raizz,

Sorry for the delay in replying, was setting up another server to duplicate via Centos to imitate Redhat environment, was EOL recently so had to reconfigure certain settings to build FIM again. Currently still in process of generating to try to duplicate the same issue.

Anyway on the log side, I saw dk-flat and he null dk-flat having the same issues, which is complaining on the files missing which is odd. It was not part of the github repository thus I'm assuming it should be generated out, had alot of critical warnings on file missing which eventually leads to error.

Whereas on the fseries-dk.log , seems like the process is killed early thus resulting in error

Let me try to duplicate on my side, and if able to I will escalate this to our OFS team to check what is the issue on the FIM generation.

Meanwhile i will continue to analyze to logs to see any other build issues.

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

I'm also debugging the issue on my end. Please keep me updated on any progress or findings from your side as well.

Let's stay in touch as we work through this.

Best regards,
Raizz

[khtan]

Hi Raizz, 

Sorry it took some time to get the build working due to my system config but I manage to build the FIM successfully on dk flat HE null.

I didnt get any critical warning error on the file missing on my log side (though there is some warnings on some other non critical files)

First thing I noticed is that your Quartus is missing 2 patches which I suspect might be the cause of it since I'm comparing the logs between mine and yours and saw some process are missing.

log:

Do install the patches

https://github.com/OFS/ofs-agx7-pcie-attach/releases/tag/ofs-2024.2-1

You should get 0.02iofs , 0.18 and 0.26 patches when you check the version "quartus_sh --version"

Could you download and install those patches first and rerun the FIM generation?

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

Thanks for your update.

I completed the patch installation of IOFS based on your previous reply and have started the FIM generation. However, I'm still facing some errors during compilation. I’ve attached the logs for your reference.

In the meantime, I've kept  fseries dk compilation again.

Thanks and Regards,
Raizz

[khtan]

Hi Raizz,

May I know whether your subsequent compilation whether encountered the same error?

Still figuring out the error in the recent log , seems like not related to compilation but other errors (internal process still running after terminated, network connection closed)

Meanwhile I'm checking with our OFS team to see any known issues on 2 F-tile. I'm also rebuilding my FIM again to see whether any issues that might pop up randomly.

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

Thanks for your response.

I encountered the same error (network connection closed) on Red Hat. In the meantime, I'm checking on Ubuntu OS and will update you soon. I’ve also attached the build log for your reference.
Please keep me updated on any progress or findings from your side as well.

Thanks and Regards,
Raizz

[khtan]

Hi Raizz,

Just an update on my side, I've rebuild flat HE null again and the dk flat with no issues 

i) ./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/fseries-dk.ofss fseries-dk:flat,null_he_lb,null_he_hssi,null_he_mem,null_he_mem_tg work_fseries-dk_flat

ii) ./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk

I was searching our database to see any reported issues, so far didn't find any reported cases that relates to the issue directly except there is one case that has the same failure but it was trying to access example design stored remotely in server during testing.  Also did not see any known issues with the release that related to this error

Error: generate_all_instances : INTERNALstill running after destroy

Error: UNAVAILABLE: Network closed for unknown reason

Could you check your Quartus license whether the license is activated (one brief comment was that the issue was due to licensing for access which i doubt in this case as you're building files locally)

May I know your kernel version as well (uname -a )? 

I notice the "Info: mem_ss: Done "mem_ss" with 2 modules, 1 files" , on my side it should have more modules and files, not 1 file. 

Attached is my build logs as well for reference

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

Thank you for your response, and I apologize for the delay.

I have checked that the license is activated in Quartus, and my kernel version is 6.1.41-dfl.
Please refer to the image below for your reference.

Thanks and Regards,
Raizz

[khtan]

Hi Raizz,

Thanks for the information, I'm also using the same 6.1.41-dfl kernel (though on Centos 8 stream).

Have you tried restarting and redo the same steps to compile? Wondering whether is there any leftover process causing the unable to terminate error.

Also how about on Ubuntu?

I'm discussing with our OFS team to check whether any known or encountered similar error in the past

Thanks

Regards

Kian

[khtan]

Hi Raizz,

One more thing , instead of using this command alone " ./ofs-common/scripts/common/syn/build_top.sh fseries-dk work_fseries-dk ", 

could you try this instead 

• ./ofs-common/scripts/common/syn/build_top.sh --ofss tools/ofss_config/fseries-dk.ofss,tools/ofss_config/hssi/hssi_8x25_ftile.ofss fseries-dk:flat work_fseries-dk_flat

  whether you still having the same issue

Thanks

Regards

Kian

[Md_Raizz]

Hi Kian,

Thank you for your response, and I apologize for the delay.

I was successfully able to compile the design by increasing the RAM size to 96GB on my host PC. Additionally, there was a missing NIOS license file that I added after increasing the RAM. I have attached the log files for your reference.

Thanks and regards,
Raizz

[khtan]

Hi Raizz,

Thanks for the update and glad that your FIM compilation now works. May I know whether does the RAM size increase alone able to solve the compilation, or need the Nios license as well . Just wondering about the root cause of that error message, as in the logs didnt seems to indicate any problems with Nios generation nor running out of memory.

Thanks

Regards

Kian

[khtan]

Hi Raizz,

Since the issue has been resolved, I will now transition this thread to community support. If you have a new question, Please login to ‘https://supporttickets.intel.com/s/?language=en_US’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

Thanks for using Altera Forum

Regards,

Kian