OpenCL environment for darby nodes on devcloud

[JSchr20]

I have been running OpenCL applications on the devcloud intel_a10gx_pac boards for several months now.  I was recently granted access to the two "darby" nodes (intel_s10sx_pac), nodes s001-n189 and s005-n008.  However, when I log in interactively to node s001-n189, for example, and try to run aoc or aocl, they're "not found".   On the arria10 nodes, one's environment was set up properly such that all the software and board models were accessible by default.  That appears not to be the case for the darby nodes.  Can someone please tell me how I should set up my environment to be able to run?

Thank you!

[DDIAKITE]

How did you set up the tools on the s10 nodes?

"tools_setup -t S10DS" works fine to set them up when you are logged interactively.

Regards!

[JSchr20]

Hi DDIAKITE, thanks for your reply.  Unfortunately, "tools_setup" is not found when I log in interactively.  (See transcript below.)  Can you please give me the full path to that command, so that I can try forcing a run of that command?

Thank you!

Transcript:

u40073@login-2:~$ qsub -q batch@v-qsvr-fpga -I -l nodes=1:darby:ppn=2 -d .
qsub: waiting for job 7103.v-qsvr-fpga.aidevcloud to start
qsub: job 7103.v-qsvr-fpga.aidevcloud ready

########################################################################
# Date: Fri Jul 17 09:55:47 PDT 2020
# Job ID: 7103.v-qsvr-fpga.aidevcloud
# User: u40073
# Resources: neednodes=1:darby:ppn=2,nodes=1:darby:ppn=2,walltime=06:00:00
########################################################################

u40073@s005-n008:~$ tools_setup -t S10DS
-bash: tools_setup: command not found

[DDIAKITE]

Hi JSchr20,

Try those command:
source /data/intel_fpga/devcloudLoginToolSetup.sh
tools_setup -t S10DS

If it works then you should add the following in your bashrc:

if [ -f /data/intel_fpga/devcloudLoginToolSetup.sh ]; then
source /data/intel_fpga/devcloudLoginToolSetup.sh
fi

And after that "tool_setup" will be found anytime without sourcing devcloudLoginToolSetup.sh manually.

Regards!

[JSchr20]

Thank you!  That has improved matters, but my code still does not run, possibly due to the gcc version I'm using in my host code (I need access to gmp).  I will look more into that and perhaps I can get something sorted out.  I at least have tools_setup running now.

[AnilErinch_A_Intel]

Hi 

Please let us know which gcc version you are using , and whether  you are able to run the code now. 

If not please let us know the error message which you are seeing while trying to compile/execute the code.

Thanks and Regards

Anil

[JSchr20]

Hi Anil,

I am still unable to run, but I'm failing on a 'higher level' now, I think.

Sorry about my slight misstatement earlier: I am using the default gcc in /usr/bin/gcc, which is version 4.8.5.  What I need, though, in addition to that, is gmp.  Previously, Intel pointed me to a shared version of that code in

/opt/.intel/inteloneapi/compiler/2021.1-beta04/linux/lib/oclfpga/linux64/lib/dspba/include

with library in

/jhub/lib

but that version often disappears even on arria10 nodes, so I eventually gave up and compiled my own copy of gmp-6.2.0 in my home directory.  Since I last posted, I recompiled gmp-6.2.0 on a darby node, and my host code now appears to run (or at least it gets farther than it did).

I now set up my darby environment, upon interactive login, according the the advice above:

source /data/intel_fpga/devcloudLoginToolSetup.sh
tools_setup -t S10DS

However, now I am seeing the following error when I try to run my host code:

u40073@s005-n008:~/calculator_s10/mult_multi_limb_wide$ bin_monster_1_4_8/host -n=10
N = 10, Nunit = 10
Initializing OpenCL
ERROR: CL_DEVICE_NOT_FOUND
Location: ../common/src/AOCLUtils/opencl.cpp:356
Query for number of devices failed

where ../common is a soft link to

/glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/hld/examples_aoc/common/

It appears I'm unable to see the actual S10 card.  Can you suggest my next step?

Thank you!

Jenn

[DDIAKITE]

Hi JSchr20,

Can you run "aocl diagnose" and see the output, it looks like the BSP is not loaded (there is no .aocx file loaded in the PAC). If that is the case then run this command:
aocl program acl0 <project>.aocx

In case you don't have a compiled .aocx, you can have a hello_world example (cp $OPAE_PLATFORM_ROOT/opencl/hello_world.aocx ) that you can copy over and try it. After that run the diagnose to see if it worked and your host may be able to found the device.

Regards,
Daouda

[JSchr20]

Hi Daouda, thanks for your message.

I ran 'aocl diagnose' and it said (among other things):

pac_ec00000 Uninitialized OpenCL BSP not loaded. Must load BSP using command:
'aocl program <device_name> <aocx_file>'
before running OpenCL programs using this device

DIAGNOSTIC_PASSED

I then programmed the bitfile by hand (my host code takes care of this on a10_pac, so it should for s10, too!):

u40073@s005-n008:~/calculator_s10/mult_multi_limb_wide$ aocl program acl0 bin_monster_1_4_8/mult_multi_limb_hw.aocx
aocl program: Running program from /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp/linux64/libexec
Program succeed.

I then tried to run my host code and got the following failure:

u40073@s005-n008:~/calculator_s10/mult_multi_limb_wide$ bin_monster_1_4_8/host -n=10
N = 10, Nunit = 10
Initializing OpenCL
Platform: Intel(R) FPGA SDK for OpenCL(TM)
Using 1 device(s)
FAILED to read auto-discovery string at byte 2: Expected version is 19, found 23
Error: The currently programmed/flashed design is no longer supported in this release. Please recompile the design with the present version of the SDK and re-program/flash the board.
acl_hal_mmd.c:1460:assert failure: Failed to initialize kernel interfacehost: acl_hal_mmd.c:1460: l_try_device: Assertion `0' failed.
Aborted

I am surprised and frustrated to see this message.  The bitfile was built on July 14th; is Intel changing versions on the devcloud so frequently that this build has gone out of date in a week?  Exasperating.

For lack of a better idea, I will rebuilt the bitfile, and I'll try it all again later.

Thanks for your help!

Jenn

[JSchr20]

It appears I may be using the wrong BSP.  I assumed the darby node was an s10_pac node (on my login node, if I type 'aoc -list-boards' I get two s10_pac choices and no other s10 choice).  However, I just checked on the darby node itself, after running:

source /data/intel_fpga/devcloudLoginToolSetup.sh
tools_setup -t S10DS

and I get a completely different BSP:

u40073@s005-n008:~$ aoc -list-boards
Board list:
pac_s10_dc
Board Package: /glob/development-tools/versions/fpgasupportstack/d5005/2.0.1/inteldevstack/d5005_ias_2_0_1_b237/opencl/opencl_bsp

I need to make local modifications to this bsp so it will use VHDL 2008; I will attempt to do that and recompile and we shall see.  Stay tuned.

[JSchr20]

Update:

I copied the BSP to my local directory.  Unfortunately, it is different enough that I can't blindly make the same modifications for VHDL 2008 as I could in the pac_s10 BSP, so my build is still failing right now.  More investigation will be necessary there.

However, meanwhile:

It seems I can ONLY compile for this BSP (locally modified or otherwise) if I am logged in interactively to a darby node.

On an fpga_compile node, if I am pointed at this BSP, I get a "aoc: Quartus is not on the path!" error.  If I'm pointed at the 'usual' pac BSPs, quartus is found. 

On a darby node, if I try to batch submit, even trying to set the environment in the batch shell the same way I would interactively, I get this error:

qsub: submit error (Job exceeds queue resource limits MSG=cannot locate feasible nodes (nodes file is empty, all systems are busy, or no nodes have the requested feature))

As a general note to Intel: this is unworkable.  The darby nodes should have a working FPGA environment on login, no additional configuration necessary.  And the fpga_compile nodes in the devcloud should have access to compatible BSPs so that one can batch compile designs for darby without having to monopolize a valuable darby node.   And the BSPs in general should allow one to specify necessary compile directives for one's own code, like "use VHDL 2008 for my stuff".  I shouldn't need to make a local BSP copy and figure out where on earth to modify it for things like that.  This is excessively painful in its current state.

[Lawrence_L_Intel]

can you send me your contact info ? I'll setup a skype to see what is going on. lawrence.landis@intel.com

Thanks,

Larry

[AnilErinch_A_Intel]

Hi ,

Hope the issue is resolved after discussions with the team.

Thanks and Regards

Anil

[JSchr20]

Thank you, yes, my environment was successfully set up and I have been able to emulate and build hardware successfully. 

Here's a summary of my notes on how to make it work at this time.  This is how I'm able to build an OpenCL design that wraps System Verilog RTL which in turn wraps encrypted VHDL 2008 RTL (which requires a local modification to the BSP):

For the darby nodes using devcloud_login, the batch job.sh looks like this (note fully-qualified paths):
#!/bin/bash
source /data/intel_fpga/devcloudLoginToolSetup.sh
tools_setup -t S10DS
aoc -board-package=/home/u40073/intel_opencl_bsp_pac_s10_dc_local -board=pac_s10_dc -DNUM_COMPUTE_UNITS=1 -DNUM_LIMBS=4 -DNUM_INTS=8 /home/u40073/calculator_s10/mult_multi_limb_wide/device/mult_multi_limb_rtl.cl -o /home/u40073/calculator_s10/mult_multi_limb_wide/bin_monster_1_4_8/mult_multi_limb_hw.aocx -I /home/u40073/calculator_s10/mult_multi_limb_wide/rtl/lib -I /home/u40073/calculator_s10/mult_multi_limb_wide/device -L /home/u40073/calculator_s10/mult_multi_limb_wide/rtl -l mult_multi_int_lib_monster.aoclib

To then run that job in batch mode:
devcloud_login -b S10PAC job.sh # submit the job
qstat batch@v-qsvr-fpga # check for jobs
watch -n 1 qstat -n -1 batch@v-qsvr-fpga # watch the job
qdel 7321.v-qsvr-fpga # example of deleting a darby job

If you get a CL_DEVICE_NOT_FOUND error, you might be able to work around it by just loading the bitfile by hand:
aocl program acl0 <project>.aocx
Then just run the host code as normal.

Thanks again for all of your help.