How can I monitor GPU usage?

[caplanr]

Hi,

I am testing a Fortan code that uses "do concurrent" for GPU offload.

I got the code to compile with the new IFX 2023, and ran the code.

The code runs very slowly, so I wanted to make sure it was runnign on teh GPU.

Is there some kind of Intel equivalent to "nvidia-smi"/"nvtop"?

Somthing to show what processes are running on the GPU and their efficency?

Thanks!

 - Ron

[DiyaN_Intel]

Hi, 

Thank you for posting in Intel communities.

Can you please help us in knowing which DevCloud and the details of the GPU you are using?

Regards, 

Diya 

[caplanr]

Hello,

I am logging into guest@ssh.devcloud.intel.com  

I am starting an interactive job with:  qsub -I -- nodes=1:iris_xe_max:ppn=2

to try to get access to the Xe MAX GPUs.

When I am on the node, I am compiling and launching my code.

I would like to see some info about the GPUs, my running process, and whether or not it is using the GPUs.

Is there an equivalent command for the Intel platform like "nividia-smi" or "nvtop" for NVIDIA platforms?

 - Ron

[DiyaN_Intel]

Hi ,

Good day to you.

>> I am logging into guest@ssh.devcloud.intel.com  

Can you please be specific about which DevCloud are you using as we have only three types of Intel DevCloud :

1.  Intel DevCloud for Edge 
2.  Intel DevCloud for oneAPI
3.  Intel DevCloud for FPGA

>> I am starting an interactive job with: qsub -I -- nodes=1:iris_xe_max:ppn=2

to try to get access to the Xe MAX GPUs.

Currently, iris_xe_max nodes are not available. We’re working on adding new Intel® Graphics in the future, as they become available on the market.

We will make an appropriate announcement when the time comes.

Please refer to the link given below for more information -  

https://community.intel.com/t5/Intel-DevCloud/Intel-Iris-Xe-Max-Graphics-to-be-retired-from-Intel-DevCloud-on/m-p/1402673

>> I would like to see some info about the GPUs, my running process, and whether or not it is using the GPUs

Since we don't have sudo permissions we can't view on DevCloud about GPU information. But you can do the same in some other system using the command: 

intel_gpu_top

Regards, 

Diya 

[caplanr]

Hi,

I am using the Intel DevCloud for oneAPI.

I wanted to test my "do concurrent" Fortran code on the Intel GPUs.

I thought that the Xe MAX nodes were the ones to use to do that.

Which nodes should I be using?

Are there nodes with the new MAX GPUs that were shown at SC22?

If not, which node should I use to test compiling and running my code on the Intel GPUs?

As for the "intel_gpu_top" - somehow allowing this or something like it on the devcloud would be really useful (otherwise I do not know if my code is actually running on the GPU or not).   The NVIDIA nvidia-smi works without "sudo" and I have used it on HPC systems like Delta at NCSA, so I would assume there should be a way to do this for Intel GPUs without sudo as well?

Thanks!

 - Ron

[DiyaN_Intel]

Hi, 

We are investigating your issue at our end and working on this internally. We will get back to you with an update soon.

Regards, 

Diya

[DiyaN_Intel · Accepted solution]

Hi, 

Sorry for the late reply.

You can try to check for the GPU usage in the Gen9 nodes in DevCloud with the below command:

1. Please check for the gen9 and gen11 nodes using this command:

pbsnodes | grep -B4 gpu

After that, you will get a list of nodes that are free and the property is gen9. 

2. Please try to access the node with gen9 which is free using the below command :

qsub -I -l nodes=<free_gen9_node>:ppn=2

3. Please try to check for the GPU usage using the below command:

intel_gpu_top

You can also check the below link for more clarity 

https://devcloud.intel.com/oneapi/documentation/job-submission/#listing-compute-node-properties

Regards,

Diya

[caplanr]

Hi,

Thanks!

This seems to be working.

FYI, on the gen11 nodes, the command does not work:

INTEL-DEVCLOUD: ~ $ intel_gpu_top

Command 'intel_gpu_top' not found, but can be installed with:

apt install intel-gpu-tools
Please ask your administrator.

 - Ron

[DiyaN_Intel]

Hi, 

We have not heard back from you. Is your issue resolved with the above solution?

If this resolves your issue, make sure to accept this as a solution. 

This would help others with similar issue. Thank you!

Regards, 

Diya 

[DiyaN_Intel]

Hi, 

In the above solution, I mentioned using gen9 nodes where you can check the GPU usage of your application.

Glad to know that your issue is resolved. 

If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards, 

Diya