Intel® DevCloud
Help for those needing help starting or connecting to the Intel® DevCloud
Announcements
The Intel sign-in experience is changing in February to support enhanced security controls. If you sign in, click here for more information.
1217 Discussions

How can I monitor GPU usage?

caplanr
Novice
402 Views

Hi,

 

I am testing a Fortan code that uses "do concurrent" for GPU offload.

 

I got the code to compile with the new IFX 2023, and ran the code.

 

The code runs very slowly, so I wanted to make sure it was runnign on teh GPU.

 

Is there some kind of Intel equivalent to "nvidia-smi"/"nvtop"?

Somthing to show what processes are running on the GPU and their efficency?

 

Thanks!

 

 - Ron

0 Kudos
1 Solution
DiyaN_Intel
Moderator
219 Views

Hi, 

 

Sorry for the late reply.

 

You can try to check for the GPU usage in the Gen9 nodes in DevCloud with the below command:

 

1. Please check for the gen9 and gen11 nodes using this command:

pbsnodes | grep -B4 gpu

After that, you will get a list of nodes that are free and the property is gen9. 

DiyaN_Intel_0-1674028815389.png

 

2. Please try to access the node with gen9 which is free using the below command :

qsub -I -l nodes=<free_gen9_node>:ppn=2

DiyaN_Intel_4-1674029469260.png

 

3. Please try to check for the GPU usage using the below command:

intel_gpu_top

DiyaN_Intel_5-1674029596388.png

 

You can also check the below link for more clarity 

https://devcloud.intel.com/oneapi/documentation/job-submission/#listing-compute-node-properties

 

Regards,

Diya

 

View solution in original post

9 Replies
DiyaN_Intel
Moderator
381 Views

Hi, 


Thank you for posting in Intel communities.

Can you please help us in knowing which DevCloud and the details of the GPU you are using?


Regards, 

Diya 



caplanr
Novice
352 Views

Hello,

 

I am logging into guest@ssh.devcloud.intel.com  

 

I am starting an interactive job with:  qsub -I -- nodes=1:iris_xe_max:ppn=2

to try to get access to the Xe MAX GPUs.

 

When I am on the node, I am compiling and launching my code.

 

I would like to see some info about the GPUs, my running process, and whether or not it is using the GPUs.

 

Is there an equivalent command for the Intel platform like "nividia-smi" or "nvtop" for NVIDIA platforms?

 

 - Ron

 

 

 

 

DiyaN_Intel
Moderator
336 Views

Hi ,

 

Good day to you.

 

>> I am logging into guest@ssh.devcloud.intel.com  

 

Can you please be specific about which DevCloud are you using as we have only three types of Intel DevCloud :

  1.  Intel DevCloud for Edge 
  2.  Intel DevCloud for oneAPI
  3.  Intel DevCloud for FPGA

 

>> I am starting an interactive job with: qsub -I -- nodes=1:iris_xe_max:ppn=2

to try to get access to the Xe MAX GPUs.

 

Currently, iris_xe_max nodes are not available. We’re working on adding new Intel® Graphics in the future, as they become available on the market.

We will make an appropriate announcement when the time comes.

Please refer to the link given below for more information -  

https://community.intel.com/t5/Intel-DevCloud/Intel-Iris-Xe-Max-Graphics-to-be-retired-from-Intel-De...

 

>> I would like to see some info about the GPUs, my running process, and whether or not it is using the GPUs

 

Since we don't have sudo permissions we can't view on DevCloud about GPU information. But you can do the same in some other system using the command: 

intel_gpu_top

 

Regards, 

Diya 

 

 

caplanr
Novice
281 Views

Hi,

 

I am using the Intel DevCloud for oneAPI.

 

I wanted to test my "do concurrent" Fortran code on the Intel GPUs.

I thought that the Xe MAX nodes were the ones to use to do that.

 

Which nodes should I be using?

Are there nodes with the new MAX GPUs that were shown at SC22?

 

If not, which node should I use to test compiling and running my code on the Intel GPUs?

 

As for the "intel_gpu_top" - somehow allowing this or something like it on the devcloud would be really useful (otherwise I do not know if my code is actually running on the GPU or not).   The NVIDIA nvidia-smi works without "sudo" and I have used it on HPC systems like Delta at NCSA, so I would assume there should be a way to do this for Intel GPUs without sudo as well?

 

Thanks!

 

 - Ron

 

DiyaN_Intel
Moderator
269 Views

Hi, 


We are investigating your issue at our end and working on this internally. We will get back to you with an update soon.


Regards, 

Diya


DiyaN_Intel
Moderator
220 Views

Hi, 

 

Sorry for the late reply.

 

You can try to check for the GPU usage in the Gen9 nodes in DevCloud with the below command:

 

1. Please check for the gen9 and gen11 nodes using this command:

pbsnodes | grep -B4 gpu

After that, you will get a list of nodes that are free and the property is gen9. 

DiyaN_Intel_0-1674028815389.png

 

2. Please try to access the node with gen9 which is free using the below command :

qsub -I -l nodes=<free_gen9_node>:ppn=2

DiyaN_Intel_4-1674029469260.png

 

3. Please try to check for the GPU usage using the below command:

intel_gpu_top

DiyaN_Intel_5-1674029596388.png

 

You can also check the below link for more clarity 

https://devcloud.intel.com/oneapi/documentation/job-submission/#listing-compute-node-properties

 

Regards,

Diya

 

caplanr
Novice
153 Views

Hi,

 

Thanks!

 

This seems to be working.

 

FYI, on the gen11 nodes, the command does not work:

 

INTEL-DEVCLOUD: ~ $ intel_gpu_top

Command 'intel_gpu_top' not found, but can be installed with:

apt install intel-gpu-tools
Please ask your administrator.

 

 - Ron

DiyaN_Intel
Moderator
167 Views

Hi, 

 

We have not heard back from you. Is your issue resolved with the above solution?

 

If this resolves your issue, make sure to accept this as a solution. 

 

This would help others with similar issue. Thank you!

 

Regards, 

Diya 

 

 

DiyaN_Intel
Moderator
138 Views


Hi, 


In the above solution, I mentioned using gen9 nodes where you can check the GPU usage of your application.


Glad to know that your issue is resolved. 


If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.


Regards, 


Diya



Reply