topic Re: mpigdb: attaching to Cannot access memory when using -gtool option for debugging in Intel® MPI Library

mpigdb: attaching to Cannot access memory when using -gtool option for debugging

xsl — Tue, 08 Mar 2022 01:38:40 GMT

This is a follow up on my previous question: https://community.intel.com/t5/Intel-oneAPI-HPC-Toolkit/Debugging-MPI-codes-with-gtool-or-gdb-hangs-nothing-happens-when/m-p/1284089

I was told the cause of the issue is that I did not use a supported OS at the time but now I am actually using a supported OS (Ubuntu 20.04.4 LTS) that is freshly installed just a couple hours ago.

I still get the same error here when I try to use -gtool option to attach GDB to my code. Here is what I used to run my code

mpirun -n 2 -gtool "gdb:0,1=attach" ./fogo

and this is what I get

mpigdb: attaching to Cannot access memory mpigdb: hangup detected: while read from [0] mpigdb: attaching to Cannot access memory mpigdb: hangup detected: while read from [1]

I have also tried to replace gdb with gdb-oneapi but got the same thing. I could use this option on our Compute Canada clusters but most of the nodes are slow for debugging / compilation.

I get the same error when running

mpirun -n 3 -gtool "gdb:0,1=attach" $I_MPI_ROOT/bin/IMB-MPI1

Any idea what is causing this issue?

Re: mpigdb: attaching to Cannot access memory when using -gtool option for debugging

SantoshY_Intel — Thu, 10 Mar 2022 07:08:34 GMT

Hi,

Thanks for reaching out to us.

We have reported this issue to the concerned development team. They are looking into your issue.

Thanks & Regards,

Santosh

Re:mpigdb: attaching to Cannot access memory when using -gtool option for debugging

SantoshY_Intel — Tue, 20 Dec 2022 10:19:57 GMT

Hi,

Could you please try Intel MPI 2021.7 and let us know if the issue persists?

Thanks & Regards,

Santosh

Re:mpigdb: attaching to Cannot access memory when using -gtool option for debugging

SantoshY_Intel — Tue, 27 Dec 2022 06:33:08 GMT

Hi,

We haven't heard back from you. Could you please try Intel MPI 2021.7 and let us know if the issue persists?

Thanks & Regards,

Santosh

Re: mpigdb: attaching to Cannot access memory when using -gtool option for debugging

SantoshY_Intel — Mon, 09 Jan 2023 10:42:54 GMT

Hi,

Please follow the below steps to resolve your issue:

Steps:

sudo sysctl -w kernel.yama.ptrace_scope=0
mpirun -gdb -n 4 ./sample

We have tried at our end & we are able to get the desired results as shown below:

Please try from your end & let us know if you face any issues.

Thanks & Regards,

Santosh

Re: mpigdb: attaching to Cannot access memory when using -gtool option for debugging

xsl — Mon, 09 Jan 2023 14:19:35 GMT

Hi Santosh,

Thanks very much for this. I have tried your solution and it does seem that it's also working on my computer. However, it won't work without first changing the settings as you suggested, which is not entirely unacceptable.

Re: mpigdb: attaching to Cannot access memory when using -gtool option for debugging

SantoshY_Intel — Tue, 10 Jan 2023 06:42:17 GMT

Hi,

The issue is specific to the systems having PTRACE scope to "1"(kernel.yama.ptrace_scope = 1).

The PTRACE system is used for debugging. With it, a single user process can attach to any other dumpable process owned by the same user. In the case of malicious software, it is possible to use PTRACE to access credentials that exist in memory (re-using existing SSH connections, extracting GPG agent information, etc).
A PTRACE scope of "0" is the more permissive mode. A scope of "1" limits PTRACE only to direct child processes (e.g. "gdb name-of-program" and "strace -f name-of-program" work, but gdb's "attach" and "strace -fp $PID" do not). The PTRACE scope is ignored when a user has CAP_SYS_PTRACE, so "sudo strace -fp $PID" will work as before.

For more details see: https://wiki.ubuntu.com/SecurityTeam/Roadmap/KernelHardening#ptrace

In general, PTRACE is not needed for the average running Ubuntu system. To that end, the default is to set the PTRACE scope to "1". This value(kernel.yama.ptrace_scope = 1) may not be appropriate for developers or servers with only admin accounts.

Thus, the issue can be simply resolved by setting kernel.yama.ptrace_scope = 0 & it it not an issue related to Intel MPI.

Thanks & Regards,

Santosh

Re: mpigdb: attaching to Cannot access memory when using -gtool option for debugging

xsl — Tue, 10 Jan 2023 12:49:11 GMT

Hi Santosh,

Thanks very much for your detailed explanation. I now understand!

Re:mpigdb: attaching to Cannot access memory when using -gtool option for debugging

SantoshY_Intel — Wed, 11 Jan 2023 06:23:43 GMT

Hi,

Thanks for the quick response.

As your issue is resolved, we are closing this thread. If you need any additional information, please post a new question as this thread will no longer be monitored

by Intel.

Thanks & Regards,

Santosh