Internal error while resolving the results file

[mjb34]

Hi

I'm trying to perform callgraph profiling on my application. I'm getting a failure that I'm sure is easy to fix but can't seem to find a solution in this forum.

Platform: Itanium 2
Vtune 3.0 for Linux

When I run:

vtl activity -c callgraph -app ./myapp -moi ./myapp run

I get:

---

Reading property files started...

Reading property files done.

Instrumentation of the various modules

Data collection started...

More Instrumentation of modules

Data collection finished...

Error - Internal error while resolving the results file

---

vtl works correctly with a simple call graph test of /bin/ls and /usr/bin/emacs. What am I doing wrong? What is the results file it is referring to?

Many thanks

MJ

[jeffrey-gallagher]

Excellent posting, mjb34. Detailed and to the point. Were you a support engineer in a previous life???

:)

A few thoughts:

0) Check out $PATH in the user account running the callgraph experiment. Also log in as root, check out $PATH there, and run the same experiment. Same error? Different error? No error then?

1) how does sampling work?

$ vtl activity -c sampling -app /bin/ls run

Same error? Different error?

If no error, what happens when you try to view the data, both ways:

$ vtl view

and

$ vtl view -gui

2) Is your kernel in use listed in the supported kernels of the Release Notes?

$ uname -a

3) I'm guessing the install had some problems, otherwiseyou couldn't be more right, you sure should be able torun a callgraph on /bin/ls.

Check out the installation error log on /var/log (I forget the name, probably install.log or vtune_install.log, something like that). That log contains all installation decisions, successes, and failures, with date/time stamps.

If you see alot of errors in there with regard to "Bistro" or "callgraph", we'll be closer to the conclusion that the install failed in some significant ways, although not completely.

Anybody else out thereever run into this before????

cheers

jdg

Message Edited by jdgallag on 07-14-2005 07:26 AM

[mjb34]

Thanks for the compliment. Your guess was very accurate - part of my job is resolving client technical queries so I appreciate brevity & clarity :)

Here's the latest update on my progress:

Regarding point 3)

I think you may have misinterpreted part of my post. Running a callgraph on both /bin/ls and /usr/bin/emacs succeeds.

There are no installation failures visible in /var/log/install.log. There are no errors regarding "bistro" and no mention of "callgraph" in the log

Regarding point 2)

yep my kernel is supported as far as I can see (Red Hat Enterprise Linux 3.0)

Regarding point 1)

Sampling works fine and both data views look normal (I can see instructions retired, cpu cycles etc.)

Regarding point 0)

I've tried to run both as a user and as root + I've checked access rights to the "global_data" directory. Also I have lots of disk space in the install and user dir locations.


It seems like the problem is either related to A) some peculiarity of my application or B) some results file/directory being inaccessible/missing.

Phew hope I've got everything in. Thanks for getting back to me so quickly on this one.

MJ

[jeffrey-gallagher]

YES! I did misunderstand. Thanks for confirming that that simple apps are WORKING, not failing.

Knowing the answers now that you gave, I'd say the most logical next steps are to focus on getting ANY kind of callgraph to run on your application. For more help regarding my suggestions that follow, don't forget:

$ man callgraph

Also, User's Guide, default location

/opt/intel/vtune/doc/users_guide/index.htm

OK, all that said, VTune callgraph generates a collector configuration text file to regulate the callgraph experiment that you are running, and this happens the first time you run the activity. You can edit those default settings by editing that file.

This file regulates which objects get instrumented, known as the instrumentation level in the docs I think. (I usually forget the name of the file, but it's in a VTune working directory: if you've never seen it, do a find command to look for files created today, for example, and track it down that way. All I can remember is, it sounds like a WINDOWS filename, which always makes me chuckle. Ends in .dll, or something like that. SORRY, not enough coffee yet today...)

ANYWAY, you can probably see where we're going with this.

FIRST, you edit the config file to make all modules MINIMAL instrumentation. See if callgraph runs without error, which it should, nothing fancy there. (And no data yet either, but we'll take care of that in the next steps.)

Then you edit that file andadd just one fully instrumented module, run callgraph again. Does it work? Edit the file again, add another module to the fully instrumented list, run callgraph again.

Do this one module at a time until the callgraph activity fails. THEN, we know which module is our problem, and that gives us something to work with. (could be more than one module that confuses callgraph, but you get the idea here. Keep adding them, if one fails, leave it out and go on. Will give us some good clues.)

This is much simpler to do than to talk about, and if I could remember where that file is, I would sure tell you, I always have to do a find command to locate the new files that have been recently created. (Just traipse around the working directory, you'll find it!)

What do you think?

cheers

jdg

Message Edited by jdgallag on 07-14-2005 02:17 PM

Message Edited by jdgallag on 07-14-2005 02:18 PM

Message Edited by jdgallag on 07-14-2005 02:21 PM

Message Edited by jdgallag on 07-14-2005 02:27 PM

Message Edited by jdgallag on 07-14-2005 02:28 PM

Message Edited by jdgallag on 07-14-2005 02:34 PM

[mjb34]

Ok jdg,

I think I have diagnosed my problem (or at least ONE of the problems causing failure). It's to do with pthreads.

My app relies on linuxthreads rather than the newer default NPTL (Native POSIX Threading Library). To enable this I need to set the environment variable LD_ASSUME_KERNEL=2.4.1. When "configuring" a callgraph activity on my app I get:

vtl activity -c callgraph -app ./myapp -moi ./myapp

Error in module "libpthread-0.10.so" - File is not valid module

and I can go no further.


vtl activity -c sampling -app ./myapp -moi ./myapp

works fine!


If my app uses NPTL then I get the error mentioned in my first post:

"Internal error while resolving the results file"

and this is because my app seg faults rather than terminating gracefully and clearly vtl doesn't catch this.

So it appears that to use vtl either:

a) I fix my app so it doesn't seg fault with NPTL libpthread-0.60.so

or

b) the vtl install is fixed to work with linuxthreads (could I somehow rebuild the kernel file?)


You can test this for yourself by downloading any pthreads apps such as "pthreadbench" and running the tests through "vtl -c callgraph" with and without LD_ASSUME_KERNEL set. I can callgraph the app fine when LD_ASSUME_KERNEL is *unset*.

If you or anyone can think of a workaround then I'd be really grateful.

Thanks

MJ

[jeffrey-gallagher]

INTERESTING stuff MJ!

Quick question about this part of your posting:

------------------------------------------------------------------

vtl activity -c callgraph -app ./myapp -moi ./myapp

Error in module "libpthread-0.10.so" - File is not valid module and I can go no further.

vtl activity -c sampling -app ./myapp -moi ./myapp

works fine!
-----------------------------------------------------------------

I'm sure I'm just being dense but I don't understand what the diff was, the command lines look the same, so, can you clarify a bit more what you mean there?

cheers

jdg

[mjb34]

Basically what that part of the posting means is that setting up a *sampling* activity works just fine while setting up a *callgraph* activity does not and fails during instrumentation of libpthread.

This seems to imply that my problem is not due to installation errors.

MJ

[jeffrey-gallagher]

YES yes yes. I actually misread that statement, am not used to seeing -moi in sampling activities! Thanks for the clarification, perfectly obvious!

Of course the two technologies, sampling and callgraph are very very different: callgraph doesn't require sampling collectors, etc.

Even though we suggest normally NOT doing this, can you duplicate the callgraph experiment logged in as root? Same error?

[mjb34]

Thanks for pointing out -moi is not necessary for sampling. Unfortunately I get the same error when logged in as root:

Error in module "libpthread-0.10.so" - File is not valid module

[jeffrey-gallagher]

Thanks for trying, and the quick turnaround. I'm filtering your linuxthread discussion to the engineers who will find it the most interesting. Either they or I will post the results here. Stay tuned.

cheers

jdg

[jeffrey-gallagher]

Ok. Engineering replied that for IA32 linux, everything worked for them. Details follow.

-------------------------------------

-- VTune 3.0 on RH3/IA-32 OK.

-- profiled the app you suggested, pthreadbench LD_ASSUME_KERNEL=2.4.1 set, was ok in all cases. libpthread-0.10.so was successfully profiled, as well as pthread applications and results were available.

---------------------------------------------

So, we know it works. That's pointing to the specifics of your setup now. Engineeringhas asked for more information about your system: OS,glibc version, the value of LD_LIBRARY_PATH.

Probably anything you could think of might be useful with regard to a non-default OS installation, for example any upgrades/downgrades that may be on board.

cheers

jdg

Message Edited by jdgallag on 07-19-2005 08:27 AM

Message Edited by jdgallag on 07-19-2005 08:27 AM

Message Edited by jdgallag on 07-19-2005 08:27 AM

[mjb34]

Ok here are the details. Probably the most critical aspect is that I am running IA64 Linux on 2 x Itanium 2 processors.

I'm running a modified 2.4.21 kernel (RHE 3) with glibc 2.3.2. In my LD_LIBRARY_PATH I have placed vtune/lib first and similarly vtune/bin for PATH.

MJ

[jeffrey-gallagher]

I'll report that and let you know what the reply is.

cheers

jdg

[Ekaterina_L_Intel]

Hi,

We tried VTune 3.0 on RHEL3.0/IA-64 and again: no such problems observed

We tried pthreadbench in any cases: w/ and w/o LD_ASSUME_KERNEL=2.4.1 set.

Please try to remove below temporary, cache and project folders and run vtl again:

rm -fr /tmp/$USER_vtune

rm fr [vtune install dir]/vtune/global_data/ISM/*

rm -fr /home/$USER/VTune

If you still meet the issue - please let us know.

BTW: there is no vtune/lib folder in the 3.0 please provide full LD_LIBRARY_PATH and PATH

Regards,

Katya

Message Edited by elogovsk on 07-20-2005 08:58 AM