Re: vcs/collectunits1/tmu/src/tmu.c:568 alloc_buf: Assertion 'buf->head != ((void *)0)' failed.

etheory · ‎05-23-2022

Dear Intel,

My name is Luke Emrose and I am the Technical Lead for Rendering at Animal Logic.

I have been experiencing some issues with vtune lately whilst profiling our proprietary renderer that have been rather frustrating.

I get the following error now, every single time I run a test:

vcs/collectunits1/tmu/src/tmu.c:568 alloc_buf: Assertion 'buf->head != ((void *)0)' failed.

and nothing I've done seems to fix the issue.

My machine is:

32 core 64 hyperthreads

vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
stepping : 7
microcode : 0x5003102
cpu MHz : 1244.854
cache size : 22528 KB

vtune version:

Intel oneAPI VTune Profiler 2021.4.0

This is whilst running a render test that runs for 40 minutes, and, up until this week, ran about 1 out of 2 times without an error, but now stubbornly refuses to run.

In general, vtune has been extremely unstable and unreliable when used for profiling on our end.

Any assistance you could provide would be most useful, thank you.

Regards,

Luke

etheory · ‎05-23-2022

I just tried upgrading to Intel VTune Profiler 2022.2.0 and the errors continue, now I get:

AMPLXE_TPSSCOLLECTOR: isa:264: (cutoff_level != (tpss_address_t)0) :

Assertion failed: isa:264: (cutoff_level != (tpss_address_t)0) : . Please contact the technical support.

Assertion failed: isa:264: (cutoff_level != (tpss address t)0) : . Please contact the technical support.

And am unable to collect any data at all with VTune 2022.2.0 which is quite surprising.

etheory · ‎05-23-2022

Rebooting my machine seems to have made a difference for some reason....

Trying again.

etheory · ‎05-24-2022

OK, now I get a different error:

vcs/collectunits1/tmu/src/tmu.c: 421 write_trace: Assertion 'Cannot write magic record to trace' failed.

Though this time, with 2022.2.0 it was able to go through the entire render before failing.

JyothisV_Intel · ‎05-25-2022

Hi,

Good day to you.

Thanks for posting in Intel Communities.

We are extremely sorry that the Intel VTune Profiler is causing issues with your workflow.

To clearly assist you with your issue, can you share the output of the following commands:

$ sudo <vtune-install-dir>/sepdk/src/insmod-sep -r

$ sudo <vtune-install-dir>/sepdk/src/insmod-sep -q

$ sudo <vtune-install-dir>/bin64/vtune-self-checker.sh

Eg.:

sudo /opt/intel/oneapi/vtune/latest/sepdk/src/insmod-sep -r
sudo /opt/intel/oneapi/vtune/latest/sepdk/src/insmod-sep -q
sudo /opt/intel/oneapi/vtune/latest/bin64/vtune-self-checker.sh

To replicate the issue from our side, can you share a sample reproducer code along with the exact steps or commands that you followed to do the analysis.

Regards,

Jyothis V James

etheory · ‎05-25-2022

Thanks Jyothis V James,

The shell I'm running this from is also printing the following error, in case it helps:

Gtk-Message: 20:14:54.796: GtkDialog mapped without a transient parent. This is discouraged.
config://viewpoint/include/timeline.xsl?id=timelinePane&displayName=TimelineWindow&globalEventAreaName=CPUUsageOverTime&description=HotspotsTimelineWindowDescription&cpuDataQuery=CPUTime&samplePointsQuery=CPUSample&samplePointsTooltipQuery=CPUFunctionFull&cpuDataCumulativeQuery=CPUTimeAllCoresCumulative&taskQuery=Task&helpKeyword=configs.timeline_f1106&contextSwitches=false&waits=true&cpuOverheadAndSpinTimeQuery=CPUOverheadAndSpinTimeCapped&cpuOverheadAndSpinTimeCumulativeQuery=CPUOverheadAndSpinTimeAllCoresCumulative&rowByPrefix=CPU&:2: parser error : Start tag expected, '<' not found

As for the output you requested:

sudo ./insmod-sep -r

Creating group vtune ... done

Warning: the following driver(s) were not found loaded in the kernel: sep5.

Warning: no vtsspp driver was found loaded in the kernel.

Warning: no socwatch driver was found loaded in the kernel.

Warning: the following driver(s) were not found loaded in the kernel: socperf3.

The PAX service is not loaded anymore.
--------------------- Loading PAX driver ---------------------
Checking for PMU arbitration service (PAX) ... not detected.
Attempting to start PAX service ...
Executing: insmod ./pax/pax-x32_64-3.10.0-1127.el7.x86_64smp.ko
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "660" ... done.
The pax driver has been successfully loaded.

--------------------- Loading SOCPERF driver ---------------------
Checking for socperf driver ... not detected.
Attempting to start socperf service ...
Executing: insmod ./socperf/src/socperf3-x32_64-3.10.0-1127.el7.x86_64smp.ko
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "660" ... done.
The socperf3 driver has been successfully loaded.

--------------------- Loading SEP driver ---------------------
Executing: insmod ./sep5-x32_64-3.10.0-1127.el7.x86_64smp.ko
Setting group ownership of devices to group "vtune" ... done.
Setting file permissions on devices to "660" ... done.
The sep5 driver has been successfully loaded.

--------------------- Loading VTSSPP driver ---------------------
Checking for vtsspp driver ... not detected.
Executing: insmod ./vtsspp/vtsspp-x32_64-3.10.0-1127.el7.x86_64smp.ko gid=1001 mode=0660 ksyms="ffffffffab520400"
The vtsspp driver has been successfully loaded.

--------------------- Loading SOCWATCH driver ---------------------
Checking for socwatch driver ... not detected.

Executing: insmod ./socwatch/drivers/socwatch2_14-x32_64-3.10.0-1127.el7.x86_64smp.ko

Setting group ownership of device file to group "vtune" ... done.
Setting file permissions of device file to "660" ... done.

The socwatch driver has been successfully loaded.

NOTE:

The driver is accessible only to users under the group vtune.
Please add the users to the group 'vtune' to use the tool.

To change driver access group, reinstall the driver using -g <desired_group> option.

NOTE:
The driver is accessible only to users under the group vtune.
Please add the users to the group vtune to use the tool.

To change driver access group, reload the driver using -g <desired_group> option.

sudo ./insmod-sep -q

pax driver is loaded and owned by group "vtune" with file permissions "660".
socperf3 driver is loaded and owned by group "vtune" with file permissions "660".
sep5 driver is loaded and owned by group "vtune" with file permissions "660".

socwatch driver is loaded and owned by group "vtune" with file permissions "660".

vtsspp driver is loaded and owned by group "vtune" with file permissions "660".

sudo ./vtune-self-checker.sh

HW event-based analysis (counting mode) (Intel driver)
Example of analysis types: Performance Snapshot
Collection: Ok
Finalization: Ok...
Report: Ok

Instrumentation based analysis check
Example of analysis types: Hotspots and Threading with user-mode sampling
Collection: Ok
Finalization: Ok...
Report: Ok

HW event-based analysis check (Intel driver)
Example of analysis types: Hotspots with HW event-based sampling, HPC Performance Characterization, etc.
Collection: Ok
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Finalization: Ok...
Report: Fail
vtune: Error: 0x40000024 (No data) -- No data is collected. Possible reasons:

HW event-based analysis check (Intel driver)
Example of analysis types: Microarchitecture Exploration
Collection: Ok
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Ok

HW event-based analysis with uncore events (Intel driver)
Example of analysis types: Memory Access
Collection: Ok
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Ok

HW event-based analysis with stacks (Perf)
Example of analysis types: Hotspots with HW event-based sampling and call stacks
Collection: Ok
vtune: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location.
Finalization: Ok...
vtune: Warning: The result contains a lot of raw data. Finalization may take a long time to complete.
vtune: Warning: Cannot read load addresses of sections from `/sys/module/symev-custom-3-10-0-1127-el7-x86-64/sections'. This may affect the correctness of symbol resolution for `/lib/modules/3.10.0-1127.el7.x86_64/kernel/drivers/char/symev-custom-3-10-0-1127-el7-x86-64.ko'. Make sure this directory exists and all files in this directory have read permissions.
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Ok

HW event-based analysis with context switches (Intel driver)
Example of analysis types: Threading with HW event-based sampling
Collection: Ok
vtune: Warning: To enable hardware event-based sampling, VTune Profiler has disabled the NMI watchdog timer. The watchdog timer will be re-enabled after collection completes.
Finalization: Ok...
vtune: Warning: Cannot locate debugging information for the Linux kernel. Source-level analysis will not be possible. Function-level analysis will be limited to kernel symbol tables. See the Enabling Linux Kernel Analysis topic in the product online help for instructions.
Report: Ok

Checking DPC++ application as prerequisite for GPU analyses: Fail
Unable to run DPC++ application on GPU connected to this system. If you are using an Intel GPU and want to verify profiling support for DPC++ applications, check these requirements:
* Install Intel(R) GPU driver.
* Install Intel(R) Level Zero GPU runtime.
* Install Intel(R) oneAPI DPC++ Runtime and set the environment.

The check observed a product failure on your system.
Review errors in the output above to fix a problem or contact Intel technical support.

The system is ready for the following analyses:
* Performance Snapshot
* Hotspots and Threading with user-mode sampling
* Microarchitecture Exploration
* Memory Access
* Hotspots with HW event-based sampling and call stacks
* Threading with HW event-based sampling

The following analyses have failed on the system:
* Hotspots with HW event-based sampling, HPC Performance Characterization, etc.
* GPU Compute/Media Hotspots (characterization mode)
* GPU Compute/Media Hotspots (source analysis mode)

Log location: /tmp/vtune-tmp-root/self-checker-2022.05.25_22.04.28/log.txt

Unfortunately I cannot provide you with any replication steps as this is whilst profiling proprietary internal technology.

I could potentially organize some kind of meeting to facilitate such a thing if that would be possible?

Regards,

Luke

etheory · ‎05-25-2022

Some feedback also to make vtune more user friendly.

1.) I cannot select text and copy/paste it in any part of the collection log interface. This is pretty frustrating.

2.) The scroll-bars for the collection log only appear on mouse-over, which sometimes makes them impossible to find.

3.) When the application output is collected using the preference setting "Application output destination - Product output window", there is no way to go directly to the output data in the .txt file it is collected in. And as above you can't copy any of the data or select it, which is something I often want to do.

JyothisV_Intel · ‎06-13-2022

Hi,

Good day to you.

Sorry for the delay. We are checking on this internally.

Thanks and Regards,

Jyothis V James