Vulkan App exits with GPA

GregConnected · ‎09-22-2022

I have a Vulkan application I'd like to profile with GPA Trace. When I try to start the application from Graphics Monitor, the application starts, then exits almost immediately with no visible error from either itself or Graphics Monitor.

However the application logs this error from vulkan validation layers:

Vulkan (Error) : Validation Error: [ VUID-vkCmdResetQueryPool-firstQuery-00797 ] Object 0: handle = 0xa7c5450000000023, type = VK_OBJECT_TYPE_QUERY_POOL; | MessageID = 0xd66ad571 | VkCmdResetQueryPool(): In Query VkQueryPool 0xa7c5450000000023[] the sum of firstQuery (10) + queryCount (2)is greater than the queryPool size (11). The Vulkan spec states: The sum of firstQuery and queryCount must be less than or equal to the number of queries in queryPool (https://vulkan.lunarg.com/doc/view/1.3.216.0/windows/1.3-extensions/vkspec.html#VUID-vkCmdResetQueryPool-firstQuery-00797)

I can run this app in DX11, OpenGL 4.5 and Vulkan. The test system contains an integrated Intel HD Graphics GPU and an nVidia Quadro GPU.

If I run with DX11 or GL on nVidia, the app runs and I see the profile overlay display. Running on intel GPU in any mode fails shortly after start.

Pamela_H_Intel · ‎11-03-2022

GregC - Have you tried any analysis of your workload with GPA Framework (our command line tool)? If so, did you run into any issues? Another thing you can try is capturing either a trace or a frame via System Analyzer.

Are you using VK_KHR_fragment_shading_rate anywhere in your workload? We realized recently that we were missing support for that, so we added it in GPA Framework (released 30.September), but it is not available yet in our GUI tools. So in this case, trying System Analyzer captures likely will not work.

Pamela

GregConnected · ‎11-03-2022

Thanks for responding Pamela. Ironically I was just about to bump that old post today, as I had not heard back or made any progress.

I have not tried the GPA Framework command line tool. I find files gpa-injector.exe and gpa-player.exe installed, so I can try using them.

I have also not tried System Analyzer recently. I will aim to do so soon.

No, I'm not using VK_KHR_fragment_shading_rate.

GregConnected · ‎11-07-2022

Ran gpa-injector.exe on Windows with layers "capture" and "hud-layer".
The app started and ran, and HUD was visible over app.

Log shows:

[GPA] I 60268 GPA_LAYER_PATH environment variable is not set or does not exist, using default layers only
[GPA] I 60268 Found layer: C:\Program Files\IntelSWTools\GPA\Streams\layers\hud-layer-x64.dll
[GPA] I 60268 HUD Layer loaded!
[GPA] I 60268 Found layer: C:\Program Files\IntelSWTools\GPA\Streams\layers\capture-x64.dll
[GPA] I 60268 Loading layer "d3d12-state-tracker" due to requirement from layer "capture"
[GPA] I 60268 Found layer: C:\Program Files\IntelSWTools\GPA\Streams\layers\d3d12-state-tracker-x64.dll
[GPA] I 60268 PageTracker: enabling default tracking
[GPA] I 60268 Capture layer loaded
[GPA] I 60268 Hooking Win32...done
[GPA] I 60268 Hooking Vulkan...done
[GPA] W 60268 Vulkan version mismatch detected! Expected 1.0.170, have 1.3.224
[GPA] I 60268 Unhooking Vulkan...done
[GPA] I 60268 Unhooking Win32...done
[GPA] I 60268 UNLOADED
...repeating...
[GPA] E 96636 Unrecognizable Vulkan structure detected in the pNext chain. Try updating GPA Framework or building GPA Framework with latest Vulkan SDK installed.

I certainly am using some Vulkan 1.3 features.

I am using GPA installer and version:
gpa_22.2.1654744802_release-external_x64_standalone.msi
Which appears to be the latest available.

I don't see any capture file generated. Guessing the Vulkan version mismatch is bad. Plan to try System Analyzer next.

Edit: More observations:

When using nVidia GPU, the HUD appears and app runs until I exit it.

When using intel GPU, the app exits with no error in app log or GPA injector log. (No new errors or messages vs log with nVidia, posted above. The last line from top section is the "version mismatch detected".)

GregConnected · ‎11-15-2022

I can get some information from System Analyzer, but I really wanted to capture some frames with Graphics Monitor and Graphics Trace Analyzer. Specifically I'd like to compare frames from DX11 with Vulkan.

GregConnected · ‎11-29-2022

To clarify... This issue remains unresolved. I am not able to analyze my Vulkan 1.3 application.

Pamela_H_Intel · ‎11-30-2022

GregConnected,

So you attempt to capture a Frame via Graphics Monitor:

You can capture running the DX11 and OpenGL versions?
You cannot capture running the Vulkan version?
The capture window doesn't even open (for Vulkan only)?
Please clarify whether you are capturing Frame, Trace or Stream.

But you can successfully attach the Vulkan app to System Analyzer? If this is the case, you should be able to capture a frame or trace. On the upper left corner of the System Analyzer window either the red dot or the camera icon should be active. The red dot is active if you chose to capture a trace (several seconds of CPU/GPU activity), the camera is active if you chose to capture a frame. Note: it is important to ensure that game is in the foreground, otherwise, the data you are capturing is not from your game . . . because nothing from your game is rendering to the screen.

Sometimes Stream capture works better than Frame capture since that is what the devs have focused on for the last couple of years. A stream is simply a set of frames. When you capture a stream and open the stream, you then select the frame or frames of interest that you want to analyze, then the frame view of Graphics Frame Analyzer opens.

I am going to run your question by the developers to see if they have seen that query count issue.

Pamela

GregConnected · ‎11-30-2022

Thanks for following up Pamela.
I've just done a full re-test via GraphicsMonitor and am seeing some different, somewhat better results.
I have not updated the intel display driver or intel tools recently, so I can only conclude that they relate to code changes on my side for improved Vulkan stability.

Here's my test matrix of APIs and GPUs on this (dual GPU) machine:

	Stream	Frame	Trace
Intel (HD 630)
DX11	yes	yes (overlay+capture)	yes
GL	fail	ran (overlay)	fail
VK	yes	ran (no overlay)	yes
nVidia (T1000)
DX11	yes	yes (overlay+capture)	yes
GL	fail	ran (overlay)	yes
VK	yes	ran (no overlay)	yes

fail = App exits soon after start, often seeing a few frames of loading screen.

I expect the lack of Frame capture in GL and VK is because it is not supported.

This was looking promising, so I captured a stream of DX11 and a stream of VK.

DX11 stream appeared to work correctly... capture -> select frame -> analyze with GFA.

Vulkan captured a stream and I could see a series of frames and thumbnails every 5sec or so. However, clicking on a frame to analyze showed:

"Playing back to selected region"

Followed by error box:
"An internal error has occurred. Please contact Intel customer support"

So that is as far as I've got. Still not able to analyze Vulkan.

Edit: I see at least part of the difference between previously reported tests... The re-tests above were all Release build, with Vulkan validation disabled. When I build a non release version, and try capture a Stream, the app (with Vulkan) exits immediately during initialization, logging (as before):

Validation Error: [ VUID-vkCmdResetQueryPool-firstQuery-00797 ] Object 0: handle = 0xa7c5450000000023, type = VK_OBJECT_TYPE_QUERY_POOL; | MessageID = 0xd66ad571 | VkCmdResetQueryPool(): In Query VkQueryPool 0xa7c5450000000023[] the sum of firstQuery (10) + queryCount (2) is greater than the queryPool size (11). The Vulkan spec states: The sum of firstQuery and queryCount must be less than or equal to the number of queries in queryPool (https://vulkan.lunarg.com/doc/view/1.3.224.1/windows/1.3-extensions/vkspec.html#VUID-vkCmdResetQueryPool-firstQuery-00797)

So with a Release build I can get one step further, but still not enough.

Pamela_H_Intel · ‎12-06-2022

Greg - I am glad you are getting better results. GTA supports OGL, but GFA does not, so that failure is no surprise.

To clarify - the table you created - yes means you could capture, but not necessarily analyze?

When you attempt to open a frame, please ensure that you have selected an Intel GPU to play back on. If you captured on NVIDIA, that's usually okay, but you won't get all the data and sometimes there are other issues, if you don't play back on Intel . . . that means . . . you don't need to disable your other GPUs, but you need to select an Intel GPU in GFA before attempting to open a frame.

Huh. I don't see a way to attach an image in my interface. I'm going to send this, then try to send you an image of what I am talking about (select GPU in upper left corner of GFA).

But first, caveat: though it usually does work, and we spend time and effort to make GPA work with 3rd party GPUs, it is not guaranteed. So you might need to disable your other GPUs before capturing, and capture on an Intel GPU.

Pamela_H_Intel · ‎12-06-2022

So, this is the top left corner of GFA. When playing back (i.e. opening a frame for analysis) in GPA's GFA, always choose an Intel GPU to play back on, otherwise you won't get all the hardware counters (data).

GregConnected · ‎12-06-2022

Thanks Pamela, I'll try selecting GPU from GFA and get back to you later.

RE Table, yes, it is a you said. The 'yes' means I could capture but not necessarily analyze the captured result.

I also realized later that my apparent new success was actually that capturing from release build worked more successfully than non-release builds, otherwise the table would have fewer 'yes' results.

Worth noting, it is difficult to select GPU for application to run on. I typically must select the adapter via API and then only use the built in display (not external monitor connected via HDMI). Otherwise the other adapter is used, or performance is near halved as it composites the result with the other adapter for desktop display.

Pamela_H_Intel · ‎02-27-2023

Greg - I haven't heard from you. Is it working now?

GregConnected · ‎02-27-2023

No change since last report Pamela...

Release build can capture a trace, but the Trace Analyzer won't view it, instead it shows "An internal error has occurred. Please contact Intel customer support".

Non-Release builds fail to capture a trace with log file reporting:

[GPA] W 60268 Vulkan version mismatch detected! Expected 1.0.170, have 1.3.224
[GPA] E 96636 Unrecognizable Vulkan structure detected in the pNext chain. Try updating GPA Framework or building GPA Framework with latest Vulkan SDK installed.

I had kind of given up on it. I do see a newer GPA v22.4 exists, I am currently using v22.2. I will have to retest when I have time.

GregConnected · ‎02-27-2023

Back again. I've installed the latest GPA v22.4 and am having some success...

Non-Release build fails to capture a trace. This time the log file reports nothing, as if .exe terminated unexpectedly. If I launch on nVidia instead of Intel GPU, I do get a log error:

Validation Error: [ VUID-vkCmdResetQueryPool-firstQuery-00797 ] Object 0: handle = 0xa7c5450000000023, type = VK_OBJECT_TYPE_QUERY_POOL; | MessageID = 0xd66ad571 | VkCmdResetQueryPool(): In Query VkQueryPool 0xa7c5450000000023[] the sum of firstQuery (10) + queryCount (2) is greater than the queryPool size (11). The Vulkan spec states: The sum of firstQuery and queryCount must be less than or equal to the number of queries in queryPool (https://vulkan.lunarg.com/doc/view/1.3.236.0/windows/1.3-extensions/vkspec.html#VUID-vkCmdResetQueryPool-firstQuery-00797)

Note this only occurs when launched from Graphics Monitor, not under normal usage.

Release build however works fine, I captured a trace and loaded the capture into Trace Analyzer without error.

(Release build running on nV also works but captures very little as expected.)

Pamela_H_Intel · ‎04-12-2023

Greg - thanks for the info. This is a strange problem - where your release build works better than your dev build in GPA. Is there any chance, for your non-release build, you are on a platform with Dev Mode toggled off?

I will be talking to the GPA Product Owner about this to find out if he has any information or workarounds. I will also talk to our Vulkan expert. He made some changes a couple of releases ago when a customer helped us figure out that a couple of missing Vulkan API functions caused errors, which of course shouldn't happen. Missing a function definition should no longer cause an issue, but maybe there is something else wonky. They may need your build to explore the issues.

If you've downloaded GPA 23.1 yet, let me know if it works any better. We have rebuilt our team after having lost our Russian colleagues a year ago, so we are "catching up" on our roadmap and fixes and have regained some of our tech expertise that was missing for awhile.

GregConnected · ‎04-12-2023

Thank you for following up Pamela and raising the issue with developers. Now to answer your questions...

The release and no-release builds are being developed and tested on the same machine, with Developer Mode enabled.

I just re-tested for GPA 23.1 and the behavior is similar...

Release build works fine: Application launches from GPA, Trace can be captured. Trace can be opened in Trace Analyzer.

Non-Release build fails: Application does not successfully launch from GPA. Application terminates abnormally during startup. This time no vulkan error messages are present in log. (With previous versions GPA, application logged various vulkan error messages.)

The vulkan code used by the application has not been significantly updated since Dec 2022.

Pamela_H_Intel · ‎04-13-2023

While waiting for assessments/thoughts/hints from the dev team . . .

Regarding your statement from 27.Feb.: (Release build running on nV also works but captures very little as expected.)

One other thing you can try (though this doesn't explain dev build vs. release build) capture a stream on NVIDIA and open it on the Intel GPU (using that upper left corner dropdown). This usually works because a capture is capturing the calls and parameters. It is not until you open a frame that it actually executes the calls, calculating all the rendering data.

Pamela_H_Intel · ‎04-13-2023

Greg - the dev team reported back to me:

Because your app, the environment and GPA might all have different settings that interfere with each other, we need to address our logic that checks and enables options. Our Vulkan expert says that it is on his TODO list and he will raise its priority. No guarantees this will enable you, but it's a good starting point.
He also says "thank you" for reporting the error messages. We definitely need to address the version mismatch error.

Meanwhile, are you able to get the info you need? As of 27.Feb. you were able to open and analyze a trace, correct? What about a stream?

FYI - I don't think this will help because I don't think you are using multiple APIs, but just in case . . . you can enable "Extended Profiling Modes" in Settings - upper right in Graphics Monitor. Then, when you choose Stream from the dropdown on the bottom right of Graphics Monitor, choose Stream [VK]. This can help when there are other APIs involved in some way, to ensure that you capture only VK calls.

GregConnected · ‎04-13-2023

@Pamela_H_Intel wrote:

Greg - the dev team reported back to me:

...

Meanwhile, are you able to get the info you need? As of 27.Feb. you were able to open and analyze a trace, correct? What about a stream?

Thank you for the update and information Pamela.

While I can now see Release build trace results (and I believe I did a stream test last time), I've been busy on other projects and not spent time analyzing the result yet. My goal is to compare the same frame in DX11 and VK to observe differences.

Pamela_H_Intel · ‎04-13-2023

Okay. When you get back to it, let me know if there is any particular data that you can't see . . . so maybe we can find a workaround.

GregConnected · ‎04-16-2023

Pamela, I tried some analyzing today with little joy, same road block as last time...

Captured a Release build Stream and Frame, but can't analyze them with Graphics Frame Analyzer.

Clicking a frame in GFA shows "Playing back to selected region" (please wait animation) for some seconds, then

---
Error
An internal error has occurred. Please contact Intel customer support.
OK
---

Checking device selection top left shows correct device...

Machine is a Dell Precision laptop with Intel UHD 630 and nVidia T1000 GPUs.

Repeated the test with both external monitor and only laptop display.

So the only analyzing I can do at present, is Release build capture a Trace then view that in Graphics Trace Analyzer.