When stencil test is enabled in the pipeline state in Direct3D 12, the right and the bottom edges of the viewport are clamped to the right and the bottom edges of the render target.
Originally, this issue was reported on GitHub for the Xenia Xbox 360 emulator, which uses a 8192x8192 viewport (the maximum size of a render target on the Xbox 360) to emulate drawing directly in screen coordinates without a viewport, that is possible on the Xbox 360, without vertex position precision loss caused by the division (it's simple exponent biasing for power of two), but render targets are generally much smaller than that size — so the clear rectangle drawn by the game becomes very small.
The viewport, though, doesn't seem to be clamped to the left and the top edges — negative TopLeftX and TopLeftY are handled as normal.
Here is a screenshot comparing a rectangle drawn with stencil testing disabled ("culled", in red — with all culling disabled by PIX) and one with stencil enabled ("visible", that passed all tests — in green). The render target is 1280x512, and the viewport is at -16, -96, with the size 1536x768 (the right edge at 1520, 672).
The vertex shader generates a rectangle covering the upper-left quadrant of the viewport — that is, from -16, -96, to 752, 288). This is correct for the red rectangle, and matches the behavior on Nvidia and WARP.
However, the green rectangle, that has stencil test enabled, has its bottom-right corner at 632, 208. This corresponds to the center of a viewport between -16, -96 and 1280, 512 (the right and the bottom boundaries of the render target):
-16 + (1280 - (-16)) / 2 == 632
-96 + (512 - (-96)) / 2 == 208
I've tried changing various other state parameters: whether there is a color render target, depth test enabled/disabled, depth/stencil target format, back face culling, front face winding order, DepthClipEnable, MSAA sample count — they all seem to have no effect. However, DepthStencilState.StencilEnable = TRUE appears to be enough to trigger this bug, regardless of the masks, comparison functions, and operations.
In the attachment, there's source code for a reproduction C++ console application, which runs the test on every adapter, and writes the red component of the color value for the bottom-right pixel (which should be 255 if the rectangle is large enough, 0 — the clear color — otherwise) of the intended rectangle to stderr. If launched from PIX, GPU captures will automatically be created for every adapter the test was executed on. You can experiment with the `viewport` values there.
The issue is reproducible on the Intel UHD Graphics 630 on Windows 11 64-bit with the 31.0.101.2121 driver (latest at the point of the report) and older versions throughout at least a couple of years.
The bug doesn't happen, however, on the Intel HD Graphics 4400 with the 15.40.42.5063 driver (the last driver supporting Direct3D 12 there) and older drivers for it.
I haven't verified this behavior on other GPU APIs though, but it may be worth checking them too (especially D3D11 possibly).
Link Copied
Hello @Triang3l
I was able to replicate the issue on the UHD 630, but not on the UHD 770 or ARC 770
When viewing the results of the program running on the console, all the devices returned a 255 color value (except for the UHD 630) which was expected, which indicates to me that the problem is tied to that specific version of the driver for the UHD 630 (101.2121) as you mentioned before.
The only thing I couldn't replicate, was to make PIX show the comparison of the rectangle drawn with the stencil disabled (red) and the other with the stencil enabled (green) on the same screen. All GPU captures taken showed only a blank rectangle.
If you can share a guide, step by step on how to show both rectangles, I would really appreciate it.
Something important to mention is the newer processor generations (11th and up) are the common target for fixes. Older products are limited to legacy updates (critical fixes and security vulnerabilities) and have a quarter-year release cadence.
Either way, I'm going to be sure to communicate the impact of this issue and hopefully hear back soon.
