Which Gstreamer plugin are you using, is it this one in the GitHub?
You can check this document for DMA buffer usage, you might need to add "io-mode=5":
Could you report this issue directly in the GitHub? this plugin is supported by different Intel team, it would be speed up the resolution if you can ask them directly.
The GStreamer is not part of the MediaSDK so it takes time for me to do the investigation.