After running decoder and VPP (For color conversion and Scaling), SyncOperation for Color conversion starts Failing with MFX_WRN_IN_EXECUTION. I had a timeout of 200 ms, which I increased to 1000 ms, but still same result.
Since the application is a real-time application, can't wait and need to drop frame in case it can't be processed. How to handle MFX_WRN_IN_EXECUTION? How to to flush the buffer that's queued for processing, because after sometime application runs out of allocated buffers.
Media Server Studio 2017 is used.
There is no way to flush already queued tasks. Application is expected call SyncOperation until it successfully returns MFX_ERR_NONE.
200 ms is completely unexpected. With HW MSDK implementation color conversion with scaling for one frame usually takes up to 10 ms.
A few questions to better understand your case:
Do you use HW or SW MSDK implementation ? Which decoder? What are input and output VPP color formats ? Resolution?
What is AsyncDepth value used in initialization of both decoder and VPP? Do decoder and VPP belong to the same MSDK session?
What is the memory type between decoder and VPP and from VPP: system, video, opaque?
If you call SyncOperation after both decoder and VPP, where do you see the glitch? In decoder or VPP?
If you set the timeout e.g. in 10 sec, when SyncOperation returns control?
Can you reproduce a similar issue with sample_decode? sample_decode has VPP support for scaling.
One correction it's VPP and Encoder and not Decoder/VPP
To Answer your questions
Overall application is following
Capture from camera 1920x1080 @ 30 fps using V4L2 APIs in UYVY format. One session is created to convert UYVY to BGRA. In and out memory are video memory. Then BGRA frame is mapped to CPU space and some pre-processing over it is done (like drawing, text, various cv analysis etc.). After this another media SDK session is given to convert from BGRA to NV12 and Encode to h.264 (This color-conv and encode happens in same session).
Both the sessions are using HW implementation.
Session 1: Color Conversion from UYVY to BGRA - uses AsyncDepth of 65
Session 2: Color Conversion from BGRA to NV12 and Encoder uses AsyncDepth of 5
SyncOperation fails for Color Conversion of Session 1 - which is doing only UYVY to BGRA. This fails after sometime around 40-60 minutes of operation (Roughly after application has processed about 70000 to 90000 frames).
Not able to reproduce it on sample code.
One recommendation: if your app does live-streaming, it's better to use AsyncDepth=1 because bigger AsyncDepth values may improve performance but also will increase latency.
Now I understand you pipeline much better. However to tell you the truth it's very hard make any conclusion without a real reproducer. The fact that the issue happens after 40-60 minutes makes me think that it might a low level issue (e.g. in KMD).
Therefore a few more questions:
on Ubuntu, do you have 4.4 kernel with patches from MSS package?
When you say that you use V4L2 and send in VPP1 surfaces in video memory, do you mean that that create vaSurfaceID's from V4L2 buffers (without memory copy) or you lock/unlock VPP input video memory and manually load data from camera?
Do you join session1 and session2 via MFXJoinSession() call?
It's 4.4 Kernel with kernel patches applied which came along with media server package.
Currently V4L2 buffer is copied to a vaSurface. Though I was looking for a way to avoid this copy - Do you have any suggestion on this as well (Any sample would help).
Session 1 and Session 2 are not joined. They are used independent of each other.
There was a typo - AsyncDepth is 5 and not 65 for session 1.
>> Session 1: Color Conversion from UYVY to BGRA - uses AsyncDepth of 65
Few more observations:
1. I changed AsyncDepth to 1 for both the sessions. Now this error comes after 20+ hrs.
2. If I capture at 10 FPS with 1280x720 resolution - This error didn't occur for more than 48+ hrs. So possibly issue happens after number of frames being processed.
- Reducing AsyncDepth to 1 helps and makes it run for longer
- Also I moved to yocto from ubuntu and had made it run much longer (More than 48+ hrs with Full HD @ 30 FPS). Looks like the kernel patches for generic kernel possibly have some issues. In intel's yocto build these patches are already included.