H264 Decoder Missing Frames

[Gilad_B_]

I am decoding an H264 video with 194 frames, using the latest Intel MSDK. The problem I am facing is that I am receiving a MFX_ERR_MORE_DATA from DecodeFrameAsync before I decoded all the frames (at frame 188).

Allow me to elaborate: I am constantly decoding frames using AsyncDepth = 1. At each call to my decoding function I am appending data from the file to the mfxBitstream. After that I am calling DecodeFrameAsync, and then immediately to SyncOperation.

Frames 1-188 are decoded correctly - after comparison with VirtualDub output. However, whereas VirtualDub successfully reads 194 frames, I can only read 188. I have tried the same thing with a different video, and again, I have less frames than there are in the video.

This is taken directly from a sample, and I prefer to have this control over decoding for debugging purposes.

So my question is: What could cause this? How to overcome this issue? 

I have not attached any code to make the question short. If there is any code I should attach to help move the solution along, please let me know.

Thank you.

[Sravanthi_K_Intel]

Hello Gilad,

To understand your question - you are able to decode 188 frames only (and not 194) using sample decode from MSDK, right? And the last status you see before the application exits is MFX_ERR_MORE_DATA , right?

Would you kindly provide the input stream that is causing this issue, so that we can reproduce it on our end? Also, if you made any modifications to the sample code - would be great if you can attach the source as well. Any information that can ease reproducing the result on our end would be helpful.

[Gilad_B_]

Hello Sravanthi,

I'm afraid I was not clear enough. My code is derived from the sample. The sample itself is decoding the stream perfectly. I know that the problem is somewhere in my code, and I believe that I am did not understand something as I should have. 

The problem, I suspect, is that there are more frames cached even after I receive MFX_ERR_MORE_DATA from DecodeAsync, even though I made sure to call the synchronization event after every call and set AsyncDepth = 1.

Any ideas?

[Sravanthi_K_Intel]

Hi Gilad,

Thanks for the clarification. The MFX_ERR_MORE_DATA occurs when the function needs additional bitstream data for decoding the frame. So, can you ensure your input and output resolutions are correct, and so is your function to load the bitstream from the file?

Since you mention your implementation is a modification of the sample_decode, it would be great if you could send in the source file (and input stream) - I can try to reproduce the issue here (or debug the issue).

[Gilad_B_]

Hi Sravanthi,

It is a modification of a different sample, one that enables texture sharing with OpenGL: https://software.intel.com/en-us/articles/texture-sharing-from-intel-media-sdk-to-opengl

I will be able to send you some snippets tomorrow if you believe it would help.

Thanks!

[Sravanthi_K_Intel]

In your implementation, do you have a draining loop for the encoder? If you look at our tutorials or samples (http://software.intel.com/sites/default/files/mediasdk-tutorials-0.0.3.zip, additional details here - https://software.intel.com/en-us/articles/media-sdk-tutorial-tutorial-samples-index), you will find the drain loop following the main loop - that completes the remaining frames under progress. Lack of this loop can cause such issues.

The article mentioned above is beyond our support scope, more so external article such as this. In general, we recommend using our samples or tutorials to understand how to develop simple applications using MSDK, and modifying them to your application use-case.

[Gilad_B_]

The sample decoder you supply manages to decode the video correctly. Therefore, I assumed that the encoding process was correct. Am I wrong to assume that?

[Sravanthi_K_Intel]

Ah, I see where I could have confused you. "In your implementation, do you have a draining loop for the encoder" --> "In your implementation, do you have a draining loop for the DECODER". My bad!

Since your code is derived from the sample, and you suspect the issue could be in your code - I was making sure you got all the necessary loops in place. For example, in our decoder tutorials and samples, you will see two while() loops - one is the main processing loop, second is the draining loop. The second while() loop (draining loop) is important to have so that the residual frames being processed can complete.

Anyway, discussing code in text form can cause many confusions. If you can reproduce your issue by modifying MSDK samples or tutorials and attach the source code, it can be helpful for us to debug.

[Gilad_B_]

I think you've hit the nail on the target. What do you mean by "draining loop"? How do I do this?

[Sravanthi_K_Intel]

The following section in this article explains about the framework expected in MSDK, and what the main processing loop and draining loop constitute. https://software.intel.com/en-us/articles/framework-for-developing-applications-using-media-sdk#Processing Loop

In our tutorials (for example, simple_2_decode_vmem: simple_decode_vmem.cpp), you will find reference to the drain loop as well. For convenience, I have provided the code snippet that shows them in use for the decoder - make sure your code has them too. Hope this helps.

    //
    // Stage 1: Main decoding loop
    //
    while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_DATA == sts || MFX_ERR_MORE_SURFACE == sts) {
        if (MFX_WRN_DEVICE_BUSY == sts)
            MSDK_SLEEP(1);  // Wait if device is busy, then repeat the same call to DecodeFrameAsync

		mfxGetTime(&tStart_1);
        if (MFX_ERR_MORE_DATA == sts) {
            sts = ReadBitStreamData(&mfxBS, fSource);       // Read more data into input bit stream
            MSDK_BREAK_ON_ERROR(sts);
        }
		mfxGetTime(&tEnd_1);
		elapsed_1 += TimeDiffMsec(tEnd_1, tStart_1) / 1000;

        if (MFX_ERR_MORE_SURFACE == sts || MFX_ERR_NONE == sts) {
            nIndex = GetFreeSurfaceIndex(pmfxSurfaces, numSurfaces);        // Find free frame surface
            MSDK_CHECK_ERROR(MFX_ERR_NOT_FOUND, nIndex, MFX_ERR_MEMORY_ALLOC);
        }
        // Decode a frame asychronously (returns immediately)
        //  - If input bitstream contains multiple frames DecodeFrameAsync will start decoding multiple frames, and remove them from bitstream
        sts = mfxDEC.DecodeFrameAsync(&mfxBS, pmfxSurfaces[nIndex], &pmfxOutSurface, &syncp);

        // Ignore warnings if output is available,
        // if no output and no action required just repeat the DecodeFrameAsync call
        if (MFX_ERR_NONE < sts && syncp)
            sts = MFX_ERR_NONE;

        if (MFX_ERR_NONE == sts)
            sts = session.SyncOperation(syncp, 60000);      // Synchronize. Wait until decoded frame is ready

        if (MFX_ERR_NONE == sts) {
            ++nFrame;
            if (bEnableOutput) {
                // Surface locking required when read/write video surfaces
                sts = mfxAllocator.Lock(mfxAllocator.pthis, pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                MSDK_BREAK_ON_ERROR(sts);

                sts = WriteRawFrame(pmfxOutSurface, fSink);
                MSDK_BREAK_ON_ERROR(sts);

                sts = mfxAllocator.Unlock(mfxAllocator.pthis, pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                MSDK_BREAK_ON_ERROR(sts);

                printf("Frame number: %d\r", nFrame);
                fflush(stdout);
            }
        }
    }

    // MFX_ERR_MORE_DATA means that file has ended, need to go to buffering loop, exit in case of other errors
    MSDK_IGNORE_MFX_STS(sts, MFX_ERR_MORE_DATA);
    MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);

    //
    // Stage 2: Retrieve the buffered decoded frames
    //
    while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_SURFACE == sts) {
        if (MFX_WRN_DEVICE_BUSY == sts)
            MSDK_SLEEP(1);  // Wait if device is busy, then repeat the same call to DecodeFrameAsync

        nIndex = GetFreeSurfaceIndex(pmfxSurfaces, numSurfaces);        // Find free frame surface
        MSDK_CHECK_ERROR(MFX_ERR_NOT_FOUND, nIndex, MFX_ERR_MEMORY_ALLOC);

        // Decode a frame asychronously (returns immediately)
        sts = mfxDEC.DecodeFrameAsync(NULL, pmfxSurfaces[nIndex], &pmfxOutSurface, &syncp);

        // Ignore warnings if output is available,
        // if no output and no action required just repeat the DecodeFrameAsync call
        if (MFX_ERR_NONE < sts && syncp)
            sts = MFX_ERR_NONE;

        if (MFX_ERR_NONE == sts)
            sts = session.SyncOperation(syncp, 60000);      // Synchronize. Waits until decoded frame is ready

        if (MFX_ERR_NONE == sts) {
            ++nFrame;
            if (bEnableOutput) {
                // Surface locking required when read/write D3D surfaces
                sts = mfxAllocator.Lock(mfxAllocator.pthis, pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                MSDK_BREAK_ON_ERROR(sts);

                sts = WriteRawFrame(pmfxOutSurface, fSink);
                MSDK_BREAK_ON_ERROR(sts);

                sts = mfxAllocator.Unlock(mfxAllocator.pthis, pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                MSDK_BREAK_ON_ERROR(sts);

                printf("Frame number: %d\r", nFrame);
                fflush(stdout);
            }
        }
    }

    // MFX_ERR_MORE_DATA indicates that all buffers has been fetched, exit in case of other errors
    MSDK_IGNORE_MFX_STS(sts, MFX_ERR_MORE_DATA);
    MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);

[Gilad_B_]

But why is that required even though AsyncDepth = 1? I thought that was the whole point of per-frame synchronization?

[Sravanthi_K_Intel]

I did not notice the AsyncDepth remark, otherwise draining loop would have been a known issue causing the behavior. At this point, I have too little information and cannot speculate what else could be missing from your code -  like mentioned earlier, if you can modify existing MSDK samples/tutorials to reproduce your bug, please send the code along and I can help you.