H.264 decoder and RestoreErrorRect

[Emmanuel_W_]

Hi,

I was wondering what exactly is the motivation behind the RestoreErrorRect function in the H.264 decoder sample. When some MacroBlocks are missing from the stream RestoreErrorRect is called which fill the missing MBs with gray.
This condisderably degrage the following pictures and most often than not is worse than not doing anything.

Emmanuel

[Chao_Y_Intel]

Hello Emmanuel,

Some comment from our expert:

RestoreRectError function try to find nearest frame and coped data from it. If such frame doesnt exist (for example error occurred at first frame and there is no any other valid frame) functions fill missed data by gray color. I guess it is optimal behavior. Gray color is much better then garbage

Thanks,
Chao

[Emmanuel_W_]

Quoting - Chao Y (Intel)

Hello Emmanuel,

Some comment from our expert:

RestoreRectError function try to find nearest frame and coped data from it. If such frame doesnt exist (for example error occurred at first frame and there is no any other valid frame) functions fill missed data by gray color. I guess it is optimal behavior. Gray color is much better then garbage

Thanks,
Chao

Hi Chao,

Thanks for the update. The problem is a little more complex than I fist describe, and the RestoreErrorRect function might not be the direct culprit (I will try to provide a stream that demonstrate the issue).

The scenario is simple,I decode a stream that contains some loss, the picture are decoded and eventually I get a gray flash even though I had some previously decoded pictures.

So far I have found that inside DecodeSegmentCAVLC_Single, there is no check to see if we have reach the end of a slice. This in turn caused an exception when decoding a MB which cause the decoding of the frame to stop. I suspect also I havn't been able to reproduce it exactly that if this happens on an IDR, and the next frame also contains loss and call RestoreErrorRect I would get the gray haze.

Inside DecodeSegmentCAVLC_Single I have added the following check which seems to solve the issue:

[cpp]if ((sd->m_MBSkipCount <= 1) &&
                ((true == sd->m_pSlice->IsSizeUnknown()) || (sd->m_pBitStream->BytesLeft()<=3)))
            {
                // Check for new slice header, which can change the next MB
                // to be decoded
                if (false == sd->m_pBitStream->More_RBSP_Data())
                {
                    // align bit stream to start code
                    nCurMBNumber += 1;
                    umcRes = UMC_ERR_END_OF_STREAM;
                    break;
                }

            }   // check for new slice header[/cpp]

The checks with BytesLeft is my own addition, 3 is to account for the padding added by the NAL splitter.

You will find attched a 264 file that demonstrate the DecodeSegmentCAVLC_Single issue. The first frame is missing the third slice. With the current implementation I get the withoutcheck.jgp result. With my patch I get the checkend.jpg.

Finally to go back on RestoreErrorRect, it seems to be missing the last row of MB. The offset of the end of the copy seems to be taken from the top left of the last MB instead of the bottom right.

I will try to provide you more information

Emmanuel

[Emmanuel_W_]

After more investigation, the gray haze happens when loosing a slice in IDRs that are in the middle of the stream. I understand that it is not always applicable but in my situation (video conferencing), using the last output picture to conceal the IDR is usually preferable even in the case of scene change or resolution change.

To recap:
_ It would be nice if we could have an option to use the last output frame for concealment in case of loss in an IDR
_ If we lose one slice, the decoding of the frame stops and all the remaining slices are not decoded (see above)
_ RestoreErrorRect contains a bug where the first and last line of MB in the concealed slice are not dealt correctly. The function assume a linear chunk of memory when there are really between 1-3 rectangles to conceal depending on the start and last MB to conceal.

Emmanuel

[Chao_Y_Intel · Accepted solution]

Hi Emmanuel,

For case of loss in an IDR, Our engineer team fixed RestoreErrorRect function and make possible to using previous frame This change will be available in the following release.

And also about you mentioned:
(sd->m_pBitStream->BytesLeft()<=3)

3 bytes is too much for checking end of slice. Our engineer think that comparing with zero may be enough. They will check it and change it in future release.


Thanks,
Chao

[Emmanuel_W_]

Quoting - Chao Y (Intel)

Hi Emmanuel,

For case of loss in an IDR, Our engineer team fixed RestoreErrorRect function and make possible to using previous frame This change will be available in the following release.

And also about you mentioned:
(sd->m_pBitStream->BytesLeft()<=3)

3 bytes is too much for checking end of slice. Our engineer think that comparing with zero may be enough. They will check it and change it in future release.


Thanks,
Chao

Hi Chao,

Thanks for the update.
The reason for the 3 bytes is that when the NAL is copied into the bitstream buffer, 3 bytes are added to the size of the bitstreambuffer. I believe this is so the buffer can be accessed aligned on 32 bits boundary.

Emmanuel

[Emmanuel_W_]

Quoting - Chao Y (Intel)

Hi Emmanuel,

For case of loss in an IDR, Our engineer team fixed RestoreErrorRect function and make possible to using previous frame This change will be available in the following release.

And also about you mentioned:
(sd->m_pBitStream->BytesLeft()<=3)

3 bytes is too much for checking end of slice. Our engineer think that comparing with zero may be enough. They will check it and change it in future release.


Thanks,
Chao

Glad to see that it is fixed in the latest update.

Thanks,

Emmanuel