Problem with H.264 encoder when using SLICE_CHECK_LIMIT

[Emmanuel_W_]

Hi,

Sorry I am reposting my finding because somehow I manage to delete my thread.

I have found I believe two issues with the latest H.264 encoder (6.0) and specifying a maximum slice size (and using the preprocessor SLICE_CHECK_LIMIT).

The first one is when rolling back the last macro block when the slice exceed the maximum allowed size. In that scenario two variables m_iLastXmittedQP and m_uSkipRun don't seem to be reset correctly.

A proposed fix is as follow in umc_h_core_enc_templ.cpp.h line 900

[cpp]#ifdef SLICE_CHECK_LIMIT
        Ipp32s size = (H264BsBase_GetBsOffset(&pBitstream->m_base) >> 3) + 3/*Start code*/ + 5;
        if (core_enc->m_MaxSliceSize && core_enc->m_MaxSliceSize < (Ipp32u)size){
            //Increase slice_id
            for (Ipp32u mb = uMB; mb < uFirstMB + uNumMBs; mb++) core_enc->m_pCurrentFrame->m_mbinfo.mbs[mb].slice_id++;
            curr_slice->m_MB_Counter--;
			curr_slice->m_iLastXmittedQP = iLastQP;		// Restore the last xmitted QP
            curr_slice->m_uSkipRun = uSaved_Skip_Run;	// Restore the last skipRun

            //Restore bitstream
            if (core_enc->m_PicParamSet.entropy_coding_mode){
                H264ENC_MAKE_NAME(H264BsReal_RestoreCABACState)(pBitstream);
                H264ENC_MAKE_NAME(H264BsReal_EncodeFinalSingleBin_CABAC)(pBitstream, 1);
            }else{
                H264BsBase_SetState(&pBitstream->m_base, pStartBits, uStartBitOffset);
                //Reset unused bits
                *pStartBits = *pStartBits & (0xff<<(8- uStartBitOffset));
            }
        }
#endif[/cpp]

The second issue leads to random crashes and a poorer encode quality.
Even though when specifying a maximum size for a slice the number of slice is set to 1, the slice_id parameter is incremented for the MBs as slices are beeing generated.

Line 904 of umc_h264_gen_enc_tmpl.cpp.h

[cpp]for (Ipp32u mb = uMB; mb < uFirstMB + uNumMBs; mb++) core_enc->m_pCurrentFrame->m_mbinfo.mbs[mb].slice_id++;[/cpp]

Unfortunaltely, at least 2 spots in the code uses the slice_id to index into tables that have been allocated with num_slices elements.

Line 654 of umc_h264_deblocking_tmpl.cpp.h

[cpp]H264SliceType *curr_slice = &core_enc->m_Slices[core_enc->m_pCurrentFrame->m_mbinfo.mbs[MBAddr].slice_id];[/cpp]

Line 451 of umc_h264_enc_cpb_tmpl.h (called from many many places with slice_id as the first parameter)

[cpp]inline
EncoderRefPicListStructType* H264ENC_MAKE_NAME(H264EncoderFrame_GetRefPicList)(
    H264EncoderFrameType* state,
    Ipp32s SliceNum,
    Ipp32s List)
{
    EncoderRefPicListStructType *pList;

    if (List == LIST_0)
        pList = &state->m_pRefPicList[SliceNum].m_RefPicListL0;
    else
        pList = &state->m_pRefPicList[SliceNum].m_RefPicListL1;

    return pList;

}[/cpp]

If I set the slice_id or SliceNum in those two spots to be 0 when in that particular mode, the encoding quality seems to be fine and I do not crash anymore but I am not sure this is the right way to fix the issue.

Emmanuel

[Emmanuel_W_]

There is also a problem when using CABAC and SLICE_CHECK_LIMIT at the same time, the state of the CABAC stream doesn't seem to be setup properly after generating a new slice causing a corrupted stream.

[Emmanuel_W_]

I guess I solved my own problems ;)

There is indeed a bug in the state of CABAC in that mode.

When hitting the condition where the last macro block need to be rolled back, the Cabac stream need to be restored to where it was last save but the end_of_slice_flag should not be set right away(it will be set a little later shortly before exiting the compress_slice function).

Moreover, the state of the CABAC stream should not be saved again (we are going to terminate the stream right away).

Emmanuel