SEI are out of order

[Rosen__Daniel]

I'm adding user_data_registered_itu_t_t35 SEI messages during an H.264 encoding, and the SEI messages are being stored in the bitstream in the display order instead of encoding order.  The encoder is set to accept frames in display order, and the encoded frames are being encoded correctly.  

Shouldn't the SEI messages associated with each frame also be in display order?    Tracked the problem by adding a sequence byte to each SEI buffer, and the sequence increases for each SEI NAL.

Any idea on what I might be doing wrong?

[Harshdeep_B_Intel]

Hi,

Its great, that you have traced the problem, but we need more information to look further into the issue. If you can please follow the format  which will provide us more useful information about your environment set up. Also it will be helpful if you can send in the logs using these tools and information, if you are using a sample or tutorial to develop your application.

Thanks,

[Rosen__Daniel]

Here's the information you requested:

The following versions of Media SDK API are supported by platform/driver:

        Version Target  Supported       Dec     Enc
        1.0     HW      Yes             X       X
        1.0     SW      Yes             X       X
        1.1     HW      Yes             X       X
        1.1     SW      Yes             X       X
        1.3     HW      Yes             X       X
        1.3     SW      Yes             X       X
        1.4     HW      Yes             X       X
        1.4     SW      Yes             X       X
        1.5     HW      Yes             X       X
        1.5     SW      Yes             X       X
        1.6     HW      Yes             X       X
        1.6     SW      Yes             X       X
        1.7     HW      Yes             X       X
        1.7     SW      Yes             X       X
        1.8     HW      Yes             X       X
        1.8     SW      Yes             X       X

Graphics Devices:
        Name                                         Version             State
        NVIDIA GeForce GTX 650                       9.18.13.3221        Active
        Intel(R) HD Graphics 4000                    10.18.10.3621       Active

System info:
        CPU:    Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
        OS:     Microsoft Windows 8.1 Pro
        Arch:   64-bit

Installed Media SDK packages (be patient...processing takes some time):
        Intel« Media SDK Video Encoding Sample 5.0.461.91752

Our code that is creating the SEI.  I've narrowed it down to illustrate the problem.  I believe you could use it, almost as is, by adding it to the simple encoding tutorial:

		mfxStatus status;
		{
		mfxEncodeCtrl		mfxEC;
		memset(&mfxEC, 0, sizeof(mfxEC));
		mfxU8 sei_data[4] = { 0x04, 0x02, 0x00, 0xff };
		sei_data[2] = m_userSEIDataNum++ % 0xff;	// Store increasing value

		mfxPayload seiPayload;
		memset( &seiPayload, 0, sizeof(seiPayload) );
		seiPayload.Type = 4;
		seiPayload.BufSize = sizeof(sei_data);
		seiPayload.NumBit = seiPayload.BufSize * 8;
		seiPayload.Data = sei_data;

		mfxPayload *seiPayloads[1];
		seiPayloads[0] = &seiPayload;
		mfxEC.Payload = seiPayloads;
		mfxEC.NumPayload = 1;
		status = m_pmfxEncoder->EncodeFrameAsync(&mfxEC, pSurface, &mfxBS, &mfxSP);
		}

Below is a dump of the NALs from the encoded output file only showing the SEI data and a portion of the slice headers.  Notice how the first byte of the SEI data increments by one for each frame, but the pic_order_cnt_lsb jumps around reflecting the standard temporal reordering.  The order of the SEI data should match the same temporal encoding order of the frames.

***  payload_type: 4 user_data_registered_itu_t_t35
   payload_size: 2
      0:  x02 xff  <<<< 2 is OK.
    string is ".."
 NAL type: 5 - Coded slice of IDR picture, ref: 1, length: 4590, start bytes: 3
   first_mb_in_slice: 0
   slice_type: 7 (I)
   pic_parameter_set_id: 0
   frame_num: 0 (8 bits)
   idr_pic_id: 0
   pic_order_cnt_lsb: 0

*** payload_type: 4 user_data_registered_itu_t_t35
   payload_size: 2
      0:  x03 xff  <<<< Should be 5 not 3.
    string is ".."
 NAL type: 1 - Coded slice of a picture, ref: 1, length: 1193, start bytes: 3
   first_mb_in_slice: 0
   slice_type: 5 (P)
   pic_parameter_set_id: 0
   frame_num: 1 (8 bits)
   pic_order_cnt_lsb: 6

*** payload_type: 4 user_data_registered_itu_t_t35
   payload_size: 2
      0:  x04 xff  <<< Should be 3 not 4.
    string is ".."
 NAL type: 1 - Coded slice of a picture, ref: 0, length: 256, start bytes: 3
   first_mb_in_slice: 0
   slice_type: 6 (B)
   pic_parameter_set_id: 0
   frame_num: 2 (8 bits)
   pic_order_cnt_lsb: 2

*** payload_type: 4 user_data_registered_itu_t_t35
   payload_size: 2
      0:  x05 xff   << Should be 4 not 5
    string is ".."
 NAL type: 1 - Coded slice of a picture, ref: 0, length: 543, start bytes: 3
   first_mb_in_slice: 0
   slice_type: 6 (B)
   pic_parameter_set_id: 0
   frame_num: 2 (8 bits)
   pic_order_cnt_lsb: 4

*** payload_type: 4 user_data_registered_itu_t_t35
   payload_size: 2
      0:  x06 xff
    string is ".."
 NAL type: 1 - Coded slice of a picture, ref: 1, length: 2006, start bytes: 3
   first_mb_in_slice: 0
   slice_type: 5 (P)
   pic_parameter_set_id: 0
   frame_num: 2 (8 bits)
   pic_order_cnt_lsb: 12

[Sravanthi_K_Intel]

Hello there, Thanks for your question on SEI. Have you had a chance to look at this article on how to insert and retrieve closed-caption data from encoded streams - https://software.intel.com/en-us/blogs/2014/08/18/how-to-add-closed-caption-messages-in-avc-and-mpeg2-streams. ;This article will give you a flavor for how to retrieve and view SEI/CC messages. Regarding, display/encoded order - let me get back to you with more details if the above article is not clear.

[Rosen__Daniel]

Thanks for getting back.  Yes, we used your article as a prototype for our code.  My sample above is a stripped down version of your code simply to illustrate the display / encode order issue.  BTW, I commented on your article as there are some errors in it that you might want to correct.

[Sravanthi_K_Intel]

One more thing - If you do not "explicitly" specify encoding in encode-order, display order encoding is performed. To enforce encode order, you need to specify the frame type in the mfxEncodeCtrl structure (first parameter to the EncodeAsync function). Before that, the mfxInfoMFX structure should also specify the EncodeOrder field. You will find much of this information on Display Order/ Encode Order in the mediasdk-man.pdf (p60,p61, found in INSTALL_PATH/doc/ folder). Let me know if you have more questions.

[Sravanthi_K_Intel]

Thanks for your comments on the article, and greatly appreciate you reporting the issue on the article. I will surely take a look into it! Thanks again.

[Rosen__Daniel]

We are doing display order encoding as specified in our initialization routines: 

	m_mfxEncParams.mfx.EncodedOrder				= 0; // binary flag, 0 signals encoder to take frames in display order

That is working fine as we've verified it by burning in a frame number into the source video before encoding.  Is there another setting that sets the order for the SEI?

[Sravanthi_K_Intel]

Hello there - The SEI message insertion follows the order of encoding. So, if you specify display order and call EncodeFrameAsync, then SEI messages will appear in that order. Basically, you cannot encode in display order and have SEI messages inserted in another order.

I went back and re-read your first message and here:

"I'm adding user_data_registered_itu_t_t35 SEI messages during an H.264 encoding, and the SEI messages are being stored in the bitstream in the display order instead of encoding order.  The encoder is set to accept frames in display order, and the encoded frames are being encoded correctly."  --> YES, since you are encoding in display order, SEI message insertion will also follow that order.

"Shouldn't the SEI messages associated with each frame also be in display order?"  --> Yes, your first statement says you are already seeing SEI messages "in display order". This is the expected behavior.

If I am misunderstanding the question, please rephrase it. Your first message in this thread (as pointed above), seems to have contradicting statements (what you are seeing and what you want seem to be the same). Thanks.

[Rosen__Daniel]

Perhaps I wasn't clear above.  We are feeding the H264 encoder in display order like this:

Uncompressed frame 1, SEI data 1
Uncompressed frame 2, SEI data 2
Uncompressed frame 3, SEI data 3
Uncompressed frame 4, SEI data 4

Encoder is returning the video correctly, but not the SEI.  
Compressed frame 4, SEI data 1
Compressed frame 1, SEI data 2
Compressed frame 2, SEI data 3
Compressed frame 3, SEI data 4

The above is an example, but what is happening is that the encoder is returning video frames in encoded order, which is correct, but SEI data in display order, which isn't correct.  The SEI data in the encoded frame should be the one associated with the original source frame.

Thanks.

[Sravanthi_K_Intel]

Hello there - I tried to reproduce the issue you mention above but was not able to. I am able to view the SEI messages are associated with the correct encoded frame. Some suggestions:

- Can you use GetPayload() function as mentioned in the SEI article, and check if correct messages are being displayed for the frames?

- Can you send us your application code to reproduce the issue?

Thanks.

[Rosen__Daniel]

Hello there - I tried to reproduce the issue you mention above but was not able to. I am able to view the SEI messages are associated with the correct encoded frame

How did you reproduce and with which version of the driver?  Will attempt to match here.  Can you send me your application to try out?

Can you use GetPayload() function as mentioned in the SEI article, and check if correct messages are being displayed for the frames

Is GetPayload() a decode function?  We aren't using the QS decoder yet.  

Can you send us your application code to reproduce the issue?

Unable to send our entire application as it's very large, and uses a number licensed of 3rd party libraries.  I'll see if we can create a sample application for you.  Once we have the information above, will try to duplicate what you did.

[Rosen__Daniel]

Any updates on this? Can we get your test application to try here?

[Sravanthi_K_Intel]

Hello there - Yes, I will send my application in a PM today, and in parallel, check with some experts as well. It has been a supremely swamped week here. Thanks.

[Rosen__Daniel]

Thanks so very much.

[Rosen__Daniel]

Hi Sravanthi, Did you ever send me your test application?  We really want to either fix it on our end or have it escalated if it's actually a bug in the H.264 encoder.

[Sravanthi_K_Intel]

Just sent you a PM with the code. Note that the encoded SEI can be retrieved using  the GetPayload() function as explained in the article. My sincere apologies for the delay - until yesterday, we were all in meetings with limited access and time for this. Next time, we will communicate such events earlier.

[Rosen__Daniel]

Any updates on this?  Thanks.

[Sravanthi_K_Intel]

I responded via PMs on the latest on this. We are able to reproduce the behavior you are observing and are consulting the experts to find more. Will get back to you once we hear more. Meantime, can you please try using the GetPayload() function to retrieve the SEI messages?

[Rosen__Daniel]

Sorry, don't understand what the benefit of using GetPayload() would be.  The people use the files created by our software, VideoReDo, use all kinds of players.  We want to create industry compliant files.

[Sravanthi_K_Intel]

Okay, now I understand the use-case you are targeting here. And yes, for that use-case GetPayload() may not work so well. Thanks for your patience again, we are discussing this internally and we are getting close to finding out the answer here. Thanks for bringing this to our notice.