- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm adding user_data_registered_itu_t_t35 SEI messages during an H.264 encoding, and the SEI messages are being stored in the bitstream in the display order instead of encoding order. The encoder is set to accept frames in display order, and the encoded frames are being encoded correctly.
Shouldn't the SEI messages associated with each frame also be in display order? Tracked the problem by adding a sequence byte to each SEI buffer, and the sequence increases for each SEI NAL.
Any idea on what I might be doing wrong?
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Its great, that you have traced the problem, but we need more information to look further into the issue. If you can please follow the format which will provide us more useful information about your environment set up. Also it will be helpful if you can send in the logs using these tools and information, if you are using a sample or tutorial to develop your application.
Thanks,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here's the information you requested:
The following versions of Media SDK API are supported by platform/driver: Version Target Supported Dec Enc 1.0 HW Yes X X 1.0 SW Yes X X 1.1 HW Yes X X 1.1 SW Yes X X 1.3 HW Yes X X 1.3 SW Yes X X 1.4 HW Yes X X 1.4 SW Yes X X 1.5 HW Yes X X 1.5 SW Yes X X 1.6 HW Yes X X 1.6 SW Yes X X 1.7 HW Yes X X 1.7 SW Yes X X 1.8 HW Yes X X 1.8 SW Yes X X Graphics Devices: Name Version State NVIDIA GeForce GTX 650 9.18.13.3221 Active Intel(R) HD Graphics 4000 10.18.10.3621 Active System info: CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz OS: Microsoft Windows 8.1 Pro Arch: 64-bit Installed Media SDK packages (be patient...processing takes some time): Intel« Media SDK Video Encoding Sample 5.0.461.91752
Our code that is creating the SEI. I've narrowed it down to illustrate the problem. I believe you could use it, almost as is, by adding it to the simple encoding tutorial:
mfxStatus status; { mfxEncodeCtrl mfxEC; memset(&mfxEC, 0, sizeof(mfxEC)); mfxU8 sei_data[4] = { 0x04, 0x02, 0x00, 0xff }; sei_data[2] = m_userSEIDataNum++ % 0xff; // Store increasing value mfxPayload seiPayload; memset( &seiPayload, 0, sizeof(seiPayload) ); seiPayload.Type = 4; seiPayload.BufSize = sizeof(sei_data); seiPayload.NumBit = seiPayload.BufSize * 8; seiPayload.Data = sei_data; mfxPayload *seiPayloads[1]; seiPayloads[0] = &seiPayload; mfxEC.Payload = seiPayloads; mfxEC.NumPayload = 1; status = m_pmfxEncoder->EncodeFrameAsync(&mfxEC, pSurface, &mfxBS, &mfxSP); }
Below is a dump of the NALs from the encoded output file only showing the SEI data and a portion of the slice headers. Notice how the first byte of the SEI data increments by one for each frame, but the pic_order_cnt_lsb jumps around reflecting the standard temporal reordering. The order of the SEI data should match the same temporal encoding order of the frames.
*** payload_type: 4 user_data_registered_itu_t_t35 payload_size: 2 0: x02 xff <<<< 2 is OK. string is ".." NAL type: 5 - Coded slice of IDR picture, ref: 1, length: 4590, start bytes: 3 first_mb_in_slice: 0 slice_type: 7 (I) pic_parameter_set_id: 0 frame_num: 0 (8 bits) idr_pic_id: 0 pic_order_cnt_lsb: 0 *** payload_type: 4 user_data_registered_itu_t_t35 payload_size: 2 0: x03 xff <<<< Should be 5 not 3. string is ".." NAL type: 1 - Coded slice of a picture, ref: 1, length: 1193, start bytes: 3 first_mb_in_slice: 0 slice_type: 5 (P) pic_parameter_set_id: 0 frame_num: 1 (8 bits) pic_order_cnt_lsb: 6 *** payload_type: 4 user_data_registered_itu_t_t35 payload_size: 2 0: x04 xff <<< Should be 3 not 4. string is ".." NAL type: 1 - Coded slice of a picture, ref: 0, length: 256, start bytes: 3 first_mb_in_slice: 0 slice_type: 6 (B) pic_parameter_set_id: 0 frame_num: 2 (8 bits) pic_order_cnt_lsb: 2 *** payload_type: 4 user_data_registered_itu_t_t35 payload_size: 2 0: x05 xff << Should be 4 not 5 string is ".." NAL type: 1 - Coded slice of a picture, ref: 0, length: 543, start bytes: 3 first_mb_in_slice: 0 slice_type: 6 (B) pic_parameter_set_id: 0 frame_num: 2 (8 bits) pic_order_cnt_lsb: 4 *** payload_type: 4 user_data_registered_itu_t_t35 payload_size: 2 0: x06 xff string is ".." NAL type: 1 - Coded slice of a picture, ref: 1, length: 2006, start bytes: 3 first_mb_in_slice: 0 slice_type: 5 (P) pic_parameter_set_id: 0 frame_num: 2 (8 bits) pic_order_cnt_lsb: 12
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello there, Thanks for your question on SEI. Have you had a chance to look at this article on how to insert and retrieve closed-caption data from encoded streams - https://software.intel.com/en-us/blogs/2014/08/18/how-to-add-closed-caption-messages-in-avc-and-mpeg2-streams. ;This article will give you a flavor for how to retrieve and view SEI/CC messages. Regarding, display/encoded order - let me get back to you with more details if the above article is not clear.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for getting back. Yes, we used your article as a prototype for our code. My sample above is a stripped down version of your code simply to illustrate the display / encode order issue. BTW, I commented on your article as there are some errors in it that you might want to correct.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
One more thing - If you do not "explicitly" specify encoding in encode-order, display order encoding is performed. To enforce encode order, you need to specify the frame type in the mfxEncodeCtrl structure (first parameter to the EncodeAsync function). Before that, the mfxInfoMFX structure should also specify the EncodeOrder field. You will find much of this information on Display Order/ Encode Order in the mediasdk-man.pdf (p60,p61, found in INSTALL_PATH/doc/ folder). Let me know if you have more questions.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for your comments on the article, and greatly appreciate you reporting the issue on the article. I will surely take a look into it! Thanks again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are doing display order encoding as specified in our initialization routines:
m_mfxEncParams.mfx.EncodedOrder = 0; // binary flag, 0 signals encoder to take frames in display order
That is working fine as we've verified it by burning in a frame number into the source video before encoding. Is there another setting that sets the order for the SEI?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello there - The SEI message insertion follows the order of encoding. So, if you specify display order and call EncodeFrameAsync, then SEI messages will appear in that order. Basically, you cannot encode in display order and have SEI messages inserted in another order.
I went back and re-read your first message and here:
"I'm adding user_data_registered_itu_t_t35 SEI messages during an H.264 encoding, and the SEI messages are being stored in the bitstream in the display order instead of encoding order. The encoder is set to accept frames in display order, and the encoded frames are being encoded correctly." --> YES, since you are encoding in display order, SEI message insertion will also follow that order.
"Shouldn't the SEI messages associated with each frame also be in display order?" --> Yes, your first statement says you are already seeing SEI messages "in display order". This is the expected behavior.
If I am misunderstanding the question, please rephrase it. Your first message in this thread (as pointed above), seems to have contradicting statements (what you are seeing and what you want seem to be the same). Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Perhaps I wasn't clear above. We are feeding the H264 encoder in display order like this:
Uncompressed frame 1, SEI data 1
Uncompressed frame 2, SEI data 2
Uncompressed frame 3, SEI data 3
Uncompressed frame 4, SEI data 4
Encoder is returning the video correctly, but not the SEI.
Compressed frame 4, SEI data 1
Compressed frame 1, SEI data 2
Compressed frame 2, SEI data 3
Compressed frame 3, SEI data 4
The above is an example, but what is happening is that the encoder is returning video frames in encoded order, which is correct, but SEI data in display order, which isn't correct. The SEI data in the encoded frame should be the one associated with the original source frame.
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello there - I tried to reproduce the issue you mention above but was not able to. I am able to view the SEI messages are associated with the correct encoded frame. Some suggestions:
- Can you use GetPayload() function as mentioned in the SEI article, and check if correct messages are being displayed for the frames?
- Can you send us your application code to reproduce the issue?
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello there - I tried to reproduce the issue you mention above but was not able to. I am able to view the SEI messages are associated with the correct encoded frame
How did you reproduce and with which version of the driver? Will attempt to match here. Can you send me your application to try out?
Can you use GetPayload() function as mentioned in the SEI article, and check if correct messages are being displayed for the frames
Is GetPayload() a decode function? We aren't using the QS decoder yet.
Can you send us your application code to reproduce the issue?
Unable to send our entire application as it's very large, and uses a number licensed of 3rd party libraries. I'll see if we can create a sample application for you. Once we have the information above, will try to duplicate what you did.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any updates on this? Can we get your test application to try here?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello there - Yes, I will send my application in a PM today, and in parallel, check with some experts as well. It has been a supremely swamped week here. Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks so very much.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sravanthi, Did you ever send me your test application? We really want to either fix it on our end or have it escalated if it's actually a bug in the H.264 encoder.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just sent you a PM with the code. Note that the encoded SEI can be retrieved using the GetPayload() function as explained in the article. My sincere apologies for the delay - until yesterday, we were all in meetings with limited access and time for this. Next time, we will communicate such events earlier.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any updates on this? Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I responded via PMs on the latest on this. We are able to reproduce the behavior you are observing and are consulting the experts to find more. Will get back to you once we hear more. Meantime, can you please try using the GetPayload() function to retrieve the SEI messages?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry, don't understand what the benefit of using GetPayload() would be. The people use the files created by our software, VideoReDo, use all kinds of players. We want to create industry compliant files.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Okay, now I understand the use-case you are targeting here. And yes, for that use-case GetPayload() may not work so well. Thanks for your patience again, we are discussing this internally and we are getting close to finding out the answer here. Thanks for bringing this to our notice.
![](/skins/images/45C6C2D737ED71F4C51F0145C8CB1E9C/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page