H.264 error when decoding

[JonasJensen]

Intel OneVPL version: 2022.0.0

Hi, I am trying to write a program that needs to be able to decode H.264. I can decode most sources (network, storage etc) but have trouble decoding H.264 from a specific source. 
To troubleshoot I have tried to decode it through the sample projects and this also fails on this particular file.  FFPlay can play it and so can other decoders I have tried (E.g. Streamcoders). 
I have tried to analyze it with SpecialVH264, but couldn't see anything obvious that was missing or wrong. 

I have attached a video file in the H.264 format that I cannot decode. 

Is it possible to get OneVPL to decode this file and if so, what should be configured to get it working? 

Thanks,
Jonas

[JananiC_Intel]

Hi,

Thanks for posting in Intel forums.

In oneVPL decode sample , only h265 file is supported. In that case, we suggest you to convert h264 file to h265 file with either ffmpeg or media sdk multi transcode and then decode using oneVPL .For ffmpeg use the below command.

ffmpeg -i user.h264 -c:v libx265 -vtag hvc1 -c:a copy output.mp4 

To convert h264 file to h265 with Media SDK transcode use the below command.

./sample_multi_transcode.exe -i::h264 user.h264 -o::h265 out1.h265

Please find the below links for reference.

https://github.com/oneapi-src/oneAPI-samples/tree/master/Libraries/oneVPL/hello-decode

https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/samples/readme-multi-transcode_linux.md

Try this and let us know the updates.

Regards,

Janani Chandran

[JonasJensen]

Hi Janani, 

You are correct that the "hello-decode.exe" example only supports H265. The sample I used was decvpp_tool.exe which also works with H264. 

I have made a C++ library that can decode most H264 sources, except for the one I have supplied as an attached file. 
Most decoders can decode the file fine, but not Intel OneVPL and I am searching for help figuring out why that is. 

Thank you,
Jonas

[JananiC_Intel]

Hi,

Thanks for the update.

As per the onevpl github page , this decvpp_tool sample  does not support H.264 file. Refer the below link to know the details.

https://github.com/oneapi-src/oneVPL/tree/master/examples/hello/hello-decvpp

Regards,

Janani Chandran

[JonasJensen]

Hi again Janani,

I do not use hello-decvpp, I use decvpp_tool.exe. It does work fine with H264. I have attached a screenshot of a decode/vpp of Big Buck Bunny as a raw h264 file, where you can see it play in FFPlay afterwards. 
I have also added another image showing the input parameters available when using the tool.

I still cannot figure out why the first file I sent does not work in any of Intel OneVPLs samples or my own custom C++ library. 

Best regards and thank you,
Jonas 

[JananiC_Intel]

Hi,

Sorry for the confusion. We will update you regarding this. Meanwhile could you share your OS details and command used to run decvpp_tool.exe ?

Regards,

Janani Chandran

[JonasJensen]

Hi, 

No problem. Thanks. 
Yes of course, here are my details:

OS: 
Windows 10 Pro | Version 21H1 | OS build 19043.1586

Command:
decvpp_tool h264 -sw -i out.h264 -o dec.raw -vpp_num 2 -vpp_params 320x240_i420,640x480_bgra -vpp_out o1.raw,o2.raw

Best regards,
Jonas 

[Ying_Guo_VPL]

Hi Jonas,

Thank you bringing this issue to our attention and provide the content file and other information. We found that the out.h264 file you provided is corrupted. This make it difficult to decode. Admittedly our cpu implementations also require some improvement so that it will be more forgiving to corrupted files.

 We acknowledge this limitation in our release note, 

• Decode input bitstream buffer size must be large enough to hold several frames. Buffer sizes that are too small may cause issues. The necessary minimum size is stream dependent but enough for 10 frames is a conservative estimate.

If you like, you may provide information on how that content file was generated, for example, giving a command line. Please do so by continue replying in this thread. We may inform you when an improvement is made.

[JonasJensen]

Hi Ying Guo,

Thank you for the feedback. 

Is there a workaround for this issue, or can you provide more information about which part of the file is corrupted?

My C++ code is based on the provided examples and I have defined the buffersize like this:
"#define BITSTREAM_BUFFER_SIZE 2000000"

Best regards,
Jonas 

[Ying_Guo_VPL]

Hi Jonas,

When playback your video, ffplay emits some repeated errors like this:

libpostproc 55. 9.100 / 55. 9.100
[h264 @ 000001dfe13e1700] non-existing PPS 0 referenced 0B f=0/0
Last message repeated 4 times
[h264 @ 000001dfe13e1700] decode_slice_header error
[h264 @ 000001dfe13e1700] no frame!
[h264 @ 000001dfe13e1700] non-existing PPS 0 referenced
[h264 @ 000001dfe13e1700] decode_slice_header error
[h264 @ 000001dfe13e1700] no frame!

I cannot be sure exactly where but possibly around 90th frame (see screenshot) when a remote control in transparent state was waving in the foreground  while a door is seen at background. Were you trying to create a special effect?

The problem we see is that decode may fail with the corrupted input bitstream. There’re some limitations to cover various types of corrupted input stream. It would help if you could explain how you encoded the video, for example, if you encoded it using our example, or which tool and the command line used. Having a variety of similar instances of corruption or knowing the way to create them would be helpful to understand a pattern.

Thanks

ying

[JonasJensen]

Hi Ying Guo,

Interesting. I did not try to create a special effect. The remote control was waved in front of the camera to try and make a steady motion in front of the camera. I recorded that video for sharing here.
The recording is made through a Windows Program that uses Streamcoders internally. 

I can try and figure out what version of Streamcoders that is being used by the program Monday. 
Monday I can also create more videos that I can share. Is there any particular type of video that would be more helpful? Motion? video without movement? or a particular length?

Thanks,
Jonas

[Ying_Guo_VPL]

Variety of video encoded in different ways would be good. Contents might have not made the difference. But try encode the same shot again using the exact same method how you created the last sample. When you encode other videos, please include a couple that use our encoders, e.g. hello-encode and sample-encode samples. When you build our repositories, their executables will be in the %VPL_INSTALL_DIR%\bin directory. Let me know if you have problem building our oneVPL and oneVPL-cpu. Thanks 

[JonasJensen]

Hi again,

I have attached a zip file with two files inside. 
One is the RAW bgr24 video that I can play in FFPlay by using this command: ffplay -f rawvideo -pixel_format bgr24 -video_size 320x240 -framerate 20 RAW.raw

The other is the StreamcodersEncoded that I can play in FFPlay by using this command: ffplay -f h264 StreamcodersEncoded.h264

I do not know how I can encode the RAW bgr24 video with the examples, as the video format is BGR24. As far as I can see the encode examples only works with yuv or rgb3. 
What example should I use and can you help me with the correct command? 

sample_encode.exe.
My command: sample_encode.exe h264 -i RAW.raw -o sample_encode_h264.h264 -f 20 -w 320 -h 240 -sw

Error: 

sample_vpp.exe.
My command: sample_vpp.exe -lib sw -sw 320 -sh 240 -sf 20 -scc rgb3 -dcc i420 -i RAW.raw -o test.hyuv

Thanks

[Ying_Guo_VPL]

Thanks. I will take a look and reply.

[Ying_Guo_VPL]

Hi Jonas,

First I want to give you a reply to the initial issue that prompted you started this thread of discussion that you could not decode one video clip in h264. Thanks for your patience. Understand that you are able to decode many other contents except for this content. I am attaching the decode result using an experimental method which shows where your input content is corrupted. We may work on a solution depending on our plans and priorities.

Meanwhile we will add the following text in our release note:

"CPU implementation decode error recovery is limited.  Decode of corrupted bitstreams may fail."

I will review the attachment you provided in your latest reply to see if they are impacted in the same way by the initial issue mentioned above.

Thanks

[JonasJensen]

Hi Ying Guo,

Yes, I can decode many other contents. 

Thank you for the attachment. Can you tell me how to play it correctly? 
I've tried with: "ffplay -f rawvideo -pixel_format bgr24 -video_size 320x240 -framerate 20 customer_content_hw_out.raw" but it shows the image as greyscale and with 6 instances (See attached). 

In our program we can detect when the content comes from this specific source. Would it be possible to correct the corruption in any way programmatically as a workaround for now? 

I appreciate the feedback and hope it is something that can be fixed. Before we can fully adopt Intel OneVPL into our program we need to be able to decode the corrupted bitstreams. 

Thanks,
Jonas

[Ying_Guo_VPL]

Hi Jonas,

Attached please see the second result video I decoded using my method. To playback the file I decoded, please use this command line:

ffplay -f rawvideo -video_size 320x240 -pixel_format yuv420p xxxx_out.raw -autoexit
where xxxx_out.raw should be replaced with the filenames.

If you playback the source videos you sent to us, using command line: 'ffplay -f h264 c:\content\StreamcodersEncoded.h264 -autoexit'

you would see the errors like the following:
[h264 @ 0000025a33e598c0] non-existing PPS 0 referenced 0B f=0/0
Last message repeated 3 times
[h264 @ 0000025a33e598c0] decode_slice_header error
[h264 @ 0000025a33e598c0] no frame!
[h264 @ 0000025a33e598c0] non-existing PPS 0 referenced
[h264 @ 0000025a33e598c0] decode_slice_header error
[h264 @ 0000025a33e598c0] no frame!
[h264 @ 0000025a33e598c0] non-existing PPS 0 referenced
[h264 @ 0000025a33e598c0] decode_slice_header error
[h264 @ 0000025a33e598c0] no frame!
[h264 @ 0000025a33e598c0] non-existing PPS 0 referenced
Last message repeated 2 times
[h264 @ 0000025a33e598c0] decode_slice_header error
[h264 @ 0000025a33e598c0] no frame!
[h264 @ 0000025a33e598c0] non-existing PPS 0 referenced
[h264 @ 0000025a33e598c0] decode_slice_header error
[h264 @ 0000025a33e598c0] no frame!
[h264 @ 0000025a33e598c0] non-existing PPS 0 referenced
These go on and on. 

Our decode catches these errors and stops decoding. That is the behavior you see of our decoder.  The error was seen by ffplay as well which indicates that there is something wrong with your source video. The visual artifacts in the previous source video may be or may not be related to the "decode_slice_header error". It is the decode_slice_header error makes our decoder stop decoding.

To make this clear, our decoder is not at fault about the error in source videos. We think the proper behavior is to stop when the source video emits errors. But we might be willing to make our decoder tolerant, like what you see with the two video I attached today and the before. To handle those error effectively we will need some careful review so that it doesn't break other users. Meanwhile you may want to address the problem with the tool you used to encode. If you encode the video using our encoder, we will be responsible to address any issue in our tools.

Thanks...ying

[JonasJensen]

Hi Ying, 

Thank you for the explanation. 

You are very clear and I do not blame Intel at all for our bad source. We would be really happy if the decoder could be tolerant for these kinds of errors. (Maybe by setting an optional flag when initializing the decoder?). 

We have addressed problem, but unfortunately we are in a situation where we have many terabytes of data that are created by the bad source. We didn't notice this since we have been using Streamcoders for both encoding and decoding until recently where we are trying to rely only on Intel OneVPL.

For now we have to use a hybrid of Streamcoders and Intel OneVPL which I think is unfortunate, so I really hope that the decoder can become tolerant so we can use only Intel OneVPL. 

As a side note. I have found the version of Streamcoders that is making the bad source. It is Streamcoders MediaSuite.NET version 1.5.983(p2). 
If it could be helpful for your team I can provide code snippets of how we instantiated the Streamcoders H264Encoder, but I would prefer if we could discuss that over a private channel instead of this open forum. 

Thanks,
Jonas.

[Ying_Guo_VPL]

After careful consideration, we determined we cannot publish a workaround to address the header corruption in your source video. As noted earlier in this thread, the following text will be added to our release notes:
"CPU implementation decode error recovery is limited. Decode of corrupted bitstreams may fail."

At this time the CPU implementation's decode error recovery is deliberately simplistic. The decode loop simply aborts if any problems are found. It is up to developers using this implementation to extend the source with decode error recovery suitable for their use case and content. We do not have plans to significantly improve the CPU implementation's decode error recovery robustness in the near future.

The problem in your input source video raises an AVERROR_INVALIDDATA error. It is detected in This line

To start debugging, you will need a sample app to call that code. Try modifying the hello-decode sample and allow h264 input video (currently only allows h265 video) or start with decvpp_tool you were using so that it does not stopped by above mentioned block of code.

To debug our solution, you would build our oneVPL repo in debug build and build our oneVPL-cpu repo in debug build following our Build/Install Instructions except:

1. Replace 'cmake --build . --config Release --target install' with 'cmake --build . --config Debug --target install'

2.Replace

script\bootstrap
script\build
script\install

with

script\bootstrap gpl
script\build debug gpl
script\install debug gpl

3. Run %VPL_INSTALL_DIR%\etc\vpl\vars.bat
4. Go to _build directory in the oneVPL, and run "vpl.sln"

---

[JonasJensen]

Hi Ying and Janani,

Thank you for the update. I am very sad to hear that it cannot be added to OneVPL. But thank you very much for the comprehensive guide in how to build the source code so I might be able to remedy it myself. 

Are you still interested in the code snippets? I have gotten a mail from Janani, but I haven't had time this week to send the snippet. If you are interested just let me know and I will send it through the private channel.

Best regards and many thanks,
Jonas

[Ying_Guo_VPL · Accepted solution]

Not for the moment. For now, I recommend the first step would be ramping up on our code by stepping through it. Please follow the build instructions to clone repos, build in debug config, open Visual Studio place a few breakpoints in the Hello-Decode project and initially using h265 source video. We have test content in the content directory for testing. Place a break point in this line. Then step into it. Sometime you need to step into a few times to reach the library code. Trying this out would ramp up very quickly. Have fun.