These flash files:
Exhibit stagnate timestamp behavior when calling DecodeFrameAsync(). I am using 22.214.171.12471 (2.0 gold)
The test case looks like this:
sts = m_pmfxDEC->DecodeFrameAsync(&m_mfxBS, &(m_pmfxSurfaces[nIndex]), &pmfxOutSurface, &syncp);
. (same is in pipeline sample code)
if (MFX_ERR_NONE == sts)
What happens is thatprior toa *MFX_WRN_DEVICE_BUSY interval in the m_mfxBS structure the DataLength queues up about 4 frames, when this happensand as the loop starts to process these... the pmfxOutSurface->Data.TimeStamp will have the last pts for all 4 frames... so for example:
if the m_pmfxSurfaces should have had
 = 936
 = 969
 = 1003
 = 1036
 = 1070
 = 1103
This is what I get
 = 936 (good)
 = 1070 (bad)
 = 1070
 = 1070
 = 1070
 = 1103
Based from this forum entry
I'd expect mpfxOutSurface->Data.TimeStamp to work properly, and the code in here is exact to the example. Is there something that I am missing? Are these files correct? They work with all other decoders including ffmpeg. Is this PTS duplication expected behavior? Is there a way to make this work? I sure would hate to resolve to hacking a solution! I hope to hear a reply. I hope Nina gets this message.
*MFX_WRN_DEVICE_BUSY- The way I determined this to be true is that I can manipulate the time by injecting a sleep... the actual queuing of the frames may take place in a different frame range interval, and the same set of frames that I test (over and over again)succeed just fine.
I've attached a code snip of the current function as it is virtually identical to the pipeline sample code as it (to the best of my knowledge) upon receiving MFX_WRN_DEVICE_BUSY will wait and repeat the same call of DecodeFrameAsync without making changes to the input bitstream. It should be noted that the accumulation within the bitstream happens BEFORE I saw this message, and just right before (as this was consistent).
I never store more than 1 frame into the input bitstream, but rather this happens implicitly because the frame will not process until it gets the 3 or 4 future frames accumulated into the bitstream. This "implicit" functionality is identical to the CSmplBitstreamReader::ReadNextFrame(), where the DecodeFrameAsync() manages the DataLength, and DataOffset accordingly.
The key here is why does it not always transfer the input bitstream packet to DataOffset consistently when I call DecodeFrameAsync, is this expected behavior?It willskip thistransfer and instead add toDataLengthabout every 10 frames or so and not necessarily in the same group of frames. The rest of what you said in regards to one frame / timestampin bitstream per successful DecodeFrameAsync()frame creationmakes sense as I now can see how to safely create a work-around.
For now I propose to make a work-around by assigning the *correct* time-stampprior to the DecodeFrameAsync call. I can determine this by keeping a count of how many frames have accumulated in the bitstream per frame session. This probably is not the cleanest solution but should be effective.
Thanks for telling me when the timestamp gets assigned to the mpfxOutSurface->Data. I'll post back if this works. Let me know if you have any insight on why theinput bitstreamframes accumulate like this.
Here is a solution that is working against all my test media files. This should illustrate the problem, but also it would be good to review as I am making some assumptions on the behavior of the inputbitstream. The idea here is when the input bitstream queue's up, so does the timestamps. Ideally, it would be great if the SDK could manage this queue for me... assuming this can be expected behavior.
Does the AVFrame *picture constain whole frame each time?
Yes, in fact one of the tests that I have done is physically copy the bitstreammemory of a time it *fails against the time it succeeded to verify they were identical to exonerate the FLV muxer.
*fails - meaning it accumulated when it should have consumed it.
Could you debug to see which status is returned from DecodeFrameAsync when bitstream accumulation happens?
It always returns MFX_ERR_NONE for all 4 frames(just verified as I type this)
And in any case the data shouldn't accumulate in the bitstream, decoder can leave only a few irrelevant bytes
For now of all the files I have tested (e.g. other flash files, mp4, m2t) I have not seen this problem there is something about these group of files (see initial post with link)
It would be great if you could model your application behaviour with sample_decode or maybe DirectShow filters. I need a reproducer to understand what exactly is happenning. Could you please try?
I have modeled the application as close to the sample_decode as much aspossible. If you need help in getting together some code to reproduce let me know. I am not using any direct-show code.
Btw, you shouldn't alter working surface timestamps if the surface is locked - and likely it would be locked after DecodeFrameAsync code. But this might be not relevant to the problem - just for your information.
I presume you saw this comment:
//This serves no purpose but is very useful for debugging
I have commented this out by default inmy currentrunning build... I just used it temporarily to help read what was going on during this issue.
I got to thinking that there may be some other variables involved which may make it harder to reproduce on your end (e.g. subtle difference in avcc->annex-b conversion). As a fall-back I could submit a raw element dump in annex-b form which should be reproduceable with the sample_decode executablewith the exception that the read next frame may not advance in frame packets (this may be necessary). Let me know if we need to go that route. basically for me the ffmpeg flv demux will present packets (i.e. the entire frame) the audio is pruned out to ACC codec, and the video first gets converted to annex-b and then submitted to the DecodeFrameAsync(). The conversion to annex-b is straight forward as it simply converts the entire frame. The logic for appending the frames is identical to the sample_decode in regards to interpreting the DataLength and DataOffset.
Yes this explains the issue and I have "partially" confirmed that it works. I say partially as my workflow iscurrentlyincompatibleto this suggested change given itsdemux environment. My actual solution would involve either queueing timestamps (as I currently do)or queueing extended locked surfaces. I agree with the suggested action item listed above. Thanks for your help on this matter.
Also I want to present a birds eye view of my work flow here in hopes that it gives perspective on how I ran into this issue.
The current workflow in a simple model is the case where I simply wish to obtain the next frame sequentially as if I were to play the video:
Inside this function it looks something like this:
UncompressedFrame=decode (next compressed frame);
This is a real over-simplified model that is only somewhat accurate, where it always gives a compressed frame to get an uncompressed frame. With some mpeg2 type codecs this model could work around p and b frames by keeping a lean queue internally.
As we see with this model using the intel codec there are some times when we need some form ofinput queue controluntil the codec says it is ready for more compressed frames.
I propose the solution be a "smart" input bit stream that can manage this for me. I'dalso attempt toencapsulate the data length and data offset, wherethe client codewouldn't need to manage them.The interfacewould be as simple as adding "packets" into it, and it work with when to submit to the current working bitstream as well as the time stamps that they corresponded to.
If this solution seems to be in the right direction, and if others could benefit from this let me know, and I'd be happy to submit it.
Ok will do, but at the moment I am going to have to put this on the back burner as some new h264stress media has come to me which challenges this codec. I'm not going to go into details about it here except to say keep an eye out for me during this week (I'll want to confirm exactly where the problem is before I post). Once I get these resolved (hopefully) I can finish this.
Let's talk a minute about the case where a frame gets consumed and leaves several bytes that look like a part of a start code. If I loop again andcall DecodeFrameAsync with just this... is it safe to say that it should have yielded a return MFX_ERR_MORE_DATA? And would it still keep this memory in the bitsteam? The follow up to this would be what would be the consequence ofsubmitting these bytes, with the next frame appendedand calling DecodeFrameAsync()? I'd like to find such a file and test this. The reason why I have that logic in there is that every file I have (except for the flash files in this case)... consume on the first call to DecodeFrameAsync()... this means I save an extra memcopy of the input stream for most of the clips I have tested. Yes, now days it's probably not a significant performance gain, but its the idea of saving extra work that has me fighting to keep it in our code. ;)
I agree with your standing on sticking with the spec, as I'd probably do the same if I were you, and therefore agree this aspect of the case is closed. The questions you have answered for me have been of great value to me and our company, and I'll take responsibility for any corner cases that may crop up.
I am concerned about the Cannon Vixia, as I have spent the past hour verifying exactly what I have submitted to DecodeFrameAsync (i.e. full complete frames). Let me know if we should open a separate case for this.