Simultaneously Decoding Video and Audio

[franknatoli]

When reading an MPEG-4 or MPEG-2 stream, with a video track and an audio track, can a single FIOReader and Splitter and MediaData (input) be used? That is, can Splitter::GetNextData be called first for the video track, and then for the audio track, on the same MediaData (input) object? So far my observation is "no", the Splitter::GetFrame for the video track succeeds but the Splitter::GetFrame for the audio track fails with -999 VM_OPERATION_FAILED.

If not, what needs to be duplicated? Can I work with one FIOReader and one Splitter but two MediaData objects, one for video and one for audio?

Thanks for your time.

[franknatoli]

Suffice to say, implementing an audio decode and render thread is actually more complicated than video decode and render. Single stepped through simple_player.cpp and now understand the data handling.

Question regarding synchronization of video and audio using a single splitter, which is what simple_player.cpp does: does the splitter process the input video+audio file sequentially, hence the audio thread is starved for data until audio data is serially encountered by the single threaded splitter, and conversely the video thread is starved for data until video data is serially encountered by the single threaded splitter?

Asked another way, assume the video thread has decoded some frames, and therefore appears to be running ahead of the audio thread. If the video thread is paused by a synchronization thread, but the video thread has not hit a UMC_ERR_NOT_ENOUGH_DATA return from the splitter, will the audio thread (now running with the video thread paused) be also unable to get any data from the splitter?

If I'm right, and the splitter effectively controls the synchronization by doling out data to the video thread or audio thread as the data is sequentially encountered in the file, then why is there any need for a separate synchronization thread?