Problem with muxing MPEG2 video and audio

stsybikov — Fri, 04 Jan 2013 17:03:58 GMT

Hi Intel Gurus,

I have a trouble to mux YUV420 video with audio to MPEG2. It's no issue to mux just the video. Works like a charm. But when I try to use PutAudioData I receive Unhandled exception error: Access violation reading location 0x0000000.

This is extraction from my code:

////////////////////////////
// Video Info
UMC::VideoStreamInfo VideoInfo;
memset(&VideoInfo, 0, sizeof(VideoInfo));
VideoInfo.streamPID    = 0;
VideoInfo.stream_type   = UMC::MPEG2_VIDEO;
VideoInfo.color_format   = UMC::YUV420;
VideoInfo.interlace_type  = UMC::PROGRESSIVE;
VideoInfo.aspect_ratio_height = 9;
VideoInfo.aspect_ratio_width = 16;
VideoInfo.clip_info.height  = 1080;
VideoInfo.clip_info.width  = 1920;
VideoInfo.bitrate    = 10000000;
VideoInfo.framerate    = 24;
VideoInfo.disp_clip_info.height = 1080;
VideoInfo.disp_clip_info.width = 1920;

//////////////////////////////
// Audio Info
UMC::AudioStreamInfo AudioInfo;
memset(&AudioInfo, 0, sizeof(AudioInfo));
AudioInfo.streamPID   = 1;
AudioInfo.stream_type  = UMC::AAC_AUDIO;
AudioInfo.channels   = 2;
AudioInfo.bitPerSample  = 16;
AudioInfo.bitrate   = 24;
AudioInfo.sample_frequency = 44100;

UMC::MPEG2MuxerParams MuxerParams;
MuxerParams.m_lpDataWriter = &Writer;
MuxerParams.m_SystemType = UMC::MPEG2_PURE_VIDEO_STREAM;
MuxerParams.m_nNumberOfTracks = 2;

MuxerParams.pTrackParams   = new UMC::TrackParams[MuxerParams.m_nNumberOfTracks];
// Track 1
MuxerParams.pTrackParams[0].type  = UMC::VIDEO_TRACK;
MuxerParams.pTrackParams[0].info.video = &VideoInfo;
MuxerParams.pTrackParams[0].bufferParams.m_prefInputBufferSize = 2000000;
// Track 2
MuxerParams.pTrackParams[1].type  = UMC::AUDIO_TRACK;
MuxerParams.pTrackParams[1].info.audio = &AudioInfo;
MuxerParams.pTrackParams[1].bufferParams.m_prefInputBufferSize = 20000000;
MuxerParams.pTrackParams[1].bufferParams.m_prefOutputBufferSize = 20000000;

UMC::MPEG2Muxer Muxer;
Muxer.Init(&MuxerParams);

/////////////////////////////////////////////
// MPEG2 Encoder - video
UMC::MPEG2EncoderParams videoParams;
videoParams.info.clip_info.height  = 1080
videoParams.info.clip_info.width  = 1920;
videoParams.info.bitrate    = 1000000;
videoParams.info.color_format   = UMC::YUV420;
videoParams.info.framerate    = 24;
videoParams.info.stream_type   = UMC::MPEG2_VIDEO;
videoParams.info.aspect_ratio_height = 9;
videoParams.info.aspect_ratio_width  = 16;
videoParams.numThreads     = 1;

UMC::MPEG2VideoEncoder Mpeg2Encoder;
Mpeg2Encoder.Init(&videoParams);

////////////////////////////////////////////
// AAC Encoder - audio
  UMC::AACEncoderParams audioParams;
  audioParams.ns_mode      = 1;
  audioParams.stereo_mode     = UMC_AAC_JOINT_STEREO;
  audioParams.m_info_in.channels   = 2;
  audioParams.m_info_in.sample_frequency = 44100;
  audioParams.m_info_in.bitrate   = 24000;
  audioParams.audioObjectType   = UMC::AOT_AAC_LC;
  audioParams.outputFormat    = UMC::UMC_AAC_ADTS;
  audioParams.m_info_out.channels   = 2;
  audioParams.m_info_out.bitrate   = 24000;
  audioParams.m_info_out.sample_frequency = 44100;

UMC::AACEncoder AacEncoder;
AacEncoder.Init(&audioParams);

UMC::VideoData DataVideoIn;
DataVideoIn.Init( 1920, 1080, UMC::YUV420,8);

Ipp8u* pOutputVideoData = ippsMalloc_8u(3110400);
Ipp8u* pOutputAudioData = ippsMalloc_8u(4096);

UMC::MediaData DataVideoOut;
UMC::MediaData MuxVideoData;
UMC::AudioData DataAudioIn;
UMC::AudioData DataAudioOut;
UMC::MediaData MuxAudioData;

DWORD dwEvent;
int nEncodedVideoFrames = 0;
int nEncodedAudioFrames = 0;

//////////////////////////////////////////////////////

Now I use WaitForMultipleObjects to catch the video and buffers

///////////////////////////////////////////////
// Audio Part

UMC::Status status;

DataAudioIn.SetBufferPointer(pAudioFrameBuffer, nAudioBufferSize);
DataAudioIn.SetDataSize(nAudioBufferSize);

DataAudioOut.SetBufferPointer(pOutputAudioData, 100000000);

  status = AacEncoder.GetFrame(&DataAudioIn, &DataAudioOut);
  if (status == UMC::UMC_OK)
  {
      nEncodedAudioFrames++;

      int nDataSize = DataAudioOut.GetDataSize();
      MuxAudioData.SetBufferPointer(pOutputAudioData, nDataSize);

      memcpy(MuxAudioData.GetDataPointer(), DataAudioOut.GetDataPointer(), nDataSize);

      MuxAudioData.SetDataSize(nDataSize);
      MuxAudioData.SetTime(nEncodedAudioFrames*((double)1.0)/(44100/1024.0));

      do
      {
         status = Muxer.PutAudioData(&MuxAudioData);       // I receive Unhandled exception right here
         if (UMC::UMC_ERR_NOT_ENOUGH_BUFFER == status)
            vm_time_sleep(5);

}while (UMC::UMC_ERR_NOT_ENOUGH_BUFFER == status);
}

Audio Part works fine with MPEG4 muxing. The difference only that during initialization I used parameters specific to MPEG4.

Thank you in advance

Sergey Tsybikov

Smith&Nephew

One more comment:

stsybikov — Tue, 08 Jan 2013 18:22:54 GMT

One more comment: In the previous sample I defined UMC::MPEG2MuxerParams MuxerParams; MuxerParams.m_SystemType = UMC::MPEG2_PURE_VIDEO_STREAM; This setting allow me to mux only video and I received the crash for audio. If I defined MuxerParams.m_SystemType = UMC::MPEG2_PURE_AUDIO_STREAM the mux of audio is fine, but the crash for video mux. Does anyone know what parameter I have to use to combine audio and video together. I tried almost avery value from SystemStreamType but no success. Thank you Sergey Tsybikov

It probably won't be as

Jeffrey_M_Intel1 — Tue, 08 Jan 2013 23:21:02 GMT

It probably won't be as simple as a parameter change. Many problems have been discovered with using audio and video together in UMC's muxers. As samples, their original purpose was to facilitate viewing output from the sample video codecs since many players only work with container formats. While it may be possible to use them as a base (probably with a lot of work to get started, and then later to harden them) they aren't an ideal starting point for anything beyond what they were originally written for. This has been discussed at great length internally, but for now there is not much more to say than that it is unlikely additional work to bring the muxers up to "turnkey" level will be prioritized. This article may be of interest as an alternative, since the method described could be applied to a wide variety of video codecs: http://software.intel.com/en-us/articles/integrating-intel-media-sdk-with-ffmpeg-for-muxdemuxing-and-audio-encodedecode-usages Best regards, Jeff

Hi Jeff,

stsybikov — Thu, 10 Jan 2013 15:48:27 GMT

Hi Jeff, Thank you for your reply. The alternative that you suggested doesn't work for me. We have our own medical camera and the camera process is feeding the memory buffers with video and audio data. This is not a file, buffers with single video and audio frames. So my idea is to mux the data to the mpeg file based on user request (mpeg1/2/4). MPEG4 muxer works great but MPEG2 does not. I have to find the solution to mux video and audio data to mpeg2. Thank you. Sergey Tsybikov Smith&Nephew Endoscopy

topic Hi Jeff, in Intel® Integrated Performance Primitives

Problem with muxing MPEG2 video and audio

One more comment:

It probably won't be as

Hi Jeff,