MP4 muxer with H264 encoder - debug vs release

[Madhan_S]

Hi,

I have some grayscale frames and I need to create an mp4 video with H264 encoding out of it. I am using Intel IPP Samples 7.0.1.

The application works fine in debug but doesn't in release configuration. The mp4 output in release has the correct size, duration and dimensions, but the image wont play in any player.

This is not the case in debug mode or when I do debug with breakpoints in release mode (which I guess makes the execution a bit slow for the encoder/muxer to work properly).After some investigation, it seems that due to optimizations in release mode, the execution is very fast and hence proper mp4 output is not generated. 

he documentation says that according to the H264 algorithm there could be some B frames pending in the buffer and they need flushed, so I tried passing NULL to the encoder and retrieve pending frames. But even that didnt help in release mode.

Should I try flushing the muxer? but at what stage? 

I am attaching the input 8 bit grayscale image (1024 X 1024 X 100 frames).

Please find my code below 

//initialize code for the mp4 creator
bool MP4::Initialize(const int& numFrames, const int& frameWidth, const int& frameHeight, const int& frameRate, const std::string& fullfilepath)
{
	m_width = frameWidth;
	m_height = frameHeight;
	m_bitDepth = 8;
	m_numFrames = numFrames;
	m_currentFrameIndex = 0;
	

	vm_string_strcpy(writerParams.m_file_name, fullfilepath.c_str());
	writerParams.m_portion_size = 0;
	result = writer.Init(&writerParams);

	videoInfo.clip_info.height = frameHeight;
	videoInfo.clip_info.width = frameWidth;
	videoInfo.stream_type = UMC::VideoStreamType::H264_VIDEO;
	videoInfo.color_format = UMC::ColorFormat::YUV420;
	videoInfo.interlace_type = UMC::InterlaceType::PROGRESSIVE;
	videoInfo.bitrate = 10000000;
	videoInfo.framerate = float(frameRate);
	videoInfo.streamPID = 0;
	videoInfo.duration = float(numFrames) / frameRate;

	muxerParams.m_lpDataWriter = &writer;
	muxerParams.m_SystemType = UMC::SystemStreamType::H264_PURE_VIDEO_STREAM;
	muxerParams.m_lFlags = FLAG_FRAGMENTED_AT_I_PICTURES;
	muxerParams.m_nNumberOfTracks = 1;
	muxerParams.pTrackParams = new UMC::TrackParams[1];//&videoTrackParams;
	muxerParams.pTrackParams[0].type = UMC::VIDEO_TRACK;
	muxerParams.pTrackParams[0].info.video = &videoInfo;
	muxerParams.pTrackParams[0].bufferParams.m_numberOfFrames = numFrames;
	

	result = m_MP4muxer.Init(&muxerParams);
	
	m_H264EncoderParams.key_frame_controls.method=1;
	m_H264EncoderParams.info.clip_info.height=frameHeight;
	m_H264EncoderParams.info.clip_info.width=frameWidth;
	m_H264EncoderParams.info.bitrate = 10000000;
	m_H264EncoderParams.numThreads = 1;
	m_H264EncoderParams.chroma_format_idc = 1;
	m_H264EncoderParams.numFramesToEncode = numFrames;
	m_H264EncoderParams.info.duration = (float)numFrames/frameRate;
	m_H264EncoderParams.info.framerate = frameRate;
	
	result = m_H264Encoder.Init(&m_H264EncoderParams);
	return result == UMC_OK;
}

//Add frame code for mp4 creator
bool MP4::AddFrame(void* frameData) // framedata is 8 bit grayscale image
{
	int yuvframeSize = m_width * m_height * 3/2;
	Ipp8u* yuvPixels = ippsMalloc_8u(yuvframeSize);
	memset(yuvPixels, 127, yuvframeSize); 						//neutral values for u and v components
	memcpy(yuvPixels, frameData, m_width * m_height); // grayscale to yuv completed

	UMC::VideoData uncompressedData;
	uncompressedData.Init(m_width, m_height, UMC::ColorFormat::YUV420, m_bitDepth);
	uncompressedData.SetBufferPointer(yuvPixels, yuvframeSize);
	uncompressedData.SetDataSize(yuvframeSize);

	UMC::MediaData videoData;
	result = m_MP4muxer.LockBuffer(&videoData, 0);
	result = m_H264Encoder.GetFrame(&uncompressedData, &videoData);

	Ipp64f start = (m_currentFrameIndex/(float)m_numFrames)*m_H264EncoderParams.info.duration;
	Ipp64f end = ((m_currentFrameIndex+1)/(float)m_numFrames)*m_H264EncoderParams.info.duration;
	videoData.SetTime(start,end);
	
	result = m_MP4muxer.UnlockBuffer(&videoData,0);

	ippsFree(yuvPixels);
	m_currentFrameIndex++;
	return result == UMC::UMC_OK;
}

bool MP4::::Close()
{
	UMC::Status status = UMC::UMC_OK;
	status = m_MP4muxer.PutEndOfStream(0);
	status = m_MP4muxer.Close();
	delete [] muxerParams.pTrackParams;
	return status == UMC::UMC_OK;
}

Client Code

#include <fstream>
#include <mp4.h>
using namespace std;
using namespace IntelIPP;

void main()
{
	int width = 1024;
	int height = 1024;
	int numFrames = 100;
	int frameRate = 10;
	int frameSize = width*height;

	char* rawData = new char[frameSize* numFrames];

	ifstream fxdFile;
	fxdFile.open("D:\\Temp\\testImage.raw",std::ios::binary);
	fxdFile.read(rawData, frameSize*numFrames);
	fxdFile.close();

	MP4 mp4creator;
	mp4creator.Initialize(numFrames, width, height, frameRate, "D:\\Temp\\video.mp4");

	char* frame = new char[frameSize];
	for(int i = 0; i<numFrames; i++)
	{
		memcpy(frame, rawData + i*frameSize, frameSize);
		mp4creator.AddFrame(frame);
	}

	mp4creator.Close();
}

[Madhan_S]

I am attaching the mp4 videos generated in release and debug mode. And the hex comparison between the two. Please note that mostly the frame data appears to be the same, and there is a pattern in the differences - 1 bit lesser in the non working video

[Madhan_S]

Could not find the actual reason but the stco table values were all offset by 16. 

muxerParams.m_lFlags = FLAG_FRAGMENTED_AT_I_PICTURES;

I removed explicit setting of the muxerParams flags as shown above and went along with the default initialization. And it worked !

[Ying_H_Intel]

Hi Madhan S, 

Glad to know it works finally.  We will investigate the reason if possible. 

Thanks

Ying 

[Ying_H_Intel]

Hi Madhan,

Thank you very much for the sharing.  We confirm there are several dependent bugs with the Flags, which cause the header offset size error in some condition. and right, the solution is to remove the flags setting.

Best Regards,

Ying

[AF5]

Hello,

I know it's 6 years late, but still,

were any of the issues with FLAG_FRAGMENTED_AT_I_PICTURES addressed?

If yes, is there any chance to get a diff with 7.1.1.013 samples or just the code for mp4muxer?

Thanks.

[Gennady_F_Intel]

Hello, this issue has not been addressed and I have to say that this functionality is not part of the current version of IPP anymore.