Intel® Integrated Performance Primitives
Deliberate problems developing high-performance vision, signal, security, and storage applications.

IPP 7.0 samples - mp4 muxer fixes

jachu19
Beginner
833 Views
Hi,

Intel programmers,
please fix some mp4 muxer issues mentioned below in neerest IPP 7.x samples release. I fixed (workarounded) it in my way, but Im not the author, and I would prefer to get official fix from Intel developers.

Problems found when muxing H.264 file, with many IDR frames (for example every 28 frames), framerate 29.970, 2 B frames, closed GOP.

1. uninitialized variable: stss->total_entries used
2. wrong m_nSttsEntries computing
3. stts.table - out of buffer writing

also:
- mov audio stream type is set wrong, when AAC set on input, had to set UNDEFINED,
- somethings wrong, when I set I_PICTURE flag in IDR frames, but when not setting muxer detects it itself, and works fine.
- somethings wrong when stream end with a P frame, but reordered B frames are missing (stream is cut)
- somethings wrong when pts is not set for every frame

here are my fixes, I had to remove some dir names:

Index: (...)audio-video-codecs/codec/mpeg4_mux/src/umc_mp4_mux.cpp
===================================================================
--- (...)audio-video-codecs/codec/mpeg4_mux/src/umc_mp4_mux.cpp (revision 4024)
+++ (...)audio-video-codecs/codec/mpeg4_mux/src/umc_mp4_mux.cpp (working copy)
@@ -254,6 +254,10 @@
curSample.m_frameTMype = GetPictureType(lpData, ntrak);
if (I_PICTURE == curSample.m_frameTMype)
{
+ if (m_nIDRFrames[ntrak] == 0)//modified by jachu
+ {
+ m_nHeaderSize += (8 + 4*2);
+ }
curSample.m_frameTMype = I_PICTURE;
m_nHeaderSize += m_bMoov ? 4 : 0;
m_nIDRFrames[ntrak]++;


Index: (...)audio-video-codecs/codec/mpeg4_mux/src/umc_mp4_mux_atoms.cpp
===================================================================
--- (...)audio-video-codecs/codec/mpeg4_mux/src/umc_mp4_mux_atoms.cpp (revision 4024)
+++ (...)audio-video-codecs/codec/mpeg4_mux/src/umc_mp4_mux_atoms.cpp (working copy)
@@ -60,13 +60,18 @@
for (j = 2; j < m_sTrack.m_nSamplesCount; j++)
{
pSamples[j - 1].m_nDuration = pSamples.m_nTimeStamp - pSamples[j - 1].m_nTimeStamp;
+ }
+
+ // restore initial order - m_nPosition is used here (not sure that it's always possible!!!)
+ qsort(m_sTrack.m_pSamples, m_sTrack.m_nSamplesCount, sizeof(sMuxSample), CompareByPos);//modified by jachu
+
+ for (j = 2; j < m_sTrack.m_nSamplesCount; j++)
+ {
if (pSamples[j - 1].m_nDuration != pSamples[j - 2].m_nDuration)
{
m_sTrack.m_nSttsEntries++;
}
}
- // restore initial order - m_nPosition is used here (not sure that it's always possible!!!)
- qsort(m_sTrack.m_pSamples, m_sTrack.m_nSamplesCount, sizeof(sMuxSample), CompareByPos);
}
}
return UMC_OK;
@@ -248,7 +253,7 @@
stbl = &m_headerMPEG4.trak->mdia.minf.stbl;
stbl->size_atom = 8 + stts->size_atom + stsd->size_atom + stsz->size_atom + stco->size_atom + stsc->size_atom + ctts->size_atom;

- if (m_headerMPEG4.trak->mdia.minf.is_video && stss->total_entries)
+ if (m_headerMPEG4.trak->mdia.minf.is_video && m_nIDRFrames/*stss->total_entries*/)//modified by jachu
{ /* not mandatory */
stbl->size_atom += stss->size_atom;
}
@@ -597,7 +602,7 @@
stts.flags = 0;
stts.total_entries = m_sTrack[nTrack].m_nSttsEntries;

- stts.table = _NEW_n(TM_stts_table_data, m_sTrack[nTrack].m_nSttsEntries);
+ stts.table = _NEW_n(TM_stts_table_data, m_sTrack[nTrack].m_nSttsEntries + 1);//(+1) modified by jachu

return UMC_OK;
}


0 Kudos
3 Replies
jachu19
Beginner
835 Views
2 more suggestions:

- aspect ratio support

my fix:
m_headerMPEG4.trak->tkhd.track_width = (Ipp32f)(m_pTrackParams.info.video->clip_info.width << 16);
if (m_pTrackParams.info.video->aspect_ratio_width != 0 && m_pTrackParams.info.video->aspect_ratio_height != 0)
{
m_headerMPEG4.trak->tkhd.track_width = (Ipp32f)(((m_pTrackParams.info.video->clip_info.width * m_pTrackParams.info.video->aspect_ratio_width) / m_pTrackParams.info.video->aspect_ratio_height) << 16);
}



- 33 bit PTS support (over 24h) (needed for remuxing recordings from DVB)

setting MAX_TIME to 95444 is not enogh, but fixes end of stream handling,
I rebased PTS on input, so for 99% cases it works.
it wont work when pts overflows (it is possible in streams dumped from DVB)
0 Kudos
Alexander_Modenov__I
New Contributor I
835 Views
Hi jachu19,
There are some questions and clarification to problems:
- Could you provide your h264 stream with problems? I can't reproduce the problem yet.
-mov audio stream type is set wrong, when AAC set on input, had to set UNDEFINED - The stream type is defined by your splitter or encoder so the information in muxerparameters.pTrackParams should be correct. Do not forget to addFLAG_DATA_FROM_SPLITTER if your are using splitter and use decoder initialization data from splitter as a first data block for your audio data.
-somethings wrong when pts is not set for every frame. According to the UMC manual each frame must have correct PTS. So please make correct timestamps.
- somethings wrong, when I set I_PICTURE flag in IDR frames, but when not setting muxer detects it itself, and works fine. Sorry, could your clarify this problem.
- somethings wrong when stream end with a P frame, but reordered B frames are missing (stream is cut). I need more information that happening.
0 Kudos
jachu19
Beginner
835 Views
Hi,

sorry for loooong answer delay :/

-mov audio stream type is set wrong...
I mean this function:
Ipp32s MP4Muxer::GetMP4StreamType(Ipp32s type)
{
Ipp32s tag;

if (type == MPEG4_VIDEO_QTIME) {
tag = 0x04; // Visual Stream
} else
if (type == UNDEF_AUDIO_SUBTYPE) {
tag = 0x05;
} else {
tag = type;
}
return tag;
}

when I set stream_subtype to AAC_LC_PROFILE this function sets MP4 stream type to 0x01 instead of 0x05. Fix is simple, so its a minor for me, but can be confusing for users. Many player skips this value, but realized that iTunes checks it, and don't play audio, when != 0x05

-somethings wrong when pts is not set for every frame. - I understand the manual, but have to interpolate time stamps on broadcast signals and DVD where time stamps are not present for every frame. Also minor for me since I implemented PTS interpolation ;)

- somethings wrong, when I set I_PICTURE flag...
I dont fully understand FLAG_FRAGMENTED_AT_I_PICTURES flag befaviour. The issue was there were only one sequence muxed, when I set this flag and signal I_PICTURE on IDRs. when needNewFragment returns true DisposeDataToFile is called which I dont fully understand. When I dont signal I_PICTURE needNewFragment does not detect IDR and it was my first WA. Now I set only FLAG_START_WITH_HEADER flag and signal I_PICs as usual, because I needed to enable SEI msg muxing.

And this is another issue. When SEI are present (enabled in: if (NAL_UT_SLICE == nNAL || NAL_UT_IDR_SLICE == nNAL || NAL_UT_AUXILIARY == nNAL || NAL_UT_SEI == nNAL) ) frame types are not recognized in H264.
Manual should then say that signalling picture type is mandatory.

- somethings wrong when stream end with a P frame, but...
for example: when last muxed frame has a PTS distance from previous frame = 3xframe time (reordered P frame) and suddenly stream ends (broadcast), muxer computes invalid timings for all video frames in this track. Occures problem with AV sync. My WA is setting m_nDuration of last two frames (n-1 and n-2) to value from n-3 frame (n - number of frames). But it wont work for 5 non ref B frames for example.


Another problem I was fixing lately is 4GB+ file size support. no co64 support at all (found only structure definition :) )

At the moment I fixed everythin I nead, I fixed muxer setup, but I would like to help the other using the muxer sample. I think manual should describe some more detailed mentioned flags behaviour. I suppose you assume H264 and AAC input from intel encoder which in my case is not enough.

regards
MJ

0 Kudos
Reply