I've converted transcode_VM(D3D) sample from C++ to C language calling the C level APIs. I've used same session to save context of Encoder and Decoder. In the Transcoding loop, first call to the function MFXVideoDECODE_DecodeFrameAsync returns MFX_ERR_UNDEFINED_BEHAVIOR.
I'm using Windows 7 with D3D9 device acceleration. Can someone help me with the error code.
This error - MFX_ERR_UNDEFINED_BEHAVIOR could be because of various issues. Can you please send us a reproducer and also can you tell us if the original sample worked for you. Please see this to include the necessary details for this query.
Original sample worked like a charm. I figured out my malloc function for allocating video memory had issues.
Can you please guide me how i should be allocating video memory. Because i noticed apart from explicit call to mfxAllocator.Alloc, alloc function is called multiple times implicitly (by libmfx) during decoder and encoder init.
Libmfx is a dispatcher which is linked to your application statically, use for detecting and linking the implementation library according to the machine. This is basically used in the start up to find which implementation(SW or HW accelerated) can be chosen, if you are choosing HW mode and cannot complete the operation with requested params, so it would fall back to SW. You can check the dev guide for more info or check this C:\Program Files\Intel\Intel(R) Media Server Studio 2015 R3\Software Development Kit\opensource\mfx_dispatch location for the opensource code.
You can read in depth about memory allocation / mfxFrameAllocator on MediaSDK Manual Pg26. Also, looking our simple tutorials should also help understanding the memory Allocations.
Hope this helps!