MFXMuxer_Init gives either AccessViolationException or SEHException

[Matthew_C_]

So, I am calling MFXMuxer_Init from within a function in a C dll, everything seems to execute fine until it reaches this line:

MFXMuxer_Init(streamParams, (mfxDataIO *)curBitstream, muxer);

At that point C# throws either AccessViolationException or SEHException for function that is calling it.

Any thoughts regarding what could be going on?

I have placed the code proceeding this that sets up streamParams and calls this function here: http://pastebin.com/3WVr64rP

Here is the BitstreamIO class: http://pastebin.com/t6QDjhrB

Thank you!

[Matthew_C_]

One other thing which is if this dll is called from C++, it works just fine, but if it is called from C#, then it throws the described errors.

I've tried catching an exception in C++, and it simply is not catching an exception.

Also, when I narrow it down, it seem to be occurring on this line of ffmpeg_mux_impl.c:

if(avformat_write_header(out_mux->format_context, NULL) < 0)

This is approximately line number 280.

I discovered this by inserting debugging code, the code before this line printed, the code after it didn't and C# threw an exception:

printf("out_mux->format_context: %p", out_mux->format_context);
printf("sts: %d", sts);  //Shows me that the error = 0
fflush(stdout);
    if (sts == MFX_ERR_NONE)
    {
        if(avformat_write_header(out_mux->format_context, NULL) < 0)
        {
            sts = MFX_ERR_UNKNOWN;
        }
    }
    printf("MFXMuxer_Init Checkpoint #7");

Any thoughts?  Thank you!

[RyanS · Accepted solution]

Not sure what the problem is but here are some thoughts:

- Make sure you have "Enable native code debugging" checkbox checked in your C# executable project's Debug properties and are running the debug builds. You should be able to set breakpoints and step into the native code.

- Native code access violations are distinct from C++ exceptions, here's a discussion: http://stackoverflow.com/questions/457577/catching-access-violation-exceptions

- Make sure you are running the dll's (libav*) you think you are running. Maybe the C++ and C# programs are picking up different dll's.

[Matthew_C_]

Being able to Debug using "Enable unmanaged code debugging" has proven quite useful, helped fix that problem as well as a couple others.  

And if anyone else stumbles across this post, it was my problem not the libraries, I was passing in a pointer to something instead of the actual thing, somehow it let me cast it funny.. point is check the inputs and for the MFXMuxer_Init, etc. functions and make sure all the pointers are right as well as all your mallocs, and frees, this may have also partially been causing by trying to delete something that had been 'malloced' instead of 'newed'.

Thank you!