Intel® oneAPI Threading Building Blocks
Ask questions and share information about adding parallelism to your applications when using this threading library.

task_scheduler_init in COM DLL

mrussel
Beginner
788 Views
I've got a Windows DLL thatcreatesthe task_scheduler_init as a global object with automatic thread creation.

Everything works fine except when I try to run regsvr32.exe onthe DLL. The process hangs & when I break in the debuggerthe workerthread is inside ExitThread, the main thread is calling WaitForSingleObject() inside of Arena::terminate_workers.

I can even see the problem runningregsvr32.exe on an empty DLL project (non-COM) that just has the global task_scheduler_init. Does anybody else have this problem? I'm using 2.1.2008.605 TBB.

Thanks

0 Kudos
17 Replies
Alexey-Kukanov
Employee
788 Views
Please explicitly terminate TBB in DllRegisterServer/DllUnregisterServer (and other functions as necessary):
global_tbb_init_object.terminate();

If your library provides some initialization function that must be called before library use (and presumably not called by regsvr32), then I would recommend you to construct your globalTBB init objectas deferred (passing tbb::task_scheduler_init::deferred into the constructor), and explicitly initialize it later in that function. Thus you would defer creation of TBB threads and have no problem with regsvr32.
0 Kudos
mrussel
Beginner
788 Views
Actually,in my situationthe DLL that has the task_scheduler_init is not a COM DLL but is used by one. I had simplified the picture for myposting.

My goal is to make the use of my DLL as easy as possible for the client. But it appears I will need to expose API functions to initialize/deinit TBB, is this theonly solution?

0 Kudos
Alexey-Kukanov
Employee
788 Views
Well, I could suggest you explictly call initialize() and terminate() in DllMain of your DLL, but first check that the process name isn't regsvr32. Or make TBB initialization lazy - create your task_scheduler_init object deferred, on every call that requires TBB check if it is_active() (the check is cheap - look at the header), and if not active, initialize explicitly.
0 Kudos
pvonkaenel
New Contributor III
788 Views
Well, I could suggest you explictly call initialize() and terminate() in DllMain of your DLL, but first check that the process name isn't regsvr32. Or make TBB initialization lazy - create your task_scheduler_init object deferred, on every call that requires TBB check if it is_active() (the check is cheap - look at the header), and if not active, initialize explicitly.

I'm seeing the same thing in my plug-in DLL.It plugs into an application that uses LoadLibrary(). I have a DllMain like the following:

[cpp]tbb::task_scheduler_init g_tbbinit(tbb::task_scheduler_init::deferred);


BOOL APIENTRY DllMain( HMODULE /*hModule*/,
                       DWORD  ul_reason_for_call,
                       LPVOID /*lpReserved*/ )
{
    switch (ul_reason_for_call) {
        case DLL_PROCESS_ATTACH:
            ippStaticInit();
            g_tbbinit.initialize(tbb::task_scheduler_init::automatic);
            break;
        case DLL_THREAD_ATTACH:
        case DLL_THREAD_DETACH:
            break;
        case DLL_PROCESS_DETACH:
            g_tbbinit.terminate();
            break;
    }
    return TRUE;
}

[/cpp]
The terminate() ends up hanging on a WaitForSingleObject() callin Arena::terminate_workers(). Note that the calling application is attaching to the library and detaching before any TBB calls are called, but I can see the worked threads are all created when initialize() is called.

Peter
0 Kudos
Alexey-Kukanov
Employee
788 Views
Quoting - pvonkaenel
I'm seeing the same thing in my plug-in DLL.It plugs into an application that uses LoadLibrary(). I have a DllMain like the following:

[cpp]tbb::task_scheduler_init g_tbbinit(tbb::task_scheduler_init::deferred);


BOOL APIENTRY DllMain( HMODULE /*hModule*/,
                       DWORD  ul_reason_for_call,
                       LPVOID /*lpReserved*/ )
{
    switch (ul_reason_for_call) {
        case DLL_PROCESS_ATTACH:
            ippStaticInit();
            g_tbbinit.initialize(tbb::task_scheduler_init::automatic);
            break;
        case DLL_THREAD_ATTACH:
        case DLL_THREAD_DETACH:
            break;
        case DLL_PROCESS_DETACH:
            g_tbbinit.terminate();
            break;
    }
    return TRUE;
}

[/cpp]
The terminate() ends up hanging on a WaitForSingleObject() callin Arena::terminate_workers(). Note that the calling application is attaching to the library and detaching before any TBB calls are called, but I can see the worked threads are all created when initialize() is called.

Peter


There may be a bug that causes hang. Thanks for reporting the problem.

It's correct thatworker threads are created by initialize(); that's how explicit initialization is supposed to work :)

0 Kudos
pvonkaenel
New Contributor III
788 Views


There may be a bug that causes hang. Thanks for reporting the problem.

It's correct thatworker threads are created by initialize(); that's how explicit initialization is supposed to work :)


Hi Alexey and thanks for responding,

I've tried debugging this some more, and it looks like the worker threads all end up in _endthread, but don't really go away (not sure why). Do you understand what the problem is, and do you have a suggested short term fix?

Thanks,
Peter
0 Kudos
pvonkaenel
New Contributor III
788 Views

Hi Alexey,

I have some additional information which I hope is helpful in tracking this down. I have verified that after the initialization all worker threads end up waiting on scheduler.wait_for_all() within GenericScheduler::worker_routine(). When task_scheduler_init::terminate() is called, all workers wake up, but they end up sticking within _endthreadex() on the call to EndThread(). Does this imply they are holding resources that have not been released?

As a temporary workaround, I'm allocating a task_scheduler_init instance within DllMain when the process is attached (using new), but never terminate it or free itto prevent the hang. Is this safe to do?

Thanks,
Peter
0 Kudos
Alexey-Kukanov
Employee
788 Views
Off the top of the head, I think your workaround should be safe. When TBB worker threads are "parked" having no work, they should not hold any system resources that would not be released at the process shutdown. If you encounter problems with that, let us know.

What the main thread is doing while worker threads are waiting for something in _endthreadex? If it still in the middle of shutdown or DLL unload process, it might happen that the worker threads are waiting for so-called loader lock which is global and currently held by the main thread.
0 Kudos
pvonkaenel
New Contributor III
788 Views
Off the top of the head, I think your workaround should be safe. When TBB worker threads are "parked" having no work, they should not hold any system resources that would not be released at the process shutdown. If you encounter problems with that, let us know.

What the main thread is doing while worker threads are waiting for something in _endthreadex? If it still in the middle of shutdown or DLL unload process, it might happen that the worker threads are waiting for so-called loader lock which is global and currently held by the main thread.

The main thread is waiting on WaitForSingleOject(w.thread_handle, INFINITE) within Arena::terminate_workers(). This has been triggered by the DllMain() calling task_scheduler_init::terminate() when the DLL is being detached from the application.
0 Kudos
Alexey-Kukanov
Employee
788 Views
Then that's what I was afraid of. Thanks for the information. We will see where the TBB shutdown logic is wrong and how it can be fixed.
0 Kudos
pvonkaenel
New Contributor III
788 Views
Then that's what I was afraid of. Thanks for the information. We will see where the TBB shutdown logic is wrong and how it can be fixed.

Please let me know if there's anything I can do to help: I really want to switch my DLL over to TBB, but am hesitant to check my code in before I have a way to shut down the worker threads.
0 Kudos
pvonkaenel
New Contributor III
788 Views
Quoting - pvonkaenel

Please let me know if there's anything I can do to help: I really want to switch my DLL over to TBB, but am hesitant to check my code in before I have a way to shut down the worker threads.

I've attached a solution that demonstrates the hang. Hopefully this will help with debugging the problem.

Peter
0 Kudos
Alexey-Kukanov
Employee
788 Views
Quoting - pvonkaenel
I've attached a solution that demonstrates the hang. Hopefully this will help with debugging the problem.

Peter

Thanks Peter. I will keepyou informed about investigations.
0 Kudos
pvonkaenel
New Contributor III
788 Views

One thing I just noticedin the case where my DLL is freed and the worker threads hang in ExitThread(), is the tbb.dll DllMain() DETACH_PROCESS has not been called yet. Could the resources that should be freed within __TBB_InitOnce::remove_ref() be a potential source of shutdown deadlock?
0 Kudos
pvonkaenel
New Contributor III
788 Views

I'm able to gt past the deadlock, and the worker threads all terminate properly if I change the INFINITE parameter in WaitForSingleObject() when called in Arena::terminate_workers() to some finite timeout like 100 (I tend to try and avoid INFINITE myself). Does this seem like a safe solution to the problem?

Peter
0 Kudos
pvonkaenel
New Contributor III
788 Views

In-case anyone was following this issue, the latest version 2.2 release appears to work when loaded through a module via LoadLibrary() without any other modifications. It also seems to work when compiled with /MT.

Great work TBB team!

Peter
0 Kudos
Steve_Nuchia
New Contributor I
788 Views

Reading over this thread I see a lot of speculation, let me just add a key word that will help others find the relevant info: The Windows "loader lock" is at the core of this problem. It is held while the intermediate DLL is being unloaded and it prevents the THREAD_DETACH calls being delivered to the TBB DLL when those threads are terminated from DllMain. This is a common cause of deadlock in Windows applications and you should be able to find a lot of info on it, once you know what to look for.
0 Kudos
Reply