Intel® oneAPI Threading Building Blocks
Ask questions and share information about adding parallelism to your applications when using this threading library.
2464 Discussions

Is there any way to observe the threads a task scheduler is managing?

wzpstbb
Beginner
697 Views

Hello,

We have an OpenGL application. In each frame, we run the following piece of code. 

   {
            // Pipeline scheduler observer. It observes the worker threads entering the 
            // task arena.
            PipelineObserver o(dc);

            // use TBB to execute a parallel while
            tbb::parallel_while w;
            ApplyIterator body(*this, iteratorStream, eyePosition, maxCell, &w);
            w.run(iteratorStream, body);
  }

For your reference, we implement the observer as below.

    class PipelineObserver : public tbb::task_scheduler_observer
    {
        ADrawContext& mDC;
        void operator=(const PipelineObserver&);
        PipelineObserver();
    public:
        PipelineObserver(ADrawContext& dc)
            : mDC(dc)
        {
            observe(true);
        }

        ~PipelineObserver()
        {
            observe(false);
        }

        virtual void on_scheduler_entry(bool  /* is_worker */ )
        {
            if(mDC.HasVirtualDevice())
                mDC.VirtualDevice().AcquireLocalOGLContext();
        }

        virtual void on_scheduler_exit(bool  /* is_worker */ )
        {
            if(mDC.HasVirtualDevice())
                mDC.VirtualDevice().ReleaseLocalOGLContext();
        }
    };

By doing this, we allocate and bind a local OpenGL context to each thread. However, I don't see a way to unbind the OpenGL context from the associated thread. We do implement on_scheduler_exit. But it doesn't seem to work as our expectation.

Ideally, we want to observe the threads each time the parallel_while finishes execution so that we have the opportunity to unbind the OpenGL context. At the least, we want to observe the threads when the task scheduler is terminated so that we can cleanup the OpenGL contexts properly. 

Does anyone know how to accomplish this?

Thanks

Wallace

 

0 Kudos
12 Replies
RafSchietekat
Valued Contributor III
697 Views

"But it doesn't seem to work as our expectation."

In what way, exactly?

0 Kudos
wzpstbb
Beginner
697 Views

Raf Schietekat wrote:

"But it doesn't seem to work as our expectation."

In what way, exactly?

Hi Raf,

We want each thread to be notified after each run of parallel_while(each frame). We also want each thread to be notified when the associated task scheduler is terminated. Is this possible? Thanks! 

0 Kudos
RafSchietekat
Valued Contributor III
697 Views

So "not as [your] expectation" didn't mean that it behaves differently from what you would expect after having read the documentation, but just that you want it to do something different, as explained. Well, you're not going to get those notifications, unless perhaps if you have a clear and preferably generalisable need that makes it worthwhile for them to be added to TBB, and a lot of patience. Now it becomes a matter of differentiating what you say you want and what you actually want. :-)

First I should probably warn that threads may jump around seemingly at random in the parallel execution tree. If your parallel_while over frames has nested parallel constructs, then a TBB thread may very well decide to jump to such a construct in another frame, which may play havoc with your assumptions about context.

Disregarding any complications that may arise from that, and just considering the cleaning up, would it be possible to keep those contexts in an enumerable_thread_specific instance and clean them up from the main thread, or do you have to be in the thread that created the context? It might help if you gave an indication of what operating system and library you are using to manage contexts.

0 Kudos
wzpstbb
Beginner
697 Views

Hi Raf,

We'd like to flush the GL commands manually in each thread after each run of parallel_while. I don't see a way to do this.

In addition, binding an OpenGL context to a thread may fail if the OpenGL context is bound to the other thread already. Creating the shared OpenGL contexts is very expensive. So we have a fixed number of shared OpenGL contexts only. we have to unbind the OpenGL contexts from the threads managed by one task scheduler if we want to bind them to the threads managed by the other task scheduler.( There would be different task schedulers if the parallel_while is run from different master threads). 

For cleaning up, we do have an enumerable_thread_specific instance to store the thread-local OpenGL contexts. We clean up those OpenGL contexts in the main thread. However, the cleanup may fail because the OpenGL contexts are still bound to the TBB threads. We really need to unbind the OpenGL contexts from the TBB threads when the task scheduler is terminated so that we can recyle the OpenGL contexts properly.

Thanks!

0 Kudos
RafSchietekat
Valued Contributor III
697 Views

You seem to be using "task scheduler" for the concept that is realised by a tbb::task_arena (each master thread has one). Threads may also migrate between arenas (but only when they are idle). Arenas are created and destroyed along with the associated master thread, but the "task scheduler" (as meant by TBB) is preferably kept alive throughout the program.

Do I understand correctly that only the thread running a context can unbind it? That would indeed be annoying, because the (worker) thread may have gone to sleep, or migrated to a different arena (in case there are multiple master threads in the program), and never get around to unbinding a context. You would actually have to be able to take another thread's context away from it without that thread's cooperation, and that wouldn't even be the end of it.

A workaround could be to have a pool of contexts, and bind/unbind one of them in the scope of a purely sequential piece of code (like a Body execute() function), but then you can't call any unknown code that might recursively use TBB, which is at least a maintenance problem.

(I repeat my question about which operating system and library are being used, perhaps with a pointer to the documentation. It might help to have a look directly at any relevant requirements.)

0 Kudos
wzpstbb
Beginner
697 Views

Hi Raf,

My inline comments below. Thank you!

Raf Schietekat wrote:

You seem to be using "task scheduler" for the concept that is realised by a tbb::task_arena (each master thread has one). Threads may also migrate between arenas (but only when they are idle). Arenas are created and destroyed along with the associated master thread, but the "task scheduler" (as meant by TBB) is preferably kept alive throughout the program.

[Wallace] Our client may customize the maximum threads allowed for multithreading. We explicitly construct a task_scheduler_init object if a customized thread numbers is provided. 

Do I understand correctly that only the thread running a context can unbind it? That would indeed be annoying, because the (worker) thread may have gone to sleep, or migrated to a different arena (in case there are multiple master threads in the program), and never get around to unbinding a context. You would actually have to be able to take another thread's context away from it without that thread's cooperation, and that wouldn't even be the end of it.

[Wallace] Yes, an OpenGL context can be bound or unbound by the running thread only. On windows, we bind a context to a thread using 

wglMakeCurrent(hDC, pContext);

We unbind the context using:

wglMakeCurrent(NULLNULL);

wglMakeCurrent must be called in the running thread. 

I think it is not a problem if a thread migrate to a different arena. We have handled this by implementing our own task_scheduler_observer. We bind the OpenGL context to the thread in on_scheduler_entry and unbind the context in on_scheduler_exit.

A workaround could be to have a pool of contexts, and bind/unbind one of them in the scope of a purely sequential piece of code (like a Body execute() function), but then you can't call any unknown code that might recursively use TBB, which is at least a maintenance problem.

(I repeat my question about which operating system and library are being used, perhaps with a pointer to the documentation. It might help to have a look directly at any relevant requirements.)

[Wallace] Our application is cross-platform. Currently we support Windows XP/7/8/10, Mac 10.8/10.9/10.10 and Linux. Here is the documentation about wglMakeCurrent on Windows.

https://msdn.microsoft.com/zh-cn/subscriptions/downloads/dd374387.aspx

Well, we can try to keep a master thread only. So we would have one single instance of task_scheduler_init. Can TBB notify each thread when the task scheduler is terminated so that we can unbind the OpenGL contexts and clean them up properly?

0 Kudos
RafSchietekat
Valued Contributor III
697 Views

"Our client may customize the maximum threads allowed for multithreading. We explicitly construct a task_scheduler_init object if a customized thread numbers is provided."

OK, but the name "task_scheduler_init" is historical and doesn't mean that you're starting up another task scheduler. All arenas belong to the same scheduler.

"I think it is not a problem if a thread migrate to a different arena. We have handled this by implementing our own task_scheduler_observer. We bind the OpenGL context to the thread in on_scheduler_entry and unbind the context in on_scheduler_exit."

So you have multiple application threads with their arenas, each with parallel_while() calls over a set of frames? If the code for each frame in the parallel_while() is sequential and self-contained, then I can see this working. But if you, for example, call a parallel_for() to compute a set of values that you would subsequently draw in a frame, then that master thread could decide to steal another iteration in the parallel_while(), which would be associated with a different frame buffer, and that seems potentially problematic unless you can just stack contexts because at some point that thread would return to the original iteration/frame, so I would be very careful with that.

"Well, we can try to keep a master thread only."

Using only a master thread means that you have no worker threads, and then you aren't really using TBB (although it is often useful for initial debugging to get at least that right), you're just doing traditional multithreading or sequential programming.

"So we would have one single instance of task_scheduler_init."

If you have only a single task_scheduler_init instance, then that means that on that application/master thread you have configured the number of workers, but on others the default number will still be used, I think. But I think the documentation is somewhat behind the times about this (TBB team?).

"Can TBB notify each thread when the task scheduler is terminated so that we can unbind the OpenGL contexts and clean them up properly?"

If by that you now mean the end of the program, not of an application thread, unfortunately not: "CAUTION: A process does not wait for the worker threads to clean up. Thus a process can terminate before on_scheduler_exit is invoked." But maybe we're still not quite on the same page.

0 Kudos
RafSchietekat
Valued Contributor III
697 Views

I have to correct myself about the scheduler being global rather than arena-specific. The term does seem to match the scope of task_scheduler_init, i.e., specific to an arena, going by some articles I found later on, so Wallace would be right and my previous correction of his usage of the term would be wrong.

Note that there still are structures in common between the various arenas to manage worker threads, like the market and something called the RML. Maybe there's still something that makes this all one big task scheduler, if worker thread assignment is affected by task priority range in each of the arenas competing for worker threads, and the market does seem to track priorities in the various arenas (market::my_priority_levels)?

It's all just a matter of definitions, of course, but definitions by Intel's TBB team trump my interpretation any time.

(Added) Meanwhile, are we making any progress on the original question? My current interpretation is that, because of some things I think I know about TBB, I'm seeing problems that Wallace isn't seeing, or at least not observing+interpreting as possible problems. Partly because of that, I also don't have a clear view on the situation, so all I've been writing was only to get closer together before I could hope to say anything of any real use.

 

0 Kudos
wzpstbb
Beginner
697 Views

Hi Raf,

Thanks for all the information.

If I understand correctly, currently TBB cannot notify the worker threads when the task scheduler is terminated(or at the time the task_scheduler_init object is destructed). So we don't have an opportunity to unbind the OpenGL context from the worker threads. Is this true?

Thanks,

Wallace

0 Kudos
RafSchietekat
Valued Contributor III
697 Views

That's what the documentation says...

Then again, you might keep track of the number of bound contexts, and in the main thread, after destruction of task_scheduler_init, wait for that number to drop to zero before exiting the program. A worker may not receive an entry notification, but unless the program just exits it seems that it should receive an exit notification, eventually, where unbinding could happen.

0 Kudos
wzpstbb
Beginner
697 Views

Hi Raf,

What do you mean by "... waiting for that number to drop to zero before exiting the program"?

We hold pointers to the bound contexts. In order to delete the bound contexts safely, we have to do the following two steps when we reset the system:

1. In the worker thread, unbind the OpenGL context by calling 

wglMakeCurrent(NULL, NULL);

2. In the main thread, bind the OpenGL context and delete the context.

wglMakeCurrent(hDC, pContext);

wglDeleteContext(pContext);

An alternative option is we delete the context in the worker thread by calling wglDeleteContext(pContext). I don't see a proper opportunity to do the deletion either in the worker thread or in the main thread. 

Thanks,

Wallace

0 Kudos
RafSchietekat
Valued Contributor III
697 Views

"waiting for that number to drop to zero before exiting the program": have an atomic count of bound contexts (increment before creating, decrement after deleting), and use that to just busy-wait in the main thread. The theory is that, even though exit callbacks are not guaranteed if the process just exits, you would get them by postponing process exit, if my interpretation of the documentation is correct anyway (seems worth a try). If that's really all you want to do, of course, waiting at the very end of the program, because I still don't quite see how a thread can just be allowed to steal from a different frame and get away with it.

0 Kudos
Reply