Problems with Enqueued Tasks in TBB v4.0 - Tasks Not Executed ( No Output )

[SergeyKostrov]

There are some problems with EnqueuedTasks in TBB v4.0. From time to time,it is not hard to
reproduce, tasks are not executed andthere is not output.

A couple of days ago Alexey Kukanov (Intel) informed everybody that there are some problems:

...that priority support for enqueued tasks is broken for now...

in a thread:

http://software.intel.com/en-us/forums/showthread.php?t=102159

Here is a Test-Case:

class CMyTask : public tbb::task
{
public :
CMyTask()
{
};

CMyTask( int iNumber, char *szPriorityCode )
{
m_iNumber = iNumber;
strcpy( m_szPriorityCode, szPriorityCode );
m_iPriority = -1;
};

~CMyTask()
{
};

tbb::task * execute()
{
printf( "[ Task Number %03ld ] [ Priority Code: %6s ] [ Priority: %10ld ]\\n",
( int )m_iNumber, &m_szPriorityCode[0], ( int )m_iPriority );
return ( tbb::task * )NULL;
};

void SetPriority( int iPriority )
{
m_iPriority = iPriority;
};

private:
int m_iNumber;
char m_szPriorityCode[16];
int m_iPriority;
};

void main( void )
{
printf( "Initialization started\\n" );

#if defined ( __TBB_TASK_PRIORITY )
printf( "TBB Task Priorities Enabled\\n" );
#else
printf( "TBB Task Priorities Disabled\\n" );
#endif

// Test-Case SK1
///*
for( int i = 0; i < 3; i++ )
{
CMyTask &TaskL = *new( tbb::task::allocate_root() ) CMyTask( i*10+1, "low" );
TaskL.SetPriority( ( int )tbb::priority_t::priority_low );
tbb::task::enqueue( TaskL, tbb::priority_t::priority_low );

CMyTask &TaskN = *new( tbb::task::allocate_root() ) CMyTask( i*10+2, "normal" );
TaskN.SetPriority( ( int )tbb::priority_t::priority_normal );
tbb::task::enqueue( TaskN, tbb::priority_t::priority_normal );

CMyTask &TaskH = *new( tbb::task::allocate_root() ) CMyTask( i*10+3, "high" );
TaskH.SetPriority( ( int )tbb::priority_t::priority_high );
tbb::task::enqueue( TaskH, tbb::priority_t::priority_high );
}
//*/

printf( "Initialization completed\\n" );
}

[SergeyKostrov]

As you can see I've executed a Test-Case SK1 ( see previous post )three timeswith a short delay for
about 5 seconds. Codes are not modified between runs.In a 3rd attempt tasks are not executed:

[RafSchietekat]

The original reproduction code had this issue, and I assume it will be picked up as part of that issue's cure, either implicitly (as just another symptom of the underlying problem) or explicitly. I don't see any need for a new forum thread yet, unless it also occurs without priorities (I don't think fire-and-forget means that you fire and TBB forgets)?

[RafSchietekat]

I tried it for myself (x86/Linux/g++, with both tbb40_258oss, i.e., 4.0 update 1, and tbb40_278oss, i.e., 4.0 update 2, and adapting this code): enqueued root tasks are in fact discarded when the scheduler is terminated (delaying that moment in the main thread gives the enqueued tasks the opportunity to execute), even without use of priorities.

But actually that just means that they can be considered "daemon" tasks (like Java daemon threads). To instead guarantee their eventual execution, allocate them as children and wait_for_all() in their parent (verified but without use of priorities).

Perhaps this should be explicitly documented, though. And maybe it would be more convenient if enqueued root tasks weren't "daemon" tasks unless explicitly flagged so.

[Alexey-Kukanov]

I think this happens when the main thread exits so fast that no TBB workers were able to join execution of the arena; and I think it might have nothing to do with priorities, though it needs to be checked. And if this is the case, I am not sure if it is worth fixing, since itis a very corner case then, and very unlikely to happen in real applications. That's not because I am lazy, but because the TBB shutdown code is so complicated already that adding more for sake of an artificial test seems plain wrong to me. Of course if there is a real app that suffers, we will either find ways to fix it or suggest an acceptable workaround.

[RafSchietekat]

"I am not sure if it is worth fixing"
I suggested a way out, provided proper documentation, but to me this does not seem like an innocent little bug that can be kept from view, as it can come back to bite you in unexpected ways.

[SergeyKostrov]

Quoting Alexey Kukanov (Intel)

...That's not because I am lazy, but because the TBB shutdown code is so complicated already that adding more for sake of an artificial test seems
plain wrong to me. Of course if there is a real app that suffers, we will either find ways to fix it or suggest an acceptable workaround...

I've underlined some statements that caused a couple of questions:

How somebody, not just me,could use TBB in a real software whenIntel software engineer makes
statements like this?

Is that my fault that a simple test case doesn't work properly?

[Alexey-Kukanov]

I do not think answering rhetorical questions is useful for anybody, but keeping silence would perhaps be worse.

> That's not because I am lazy
Is it bad for anybody to admit that he is lazy, or is it only bad for an Intel engineer? :)

> but because the TBB shutdown code is so complicated already
That's just the statement of the fact. Yes, it is complicated; what's wrong with that?
Try making a DLL that creates (including lazily on first use), uses, and destroys a pool of threads, and allocates many internal resources that may or may not be re-used in the future, and may be still in use when shutdown is initiated... you will find out that this is complicated, and that startup and shutdown are very complex. You have to care about deallocating resources, avoiding deadlocks (in particular when the loader lock isinvolved), shutting down the threads correctly when the DLL is unloaded but the app continues, and etc.

> that adding more for sake of an artificial test seems plain wrong to me.
So, what's wrong with the accented part? In my opinion, the test does not represent any real application behavior, because it finishes too quickly and does not use the result of the jobs it enqueued to execute in parallel. It can be useful to investigate what are the issues with task::enqueue, but then the issues should be considered the same way as any other engineering problems,and the decision on whether it should be fixed, documented, or ignored should be done after consideration of many factors. My initial assessment is that it's not worth fixing; I did not say it's not a bug, but it seems to be a low priority issue. Perhaps I should not tell it here, or should use different words, but now it's too late. Of course I can change the opinion later, as more details are clear.

> Of course if there is a real app that suffers, we will either find ways to fix it or suggest an acceptable workaround.
Again, what you see wrong here? Let me rephrase: if a customer comes to me with the real use case, not just a test, then the priority of the issue becomes higher, and we will find a way out suitable for the customer.

> How somebody, not just me,could use TBB in a real software whenIntel software engineer makes
statements like this?
I am not a psychologist, so this question is out of my area of expertise.

> Is that my fault that a simple test case doesn't work properly?
No it's not your fault. Did I say that it's your fault? No I did not. Or maybe I said that the test does not deserve attention? No I did not; in fact I said that it should be investigated.

To make the discussion more constructive, do you have a real usage scenario that suffers from the problem exposed by this test, for meto raise the priority of investigating the issue?

[Alexey-Kukanov]

Thank you Raf for the investigation. I suspected the case like this, but missed your post when was writing mine. And of course your workarounds are right to the point. By the way, maybe (i.e. I am not 100% sure, but have an educated guess) another sufficient workaround is to initialize (and so de-initialize) TBB explicitly with an instance of task_scheduler_init.

I also agree that the issue, if not possible / too hardto fix, should be documented somewhere. It's not intentional that enqueued root* tasks behave like "daemons", but I'm afraid (again, an educated guess) that it cannot be reliably fixed. We will need to investigate it though.

*Actually, I think it's not essential that the tasks are root tasks, butit's essential that the tasks are "fired-and-forgotten", i.e. the main thread leaves without waiting for the tasks to complete.