In concurrent_bounded_queue,push() sleeps if no slot is available and pop() sleeps if no data. There can be short periods of busy-waiting (spinning), but long waits are always passive. On the other hand, TBB was not designed for real-time, so I would expect surprises.
How a receiver thread selects a queue to push data into - randomly?
Would you be able to show thread stacks when they spin?
(Added 2011-10-13) Sorry, I shouldn't have tried to rush a half-baked reply: after sleeping and becoming runnable again, the thread would be inserted to the end of the list for its priority. Is Roman's advice helpful?
I'll leave it to Alexey to provide authoritative answers about the current implementation.
"I think the best thing to do is try Roman's suggestion. Although if it works, I'd still want to know why it worked."
I don't see which version and package you tried, but perhaps if _lin uses icc you might see improvement from switching to the _src package instead? Or you could wait for what Alexey has to say about that.
First, do you use bounded queue or unbounded queue?
Second, you said that the threads that spin are "worker" threads; but worker threads should pop(), not push(). Did you make some mistake somewhere?
Third, you said there are 10 receiving threads and 6 worker threads; but the ps output shows 10 threads with priority 75 (receiving?) and 10 threads with priority 50 (workers?). And how many HW threads are on the machine, by the way?
By the way, one possible reason for endless busy-waiting is zero capacity of a bounded queue. Can it happen that there is a race somewhere between setting capacity and using the queue?