can concurrent_bounded_queue deadlock if producer and consumer have different thread priorities?

[Francis_R_1]

I am debugging an issue in my code where I have a concurrent_bounded_queue two producers threads and one consumer thread. The consumer has default thread priority and the producers have low thread priority.

Sometimes I get a deadlock with these stack traces:

Thread 66 (Thread 0x7ffedffff700 (LWP 9126)):

#0  0x00007ffff2025737 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff7981f4d in pause (this=<synthetic pointer>) at src/tbb/tbb/include/tbb/tbb_machine.h:333
#2  __TBB_LockByte (flag=@0x7ffff1b37b08: 0 '\000') at src/tbb/tbb/include/tbb/tbb_machine.h:846
#3  scoped_lock (m=..., this=<synthetic pointer>) at src/tbb/tbb/include/tbb/internal/../spin_mutex.h:93
#4  tbb::internal::concurrent_monitor::notify_relaxed<tbb::internal::predicate_leq> (this=this@entry=0x7ffff1b37b08, predicate=...) at src/tbb/tbb/src/tbb/concurrent_monitor.h:223
#5  0x00007ffff798177f in notify<tbb::internal::predicate_leq> (predicate=..., this=0x7ffff1b37b08) at src/tbb/tbb/src/tbb/concurrent_monitor.h:181
#6  tbb::internal::concurrent_queue_base_v3::internal_push (this=this@entry=0x145d368, src=src@entry=0x7ffedfffe530) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:403
#7  0x00000000004a0840 in push (source=..., this=0x145d368) at ../src/tbb/tbb/include/tbb/concurrent_queue.h:263
 

Thread 64 (Thread 0x7ffee57fa700 (LWP 9124)):
#0  0x00007ffff2025737 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff7981f4d in pause (this=<synthetic pointer>) at src/tbb/tbb/include/tbb/tbb_machine.h:333
#2  __TBB_LockByte (flag=@0x7ffff1b37b08: 0 '\000') at src/tbb/tbb/include/tbb/tbb_machine.h:846
#3  scoped_lock (m=..., this=<synthetic pointer>) at src/tbb/tbb/include/tbb/internal/../spin_mutex.h:93
#4  tbb::internal::concurrent_monitor::notify_relaxed<tbb::internal::predicate_leq> (this=this@entry=0x7ffff1b37b08, predicate=...) at src/tbb/tbb/src/tbb/concurrent_monitor.h:223
#5  0x00007ffff798177f in notify<tbb::internal::predicate_leq> (predicate=..., this=0x7ffff1b37b08) at src/tbb/tbb/src/tbb/concurrent_monitor.h:181
#6  tbb::internal::concurrent_queue_base_v3::internal_push (this=this@entry=0x145d368, src=src@entry=0x7ffee57f9530) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:403
#7  0x00000000004a0840 in push (source=..., this=0x145d368) at ../src/tbb/tbb/include/tbb/concurrent_queue.h:263
 

Consumer thread:

Thread 33 (Thread 0x7fff7ffff700 (LWP 9092)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff7981a3b in futex_wait (comparand=2, futex=0x7fff7fffe960) at src/tbb/tbb/include/tbb/machine/linux_common.h:67
#2  P (this=0x7fff7fffe960) at src/tbb/tbb/src/tbb/semaphore.h:212
#3  commit_wait (thr=..., this=<optimized out>) at src/tbb/tbb/src/tbb/concurrent_monitor.h:152
#4  commit_wait (thr=..., this=<optimized out>) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:406
#5  tbb::internal::concurrent_queue_base_v3::internal_pop (this=this@entry=0x145d368, dst=dst@entry=0x7fff7fffea20) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:426
#6  0x00000000004a274b in pop (destination=..., this=<optimized out>) at ../src/tbb/tbb/include/tbb/concurrent_queue.h:269
 

Is this a known bug? Or is the problem likely to be in my code?

I am using TBB 4.1 (6100) on a Linux 64-bit machine.

[MLema2]

I suggest to never change thread priorities when using TBB..  We experienced a lot of stalling/spinning issues with this and went back to normal priority threads to see them disappear.

[Kirill_R_Intel]

It is possible that OS just doesn't give the low-priority producer a chance to execute. If there are more active threads with normal priorities than CPU cores, they consume CPU resources and low-priority thread starves. in this case, it would be not a deadlock, but very very slow progress. The consumers just wait for an item to appear in queue, the producer waits for CPU time slice to put an item.

[Francis_R_1]

Thanks for the replies.

It turned out to not be a deadlock but very long stalls. Revering to default priorities removed the problem.

[RafSchietekat]

What you could also do, I suppose, is to just use a plain mutex with a plain queue. Then the normal-priority consumer wouldn't starve the low-priority producer for CPU time by just spinning most of the time. The queue wouldn't be as scalable, but you're not using that scalability anyway. Of course, that assumes that the different priorities were used for a good reason, and that you're not entirely happy using equal priorities as a workaround.

(I wonder whether it's possible for the queue to tune itself. Maybe it should spin the same number of times as last time plus a random number, and keep a crude statistic of the time it then blocks: if that time is more or less constant, it becomes preferable to block earlier; if the total time is more or less constant, it just depends on how long that total time typically is.)