- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am debugging an issue in my code where I have a concurrent_bounded_queue two producers threads and one consumer thread. The consumer has default thread priority and the producers have low thread priority.
Sometimes I get a deadlock with these stack traces:
Thread 66 (Thread 0x7ffedffff700 (LWP 9126)):
#0 0x00007ffff2025737 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007ffff7981f4d in pause (this=<synthetic pointer>) at src/tbb/tbb/include/tbb/tbb_machine.h:333
#2 __TBB_LockByte (flag=@0x7ffff1b37b08: 0 '\000') at src/tbb/tbb/include/tbb/tbb_machine.h:846
#3 scoped_lock (m=..., this=<synthetic pointer>) at src/tbb/tbb/include/tbb/internal/../spin_mutex.h:93
#4 tbb::internal::concurrent_monitor::notify_relaxed<tbb::internal::predicate_leq> (this=this@entry=0x7ffff1b37b08, predicate=...) at src/tbb/tbb/src/tbb/concurrent_monitor.h:223
#5 0x00007ffff798177f in notify<tbb::internal::predicate_leq> (predicate=..., this=0x7ffff1b37b08) at src/tbb/tbb/src/tbb/concurrent_monitor.h:181
#6 tbb::internal::concurrent_queue_base_v3::internal_push (this=this@entry=0x145d368, src=src@entry=0x7ffedfffe530) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:403
#7 0x00000000004a0840 in push (source=..., this=0x145d368) at ../src/tbb/tbb/include/tbb/concurrent_queue.h:263
Thread 64 (Thread 0x7ffee57fa700 (LWP 9124)):
#0 0x00007ffff2025737 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007ffff7981f4d in pause (this=<synthetic pointer>) at src/tbb/tbb/include/tbb/tbb_machine.h:333
#2 __TBB_LockByte (flag=@0x7ffff1b37b08: 0 '\000') at src/tbb/tbb/include/tbb/tbb_machine.h:846
#3 scoped_lock (m=..., this=<synthetic pointer>) at src/tbb/tbb/include/tbb/internal/../spin_mutex.h:93
#4 tbb::internal::concurrent_monitor::notify_relaxed<tbb::internal::predicate_leq> (this=this@entry=0x7ffff1b37b08, predicate=...) at src/tbb/tbb/src/tbb/concurrent_monitor.h:223
#5 0x00007ffff798177f in notify<tbb::internal::predicate_leq> (predicate=..., this=0x7ffff1b37b08) at src/tbb/tbb/src/tbb/concurrent_monitor.h:181
#6 tbb::internal::concurrent_queue_base_v3::internal_push (this=this@entry=0x145d368, src=src@entry=0x7ffee57f9530) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:403
#7 0x00000000004a0840 in push (source=..., this=0x145d368) at ../src/tbb/tbb/include/tbb/concurrent_queue.h:263
Consumer thread:
Thread 33 (Thread 0x7fff7ffff700 (LWP 9092)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007ffff7981a3b in futex_wait (comparand=2, futex=0x7fff7fffe960) at src/tbb/tbb/include/tbb/machine/linux_common.h:67
#2 P (this=0x7fff7fffe960) at src/tbb/tbb/src/tbb/semaphore.h:212
#3 commit_wait (thr=..., this=<optimized out>) at src/tbb/tbb/src/tbb/concurrent_monitor.h:152
#4 commit_wait (thr=..., this=<optimized out>) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:406
#5 tbb::internal::concurrent_queue_base_v3::internal_pop (this=this@entry=0x145d368, dst=dst@entry=0x7fff7fffea20) at src/tbb/tbb/src/tbb/concurrent_queue.cpp:426
#6 0x00000000004a274b in pop (destination=..., this=<optimized out>) at ../src/tbb/tbb/include/tbb/concurrent_queue.h:269
Is this a known bug? Or is the problem likely to be in my code?
I am using TBB 4.1 (6100) on a Linux 64-bit machine.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I suggest to never change thread priorities when using TBB.. We experienced a lot of stalling/spinning issues with this and went back to normal priority threads to see them disappear.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It is possible that OS just doesn't give the low-priority producer a chance to execute. If there are more active threads with normal priorities than CPU cores, they consume CPU resources and low-priority thread starves. in this case, it would be not a deadlock, but very very slow progress. The consumers just wait for an item to appear in queue, the producer waits for CPU time slice to put an item.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the replies.
It turned out to not be a deadlock but very long stalls. Revering to default priorities removed the problem.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What you could also do, I suppose, is to just use a plain mutex with a plain queue. Then the normal-priority consumer wouldn't starve the low-priority producer for CPU time by just spinning most of the time. The queue wouldn't be as scalable, but you're not using that scalability anyway. Of course, that assumes that the different priorities were used for a good reason, and that you're not entirely happy using equal priorities as a workaround.
(I wonder whether it's possible for the queue to tune itself. Maybe it should spin the same number of times as last time plus a random number, and keep a crude statistic of the time it then blocks: if that time is more or less constant, it becomes preferable to block earlier; if the total time is more or less constant, it just depends on how long that total time typically is.)
![](/skins/images/45C6C2D737ED71F4C51F0145C8CB1E9C/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page