lazy pipeline

[nagy]

I have a producer/consumer scenario where the producer is implemented as a tbb::pipeline. However due to memory constraints the pipeline isn't allowed to produce all items at once and has to wait for the consumer to consumer items.

I previously solved this by blocking the pipeline output on the master thread (seehttp://software.intel.com/en-us/forums/showthread.php?t=73642&o=a&s=lr). However this solution has some flaws,over-subscriptionand blocking. The output blocking seems to be indicated as a bottleneck by Intel Parallel Amplifier 2011 BETA.
I have now revised the general design of my system and would like to have some advice how to adapt the tbb::pipeline to this.

Instead of having a buffer (implemented with tbb::concurrent_bounded_queue) and do blocking pushes. I now use a reactive approach where I can register a callback which is called as soon as an item is removed/consumed from the buffer, and thus avoid the need of the producers being implemented as active objects and instead invoke task::enqueue through callbacks.

What I am looking for is something like:

pipeline starts n tokens with task::enqueue (where n is the size of the buffer)

item is popped from buffer

callback is invoked which starts a token in pipeline with task::enqueue

This way I won't need an additional thread to call pipeline::run from and which is used in processing and thus causing over-subscription. I also avoid most blocking.

Is there anyway to do this with the current implementation of tbb::pipeline or will it require a new implementation?

EDIT:

On a side note.

I remember that there was a list of projects using tbb. I would like to add casparcg (www.casparcg,com) to this list as we are soon in the process of using tbb in our stable versions and have been generally very satisfied with this library so far. Where can I find this list?

[Alexey-Kukanov]

For the list of projects, is the following what you look for?
User Success Page Available:Check out who is having success using TBBhere. If you have something you would like to say about TBB let us knowby emailing us attbb-users@lists.sourceforge.net.

I don't see how the pipeline, as of now,can be used inyour new design. Pipeline implementation still uses "normal" tasks, and it can only be started with the blocking run() call. However, please take a look at the attached file - it contains a sketch of a pipeline-like setup based on task::enqueue, std::priority_queue to pass the work from the main thread to workers, and concurrent_queue to pass results back to the main thread. The main idea is that tasks serve as "worktime slots" but do not carry any data; instead, queues are used to communicate and prioritize pieces of data. I think it can be adapted to your purpose; at least, seems worth a try.