Trading App - pipeline vs parallel_for

[nevadalife]

Hi,

I'm trying to implement a trading app that parses and outputs market data and sends orders.

I'm debating on how to implement this with tbb. The app receives market data from vendor and it parses and processes the data and sends data to client processes. Few requirements

1. The data for each ticker symbol has to be processed in order, but symbols can be processed in parallel.
2. There are several stages, receive data from socket, parse the data, normalize the data and then output on socket.

So I was thinking of using multiple tbb:pipeline or using simpler parallel_for or some other tbb pattern?

Any ideas on how to organize this better with tbb?

Thanks,
David

[Alexey-Kukanov]

If you have a priori knowledge of how much data to process (e.g. all ticker symbols of interest are stored in an array), you may use parallel_for to process each piece of data (i.e. array element) independently of others.

If you do not have such a knowledge, pipeline might be a better fit.

Another possibility, which can be a better fit if the "receive data from socket" part may cause significant delays: the main thread receives data from socket and then enqueues tasks that process the data. The disadvantage is that it requires direct manipulation with tasks and so is not as convenient to use as pre-defined patterns.

UPDATE: an additional thought occurred to me: tbb::graph, currently a community preview feature, may help to organize the computation similarly to the pipeline, but more friendly to I/O delays. However, graph's API will likely change as we finalize it as a fully supported feature.