Hi,
I'm working on a message processing application that needs to process list of message with lowest latency but also handle message rates up to 40k per sec. Each message with a certain topic must be processed in order. But different topic messages can be processed in parallel.
Current approach i'm using is to use pipeline in outer loop which has 3 stages - input queuing, parsing message and separating by topic and processing stage. Each filter is run serially and processing filter does parallel_for to process topics in parallel.
This seems to be working but single pipeline is not processing fast enough causing latency on messages in input queue. Would multiple pipeline help where input data is split into 4 and then fed into 4 pipeline. So certain set topics will be processed by different pipeline. This would mean some preparsing would need to be done before hand. Let me know if there are any other suggestions.
David
