If the self-defined pipeline filter object contains a data member and the filter is set to be non-serial, the method operator() tries to write new value to the data member, is there any potential race problem on that data member?
If the filter is set to be serial,is there still any race problem? if not, is it guaranteed to be no?
There is one instance of a particular filter per pipeline; and it is used to process all tokens passing through it. So if a user defines a filter as parallel, he/she should care about making it free of data races; TBB can't help here because it is agnostic of user classes. So the answer to the first your question is Yes.
If the filter is serial, its essense is to process tokens one-by-one. So it is guaranteed that all calls to operator() are properly synchronized (serialized) within the pipeline, simultaneous calls are impossible, and there is no data races between invocations of operator().
However, one might create two instances of the same filter class, and add those to the same or different pipelines. For two separate filter instances, no guarantees like above can be provided, except for instance-specific data. Ifsomedata are shared between instances (such as static variables in the filter class), they can be accessed in parallel from different filter instances, even though the filter is serial.
I hope there should be nothing surprising and the behavior is what you would expect.