One possible solution is what we use to solve a similar problem inside the TBB scheduler. Heap-allocate the TaskManager object, and make the last task delete it. To prevent premature deletion, initialize m_noofTaskInFlight to 1, not 0. Then when the client is done with the TaskManager object, have it decrement the counter, "as if" it were another task completing (and if the count becomes zero, delete the TaskManager object).
For syntactic convenience, it may be useful to split TaskManager into two parts: a stack-allocated part that the client sees and a heap-allocated part. The destructor for the stack-allocated part can to the "as if" decrement of the counter in the heap-allocated part.