How is the user worker thread returned when returning from cilk runtime?

[Suyeon_K_]

Hi

In the Binary Interface Specification, it says when the last user thread calls __cilkrts_leave_frame() with a __cilkrts_stack_frame which has CILK_FRAME_LAST set in the flags field, the runtime will suspend all of the worker threads it created."

I have a question about when the stack frame set as CILK_FRAME_LAST gets stolen by the system worker.

If it is possible, the user worker does not have a stack frame set as CILK_FRAME_LAST, and the stack frame stolen to the system worker will be changed to CILK_FRAME_STOLEN. Then, calling __cilkrts_c_return_from_initial() on the user worker is not possible because it does not have CILK_FRAME_LAST frame any more.

How does cilk runtime ensure the user worker thread keeps the stack frame set as CILK_FRAME_LAST and returns from that frame when _cilkrts_leave_frame() is called and exits the Cilk runtime?

Thank you,

Suyeon

[Jim_S_Intel · Accepted solution]

The short answer is that a frame marked as CILK_FRAME_LAST can be stolen away, but only the user worker is allowed to resume after a cilk_sync in that frame.   Thus, the frame does get "returned" to the correct worker thread before that frame completes.   
If you haven't already, I'd check some of the comments in the runtime source related to "sync_master" and "push_next_frame", to see if that clarifies anything.

Cheers,

Jim