I did not analyse your code much, but I think parameters you pass to IPP function cause memory access of out allocated space.
By the way, for RGB to YCbCr420 conversion you need 640x480source buffer and you need output buffers like 640x480 for Y component and two 320x240 for Cb and Cr components.
We are still investigating that issue. It seems that particular version of Intel C/C++ compiler which was used to build IPP 5.3 had (or cause) some throubles with Mac OS Leopard linker. Note, there is no such issue under Mac OS Tiger.
If that will be confirmed we will provide bug fix update for IPP 5.3 libraries.