please be aware of the following:
1) UMC H.264 Encoder doesnt process RGB format, it works with YUV, specifically YV12 or i420.
2) The product you get from UMC H.264 encoder is a video only stream, means that its not multiplexed yet, though most players wouldnt play it.
a- to examine the success of your encoded video, UMC simple_player will play it for you.
b- Multiplex (mux) your H.264 video into an avi or mp4 container, then it is playable by any media player supports H.264 video