Compositing images

[Aditya_Kadambi]

This is a theory question!

My objective is to add annotations to a video frame.

Say, I take a video frame, and want to add text to it.

My decoded frame is in YUY2 space. This is what I am thinking I should do.

1. Convert YUY2 frame to RGB32.

2. The "text image" which is in RGB32 will now be overlaid/composited on the RGB32 video frame.

3. Convert the composited image to YUY2 and send it to encoder.

Is my approach correct? What are the best IPP functions to accomplish compositing. I already know how to convert between color spaces.

[SergeyKostrov]

Please take a look at ippiDrawText_8u_xxx function(s).

[Aditya_Kadambi]

Hi Sergey;

I could not find that function at all. I searched the IPP "Image and Video Processing Manual" for IPP 7.1. I did a general google search. I even searched from the search facility in this forum.

I searched for "ippiDrawText" as well as "ippiDrawText_8u". I could not find any results!

[Gennady_F_Intel]

I am not sure, but may be ippiAdd_*. will help you to compose both of these images.

[SergeyKostrov]

>>...I searched for "ippiDrawText" as well as "ippiDrawText_8u". I could not find any results!.. You're right and the function was deleted in some version of IPP, maybe in v5 or in v6. What I see that it exists in older versions of IPP. So, do you want to preseve a background of a source image?

[Aditya_Kadambi]

>>"You're right and the function was deleted in some version of IPP, maybe in v5 or in v6. What I see that it exists in older versions of IPP.

>>So, do you want to preseve a background of a source image?"

Hi Sergey;

Yes. The way it could be implemented is that the background image would be "as is". I would paste this "text image" or "speech bubble" or anything on top of it. In this operation, the part of the background image where there is no "speech bubble" not "text" would be as is. Otherwise, the "text" or "speech bubble" would show up on top.

I hope I made sense!

[SergeyKostrov]

The following set of IPP functions could be used on a final "composing" step of your processing: ... IPPAPI( IppStatus, ippiOr_8u_C1R, (const Ipp8u* pSrc1, int src1Step, const Ipp8u* pSrc2, int src2Step, Ipp8u* pDst, int dstStep, IppiSize roiSize) ) ... IPPAPI( IppStatus, ippiOrC_8u_C1R, (const Ipp8u* pSrc, int srcStep, Ipp8u value, Ipp8u* pDst, int dstStep, IppiSize roiSize) ) ... Note: It is assumed that a second image with, for example bubbles and empty background ( zeros ), already created (!). Let me know if you need a generic information on how to add some annotations with GDI, GDI+ or DirectDraw APIs.

[Aditya_Kadambi]

Hi Sergey;

Many thanks!

Yes, the second image will just have speech bubble with empty background (all zeros). It will be already created. At least for my first cut, I am not worried about how such image will be generated.

Now, I can start my project. I will post updates on my progress here!

[SergeyKostrov]

>>This is a theory question! >> >>My objective is to add annotations to a video frame... I used an ImageJ application to "model" composition of two images ( it took just a couple of minutes ) and this is how it looks like: I hope that it could help you.

[Aditya_Kadambi]

Thanks Sergey;

In this your "empty pixels" were white, right? and the image to be shown was black (black dots)?

I have started coding with black as empty with the "Or" operation and will report back.

ps: This is also like image layering operation in GIMP, right?

[Aditya_Kadambi]

Hi Sergey;

I am trying to do the same in ImageJ. I can OR an image with a constant. How did you OR these two images? (or whatever operation it is that you did?)

Which brings an interesting question.

What if my Text is in black and the rest of the "empty" pixels is white? In this case, some sort of AND operation?

[SergeyKostrov]

>>...How did you OR these two images?... Take a look at Process -> Image Calculator menu item. In my example AND operation of Image Calculator was used because the background was white. OR operation of Image Calculator needs to be used if the beackground is black ( as discussed before ).

[SergeyKostrov]

>>...In this your "empty pixels" were white, right? and the image to be shown was black (black dots)? No. Background of the image in the middle is filled with white color and black dots ( bubbles ) created after that. In overall, you see it exactly and no more transformations / corrections were done in my modelling test. I see that depending on a brightness of a background of the source image different logical operations need to be used and a histogram of the image could help here. Also, I would really consider application of DirectDraw interfaces because they perform these procedures easily and there are examples in DirectX SDK. Please take a look if you have time.

[Aditya_Kadambi]

Awesome Sergey!

I was able to composite with OR operation when the background was black and with AND operation when the background was white.

I will certainly look at directdraw interfaces as well.

I will report back progress as I work on this project.

[Aditya_Kadambi]

Awesome Sergey!

I was able to composite with OR operation when the background was black and with AND operation when the background was white.

I will certainly look at directdraw interfaces as well.

I will report back progress as I work on this project.

[SergeyKostrov]

>>... I would really consider application of DirectDraw interfaces because they perform these procedures easily and >>there are examples in DirectX SDK... [ Correction ] There is also a set of Multimedia samples in Microsoft Platform SDK. Take a look at these two: 1. [ Platform SDK dir ]\Samples\Multimedia\DirectShow\VMR\Ticker Here is a short description: ... Using the Windows XP Video Mixing Renderer, a static image is alpha blended with the image in the corner of the screen. The image moves from right to left near the bottom of the screen. ... and 2. [ Platform SDK dir ]\Samples\Multimedia\DirectShow\VMR9\Ticker Here is a short description: ... Using the DirectX 9 Video Mixing Renderer, a static image is alpha blended with the image in the corner of the screen. The image moves from right to left near the bottom of the screen. ...

[Aditya_Kadambi]

Really appreciate all the help Sergey.

[Aditya_Kadambi]

Hi Sergey;

An update.

I am targeting Windows 7 only and am also looking into Direct2D. I am wary of using directshow as it is being deprecated. I will let you know if I find something similar in Direct2D.

Interestingly Direc2D on Windows 8 offers custom effects as well as easy blending. But not on Win 7.

[SergeyKostrov]

>>...Interestingly Direc2D on Windows 8 offers custom effects as well as easy blending. But not on Win 7... Thanks and I did a quick review of the API at: msdn.microsoft.com/en-us/library/windows/desktop/dd370987(v=vs.85).aspx However, I see on an Architectural Layering of Direct2D diagram that the API is based on Direct3D and it means that it has to be supported ob Windows 7 operating systems ( OS ). I don't think Microsoft would remove support for one of the most popular Desktop OS at the moment and designed the Direct2D just for Windows 8.

[Aditya_Kadambi]

Hi Sergey;

More progress.

I updated windows 7 with "Windows 7 platform update" and now I have all the functionality Direct2D functionality including blending and builtin effects.

I will report back on the API the I will use and progress.