possible bug when resizing images with transparency

[David_N_]

Hello,

In the post here, you indicated that IPP resizing supported transparency. For the most part it seems to work okay, but there are problems around the border.

In the attached file "in.png" there are circles that are 100% opaque (alpha 255) and the rest of the image is 100% transparent (alpha value is 0). The pixel values around the circles are black with 0 alpha, and the rest of the transparent area is white with zero alpha.

When you resize the image using IppResizeLinear, based on the resize_mt example and the code here.  to 50% of the size you see a grey border (see attached resize_mt_out.png). Somehow, the black pixels are "bleeding" into the resized image, even though the black pixels are 100% transparent and should not contribute to the final resized image.

In some cases, the borders of the resized circles have some transparency, and become semi-transparent in the original color that they were (black arrow in bleedexample.png) and others are semi transparent with black (red arrow in bleedExample.png).

Is this a bug, or is there something we can do about this? It definately seems to me that pixels that were 100% transparent in the original image should not contribute anything to the resized image.

We saw the same effect with cubic filter as well
 

[Jonghak_K_Intel]

Hi David,

 could you let me know which version of IPP you are using?

 and please describe your development environment, such as OS, your platform information and compiler options you used if applicable.

[BMart1]

Hi David,

You need to premultiply your alpha before resizing the image.

Bruno

[David_N_]

@ Bruno. Thank you for the comments. Are you sure that is the case? Fiona from Intel, in the post here, seems to indicate that premultiplying is necessary for combining (merging) images, but not for resizing.

[BMart1]

I'm 100% sure.  See https://developer.nvidia.com/content/alpha-blending-pre-or-not-pre​ A web search can give you tons of other articles in the subject. It's rediscovered often.

While you are at it, I also believe you should gamma convert the input image so that the resizing is done with linear light. This however is not as noticeable. I don't bother in my programs.

[David_N_]

HI Bruno,

Thanks again for the comments. What I meant was whether or not IPP resize methods do the pre- multiplication for you.

Fiona wrote that

If you point to use premultiplied alpha that would be used for alpha composition function that when two 4 channel images cover each other, you probably need to calculate alpha(A)*RGB(A)+alpha(B)*RGB(B). But other functions, like resize, the size of alpha also need to be changed by interpolation method. it is not using premultiplied alpha, it process with alpha channel. 

Based on that, I thought that for iPP composition functions you needed to premultiply, but iPPresize did whatever premultiplication was necessary as part of the filtering/interpolation process and we did not need to do it manually.

In addition, when composing (merging) images, the IPP docs are fairly clear about how/when to premultiply, but this is not mentioned at all for resizing.

[BMart1]

You can resize a 10x10 image with ipp and by hand to confirm that ipp doesn't treat the alpha channel specially. Remember that ipp contains primitives that you are supposed to assemble into your own pipelines. You may want to operate multiple times with an image. It would be wasteful to pre multiplicate inside each operation, compared with once only at the beginning.

[David_N_]

Bruno, thanks so much for the help.

Ok, before I submit any code I will continue testing with premultiplication. But just to ensure my expectations are correct, can you confirm the following:

I am dealing with RGBA images that either have fully transparent or fully opaque pixels. After premultiplication, the fully transparent pixels should all have values of (0,0,0,0) and the fully opaque pixels will be untouched. Since I am resizing the image, I don't have a "source" and "destination" image as described in your previous link so I presume premultiplication will just do the multiplication on the source image based on its own alpha value.

So if I make a 50x50 image that is all (0,0,0,0) except for a 14 pixel red rectangle in the middle that is (255,0,0,255). This is starting off as effectively "premultiplied". Is that correct?

Then I shrink that by 3 in each dimension (i.e. ippResize to 1/3 the size). I would naively expect that inside the shrunk rectangle it would be (255,0,0,255). However, since 14 does not divide equally by 3, I would expect pixels on the border to be semi-transparent. On the border of the shrunk rectangle it would be (255,0,0,x) where x is the transparency of that pixel as determined by whatever filter I was using. It should not vary in the RGB portion, which should always be (255,0,0) but may have different values in the alpha pixel depending on the filter.

Is that the proper expectation for the behavior?

[BMart1 · Accepted solution]

I am dealing with RGBA images that either have fully transparent or fully opaque pixels. After premultiplication, the fully transparent pixels should all have values of (0,0,0,0) and the fully opaque pixels will be untouched.

Correct.

Since I am resizing the image, I don't have a "source" and "destination" image as described in your previous link so I presume premultiplication will just do the multiplication on the source image based on its own alpha value.

Correct. Note that before saving the resized image back to .png you will have to undo the premultiplication.

So if I make a 50x50 image that is all (0,0,0,0) except for a 14 pixel red rectangle in the middle that is (255,0,0,255). This is starting off as effectively "premultiplied". Is that correct?

Correct.

Then I shrink that by 3 in each dimension (i.e. ippResize to 1/3 the size). I would naively expect that inside the shrunk rectangle it would be (255,0,0,255).

Correct

However, since 14 does not divide equally by 3, I would expect pixels on the border to be semi-transparent.

Actually, proper downsampling with the antialiasing resize functions will read several input pixels to generate each output pixel. Normally linear resizing uses two pixels to generate each output pixel. Since you are shrinking by a factor of 3, ResizeAntialiasingLinear will read 6 input pixels. You can avoid this by having your non transparent pixels be a multiple of 3 and also start at a multiple of 3 and disabling antialiasing or using supersampling.  It won't look as good.

On the border of the shrunk rectangle it would be (255,0,0,x) where x is the transparency of that pixel as determined by whatever filter I was using. It should not vary in the RGB portion, which should always be (255,0,0) but may have different values in the alpha pixel depending on the filter.

No. Border pixels will be (x, 0, 0, x).

[David_N_]

No. Border pixels will be (x, 0, 0, x).

Is that because after resizing, I have a "premultiplied, resized" image. If I want to display that to a user, I have to un-premultiply it. Doing that will divide by R value to be x/x hence back to 255 all the time. Therefore, in my "real" resized image, I only have red pixels with varying degrees of transparency?

Is there an IPP function to un-premultiply an image?

[BMart1]

Yes, resizing a premultiplied image produces a premultipied image. I don't know how to un-premultiply with ipp.

[David_N_]

@Bruno,

Thank you for your help. Based on the discussion above we got everything working and there does not seem to be any bug. It is kind of non-intuitive that you need to pre-multiply to resize, and end up with a premultiplied image, and you can also premultiply to compose, but then you end up with a "normal" image.  I guess to unpremultiply you need to compose the image against another pre-multiplied image pixels that are (0,0,0,0), or do it yourself.

[BMart1]

I'm glad you got it working. I think that when you compose you also get a pre multiplied image, since the table at https://software.intel.com/en-us/node/503865 has no division. I hope someone from Intel reads this and answers how to convert to straight alpha/

[David_N_]

We did the following tests:

I took an opaque red background (255,0,0,255) and blended unpremul green (0,255,0,128) onto it:

• GDIPlus gives (127,128,0,255)
• IPP Alphacomp with ippAlphaOver gives (127,128,0,255)

Next, I took an opaque red background (255,0,0,255) with premul green (0,128,0,128):

• IPP Alphacomp with ippAlphaOver gives (127,64,0,255)
• IPP AlphaComp with ippAlphaOverPremul gives (127,128,0,255)

So it seems that when you take premultiplies background composed with a premultiplied foreground and call Alphacomp with ippAlphaOverPremul  you get the correct "non-premultiplied" image. At least that was our interpretation.

[BMart1]

Since the result has alpha=255, you cannot tell if it's pre multiplied or not. Try a (128, 0, 0, 128) background and a (0, 128, 0, 128) foreground with ippAlphaOverPremul. You should get (64, 128, 0, 192).

[David_N_]

Hi Bruno,

If you do (128,0,0,128) background and (0,128,0,128) foreground with ippAlphaOverPremul you get (64, 128, 0, 192)

If you do (256,0,0,128) background and (0,256,0,128) foreground with ippAlphaOver you also get (64, 128, 0, 192)

Strangely if you do the same in GDI+ ( semi red background (255,0,0,128) and blended unpremul green (0,255,0,128)) you get (85,170,0,192). I guess GDI+ is doing some extra processing..

[BMart1]

85 = 64*256/192

170 = 128*256/192

GDI+ is converting the result back to straight alpha.

[David_N_]

@Bruno,

I thought that (64, 128, 0, 192) is the non-premultiplied result.

If you take (256,0,0,128) background and (0,256,0,128) foreground and do the math by hand, you get (64, 128, 0, 192)

If you take (256,0,0,128) background and (0,256,0,128) foreground (both not premultiplied) and use AlphaComp with ippAlphaOver you also get (64, 128, 0, 192), again, seeming to confirm that (64, 128, 0, 192) is the not premultiplied result.

If you pass in the premultiplied version ( (128,0,0,128) background and (0,128,0,128) foreground) to AlphaComp with ippAlphaOverPremul you also get (64, 128, 0, 192), so we interpreted that AlphaComp with ippAlphaOverPremul is also returning the non-premultiplied result of (64, 128, 0, 192).

By "straight alpha" do you mean non-premultiplied alpha? If so, does that mean that you think that (64, 128, 0, 192) is a premultiplied value? If so, then I am a little confused why AlphaComp with ippAlphaOver with non-premultiplied inputs returns a premultiplied result.

[BMart1]

Yes, (64, 128, 0, 192)​ is premultiplied. You can confirm that AlphaComp returns a premultiplied image by using a (0,0,0,0) background. Then ippAlphaOver ​just premultiplies.

[David_N_]

@Bruno,

You are correct. Amazing... AlphaComp always produces pre-multiplied results. The documentation on Alpha Composition could be improved in this regard, especially for beginners. While it lists the equations, it would be helpful to mention that these equations result in getting pre-multiplied results. I guess if I was an image processing wizard, I would have recognized that.

I guess the fact that running AlphaComp against a completely opaque background results in an opaque image, so the fact that the opaque image is actually "premultiplied" is irrelevant since for opaque images pre-multiplied and native-alpha are the same.

I guess that is why there is no un-premultiply since you just AlphaComp against some opaque background to get back to native-alpha. Although if trying to save as PNG or just inspect the values of a blending operation in a "native alpha" it would be useful. I will ask this as a separate thread

Thank you so much for your help. We never would have figured all this out on our own.

[BMart1]

Glad to help.  I only recently learned this.  I'm only giving back.