Function Request - sub-pixel 2D translation

[adi_shavit]

Hi,

I'd like to see an optimized new function to translate an image with subpixel accuracy (dx,dy). Today I have to use the warpAffine (or rotate/shear) to get the same effect.
These functions do more work than I actually need (they use a warping model with more parameters e.g. angle).

This can be implemented in several ways using interpolations, convolutions etc.

Thanks,
Adi

[Vladimir_Dudnik]

Hi Adi

Is ippiResize function doesn't do what you want?

Regards,

Vladimir

[adi_shavit]

Hi,

Grrr... my (long) reply from 2 days ago didn't go through.

Basically what I said, was that ippiResize does not let me do what I want. I want ONLY sub-pixel translation, with NO scaling.
ippiResize will give me whole pixel translation when the scale is 1, but I can get this with ippiCopy. (The same goes for ippiResizeShift)

Even if it did, it also does extra work which is un-nessecary for 2D translation.

Adi

[Vladimir_Dudnik]

Hi

O-h, sorry, I didn't look into the problem deeper. It seems you need to take a look on ippiCopySubpix function family (declared in ippcv.h file).

Code:

/*F///////////////////////////////////////////////////////////////////////////////////////
//  Name: ippiCopySubpix_8u_C1R,              ippiCopySubpix_16u_C1R,
//        ippiCopySubpix_8u16u_C1R_Sfs,       ippiCopySubpix_16u32f_C1R,
//        ippiCopySubpix_8u32f_C1R,           ippiCopySubpix_32f_C1R
//
//  Purpose:   copies source image to destination image with interpolation
//
//  Returns:
//    ippStsNoErr              Ok
//    ippStsNullPtrErr         One of pointers is NULL
//    ippStsSizeErr            The width or height of images is less or equal zero
//    ippStsStepErr            The steps in images are too small
//    ippStsNotEvenStepErr     Step is not multiple of element.
//
//  Parameters:
//    pSrc                     Pointer to source image
//    srcStep                  Step in source image
//    pDst                     Pointer to destination image
//    dstStep                  Step in destination image
//    roiSize                  Source and destination image ROI size.
//    dx                       x coeff of linear interpolation
//    dy                       y coeff of linear interpolation
//    scaleFactor              Output scale factor, >= 0
//
//  Notes:
//F*/

IPPAPI(IppStatus, ippiCopySubpix_8u_C1R,        (const Ipp8u* pSrc, int srcStep,
                  Ipp8u* pDst, int dstStep, IppiSize roiSize, Ipp32f dx, Ipp32f dy))

Regards,
Vladmir

[adi_shavit]

Hi Vladmir,

Bingo! That's exactly what I wanted.

This only exists in Beta 5.0.
I haven't noticed it as it is undocumented except in the header file and additions file.

Can you explain more thoughroly what the difference is between ippiCopySubpix*() and ippiCopySubpixIntersect*()?

Thanks,
Adi

[Intel_C_Intel]

IPP contains ippiCopySubpix and ippiCopySubpixIntersect functions to extract image part with subpixel precision. Bilinear interpolation is used. The difference between functions is that ippiCopySubpix consider all pixels necessary to calculate destination pixels are in the memory but ippiCopySubpixIntersect in case that some of 4 src pixels necessary to calculate a dst pixel are outside the src image calculates a dst pixel using nearest 2 or 1 pixels of src boundary.

[adi_shavit]

Hi,

Thanks for the answer.
However, I think a sample program will be best to explain how to use these functions.

I'm trying to compare it's usage with iplRotate() and I'm getting inconsistent results.

Can you provide a sample program that shows how these 2 functions work on a ROI inside an image?
What I want to do is answer this question:

* Given an image 'src', I'd like to know what it would look like, inside some ROI, AFTER I move it by, say, dx=10.5 dy=-3.2.

How can I implement this with ippiCopySubpix() and ippiCopySubpixIntersect() ?

Also, do you plan to have 3 channel versions?

Thanks,
Adi

[Intel_C_Intel]

Adi,

There is a picture 4-4 in Volume 2 of IPP manual (two arrows have wrong beginning there). Actions for this picture are (src[6,8],dst[4,7]):

IppiSize roi={8,6};

IppiPoint_32f point={5.6,-0.3};

IppiPoint min, max;

ippiCopySubpixIntersect_8u_C1R(src, 8, roi, dst, 7, point, &min, &max);

dst[0,0]= dst[1,0]=0.4*src[0,2]+0.6*src[0,3]

dst[0,1]= dst[1,1]=0.4*src[0,3]+0.6*src[0,4]

dst[0,2]= dst[1,2]=0.4*src[0,4]+0.6*src[0,5]

dst[0,3]= dst[1,3]=0.4*src[0,5]+0.6*src[0,6]

dst[0,4]= dst[1,4]=0.4*src[0,6]+0.6*src[0,7]

dst[0,5]= dst[1,5]= src[0,7

dst[0,6]= dst[1,6]= src[0,7

dst[2,5]= dst[2,6]=0.8*src[0,6]+0.2*src[1,6]

dst[3,5]= dst[3,6]=0.8*src[1,6]+0.2*src[2,6]

dst[2,0]=0.32*src[0,2]+0.48*src[0,3]+ 0.08*src[1,2]+0.12*src[1,3]

dst[2,1]=0.32*src[0,3]+0.48*src[0,4]+ 0.08*src[1,3]+0.12*src[1,4]

dst[2,2]=0.32*src[0,4]+0.48*src[0,5]+ 0.08*src[1,4]+0.12*src[1,5]

dst[2,3]=0.32*src[0,5]+0.48*src[0,6]+ 0.08*src[1,5]+0.12*src[1,6]

dst[2,4]=0.32*src[0,6]+0.48*src[0,7]+ 0.08*src[1,6]+0.12*src[1,7]

dst[3,0]=0.32*src[1,2]+0.48*src[1,3]+ 0.08*src[2,2]+0.12*src[2,3]

dst[3,1]=0.32*src[1,3]+0.48*src[1,4]+ 0.08*src[2,3]+0.12*src[2,4]

dst[3,2]=0.32*src[1,4]+0.48*src[1,5]+ 0.08*src[2,4]+0.12*src[2,5]

dst[3,3]=0.32*src[1,5]+0.48*src[1,6]+ 0.08*src[2,5]+0.12*src[2,6]

dst[3,4]=0.32*src[1,6]+0.48*src[1,7]+ 0.08*src[2,6]+0.12*src[2,7]

min == {2,0}

max == {3,4}

ippiCopySubpixIntersect allow for cutting the centered window from the image even if the window does not placed inside the image (eg while optical flow calculating). So here the coordinates of the window center are passed as arguments.

ippiCopySubpix simply copies image with subpixel precision and does not checks for image boundaries. For the same image:

roi1={3,2};

dx=0.6;

dy=0.8;

ippiCopySubpix_8u_C1R(src+10, 8, roi1, dst, 7, dx, dy);

dst[0,0]=0.32*src[1,2]+0.48*src[1,3]+ 0.08*src[2,2]+0.12*src[2,3]

dst[0,1]=0.32*src[1,3]+0.48*src[1,4]+ 0.08*src[2,3]+0.12*src[2,4]

dst[0,2]=0.32*src[1,4]+0.48*src[1,5]+ 0.08*src[2,4]+0.12*src[2,5]

dst[1,0]=0.32*src[2,2]+0.48*src[2,3]+ 0.08*src[3,2]+0.12*src[3,3]

dst[1,1]=0.32*src[2,3]+0.48*src[2,4]+ 0.08*src[3,3]+0.12*src[3,4]

dst[1,2]=0.32*src[2,4]+0.48*src[2,5]+ 0.08*src[3,4]+0.12*src[3,5]

Alexander

[adi_shavit]

Thanks. I'll look into it.

Is the interpolation implemented using convolutions?

The sub-pixel bilinear interpolation can be achieved with a 2D seperable convolution.

Adi

[Intel_C_Intel]

Here the interpolation is calculated directly using 4, 2 or 1 nearest pixels without extra function calls. But of course the bilinear interpolation formula can be represented as the convolution with 2x2 kernel.

I'll treat you question on 3 channelsubpixel copy as the feature request

Alexander

[adi_shavit]

Hi,

If my tests are correct, then ippiCopySubpix() only does the subpixel part of the translation, i.e. for -11 and -11.

If dx or dy are bigger than or equal to 1 (resp. -1), then the integer value is trucated. This part of the motion has to be added to the src ROI.
I think this has to be stated explicitely in the documentation.

Also, a more general wrapper function will be useful, for general morion. Is this what ippiCopySubpixIntersect does?

Adi

I look forward to the 3 channel version.

[Intel_C_Intel]

Hi,

We can not change the function semantic after the release but the description will be made mor presize (that the fractional part is really used).

For general motion ippiRemap function can be used

CopySubpixIntersect uses two ROIs for the source image and for the window. It fills the intersection by interpolating of 4 nearest pixels as CopySubpix and pixels inside the window but outside the source image - by interpolating of 2 or 1 nearest border pixels. It is useful when eg image derivative is averaged by the pixel neighborhood that is not inside the image for pixels near the image border

In CopySubpix ROI describe the copied part and the caller should provide all used pixels of the source image in the memory

Alexander

[adi_shavit]

Hi Alexander,


"We can not change the function semantic after the release but the description will be made mor presize (that the fractional part is really used)."

Well, all I meant was to change the documentation.
BUT, you know, this is a new function of IPP 5.0 Beta, so it would actually be OK to change the semantics to as this is only a beta.
Any existing users would:
a. already know its a beta
b. have been surprised by the fact that it does not do what the docs say it should.

"CopySubpixIntersect uses two ROIs for the source image and for the window. It fills the intersection by interpolating of 4 nearest pixels as CopySubpix and pixels inside the window but outside the source image - by interpolating of 2 or 1 nearest border pixels. It is useful when eg image derivative is averaged by the pixel neighborhood that is not inside the image for pixels near the image border
In CopySubpix ROI describe the copied part and the caller should provide all used pixels of the source image in the memory"


Actually ippiCopySubpixIntersect() is a strange beast, and does not seem to fit nicely with the IPP design. The requirement for a "center" pixel is strange as it requires divisions by 2 and perhaps more related operations internally.
I would expect a ROI based design with e.g. top-left origin makes more sense, as it is how most ippi functions work.

Adi

[Intel_C_Intel]

Hi Adi,

The first about release, 5.0 Beta program is closed that means that 5.0 release is comingand we can't change it.

But I could explain why such semantis was chosen for ippiCopySubpix. The usual way to get the subimage in IPP style is todefine the pointer to the topleft pixel, the ROI that is the size of the subimage and the image step that is the same as for the whole image. This paradigm works OK for pixel granularity and can be generalized to subpixel granularity by different ways. We chose the pointer to the topleft pixel using in interpolation and interpolation coefficients. So the embracing subimage with integer coordinates is defined by CopySubpix arguments.

CopySubpixelIntersect is really the slightly different beast because it is intended to be used in "pixel and its neighborhood" context. It saves the caller that knows the image point coordinates with subpixel precision from calculating coordinates of the topleft neighborhood pixel and checking is the window completely inside the image or not. Optical flow algorithms are just the case. IPP implementation of the optical flowallows for tracking points near the image border or even outside the border.

[adi_shavit]

Hi,

---

akibkal1 wrote:

snip ...But I could explain why such semantis was chosen for ippiCopySubpix. The usual way to get the subimage in IPP style is to define the pointer to the topleft pixel, the ROI that is the size of the subimage and the image step that is the same as for the whole image. This paradigm works OK for pixel granularity and can be generalized to subpixel granularity by different ways. We chose the pointer to the topleft pixel using in interpolation and interpolation coefficients. So the embracing subimage with integer coordinates is defined by CopySubpix arguments.

Yes. I understand this completely.

This makes perfect sense. Whole motion is done via ROIs and sub-pixel is done via interpolation.


However, using this function to perform arbitrary motion is tricky.

It took my a long time to figure it out.



Basically, for e.g. dx = -2.5, my code looks like this:

IplROI roi2 = roi; // This will be src's new roi

roi2.xOffset += (int)dx; // move is by the whole component of dx 



dx=dx-int(dx);  // Get dx's fractional component

if (dx&lt0)

{

   roi2.xOffset -= 1; // if dx is negative, then move src's ROI back by 1

   dx = 1+dx;         // Take the complement component as positive

}



changeRoiLocal crl1(src->roi, &roi2); // Assign ROIs

changeRoiLocal crl2(dst->roi, &roi);  // Assign ROIs

IplImgWrapper srcw(src); // Create IPP adapter for src

IplImgWrapper dstw(dst); // Create IPP adapter for dst

IppStatus res = ippiCopySubpix_8u_C1R(srcw.getData(), srcw.step(),   dstw.getData(), dstw.step(), dstw.size(), dx, dy);

In the code I use some adpters to the IPL image structure.



They are quite straight forward and do not effect my point.



As you can see, this code has to give special tratment for negative motions.



They same would apply to dy.



I wonder if there is a more general way to do this without the branch.



Well, anyway, I look forward to the 3 channel version.



Thanks,



Adi

Message Edited by Adi_Shavit on 09-29-2005 04:50 PM