Optical Flow Calculation samples/information?

[tix]

Hello,

I am currently trying to use the IPP for the calculation of optical flow. I have been using the OpenCV up to now, which was a very convenient way, actually, but I would like to compare the performance of both implementations for my application.

Is there a possibility to use the IPP function from OpenCV to have OpenCV do all the troublesome intialization stuff for me? Or is there a way to directly construct a pyramid structure of the desired depth from an image? Is there any sample code which shows, how Optical Flow calculation is done? If not, can someone maybe help me with my questions? Unfortunately the documentation does not point from the parameters that one function needs as an input to the functions that create and properly initialize those parameters, so that it is difficult to understand which functions have to be called in which order.

Which function calls are actually necessary before I can start OpticalFlowPyrLK? First I initialize the two IppiPyramid* and the IppiOptFlowPyrLK variables using the ippiPyramidInitAlloc and ippiOpticalFlowPyrLKInitAlloc functions. Now I am done with the IppiOptFlowPyrLK data structure but for the IppiPyramid-structure, the documentation says, that the next step is a call to ippiPyramidLayerDownInitAlloc.From that point, it is getting difficult to understand. That function needs a gaussian kernel to be specified, which is probably defined somewhere inside the IPP-library, because there are other functions doing Gaussian filtering too, but there is no information given in that chapter of the documentation, where thekernel can be found or generated.

Theother problem is, how to get my images insidethose pyramids. The functions PyramidLayerDown and PyramidDown do not take the previously created data structures as an input but some IppiPyramidDownState_ instead. Where do I get that from?BasicallyI understand, what the individual functions do, from the documentation, but it is unclear for me, how to make them work together.

I would really appreciate any hints on how the optical flow calculation is done. Thank you in advance

Tix

[Intel_C_Intel]

Hello,

You may look into OpenCV sources :)

See cv/src/lkpyramid.cpp, the function

icvCalcOpticalFlowPyrLK_8uC1R.

There is a fragment, currentlyswitched off using

#if 0 endif. It prepares all the necessary data and calls IPP's optical flow.

Vadim

[Intel_C_Intel]

Hi, Tix,

I add the example of using IPP optical flow in OpenCV environment. It includes image read and show by OpenCV, pyramid building and optical flow calculation for grid points by IPP and pyramid layer show by OpenCV. Points to track are as inside as outside the image. You need to add your own images with fixed names.

Thanks,

Alexander

#include

"highgui.h"

#include

"cxcore.h"

#include

#include

#include

IplImage* img1 = 0, *img2 = 0, *imp = 0, *imn = 0;

void

ownCopy(IppiPyramid *pPyr1, IppiPyramid *pPyr2, int i) {

cvZero(img1);

cvZero(img2);

ippiCopy_8u_C1R(pPyr1->pImage,pPyr1->pStep,(Ipp8u*)(img1->imageData),img1->widthStep,pPyr1->pRoi);

ippiCopy_8u_C1R(pPyr2->pImage,pPyr2->pStep,(Ipp8u*)(img2->imageData),img2->widthStep,pPyr2->pRoi);

cvShowImage(

"image1",img1);

cvShowImage(

"image2",img2);

cvWaitKey(0);

}

void

optflow_IPP( const Ipp8u *prevFrame, // previous frame int prevStep, // its row step const Ipp8u *nextFrame, // next frame int nextStep, // its row step

IppiSize roiSize,

// frame size int numLevel, // pyramid level number (5) float rate, // pyramid rate (2.0f)

Ipp16s *pKernel,

// pyramid kernel int kerSize, // pyramid kernel size (5)

IppiPoint_32f *prevPt,

// coordinates on previous frame

IppiPoint_32f *nextPt,

// hint to coordinates on next frame

Ipp8s *pStatus,

// result indicators

Ipp32f *pError,

// differ ences int numFeat, // point number int winSize, // search window size (41) int numIter, // iteration number (5) float threshold) // threshold (0.0001f)

{

IppiPyramid *pPyr1, *pPyr2;

IppiOptFlowPyrLK *pOF;

ippiPyramidInitAlloc (&pPyr1, numLevel, roiSize, rate);

ippiPyramidInitAlloc (&pPyr2, numLevel, roiSize, rate);

{

IppiPyramidDownState_8u_C1R **pState1 = (IppiPyramidDownState_8u_C1R**)&(pPyr1->pState);

IppiPyramidDownState_8u_C1R **pState2 = (IppiPyramidDownState_8u_C1R**)&(pPyr2->pState);

Ipp8u **pImg1 = pPyr1->pImage;

Ipp8u **pImg2 = pPyr2->pImage;

int *pStep1 = pPyr1->pStep; int *pStep2 = pPyr2->pStep;

IppiSize *pRoi1 = pPyr1->pRoi;

IppiSize *pRoi2 = pPyr2->pRoi;

IppHintAlgorithm hint=ippAlgHintFast;

int i,level = pPyr1->level;

ippiPyramidLayerDownInitAlloc_8u_C1R(pState1,roiSize,rate,pKernel,kerSize,IPPI_INTER_LINEAR);

ippiPyramidLayerDownInitAlloc_8u_C1R(pState2,roiSize,rate,pKernel,kerSize,IPPI_INTER_LINEAR);

pImg1[0] = (Ipp8u*)prevFrame;

pImg2[0] = (Ipp8u*)nextFrame;

pStep1[0] = prevStep;

pStep2[0] = nextStep;

pRoi1[0] = pRoi2[0] = roiSize;

for (i=1; i<=level; i++) {

pPyr1->pImage = ippiMalloc_8u_C1(pRoi1.width,pRoi1.height,pStep1+i);

pPyr2->pImage = ippiMalloc_8u_C1(pRoi2.width,pRoi2.height,pStep2+i);

ippiPyramidLayerDown_8u_C1R(pImg1[i-1],pStep1[i-1],pRoi1[i-1],

pImg1,pStep1,pRoi1,*pState1);

ippiPyramidLayerDown_8u_C1R(pImg2[i-1],pStep2[i-1],pRoi2[i-1],

pImg2,pStep2,pRoi2,*pState2);

ownCopy(pPyr1,pPyr2,i);

}

ippiOpticalFlowPyrLKInitAlloc_8u_C1R (&pOF,roiSize,winSize,hint);

ippiOpticalFlowPyrLK_8u_C1R (pPyr1,pPyr2,prevPt,nextPt,pStatus,pError,

numFeat,winSize,numLevel,numIter,threshold,pOF);

ippiOpticalFlowPyrLKFree_8u_C1R(pOF);

for (i=level; i>0; i--) { if (pImg2) ippiFree(pImg2); if (pImg1) ippiFree(pImg1);

}

ippiPyramidLayerDownFree_8u_C1R(*pState1);

ippiPyramidLayerDownFree_8u_C1R(*pState2);

}

ippiPyramidFree (pPyr2);

ippiPyramidFree (pPyr1);

}

int

main( int argc, char** argv )

{

int c=-1,i,j,nfeat=0,levels=5,nx=20,ny=20,ksize=5,nlev=5,wsize=41,niter=5,shift=200;

float rate=2.0f,dx=0.0f,dy=0.0f,alpha=0.375f,thresh=1.0e-4f; char* filename1 = (char*)"OptFlowTest_Prev.bmp"; char* filename2 = (char*)"OptFlowTest_Next.bmp";

IppiPoint_32f *prev=0, *next=0;

Ipp8s *status=0;

Ipp32f *error=0;

Ipp32f *kernel;

Ipp16s *kernel_16s;

CvSize size;

IppiSize roi;

printf(

"Hot keys: " " ESC - quit the program " " o - calculate optical flow " " any - show next pyramid layer ");

cvNamedWindow(

"image1", 1 );

cvNamedWindow(

"image2", 1 );

img1 = cvLoadImage( filename1, 0 );

img2 = cvLoadImage( filename2, 0 );

size.width = roi.width = img1->width;

size.height = roi.height = img1->height;

cvShowImage(

"image1", img1 );

cvShowImage(

"image2", img2 ); if( c < 0 )

c = cvWaitKey(0);

for (;;) { if (c == 27) break; if (c == 'o') {

imp = cvCloneImage(img1);

imn = cvCloneImage(img2);

nfeat = (nx+1)*(ny+1);

prev = (IppiPoint_32f*)ippsMalloc_32f(nfeat*2);

next = (IppiPoint_32f*)ippsMalloc_32f(nfeat*2);

status = (Ipp8s*)ippsMalloc_8u(nfeat*2);

error = (Ipp32f*)ippsMalloc_32f(nfeat*2);

for (i=0; i<=ny; i++) { for (j=0; j<=nx; j++) {

prev[i*(nx+1)+j].x = (size.width +shift+shift)/(

float)nx*j-shift;

prev[i*(nx+1)+j].y = (size.height+shift+shift)/(

float)ny*i-shift;

next[i*(nx+1)+j].x = prev[i*(nx+1)+j].x + dx;

next[i*(nx+1)+j].y = prev[i*(nx+1)+j].y + dy;

}

}

kernel=(Ipp32f*)malloc(

sizeof(Ipp32f)*ksize);

kernel_16s=(Ipp16s*)malloc(

sizeof(Ipp16s)*ksize); switch (ksize) { case< /FONT> 3:

kernel[1] = alpha;

kernel[0] = kernel[2] = 0.5f*(1.0f - alpha);

break; case 5:

kernel[2] = alpha;

kernel[1] = kernel[3] = 0.25f;

kernel[0] = kernel[4] = 0.5f*(0.5f - alpha);

break; default:

Ipp32f sum = 0;

for (i=0; i

kernel = (Ipp32f)exp(alpha*(ksize/2-i)*(ksize/2-i)*0.5f);

sum += kernel;

}

for (i=0; i

kernel /= sum;

}

}

ippsConvert_32f16s_Sfs(kernel, kernel_16s, ksize, ippRndNear, -15);

const Ipp8u *src=(const Ipp8u*)(imp->imageData); const Ipp8u *dst=(const Ipp8u*)(imn->imageData); int srcStep=imp->widthStep; int dstStep=imn->widthStep;

optflow_IPP(src, srcStep, dst, dstStep, roi,

nlev, rate, kernel_16s, ksize,

prev, next, status, error, nfeat,

wsize, niter, thresh);

for (i=0; i

printf(

"%4d [%8.3f,%8.3f] -> [%8.3f,%8.3f] status %3d error %10.4f ",

i,prev.x,prev.y,next.x,next.y,status,error);

}

if (kernel_16s) free(kernel_16s); if (kernel) free(kernel); if (error) ippsFree(error); if (status) ippsFree(status); if (next) ippsFree(next); if (prev) ippsFree(prev);

}

c = cvWaitKey(0);

}

return 1;

}

[tix]

Thank you very much for your answers. The examples really explainthe working principlevery well. I hope, they will help others too, who have the same difficulties as me. :) .

[Vladimir_Dudnik]

you are welcome:)

[santoscadenas]

Hello:
My name is Jos Antonio Santos Cadenas, and I am making some research about robotics with a research group in Rey Juan Carlos University. The reason I am writing here is because we are using the ipp library in order to calculate the optical flow for the robot to percieve the environment. The problem is that I do not know how prev and next parametres work, so maybe I am using them in a wrong way; I think that this is one the reasons because the optical flow is not calculated. Another reason can be that I am not using OpenCV to capture the images, instead I am obtaining the images from an array with this format:
unsigned char image [numpix*3]; // which contains the image BGR using three unsigned char for each pixel.
My questions are if you could explain me how prev and next parametres exactly work and how could I make my image format be compatible with the one used in the example.
Thank you very much for your help,
Jos Antonio Santos Cadenas.

[Intel_C_Intel]

Hi, Jose,

pPrev const array (src) contains coordinates of points (in the previous image) to calculate the optical flow for. These coordinates are used while manipulating of the pyramid for the previous image.

pNext array (srcdst) contains probable coordinates of corresponding points in the next image. These coordinates are used while manipulating of the pyramid for the next image. The search of the new position ofi-th point starts from pNext point. Search results are returned also in pNext array. .

If you have no information about the probable position of points in the next image you should set pNext elements equal to corresponding pPrev elements. If you have some estimations (eg global motion vector) you can add it to pPrev elements and place to pNext. In this case search starts from point nearer to the result. If you have not initialized pNext search starts from unpredictable points (pNext values) and the result can be unpredictable, too.

Elements of pStatus array contain number of level where search was finished. For example, optical flow calculation for a point outside the image is possible. It is continued while the intersection of the window and the pyramid layer is not empty.

Thanks,

Alexander