From Image List to Emotion viewer

Alejandro_S_Intel1 · ‎01-12-2016

I have a bunch of images that where extracted of a .rssdk file. My set of images is almost 66% .jpg images and the rest are .png images that are not visible by the Windows picture manager. I want to feed the Emotion viewer application with this set but I don't have any idea of how handle this situation.

I started by trying to feed only the color Images (and not including the depth images that I suppose are the .png files) in the source code of the emotion viewer application in this way:

PXCSenseManager *putil;

...

putil->QueryEmotionSample()->color->Release();

putil->QueryEmotionSample()->color = g_session->CreateImage(&infoRGB,&dataRGB);

I used a list with the names of the files and I am able to see the sequence perfectly. I also tried to load the depth images and any error were thrown, but the algorithm isn't working. I suspect that the main problem is in the putil->AqcuireFrame() function but I don't have any idea of how to create a synthetic frame with my own set of images and render it to the Sense Manager. Here my code:

void SimplePipeline(HWND hwndDlg) {
   PXCSenseManager *putil= g_session->CreateSenseManager();

   if (!putil) {
       SetStatus(hwndDlg,L"Failed to create an SDK SenseManager");
       return;
   }

   /* Set Mode & Source */
   pxcStatus sts=PXC_STATUS_NO_ERROR;
   PXCCaptureManager *captureMgr = putil->QueryCaptureManager(); //no need to Release it is released with putil
   pxcCHAR* device = NULL;
   if (GetRecordState(hwndDlg)) {
       sts = captureMgr->SetFileName(g_file,true);
       captureMgr->FilterByDeviceInfo(GetCheckedDevice(hwndDlg),0,0);
   } else if (GetPlaybackState(hwndDlg)) {
       sts = captureMgr->SetFileName(g_file,false);
   } else {
       device = GetCheckedDevice(hwndDlg);
       captureMgr->FilterByDeviceInfo(device,0,0);
   }

   if (sts<PXC_STATUS_NO_ERROR) {
       SetStatus(hwndDlg,L"Failed to Set Record/Playback File");
       return;
   }

   bool stsFlag=true;

   /* Set Module */
   putil->EnableEmotion();

   /* Init */
   SetStatus(hwndDlg,L"Init Started");
   MyHandler handler(0,hwndDlg);

   if (putil->Init(&handler) >= PXC_STATUS_NO_ERROR){
       SetStatus(hwndDlg,L"Streaming");
       g_disconnected=false;

       std::fstream jpglist;
       jpglist.open("jpg_list.txt");
       std::string rgb;
       putil->AcquireFrame(true);
       int numFaces = 0;
       while (!g_stop && jpglist >> rgb) {
           if (putil->AcquireFrame(true)<PXC_STATUS_NO_ERROR) break;
           std::string dpth = "D:\\E010802\\E010802\\774694796.png";
           ////////<<<<<<<>>>>>>>>
           IplImage *imageRGB = cvLoadImage(rgb.c_str(),1);//-1 for 4 channels, 1 for 3 channels, 0 1 channel :D
           IplImage *imageDEPTH=cvLoadImage(dpth.c_str(),-1);
           PXCImage::ImageData imageD;

           std::fstream doit;
           doit.open("WeDidIt.txt",std::ios::out|std::ios::app);

           PXCImage::ImageInfo infoRGB, infoDEPTH;
           PXCImage::ImageData dataRGB, dataDEPTH;
           memset(&infoRGB,0,sizeof(infoRGB));
           memset(&dataRGB,0,sizeof(dataRGB));
           memset(&infoDEPTH,0,sizeof(infoDEPTH));
           memset(&dataDEPTH,0,sizeof(dataDEPTH));

           infoRGB.width=640;
           infoRGB.height=480;
           infoRGB.format=PXCImage::PIXEL_FORMAT_RGB24;
           infoDEPTH.width=320;
           infoDEPTH.height=240;
           infoDEPTH.format=PXCImage::PIXEL_FORMAT_RGB32;

           unsigned char *rgb_data, *depth_data;
           int rgb_pitch = imageRGB->widthStep,
               depth_pitch = imageDEPTH->widthStep;
           rgb_data = (unsigned char*)imageRGB->imageData;
           depth_data = (unsigned char*)imageDEPTH->imageData;

           dataRGB.format = PXCImage::PIXEL_FORMAT_RGB24;
           dataRGB.planes[0] = rgb_data;
           dataRGB.pitches[0] = (pxcI32)rgb_pitch;

           dataDEPTH.format = PXCImage::PIXEL_FORMAT_RGB32;
           dataDEPTH.planes[0] = depth_data;
           dataDEPTH.pitches[0] = pxcI64(depth_pitch);

           putil->QueryEmotionSample()->color->Release();
           putil->QueryEmotionSample()->color = g_session->CreateImage(&infoRGB,&dataRGB);

           //putil->QueryEmotionSample()->depth->Release();
           putil->QueryEmotionSample()->depth = g_session->CreateImage(&infoDEPTH,&dataDEPTH);

           //<<<<<<<<<<<>>>>>>>>>>
           if (putil->AcquireFrame(true)<PXC_STATUS_NO_ERROR) break;
           PXCEmotion *emotionDet=putil->QueryEmotion();

           if(emotionDet!=NULL){
               /* Display Results */
               PXCEmotion::EmotionData arrData[NUM_EMOTIONS];
               const PXCCapture::Sample *sample = putil->QueryEmotionSample();
               if (sample) DrawBitmap(hwndDlg, sample->color);
               numFaces = emotionDet->QueryNumFaces();
               for(int i=0; i < numFaces; i++){
                   emotionDet->QueryAllEmotionData(i, &arrData[0]);
                   DrawEmotion(hwndDlg, NUM_EMOTIONS, arrData);
               }
           }
           //bbb->Release();
           cvReleaseImage(&imageRGB);
           cvReleaseImage(&imageDEPTH);

           UpdatePanel(hwndDlg);

           putil->ReleaseFrame();
           doit.close();

           jpglist.close();
           break;
       }

   } else {
       SetStatus(hwndDlg,L"Init Failed");
       stsFlag=false;
   }

   putil->Close();
   putil->Release();
   if (stsFlag) SetStatus(hwndDlg,L"Stopped");
}

MartyG · ‎01-13-2016

Emotion detection has been removed from the RealSense SDK, so if your script depends on that module then that it ls likely why it would not be working. The emotion detection was a third-party component from a company called Emotient. Emotient was purchased by Apple, it was announced last week.

http://www.reuters.com/article/us-emotient-m-a-apple-idUSKBN0UL27420160107

If this deal was in the pipeline during the last quarter of 2015 then that may have been why emotion detection was refused after the first year following RealSense's release (perhaps Emotient not renewing licences, knowing that the Apple deal was being worked on).

A similar thing happened last summer with the Metaio company that provides the 3D augmented reality component of RealSense - they were purchased by Apple and immediately announced that they were ceasing issuing new licenses. As far as I know, the Metaio component is still a part of RealSense, so I don't know when their license deal with Intel expires.

Alejandro_S_Intel1 · ‎01-13-2016

Marty G. wrote:

Emotion detection has been removed from the RealSense SDK, so if your script depends on that module then that it ls likely why it would not be working. The emotion detection was a third-party component from a company called Emotient. Emotient was purchased by Apple, it was announced last week.

http://www.reuters.com/article/us-emotient-m-a-apple-idUSKBN0UL27420160107

If this deal was in the pipeline during the last quarter of 2015 then that may have been why emotion detection was refused after the first year following RealSense's release (perhaps Emotient not renewing licences, knowing that the Apple deal was being worked on).

A similar thing happened last summer with the Metaio company that provides the 3D augmented reality component of RealSense - they were purchased by Apple and immediately announced that they were ceasing issuing new licenses. As far as I know, the Metaio component is still a part of RealSense, so I don't know when their license deal with Intel expires.

Oh, I didn't know that, any ways I have an earlier version of the RealSense and It support the emotion viewer. I really like to know how to use my set of image files instead of the stream or playback options. Do you have any idea of how to do that? Regards.

MartyG · ‎01-13-2016

Yeah, I'd suspected that earlier versions of the SDK would still have features that had now been removed and would be retained even if license agreements expired. You don't tend to hear of components being retroactively removed from software after a license expires. The most common problem people have is that once they realize that they need to go back to an older version, it's too late because all the downloads of the previous edition have been withdrawn from the net by then. So it's lucky that you still have the old one. *smiles*

Anyway, I'm afraid that analyzing images isn't part of my RealSense skillset (I specialize in using RealSense with Unity, where I built my own emotion recognition system based on facial landmark angles). I'm sure someone on the forum here will be able to help though. Best of luck!