Stream desktop screen using Intel media SDK with intel atom E3825

[Prakash_D_]

      Hi,

      I am trying to stream(h264) desktop using Intel media sdk.

     The platform details are as below.

      Intel atom E3825

      Windows 7

     Media SDK 2016.

    By using sample codes provided with SDK and screen capture plugin, I am able to stream h264 elementary stream using an i7 laptop.

   The stream is over UDP. While receiving, data is copied to a ring buffer and the sample decoder program tries to render the h264 bitstream.

   But the decoder crashes after 2/3 seconds.

    1- What is the proper way of streaming desktop screen using intel SDK ? The requirement is h264 stream over udp.

    2-Again I am not able to install intel SDk on atom E3825. It says unsupported CPU.

    3-Is there any benefit of using media SDK on atom ? or should i use IPP ? 

   Thanks

[Surbhi_M_Intel]

Hi Prakash, 

Can you let us know if there is Intel integrated graphic device on your machine( Intel atom E3825)? You can capture system analyzer log which provides in depth details on your system (copy system analyzer, installed in the media sdk doirectory under tools from your i7 machine to the Atom processor)
There are only selected SKUs of the Atom line which are supported by Media SDK. 

Can you please explain your set up - so far seems like you are using an i7 laptop but will like to replace it with Atom processor or  is your receiver i.e. client will be an Atom processor and the host, where screen capture is running will be an i7? 
FYI E3285 Atom processor, which is codename as Baytrail will support till API 1.8 and screen capture plugin requires API 1.13 support. So screen capture will might not work. 

Thanks,
Surbhi

[Prakash_D_]

Hi Surbhi,

Thanks for the reply.

I want to use atom E3825 as host that will stream the desktop and machines like i7(or i5) 
as client to preview and record the stream.                                                                                                                                              

The graphics and system info as below.                                                                                                                                                    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Graphics Devices:
    Name                                         Version             State
    Intel(R) Atom(TM) Processor E3800 Series/Intel(R) Celeron(R) Processor N2920/J190036.15.0.1091        Active

System info:
    CPU:    Intel(R) Atom(TM) CPU  E3825  @ 1.33GHz
    OS:    Microsoft Windows Embedded Standard 
    Arch:    32-bit

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------                              I have attached the analyzer log.

Is the screen capture plugin is similar to "https://github.com/rdp/screen-capture-recorder-to-video-windows-free" ? It uses directshow push source filters using GDI to capture the screens. I want to know how the Intel screen capture plugin works and the possibility/procedure to make one for atom.  

Is there any h264 streaming sample codes available for media SDK ? 

Thanks again.

Prakash

[Surbhi_M_Intel]

Hi Prakash, 

From your system details, it is unclear whether your system has HD graphics in it, like I said there are limited SKUs which have Quick Video Sync support. This is the latest driver for HD graphics - https://downloadcenter.intel.com/download/24973/Intel-HD-Graphics-Driver-for-Windows-7-8-8-1-64-bit which indicates Intel(R) Celeron(R) Processor N2920 doesn't have HD graphics whereas J1900 has HD graphics in it. Since it showed the graphic device to be active so I'm counting that it has HD graphics in it. To confirm the driver and update it to the latest version, I would recommend you to download the Intel driver utility and update the driver. 

Who is the OEM/vendor of your atom system? May be you can contact them to check about graphic device installed on it, in case updating driver from Intel website doesn't work which means your machine needs customized driver update from your vendor. 

Regarding screen capture, you can check the reference manual to get most of the information. The git hub repository you have pointed looks like using ffmpeg implementation of screen capture which won't be same as Media SDK since screen capture plugin in MSDK uses HW implementation and falls back on sw implementation, if hw is unsupported. sample_decode has capture plugin already added to it, you can refer the sample to get most of it integrated in your pipeline. You might want to check performance of both implementation on your underlying platform.

Media SDK has some samples posted on our website, some of them are deprecated but can be still reffered.
Hope the information provided above helps!

Thanks,
Surbhi
 

[Prakash_D_]

      Hi Surbhi,

     Thanks. Wishing you a great year ahead. 

   The atom system has Intel HD graphics and latest drivers from OEM.

     Tried out windows BitBlt to capture screen(640x480) and used it with media SDK's encoding pipeline.  I am able to get h264 stream.                                      The CPU usage on atom for the application is 25% and  fps is 30 .  Initial screen bitmap is RGB32 then it is being convertted to NV12 in software.               How I can improve it  ?(use IPP for convertion from RGB to NV12 and DirectX for capture may be ? ) 

     As you mentioned screen capture API can be used instead of plugin(since it does'nt work on atom ), I am not getting a clear idea where to start. 

    Regards,

[Surbhi_M_Intel]

Hi Prakash, 

Thanks and wish you a great 2016! 
Your system has HD graphics in it so we can use VPP to do color conversion i.e. from RGB32 to NV12 format so that it happens on hw instead of using sw and that should improve the performance. You can refer to the MSDK manual Pg 7 to see what all color conversion are supported by VPP and we have sample_vpp and simple_vpp tutorial to show case how to set VPP pipeline. 
Another recommendation is if you can upgrade to windows 8 or 8.1, you can make use of D3D11 implementation which is not available on windows 7 and can contribute to performance bump. 

For the screen capture, my bad seems like my reply was confusing. Can you please send us the system analyzer 64 bit instead of 32 bit, if it works? 32 bit system installer doesn't report till recent API and ends at API 1.8, we just found that we miss it in Media SDK 2016 release where 64 bit system analyzer should report till recent API which is 1.17. 
If your system supports API 1.13 and beyond in HW mode, we can use hw implementation of screen capture plugin and keep up with the overall performance of the whole application. If not, then the alternates are to use DirectX capture or use screen capture sw implementation or use ffmpeg implementation indicated by you before. 

Thanks,
Surbhi

[Prakash_D_]

 Hi Surbhi,

Thanks for the reply.

The atom bay trail system has HD graphics with Directx 11.0. (Windows embedded standard 7 64 bit). It supports Media SDK API upto 1.7 for hardware and from 1.8 to 1.17 for software.  Since software takes over 50% cpu I cannot use it.  

I have conducted following tests and the CPU usages as below.

I am using the sample encode application to convert desktop screen(RGB4) to h264 and dumping it over UDP.  The media SDK encoder takes direct RGB4 input (after changing m_mfxEncParams.mfx.FrameInfo.Fourcc=MFX_FOURCC_RGB4 in pipeline_encode.cpp and modified the bit stream input for RGB4 to the frame surface in Sample_utils.cpp) for my laptop which supports 1.17 API version (hw) and converts it to h264. Here I am having significant performance improvement. In case of bay trail, the encoder is unable to take RGB4 directly and shows unsupported format. It only works with NV12 input.

Due to above issue, I am converting RGB4 to NV12 first and input it to Encoder. 

Starting from which API version, the encoder supports RGB4 input ? 

 Please find the result with extra color conversion.The code has been built with Intel compiler using /O3 and /QxSSE4.2 flags. 

1- Stream desktop(sample_encode) with software conversion of RGB4 to YUV420 takes around  17%-22% cpu

2- Stream desktop(sample_encode) with VPP conversion of RGB4 to YUV420 takes around  20%-22% cpu

3- Stream desktop(sample_encode) with IPP conversion of RGB4 to YUV420 takes around  18%-23% cpu   

Is there a way to avoid this RGB4 to NV12 conversion ? Can I capture NV12 surfaces directly using directx ? If so please provide some references. 

Regards,

Prakash Dash

[Surbhi_M_Intel]

Hi Prakash, 

Sorry for a late response, I am looking at your query and will get back to you answer on this soon. 

-Surbh

[Prakash_D_]

Hi Surbhi,

No problem.

[Surbhi_M_Intel]

I have looked into the options you have tried above and have couple of questions. 

>>For Stream desktop(sample_encode) with VPP conversion of RGB4 to YUV420 takes around  20%-22% cpu
1.  Is the CPU usage accounted above is for both VPP and encode or just VPP function? 
2. Do you happen to know if your system supports hw accelerate VPP color conversion filter ?
quickest way is to use sample_vpp or simple_vpp_vmem tutorial and do color conversion operation only, if it executes on hardware and doesn't say partial acceleration that means you system supports hardware acceleration. Check if you can skip writing o/p by blocking the write call to avoid CPU usage in your test. 
Let me know your results to further debug the issue. 
 

>>Stream desktop(sample_encode) with software conversion of RGB4 to YUV420 takes around  17%-22% cpu
The method explained in the sample  is definitely not the optimized, it was mere shown as an example to do quick color formation conversion by copying all the pixels. 

Performance of your other system i.e. laptop cannot be compared with atom, it seems like your system is a core processor and might have better graphics but more cores than atom processor. 

There might be other ways through microsoft to capture NV12 surfaces, we added screen capture support to avoid this but that is not supported on your configuration. You can also look at the screen capture devices available online if that make sense in your pipeline to capture NV12 surfaces directly and if it's worth adding. 

Thanks,
Surbhi