Intel® VPL Unveils AI-Powered Video Frame Interpolation

Furong_Zhang_Intel · ‎08-04-2024

Author

Furong Zhang, GPU Software Development Engineer, Intel Corporation

At the Forefront of Video Processing

Intel has once again positioned itself at the forefront of video processing innovation with the introduction of an AI-powered video frame interpolation feature in its Intel® Video Processing Library (Intel® VPL), specifically designed for frame rate conversion and slow-motion use scenario in various applications. This cutting-edge technology is set to redefine the way we create and enjoy video content, offering a level of smoothness and detail that surpasses traditional methods.

Video frame interpolation is a technique used to increase the frame rate of video footage by generating intermediate frames between two consecutive video frames. This process is crucial for creating smooth frame rate conversion or slow-motion effects, as it allows for a fluid transition. Until now, achieving high-quality video frame interpolation required complex algorithms and often resulted in less-than-ideal results, with artifacts and jitter that could distract from the viewing experience.

Intel's AI-powered video frame interpolation changes the game by utilizing advanced artificial intelligence algorithms to predict and construct new frames with accuracy. The AI examines the motion and video data from the original footage and intelligently fills in the gaps, creating new frames. This results in a smooth, natural-looking video effect that maintains the integrity of the motion and the clarity of the video.

Intel's AI-powered video frame interpolation is not only about improving visual quality; it's also about efficiency and accessibility. By adding this technology and feature into Intel® Video Processing Library, Intel® VPL ensures that users can take advantage of this technology across a range of Intel hardware platforms.

Moreover, the AI-driven approach to video frame interpolation means that the process can be continually refined. As the AI models are exposed to more video data and different types of motion, their ability to interpolate frames accurately will only improve, leading to even smoother and more realistic video frame interpolation effect.

The Intel® VPL API

Intel® video processing library (VPL) intel/libvpl: Intel® Video Processing Library API, dispatcher, and examples (github.com) provides access to hardware accelerated video processing capabilities on Intel® GPUs. Starting from Intel® VPL specification version 2.12.0.( Change Log — Intel® VPL documentation), AI-powered video frame interpolation was added for better video quality and visual experience. This article lists the step-by-step Intel® VPL API call with sample code, which will help to expedite the application enabling. Follow these steps to enable AI powered video frame interpolation on Intel® Graphics in the application via Intel® VPL API based on your needs.

There are 3 new definitions in API for AI powered video frame interpolation.

The extended buffer ID MFX_EXTBUFF_VPP_AI_FRAME_INTERPOLATION was defined to indicate the extended buffer is an AI video frame interpolation extended buffer.
The extended buffer “mfxExtVPPAIFrameInterpolation” was defined to indicate the details of AI powered frame interpolation.
The enumerator mfxAIFrameInterpolationMode was defined to indicate the mode of AI powered frame interpolation.

With the above 3 new definitions, the application developer can use the same approach which uses VPP filters, includes querying capability, initializing, processing frame by frame.

As the programming guide Video Processing Procedures — Intel® VPL documentation describes, the first step is to configure AI video frame interpolation VPP filter.

// configure AI frame interpolation vpp filter
 mfxExtVPPAIFrameInterpolation aiFrameInterpolation = {};
 aiFrameInterpolation.Header.BufferId      = MFX_EXTBUFF_VPP_AI_FRAME_INTERPOLATION;
 aiFrameInterpolation.Header.BufferSz      = sizeof(mfxExtVPPAIFrameInterpolation);
 aiFrameInterpolation.FIMode               = MFX_AI_FRAME_INTERPOLATION_MODE_DEFAULT;
 aiFrameInterpolation.EnableScd            = 1;
 mfxExtBuffer * ExtParam[1] = { (mfxExtBuffer *)&aiFrameInterpolation };

Except mfxExtVPPAIFrameInterpolation filter, the developer needs to specify the frame rate. Without the correct frame rate setting, the Intel® VPL will return status code MFX_ERR_UNSUPPORTED.

 init_param.NumExtParam   = 1;
 init_param.ExtParam      = ExtParam;
 init_param.vpp.In.FrameRateExtN = 30;
 init_param.vpp.In.FrameRateExtD = 1;
 init_param.vpp.Out.FrameRateExtN = 60;
 init_param.vpp.Out.FrameRateExtD = 1;

With those 2 configurations, the developer can use the same code as other VPP filters to query, initialize and process frame by frame. The developer needs to pay attention to the return value of the API MFXVideoVPP_RunFrameVPPAsync as the below code shows. If the return value of the API MFXVideoVPP_RunFrameVPPAsync is not handled as the API definition describes, Intel® VPL will have unexpected behavior.

sts = MFXVideoVPP_QueryIOSurf(session, &init_param, response);
sts = MFXVideoVPP_Init(session, &init_param);

// The below code follows the video processing procedure, not specific to AI frame interpolation.
allocate_pool_of_surfaces(in_pool, response[0].NumFrameSuggested);
allocate_pool_of_surfaces(out_pool, response[1].NumFrameSuggested);
mfxFrameSurface1 *in=find_unlocked_surface_and_fill_content(in_pool);
mfxFrameSurface1 *out=find_unlocked_surface_from_the_pool(out_pool);
for (;;) {
   sts=MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
   if (sts==MFX_ERR_MORE_SURFACE || sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
      process_output_frame(out);
      out=find_unlocked_surface_from_the_pool(out_pool);
   }
   if (sts==MFX_ERR_MORE_DATA && in==NULL)
      break;
   if (sts==MFX_ERR_NONE || sts==MFX_ERR_MORE_DATA) {
      in=find_unlocked_surface_from_the_pool(in_pool);
      fill_content_for_video_processing(in);
      if (end_of_stream())
         in=NULL;
   }
}
MFXVideoVPP_Close(session);
free_pool_of_surfaces(in_pool);
free_pool_of_surfaces(out_pool);

BTW, the application developer also needs to pay attention to configure mfxextvideosignalinfo, the interpolated video frames need to have the same range setting ”VideoFullRange” in the structure mfxextvideosignalinfo as the original video frames, otherwise, the whole video frame sequence with the original video frames and interpolated video frames will have flicking issue.

Video Frame Interpolation Aided by AI

In conclusion, Intel® VPL’s introduction of AI-powered video frame interpolation for slow-motion and frame rate conversion is a significant advancement in video processing technology. By leveraging the power of artificial intelligence, Intel® VPL is providing content creators with a tool that can transform ordinary footage into extraordinary interpolated video sequences with unparalleled smoothness and clarity. As this technology becomes more integrated into video production workflows, we can expect to see a new wave of captivating slow-motion or frame rate conversion content that pushes the boundaries of what is visually possible.

Start Using It

The GitHub repositories for Intel VPL are the best place to find the latest updates and the recommended location to obtain Intel VPL:

For Linux*:

Acknowledgement

I would like to extend my heartfelt thanks to my Intel colleague Rob (Mueller-Albrecht, Rob) for his insightful reviews to this article.

Also sincere appreciation to my Intel colleagues Russell, Jay, Jian, Colan, Noah and Ben Lin for the great support.

Legal Information

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications, and roadmaps.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel, the Intel logo, Intel® are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.