FFmpeg qsv_av1

[shinigami_28]

I am having a hard time enabling qsv hw acceleration on latest ffmpeg 6.0, on a headless Ubuntu 22.04 server, 6.5.5 kernel, Intel ARC 770. I successfully compiled oneVPL, oneVPL-gpu and Intel MediaSDK, but I keep getting errors like "No VA display found for device".

Any help please?

[shinigami_28]

ffmpeg -hwaccels | grep vaapi

returns nothing, but

ffmpeg -encoders | grep vaapi

retuns

V....D av1_vaapi AV1 (VAAPI) (codec av1)
V....D h264_vaapi H.264/AVC (VAAPI) (codec h264)
V....D hevc_vaapi H.265/HEVC (VAAPI) (codec hevc)
V....D mjpeg_vaapi MJPEG (VAAPI) (codec mjpeg)
V....D mpeg2_vaapi MPEG-2 (VAAPI) (codec mpeg2video)
V....D vp8_vaapi VP8 (VAAPI) (codec vp8)
V....D vp9_vaapi VP9 (VAAPI) (codec vp9)

My FFMpeg configuration is:

--enable-nonfree --enable-version3 --ld=g++ --extra-libs='-lpthread -lm -lz -ldl' --pkg-config-flags=--static --enable-gpl --enable-vaapi --enable-libvpl --enable-libsrt --enable-libvmaf --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libzvbi --enable-libx264 --enable-libx265 --enable-openssl --enable-libvidstab --enable-pic --enable-shared --enable-parser=mpegaudio

I added --enable-vaapi and exported PKG_CONFIG_PATH to where my libva.pc is located, but it doesn't seem to help.

Maybe I should add some missing flags during libva compilation?

[SreedeviK_Intel]

Hi,

Thank you for sharing the information with us.

Please find below our observations and kindly share the details as mentioned:

1. if libva is available in the environment, usually 

ffmpeg -hwaccels | grep vaapi

should returns vaapi.

Can you please double-check and share the output of the below two commands?

ffmpeg -hwaccels

cat config.h | grep -i vaapi

2. Also, we noticed that you had ran vainfo with sudo but ran FFmpeg without sudo. Hence, can you please try running FFmpeg with sudo too? As you don't have permission to access /dev/dri/renderD128 or /dev/dri/card0, so sudo is required when running FFmpeg.

3. If the issue still persists, please try running FFmpeg command in verbose mode (see the command below) and share the whole output with us.

$ ffmpeg -y -loglevel verbose -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i input.ts -c:v av1_qsv -preset slow -b:v 512k -bufsize 512k -maxrate 1024k -c:a libopus -vbr off -ac 2 -b:a 48k -f Matroska output.mkv

---

@shinigami_28 wrote:

Maybe I should add some missing flags during libva compilation?

---

As soon as

./configure --enable-vaapi 

works when building FFmpeg, no extra flags are required during libva compilation.

Kindly let us know if this resolves your issue.

Regards,

Sreedevi

[shinigami_28]

ffmpeg -hwaccels

ffmpeg version 6.1 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
configuration: --enable-nonfree --enable-version3 --ld=g++ --extra-libs='-lpthread -lm -lz -ldl' --pkg-config-flags=--static --enable-gpl --enable-vaapi --enable-libvpl --enable-libsrt --enable-libvmaf --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libzvbi --enable-libx264 --enable-libx265 --enable-openssl --enable-libvidstab --enable-pic --enable-shared --enable-parser=mpegaudio
libavutil 58. 29.100 / 58. 32.100
libavcodec 60. 31.102 / 60. 33.100
libavformat 60. 16.100 / 60. 17.100
libavdevice 60. 3.100 / 60. 4.100
libavfilter 9. 12.100 / 9. 13.100
libswscale 7. 5.100 / 7. 6.100
libswresample 4. 12.100 / 4. 13.100
libpostproc 57. 3.100 / 57. 4.100
Hardware acceleration methods:
vaapi
qsv

Running ffmpeg with sudo doesn't make any difference.

Running:

ffmpeg -y -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i input.mp4 -c:v av1_qsv -preset slow -b:v 512k -bufsize 512k -maxrate 1024k -c:a libopus -vbr off -ac 2 -b:a 48k -f Matroska output.mkv

always returns errors, both with H264 incapsulated in a ts container and H264 incapsulated in a mp4 container:

TS:

[av1_qsv @ 0x5579e445d340] Invalid FrameType:0.=00:11:10.49 bitrate= 566.1kbits/s speed=17.4x
[vost#0:0/av1_qsv @ 0x5579e4581100] Error submitting video frame to the encoder
Error while filtering: Invalid data found when processing input
[h264_qsv @ 0x5579e4529f00] A decode call did not consume any data: expect more data at input (-10)

MP4:

[h264_qsv @ 0x559e0179c480] More data is required to decode header
Assertion consumed != (-(11)) failed at libavcodec/decode.c:459

[SreedeviK_Intel]

Hi,

Thank you for sharing the details with us.

As requested above, can you please try running FFmpeg command in verbose mode and share the whole output with us.

$ ffmpeg -y -loglevel verbose -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i input.ts -c:v av1_qsv -preset slow -b:v 512k -bufsize 512k -maxrate 1024k -c:a libopus -vbr off -ac 2 -b:a 48k -f Matroska output.mkv

Regards,

Sreedevi

[shinigami_28]

Sure, no problem.

[AVHWDeviceContext @ 0x55ac12e94a40] libva: VA-API version 1.21.0
[AVHWDeviceContext @ 0x55ac12e94a40] libva: User requested driver 'iHD'
[AVHWDeviceContext @ 0x55ac12e94a40] libva: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x55ac12e94a40] libva: Found init function __vaDriverInit_1_21
[AVHWDeviceContext @ 0x55ac12e94a40] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55ac12e94a40] Initialised VAAPI connection: version 1.21
[AVHWDeviceContext @ 0x55ac12e94a40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.4.1 (94a4f8d16).
[AVHWDeviceContext @ 0x55ac12e94a40] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x55ac12e94540] Use Intel(R) oneVPL to create MFX session, API version is 2.9, the required implementation version is 1.3
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55ac12e94540] Initialize MFX session: implementation version is 2.7
[h264 @ 0x55ac13116f80] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x55ac13116f80] mmco: unref short failure
Last message repeated 1 times
[h264 @ 0x55ac13116f80] number of reference frames (0+5) exceeds max (4; probably corrupt input), discarding one
[h264 @ 0x55ac13116f80] Increasing reorder buffer to 2
[h264 @ 0x55ac13116f80] mmco: unref short failure
Last message repeated 1 times
[mpegts @ 0x55ac13126580] max_analyze_duration 5000000 reached at 5024000 microseconds st:1
[vist#0:0/h264 @ 0x55ac12faf740] WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv".
Input #0, mpegts, from 'input.ts':
Duration: 03:07:48.11, start: 1.400000, bitrate: 4120 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, top first, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 25 fps, 50 tbr, 90k tbn
Stream #0:1[0x101](ita): Audio: eac3 (EAC3 / 0x33434145), 48000 Hz, stereo, fltp, 128 kb/s
[out#0/matroska @ 0x55ac1305e540] No explicit maps, mapping streams automatically...
[vost#0:0/av1_qsv @ 0x55ac13081700] Created video stream from input stream 0:0
[h264_mp4toannexb @ 0x55ac1305d880] The input looks like it is Annex B already
[aost#0:1/libopus @ 0x55ac13117d00] Created audio stream from input stream 0:1
Stream mapping:
Stream #0:0 -> #0:0 (h264 (h264_qsv) -> av1 (av1_qsv))
Stream #0:1 -> #0:1 (eac3 (native) -> opus (libopus))
Press [q] to stop, [?] for help
[graph_1_in_0_1 @ 0x55ac1312afc0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:stereo
[format_out_0_1 @ 0x55ac12fbb3c0] auto-inserting filter 'auto_aresample_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_1'
[auto_aresample_0 @ 0x55ac130d0900] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:flt r:48000Hz
[AVHWDeviceContext @ 0x7fc9c0002980] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.4.1 (94a4f8d16).
[AVHWDeviceContext @ 0x7fc9c0002980] Driver not found in known nonstandard list, using standard behaviour.
[h264_qsv @ 0x55ac13054a00] Decoder: output is video memory surface
[h264_qsv @ 0x55ac13054a00] Use Intel(R) oneVPL to create MFX session with the specified MFX loader
[AVHWDeviceContext @ 0x7fc9c000c100] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.4.1 (94a4f8d16).
[AVHWDeviceContext @ 0x7fc9c000c100] Driver not found in known nonstandard list, using standard behaviour.
[h264_qsv @ 0x55ac13054a00] Decoder: output is video memory surface
[h264_qsv @ 0x55ac13054a00] Use Intel(R) oneVPL to create MFX session with the specified MFX loader
[graph 0 input from stream 0:0 @ 0x55ac13050980] w:1920 h:1080 pixfmt:qsv tb:1/90000 fr:25/1 sar:1/1
[av1_qsv @ 0x55ac1400cac0] Using input frames context (format qsv) with av1_qsv encoder.
[av1_qsv @ 0x55ac1400cac0] Encoder: input is video memory surface
[av1_qsv @ 0x55ac1400cac0] Use Intel(R) oneVPL to create MFX session with the specified MFX loader
[av1_qsv @ 0x55ac1400cac0] Using the variable bitrate (VBR) ratecontrol method
[av1_qsv @ 0x55ac1400cac0] profile: av1 main; level: 40
[av1_qsv @ 0x55ac1400cac0] GopPicSize: 65535; GopRefDist: 1; GopOptFlag:; IdrInterval: 0
[av1_qsv @ 0x55ac1400cac0] TargetUsage: 3; RateControlMethod: VBR
[av1_qsv @ 0x55ac1400cac0] BufferSizeInKB: 64; InitialDelayInKB: 48; TargetKbps: 512; MaxKbps: 1024; BRCParamMultiplier: 1
[av1_qsv @ 0x55ac1400cac0] NumRefFrame: 2
[av1_qsv @ 0x55ac1400cac0] IntRefType: 0; IntRefCycleSize: 0; IntRefQPDelta: 0; IntRefCycleDist: 0
[av1_qsv @ 0x55ac1400cac0] MaxFrameSize: 0;
[av1_qsv @ 0x55ac1400cac0] BitrateLimit: unknown; MBBRC: unknown; ExtBRC: unknown
[av1_qsv @ 0x55ac1400cac0] VDENC: ON
[av1_qsv @ 0x55ac1400cac0] BRefType: off
[av1_qsv @ 0x55ac1400cac0] PRefType: default
[av1_qsv @ 0x55ac1400cac0] MinQPI: 0; MaxQPI: 0; MinQPP: 0; MaxQPP: 0; MinQPB: 0; MaxQPB: 0
[av1_qsv @ 0x55ac1400cac0] FrameRateExtD: 1; FrameRateExtN: 25
[av1_qsv @ 0x55ac1400cac0] NumTileRows: 1; NumTileColumns: 1; NumTileGroups: 1
[av1_qsv @ 0x55ac1400cac0] WriteIVFHeaders: OFF
[av1_qsv @ 0x55ac1400cac0] LowDelayBRC: unknown
[av1_qsv @ 0x55ac1400cac0] MaxFrameSize: 0;
Output #0, matroska, to 'output.mkv':
Metadata:
encoder : Lavf60.17.100
Stream #0:0: Video: av1, 1 reference frame (AV01 / 0x31305641), qsv(tv, bt709, top coded first (swapped), left), 1920x1080 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 512 kb/s, 25 fps, 1k tbn
Metadata:
encoder : Lavc60.31.102 av1_qsv
Stream #0:1(ita): Audio: opus ([255][255][255][255] / 0xFFFFFFFF), 48000 Hz, stereo, flt, delay 312, 48 kb/s
Metadata:
encoder : Lavc60.31.102 libopus
[av1_qsv @ 0x55ac1400cac0] Invalid FrameType:0.=00:11:03.61 bitrate= 568.8kbits/s speed=17.2x
[vost#0:0/av1_qsv @ 0x55ac13081700] Error submitting video frame to the encoder
Error while filtering: Invalid data found when processing input
[vist#0:0/h264 @ 0x55ac12faf740] Decoder thread received EOF packet
[h264_qsv @ 0x55ac13054a00] A decode call did not consume any data: expect more data at input (-10)
Last message repeated 2 times
[vist#0:0/h264 @ 0x55ac12faf740] Decoder returned EOF, finishing
[vist#0:0/h264 @ 0x55ac12faf740] Terminating decoder thread
[aist#0:1/eac3 @ 0x55ac13021180] Decoder thread received EOF packet
[aist#0:1/eac3 @ 0x55ac13021180] Decoder returned EOF, finishing
[aist#0:1/eac3 @ 0x55ac13021180] Terminating decoder thread

And then it hangs indefinitely.

[SreedeviK_Intel]

Hi,

Thank you for sharing the details. Please find below points and kindly share the output with us:

1. Since the new output of 'ffmpeg -hwaccels' shows vaapi is enabled, it seems that you have the right vaapi settings now.

2.

TS:

[av1_qsv @ 0x5579e445d340] Invalid FrameType:0.=00:11:10.49 bitrate= 566.1kbits/s speed=17.4x

[vost#0:0/av1_qsv @ 0x5579e4581100] Error submitting video frame to the encoder

Error while filtering: Invalid data found when processing input

For the above error, can you please try a different buf size / bitrate or do not set buffer size ? The media driver has limitation which can't support arbitrary combination of bitrate & bufsize. Please refer below commands:

$ ffmpeg -y -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i input.mp4 -c:v av1_qsv -preset slow -b:v 512k -bufsize 1024k -maxrate 1024k -c:a libopus -vbr off -ac 2 -b:a 48k -f Matroska output.mkv

$ ffmpeg -y -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i input.mp4 -c:v av1_qsv -preset slow -b:v 512k -maxrate 1024k -c:a libopus -vbr off -ac 2 -b:a 48k -f Matroska output.mkv

$ ffmpeg -y -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i input.mp4 -c:v av1_qsv -preset slow -b:v 2048k -maxrate 2048k -c:a libopus -vbr off -ac 2 -b:a 48k -f Matroska output.mkv

3.

 MP4:

[h264_qsv @ 0x559e0179c480] More data is required to decode header

Assertion consumed != (-(11)) failed at libavcodec/decode.c:459

For the above error, this is caused by a recent change in FFmpeg master, unfortunately FFmpeg 6.1 also has this issue.

Could you try FFmpeg 6.0 instead ? Or kindly try this patch:

 https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231127060151.2435971-1-haihao.xiang@intel.com/ ?

Kindly let us know if it resolves your issue and share the output if getting any errors.

Regards,

Sreedevi

[SreedeviK_Intel]

Hi,

We have not heard back from you. Could you please give us an update if your issue is resolved or not?

Regards,

Sreedevi

[SreedeviK_Intel]

Hi,

Since we didn't hear back from you, we assume that your issue is resolved.

If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards,

Sreedevi