0

Background: I'm trying to overlay a full frame image on top of an MP4, in ffmpeg, using as much hardware-assistance as possible. The inputs to ffmpeg are:

  • a stream from an MP4 file - with 1 audio and 1 video stream
  • frames from some software, which are supplied as raw RGBA frames, at the same frame size as the movie.

I struggled for some time to get something working and through various iterations, ended up with this:

ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format yuv420p 
-i movie.mp4 
-f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - 
-filter_complex [0:v]hwupload[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda[out] 
-map [out] -vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4

I picked yuv420p because looking at the overlay code ( https://github.com/FFmpeg/FFmpeg/blob/4976b102d89787d59a3aaa438b76c62aec86ad5a/libavfilter/vf_overlay_cuda.c#L94 ) - you can only overlay an image with alpha on top of an image of this type.

However, this results in a very green image! - although the overlay (the date, in this case) seems to have been overlaid with the right alpha etc.

a green image

I was wondering if the blend mode or some other aspect of the overlay was causing a problem, so attempted to reduce the problem size. I thought that perhaps just processing the video - movie->decode->encode - might shed some light, so I tried this:

ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format yuv420p -i movie.mp4 
-f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - 
-filter_complex [0:v]hwupload[mp4_stream] 
-map [mp4_stream] -vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4

This is essentially the same (I hope) but just ignoring the overlay frames - still green.

However, replacing the yuv420p with nv12: -hwaccel cuda -hwaccel_output_format nv12 results in a correctly coloured image. This is no use for my actual use case, as nv12 does not support alpha overlays.

a correctly coloured image

Versions:

ffmpeg version 4.4.2-0ubuntu0.22.04.1

Hardware: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)

Kernel: 5.15.0-71-generic #78-Ubuntu SMP

Driver: 525.105.17

How to get the output the correct colour, while still using yuv420p, so I can overlay properly on top of it.

I have an inkling that this is due to some planes of the YUV file being zero? But how to fix this in the command line ffmpeg - I'm not sure. Thanks!

1
  • I am wondering if it is the version of ffmpeg - I can see that scale_cuda was modified to add the option "format" in what appears to be 8.0.103 -(github.com/FFmpeg/FFmpeg/commit/…) but my ffmpeg appears to be using 7.110.100 - is it just too old?
    – time4tea
    May 17, 2023 at 10:37

1 Answer 1

0

So, after some (long) time - it is a problem with the old version of ffmpeg - which is the version bundled with Ubuntu 22.04.

Using a self-compiled version of ffmpeg version 6.0 with CUDA Extensions. I can run the following command line, and everything is working perfectly.

/home/richja/bin/ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format cuda 
-i movie.MP4 
-f rawvideo -framerate 10.0 -s 1920x1080 -pix_fmt rgba -i - 
-filter_complex [0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda 
-vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart overlay.MP4

The performance of this setup is frankly incredible, and can render overlays at more than 12x real-time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.