Background: I'm trying to overlay a full frame image on top of an MP4, in ffmpeg, using as much hardware-assistance as possible. The inputs to ffmpeg are:
- a stream from an MP4 file - with 1 audio and 1 video stream
- frames from some software, which are supplied as raw RGBA frames, at the same frame size as the movie.
I struggled for some time to get something working and through various iterations, ended up with this:
ffmpeg -y -report -hide_banner -loglevel verbose -hwaccel cuda -hwaccel_output_format yuv420p -i movie.mp4 -f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - -filter_complex [0:v]hwupload[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda[out] -map [out] -vcodec h264_nvenc -rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4
yuv420p because looking at the overlay code ( https://github.com/FFmpeg/FFmpeg/blob/4976b102d89787d59a3aaa438b76c62aec86ad5a/libavfilter/vf_overlay_cuda.c#L94 ) - you can only overlay an image with alpha on top of an image of this type.
However, this results in a very green image! - although the overlay (the date, in this case) seems to have been overlaid with the right alpha etc.
I was wondering if the blend mode or some other aspect of the overlay was causing a problem, so attempted to reduce the problem size. I thought that perhaps just processing the video - movie->decode->encode - might shed some light, so I tried this:
ffmpeg -y -report -hide_banner -loglevel verbose -hwaccel cuda -hwaccel_output_format yuv420p -i movie.mp4 -f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - -filter_complex [0:v]hwupload[mp4_stream] -map [mp4_stream] -vcodec h264_nvenc -rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4
This is essentially the same (I hope) but just ignoring the overlay frames - still green.
However, replacing the
-hwaccel cuda -hwaccel_output_format nv12 results in a correctly coloured image. This is no use for my actual use case, as nv12 does not support alpha overlays.
ffmpeg version 4.4.2-0ubuntu0.22.04.1
Hardware: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
Kernel: 5.15.0-71-generic #78-Ubuntu SMP
How to get the output the correct colour, while still using yuv420p, so I can overlay properly on top of it.
I have an inkling that this is due to some planes of the YUV file being zero? But how to fix this in the command line
ffmpeg - I'm not sure. Thanks!