ffmpeg to combine video + alpha video, transparency key with threshold

Question

I have two corresponding video files. One is a regular video and the other is meant to be the alpha channel for that video. I wish to combine them, and use the alpha video to lay a transparency key over the video, then output to a PNG sequence to avoid any lossy artifacts on the transparency key.

The alpha video needs a threshold filter applied to remove the gray pixels, then the main video needs the transparency color applied where the alpha video has black pixels. From trying to understand the threshold filter in ffmpeg, is almost seems like it might be possible to do this one one operation, however I am having trouble understanding how streams work.

The motivation here is that I would like to store these videos as small as possible. This could be done directly in a video editor, but I would need to keep the videos as PNG sequences to preserve the sharp transparency key. I have tried other codecs to keep this alpha data but they either aren't supported by my editor, are as large as PNG sequence and/or don't preserve the sharp transparency mask. I would like to use ffmpeg so that I can script this conversion when the frames are needed and they can remain as h264 the rest of the time.

Answer 1

ffmpeg \
-i test.mp4 -r 29.97 \
-i test_alpha.mp4 -r 29.97 \
-filter_complex "\
\
color=c=fuchsia:r=29.97,format=rgb24[fuchsia];\
color=c=black:r=29.97,format=rgb24[black];\
color=c=white:r=29.97,format=rgb24[white];\
color=c=0xe0e0e0:r=29.97,format=rgb24[gray];\
[fuchsia][0:v]scale2ref[fuchsia][i];\
[black][i]scale2ref[black][i];\
[white][i]scale2ref[white][i];\
[gray][i]scale2ref[gray][i];\
\
[1] [gray] [black] [white] threshold [mask];\
[i] [mask] alphamerge [merged];\
[fuchsia][merged]overlay=format=auto:shortest=1,setsar=1\
" \
tmp\%04d.png

I spent some time learning how filter_complex works while I wait for an answer.

This fixes the alpha video with a threshold, does an alphamerge with the source video and threholded video, then just overlays that on top of a fuchsia background. The one tricky thing was setting all of their frame rates or else the frames wouldn't line up after the threshold.

The only problem with trying to do this all with just the threshold filter is that when you set the "min" color to fuchsia, it comes out a dark gray.