I am trying to perform a video segmentation for background blurring similar to Google Meet or Zoom using FFmpeg, and I'm not very familiar with it.
Google's MediaPipe model is available as a tensorflow .pb file here (using
I can load it in python and it works as expected, though I do need to format the input frames: scaling to the model input dimension, adding a batch dimension, dividing the pixel values by 255 to have a range 0-1.
FFmpeg has a filter that can use tensorflow models thanks to dnn_processing, but I'm wondering about these preprocessing steps. I tried to read the dnn_backend_tf.c file in ffmpeg's github repo, but C is not my forte. I'm guessing it adds a batch dimension somewhere otherwise the model wouldn't run, but I'm not sure about the rest.
Here is my current command:
ffmpeg \ -i $FILE -filter_complex \ "[0:v]scale=160:96,format=rgb24,dnn_processing=dnn_backend=tensorflow:model=$MODEL:input=input_1:output=segment[masks];[masks]extractplanes=2[mask]" \ -map "[mask]" output.png
- I'm already applying a scaling to match the input dimension.
- I wrote this
[masks]extractplanes=2[mask]because the model outputs a HxWx2 tensor (background mask and foreground mask) and I want to keep the foreground mask.
The result I get with this command is the following (input-output):
I'm not sure how to interpret the problems in this output. In python I can easily get a nice grayscale output:
I'm trying to obtain something similar with FFmpeg.
Any suggestion or insights to obtain a correct output with FFmpeg would be greatly appreciated.
PS: If I try to apply this on a video file, it hits a Segmentation Fault somewhere before getting any output so I stick with testing on an image for now.