I am trying to perform a video segmentation for background blurring similar to Google Meet or Zoom using FFmpeg, and I'm not very familiar with it.

Google's MediaPipe model is available as a tensorflow .pb file here (using download_hhxwww.sh).

I can load it in python and it works as expected, though I do need to format the input frames: scaling to the model input dimension, adding a batch dimension, dividing the pixel values by 255 to have a range 0-1.

FFmpeg has a filter that can use tensorflow models thanks to dnn_processing, but I'm wondering about these preprocessing steps. I tried to read the dnn_backend_tf.c file in ffmpeg's github repo, but C is not my forte. I'm guessing it adds a batch dimension somewhere otherwise the model wouldn't run, but I'm not sure about the rest.

Here is my current command:

ffmpeg \
    -i $FILE -filter_complex \
    "[0:v]scale=160:96,format=rgb24,dnn_processing=dnn_backend=tensorflow:model=$MODEL:input=input_1:output=segment[masks];[masks]extractplanes=2[mask]" \
    -map "[mask]" output.png
  • I'm already applying a scaling to match the input dimension.
  • I wrote this [masks]extractplanes=2[mask] because the model outputs a HxWx2 tensor (background mask and foreground mask) and I want to keep the foreground mask.

The result I get with this command is the following (input-output):

Output example

I'm not sure how to interpret the problems in this output. In python I can easily get a nice grayscale output:

enter image description here

I'm trying to obtain something similar with FFmpeg.

Any suggestion or insights to obtain a correct output with FFmpeg would be greatly appreciated.

PS: If I try to apply this on a video file, it hits a Segmentation Fault somewhere before getting any output so I stick with testing on an image for now.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.