FFmpeg dnn_processing with tensorflow backend: difficulties applying a dnn filter on an image

Question

I am trying to perform a video segmentation for background blurring similar to Google Meet or Zoom using FFmpeg, and I'm not very familiar with it.

Google's MediaPipe model is available as a tensorflow .pb file here (using download_hhxwww.sh).

I can load it in python and it works as expected, though I do need to format the input frames: scaling to the model input dimension, adding a batch dimension, dividing the pixel values by 255 to have a range 0-1.

FFmpeg has a filter that can use tensorflow models thanks to dnn_processing, but I'm wondering about these preprocessing steps. I tried to read the dnn_backend_tf.c file in ffmpeg's github repo, but C is not my forte. I'm guessing it adds a batch dimension somewhere otherwise the model wouldn't run, but I'm not sure about the rest.

Here is my current command:

ffmpeg \
    -i $FILE -filter_complex \
    "[0:v]scale=160:96,format=rgb24,dnn_processing=dnn_backend=tensorflow:model=$MODEL:input=input_1:output=segment[masks];[masks]extractplanes=2[mask]" \
    -map "[mask]" output.png

• I'm already applying a scaling to match the input dimension.
• I wrote this [masks]extractplanes=2[mask] because the model outputs a HxWx2 tensor (background mask and foreground mask) and I want to keep the foreground mask.

The result I get with this command is the following (input-output):

I'm not sure how to interpret the problems in this output. In python I can easily get a nice grayscale output:

I'm trying to obtain something similar with FFmpeg.

Any suggestion or insights to obtain a correct output with FFmpeg would be greatly appreciated.

PS: If I try to apply this on a video file, it hits a Segmentation Fault somewhere before getting any output so I stick with testing on an image for now.