3

I'm looking for an ffmpeg-like tool to blur/remove/change a static background from a "talking head" video, on the command line with on-device processing.

Essentially,

ffmpeg-like-tool [--blur-background | --remove-background | --change-background] --background image.jpg --in input.mp4 --out output.mp4

Does such a tool exist yet?

1

1 Answer 1

0

There is a reason why monotone background are used, they allow one to determine where the object is, and where is the background. As far as i know no took exists, which can always determine what is the background, and what is the object that is actually being filmed.

If the background is really static (not moving in the frame, rather than simply always being behind the person), then you could attempt to extract frames and determine the pixels that keep their value above some arbitrary percentage of frames. You then can create a mask based on if the pixels of the specific frame are the same as the value of the 'stable pixels', and blur in that mask.

The the background is moving in the frame due to camera movement or changes in some manner, this becomes very difficult, and i may even guess impossible in some situations. Also even if it will work, it will need to be double checked to see that you won't get false positives. You will definitely get false negatives near the person's head.

In the even you have resources for development and really need it to be done, you could go neural network route. Be prepared to provide neural net lots and lots of training data. For that you would need to film akin to the videos that you will be processing, but doing that in front of the green screen. This will allow you to determine where the head truly is in each frame, because once you put in the background it will be the job of the neural net to figure out where the head is, and your job to tell it whether it is correct or not during the training, and to be able to do that in testing to see how well it has learnt to identify the human head.

Good luck.

1
  • 3
    But then, how do Zoom, Teams etc do to separate you from the background in a live call? I would like to use that same type of software and apply to some training videos we have where the background is distracting
    – kxtronic
    Mar 18, 2022 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.