How do you do a "Video Algorithm"?

Question

Facebook generate these anniversary videos which they allow you to share. The video has your Facebook avatar (along with certain friend's avatars) imposed onto certain objects. For example, it can put your avatar on an album cover.

This made me wonder how you do this on a mass scale like Facebook does?

Surely there is some kind of "video algorithm" that does this? What is this technique called and how can I do it?

I would imagine it would be a laugh for my friends if I could take pictures of them and impose them effortlessly onto landmarks, where their pictures are perfectly warped and tracked to the surface each time.

Before you answer I should mention that my attempt at this would be to film a template which had some well placed markers on a flat surface and then track a plain 'placeholder' texture on top of the surface. When the time comes to render a video I would just change the texture with a friend's face or something.

However that's the long way of doing it. I am more curious if there is a way for me to set this up so that I could type into a terminal prompt:

root@root:~$ myCoolVideoProgram.o pictureToImpose.jpeg

and in a matter of seconds (or however long the rendering takes) I have my funny video, without having to go through a GUI in the video editing software of my choice.

Answer 1

This is effectively what every professional editing software in existence does at its core. Video editing is always done non-destructively in any professional and most consumer contexts. What that means is that when you are editing a project, you are not actually making changes to the source files, but rather making a series of decisions about what changes should be applied to get the desired output. When it comes time to make the final rendered output, the instructions are run by the editing software on the original sources and the output is recorded to a new video file.

This series of instructions can be saved off in a variety of different formats such as EDLs (editing decision lists), Final Cut XML files (which are supported by more than just Final Cut) or any number of other formats that include the series of instructions that need to be applied. These files can be used to move editing projects between completely different editing software.

It is also possible to change the inputs that go in to these instructions. As long as they are the same size and placement, then it should carry through consistently for different inputs. This is why proxy workflows work. In proxy workflows very large and difficult to work with files (such as 4k RAW video) are replaced with easy to work with files (such as much smaller compressed 1080p video). This makes it easy to work through the edit and then only swap out with the high quality, difficult to work with footage at the end of the project.

The same principle can be applied to swapping photos or videos in to a particular spot in an overall video template. All the various effects and edits will be applied to the new source just like they were to the old, you just need a render engine capable of doing the rendering and swap automatically.

There are some commercial options available for this, but much is done with custom solutions. Depending on the complexity, it may also be possible to do it with an ffmpeg script that could be easily run by a command line, though ffmpeg would be considerably more manual to develop the template for.