Moving Crop in Video

Question

I have an HD video of a speaker giving a lecture. I would like to resize the video by cropping each frame, but the cropping rectangle needs to be centered on the speaker's head as he moves around the stage.

I can generate a file that contains the cropping rectangle's X, Y, Width, & Height for each frame in the video.

What's the best way to apply this moving crop to the video?

It should work if I..

1. Use ffmpeg to extract each frame to a folder
2. Use another program to crop each image file
3. Use ffmpeg to rebuild the video using the cropped image files

Is there a better way to do this?

Answer 1

Similar to Gyan's overlay answer, a slightly simpler version can use a basic -filter_script with the "swaprect" and "crop" filters. Move the crop target to 0,0 and then crop the target area at 0,0. The target has to be swapped to 0,0 as otherwise the other swapped rect can overlap. This worked with ffmpeg version 3.4.6-0ubuntu0.18.04.1

Make sure the frame numbers are relative to the offset passed with "-ss".

An example targeting 1280x720 crops from a 4096x2160 source. The source is 24fps and the tracking info is 8fps.

crop-filter-script:

swaprect=1280:720:0:0:1568:594:enable='between(n,0,26)',
swaprect=1280:720:0:0:1552:598:enable='between(n,27,29)',
swaprect=1280:720:0:0:1565:583:enable='between(n,30,32)',
swaprect=1280:720:0:0:1603:576:enable='between(n,33,35)',
crop=1280:720:0:0

ffmpeg -ss [start offset] -t [duration] -i input.mov -filter_script:v:0 crop-filter-script -acodec copy out.mov

Answer 2

As I'm working on the exact same challenge, I'm currently thinking of abusing the ffmpeg vidstabtransform filter with a generated trf file. But I have not worked out the details yet.

Info on the trf file format.

Answer 3

I have already changed to avconv, so i apologise if the answer may be a little different in the classic ffmpeg, but i believe that not much difference should be there.

You can have a moving crop if you can create some sort of formula between the frame number and the position of the crop; but there are no analytic tools available to do that. This means that if the the speaker moves in a predictable way you could theoretically do that without an external application.

Let's assume, for example, that the speaker is moving from top-left to bottom-right in the picture. You could do something along the lines of:

avconv -i input.ogv -vf 'crop=200:100:n:n' -c:v libtheora output.ogv

In here i'm moving 1 pixel per frame, which is very unlikely how somebody would move.

Keep in mind that there are quite a lot of math functions that are provided by the filter interface of avconv (and i'm sure ffmpeg is similar).

So let's imagine that you want to move 1 pixel per frame, but only from 200 to 350 frame. You can then do

min(max(n,200)-200, 150)

For the frames <200, max(n,200) would generate 200, then we take away 200 to make it 0, and then min will return the first part up until 350 frame, where that result would become more than 150 and the constant will begin to be returned.

Although writing such formula by hand would be quite tedious, maybe it will be helpful in some cases as well.

Having said all that, if i were to have an option of having access to the software that is able to work on images frame by frame, i would go that route.

Answer 4

Given that you have "a file that contains the cropping rectangle's X, Y, Width, & Height for each frame", you can use the filter_script option to do this in FFmpeg. The crop filter does not support timeline editing but the overlay filter does. So, if you create a blank canvas of the same resolution as your video and then overlay the video on top with co-ordinates such that your intended region is superimposed in the visible area, the goal is achieved.

Basic syntax is

ffmpeg -i in.mp4 -filter_complex_script file.txt -map "[out]" output.mp4

where file.txt looks like this:

nullsrc=WxH:r=FPS[cv];
[cv][0]overlay=-X0:-Y0:shortest=1:enable='eq(n\,0)'[b0];
[b0][0]overlay=-X1:-Y1:shortest=1:enable='eq(n\,1)'[b1];
[b1][0]overlay=-X2:-Y2:shortest=1:enable='eq(n\,2)'[b2];
...
[bm-1][0]overlay=-Xm:-Ym:shortest=1:enable='eq(n\,m)'[out]

The canvas W and H should be equal to the (fixed/static) crop dimensions and of the same FPS. Then each frame of the video is overlaid at (- X, - Y) so that the top-left corner of the desired region is at (0,0) of the canvas. Each overlay occurs for one frame. For efficiency's sake, if you have intervals where the crop region is static, you should use the between evaluator e.g.

[bi][0]overlay=-Xi:-Yi:shortest=1:enable='between(n\,1200\,1445)'[bj];

You can also evaluate using timestamp t instead of frame index.