7

I have an HD video of a speaker giving a lecture. I would like to resize the video by cropping each frame, but the cropping rectangle needs to be centered on the speaker's head as he moves around the stage.

I can generate a file that contains the cropping rectangle's X, Y, Width, & Height for each frame in the video.

What's the best way to apply this moving crop to the video?

It should work if I..

  1. Use ffmpeg to extract each frame to a folder
  2. Use another program to crop each image file
  3. Use ffmpeg to rebuild the video using the cropped image files

Is there a better way to do this?

  • ImageMagick could do the cropping if you have numerical description of the crop frame. I'd suggest: export as image sequence using ffmpeg, batch process using ImageMagick and remux as a movie in ffmpeg. You'll need some sort of shell scripting to glue it all together probably. If you post an example of the file with the crop info I can work out the details for you. – stib Apr 29 '15 at 2:43
  • May I ask how you generate that file? I'm working on exactly the same issue. I have a proof of concept using the ffmpeg tblend subtract filter and then the cropdetect filter. Wondering if you have a better way. – Jannes May 2 '15 at 1:51
1

I have already changed to avconv, so i apologise if the answer may be a little different in the classic ffmpeg, but i believe that not much difference should be there.

You can have a moving crop if you can create some sort of formula between the frame number and the position of the crop; but there are no analytic tools available to do that. This means that if the the speaker moves in a predictable way you could theoretically do that without an external application.

Let's assume, for example, that the speaker is moving from top-left to bottom-right in the picture. You could do something along the lines of:

avconv -i input.ogv -vf 'crop=200:100:n:n' -c:v libtheora output.ogv

In here i'm moving 1 pixel per frame, which is very unlikely how somebody would move.

Keep in mind that there are quite a lot of math functions that are provided by the filter interface of avconv (and i'm sure ffmpeg is similar).

So let's imagine that you want to move 1 pixel per frame, but only from 200 to 350 frame. You can then do

min(max(n,200)-200, 150)

For the frames <200, max(n,200) would generate 200, then we take away 200 to make it 0, and then min will return the first part up until 350 frame, where that result would become more than 150 and the constant will begin to be returned.

Although writing such formula by hand would be quite tedious, maybe it will be helpful in some cases as well.

Having said all that, if i were to have an option of having access to the software that is able to work on images frame by frame, i would go that route.

1

As I'm working on the exact same challenge, I'm currently thinking of abusing the ffmpeg vidstabtransform filter with a generated trf file. But I have not worked out the details yet.

Info on the trf file format.

1

Given that you have "a file that contains the cropping rectangle's X, Y, Width, & Height for each frame", you can use the filter_script option to do this in FFmpeg. The crop filter does not support timeline editing but the overlay filter does. So, if you create a blank canvas of the same resolution as your video and then overlay the video on top with co-ordinates such that your intended region is superimposed in the visible area, the goal is achieved.

Basic syntax is

ffmpeg -i in.mp4 -filter_complex_script file.txt -map "[out]" output.mp4

where file.txt looks like this:

nullsrc=WxH:r=FPS[cv];
[cv][0]overlay=-X0:-Y0:shortest=1:enable='eq(n\,0)'[b0];
[b0][0]overlay=-X1:-Y1:shortest=1:enable='eq(n\,1)'[b1];
[b1][0]overlay=-X2:-Y2:shortest=1:enable='eq(n\,2)'[b2];
...
[bm-1][0]overlay=-Xm:-Ym:shortest=1:enable='eq(n\,m)'[out]

The canvas W and H should be equal to the (fixed/static) crop dimensions and of the same FPS. Then each frame of the video is overlaid at (- X, - Y) so that the top-left corner of the desired region is at (0,0) of the canvas. Each overlay occurs for one frame. For efficiency's sake, if you have intervals where the crop region is static, you should use the between evaluator e.g.

[bi][0]overlay=-Xi:-Yi:shortest=1:enable='between(n\,1200\,1445)'[bj];

You can also evaluate using timestamp t instead of frame index.

  • I tried implementing this solution to see how it goes, just wanted a brute version at the beginning for testing so I haven't implemented between, I just added the position for each frame, for the first 3000 frames. My ffmpeg memory is now at 36GB and going with 1 frame per minute :( I guess this isn't the best solution for per frame haha – Eek Oct 3 '16 at 10:11
  • I'm currently out of town, but look into using trim filter to isolate one frame and output to Png. So n commands for n frames. Then concat the images to a video – Gyan Oct 3 '16 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.