Very Shaky camera but a constant background

Question

I recorded a video of a speaker at a graduation ceremony using my tablet. The first half of the recording, I held the tablet in my hand so it's shaky and just terrible. I finally realized that I could capture the scene I wanted just fine by using the tablet's stand and placing it on the ground.

Throughout the entire video, the background never changes, the speaker is at a podium and it can act as a focal point...other than the speaker, there is no motion in the scene (other than audience heads in the foreground.

I tried to use After Effects Motion Warp but it looks horrid but it would work perfectly if I could select a "static background" from one frame and apply that throughout the rest of the video.

Is something like this possible?

Answer 1

While it's fairly easy for it to figure out the camera position from the shot, it is not nearly as easy for it to identify subjects cleanly. In order to do what you are talking about, the subject would have to be identified and extracted from the background without any extra bits coming along.

Computer vision algorithms simply aren't that advanced yet and such work is always either done by rotoscoping (manually outlining the subject on each frame) or using a chroma or luma key backdrop that can be removed. Stabilization works by identifying straight lines and tracking them generally. The border fill in motion warp then works by attempting something similar to what you are talking about by looking at what appeared to be there in previous frames, but it doesn't really work all that well yet either as it is a new technology.

The same problems that prevent them figuring out accurately what was there in previous frames are what prevent separating the subject from the background (the ability to identify, with precise accuracy, the same object in one image to another. You would end up with a subject that hovered oddly on the background and had a ring of background around them and/or left bits of them behind if you tried it with current level of technology.

Using motion warp and applying a crop or bypassing the auto-crop and manually cropping and moving around the frame is probably your best bet. This is how I usually handle stabilization issues.

Answer 2

Deshaker which is a filter that can be applied in Virtual Dub, could fix the original shaking some, on mostly default settings (where the background moves).

Dehaker also has many many settings possible, you could mask out areas , as long as the person moving was then larger than the masked out area, and you got the settings all correct, it might be able to fix some of that up for you.

In most situations of any EIS (electronic image stabilization) It is cropping up a smaller portion of the whole video, and the methods for filling in where there is no picture at all (borders) are not great.

I find that setting everything and testing by looking at the vectors, using a excessive ammount of smoothing, and then using only about 1/2 of what would completly fix it , works ok to make the pictures a little more viewable.