Why deshaken video looks "wobbly" and how to fix it

Question

When I filter a shaky video through deshaking filter (that fixes frames positions and rotations), the resulting video looks "wobbly" (example gif), i.e. while there are no per-frame displacements or tilt, there is still per-frame deformation.

How is it called and how do I get rid of it?

Note: The question is not about motion blur, it is about geometrical deformation that is not rotation and moves.

Update: I have experimented a bit with various videos and found out the case where the problem is visible the most.

If camera is held at right angle to the object (), I get only motion blur after deshaking.

But of camera is held at angle to the surface (), then I got this type of distorsion when moving the camera: .

Answer 1

The problem is that this is what the original footage looked like. While you can stabilize the objects themselves by tracking, you still have motion blur from the movement because the objects moved within the time each frame was being captured. Additionally, you also have perspective changes if the camera is moving in 3d space in addition to having the angle change.

This motion blur appears regardless of the stabilization as the stabilization simply repositioned the video to keep objects in the same place, but you can't get rid of the motion blur without completely synthesizing the image to produce something that resembles what the frame would have looked like without any motion blur. This is both exceedingly difficult and also extremely likely to introduce as many artifacts as it fixes.

Similarly, the perspective issues result in a shifting perspective without corresponding movement as the camera position changes are counteracted by providing a perspective correction, but the contents of the video are still viewing from the wrong angle, so what portion of the surface is visible changes very disconcertingly.

This is most obvious if we take a hypothetical situation where we have a camera looking at a dice. We start looking straight down at two dots. Then, after a few seconds, the camera suddenly gets jerked around to the front and we are looking at 4 dots. To smooth that camera motion, we would need to make it look like the camera movement was smooth between the two positions, however, we don't actually know the 3d geometry of the dice, we just know that it shifted relative to other things in the scene. We can't simply apply some distortion to each frame, because half of our frames only show the 2 dots and half our frames only show the 4 dots.

Most image stabilization is limited to working with whatever detail is available in frame and warping and adjusting it (and in advanced cases, independently warping individual objects on screen). We can shift the perspective of the image so it seems kind of like the camera is moving, but we'll still see the two dots until suddenly it changes to 4 dots. The camera movement will feel a bit more smooth, but the perspective was telling us we were looking at half way between the side and the top while the number of dots shows us to be looking at the top. This disjoint can't be fixed without synthesizing new data, which is super hard.

The best bet to avoid motion blur is to use the fastest possible shutter speed when capturing footage. This will reduce the amount of motion blur present and help make stabilized footage look much better. After the fact, it is exceedingly far more difficult to correct for.

For perspective, there isn't much you can do other than trying to limit the amount of changes in position of the camera.

There are some interesting bits of research by organizations like Microsoft Research (there is also an interesting tidbit they have here about what the currently available state of the art is) can build point clouds that can be used to correct for perspective issues if the video is complete enough, but such research is exceedingly ground breaking and mind numbingly complex, so you won't find it in the open source world yet (or even the commercial one quite yet.)

Answer 2

Since you've not accepted the other two answers (which I thought were very good) because they focus on motion blur and parallax/ perspective, I'm guessing that the "wobble" you're talking about due to the rolling shutter effect that CMOS sensors introduce to footage.

In a nutshell, CMOS sensors scan an each frame of an exposure very quickly from top to bottom. While the entire exposure occurs within the 1/60th of a second it takes to capture a 30fps 180° shutter image, for example, the capture is not instantaneous. This causes the image to skew temporally, an effect which is most visibly noticeable when the camera pans quickly or shakes.

The best way to avoid this problem is to use a camera with "global shutter". CCD sensors in old camcorders did not suffer from rolling shutter, but because CMOS outpaced the development of CCD and because not many people noticed rolling shutter, global shutter cameras are now a niche market.

Congratulations. You fall squarely within this niche. But consider yourself in good company. One of the primary markets for global shutter cameras is the VFX industry. This is because rolling shutter is A) particularly problematic in matchmoving software like Boujou and B) particularly noticeable after software stabilization.

The trouble with rolling shutter, is that it doesn't actually cause a perspective shift, although it may appear to. For example, point your cell phone camera out the driver's side window of a moving car and snap a picture (while someone else is driving, obviously). The image will slant in one direction. Now snap a picture out the passenger window. The image will slant in the opposite direction. So while you could easily correct this "slant" in Photoshop with edit->transform->skew or edit-transform->perspective, there's no way for software to perform the correction automatically, because it doesn't know which window you were shooting from.

Another example: Take a picture of a spinning propeller, and the blades will appear to curve, not slant. Rolling shutter distorts images in ways that post software can't compensate because the variables include camera motion, scene motion, and direction of scan.

I realize this information doesn't help fix footage you've already shot, but hopefully it will help point you in the right direction. I'm not too familiar with many software packages to fix this problem. After Effects has a "rolling shutter" plugin which may be sufficient for your needs. Perhaps someone can recommend some others.

Answer 3

When a camera is recording video, it takes many still photographs per second.

For each of these photographs, the camera opens its shutter (either mechanical or electronic), and allows light to hit the sensor for an amount of time. During this time, any object that moves, or any motion in the camera, is captured as motion blur. Imagine watching an athlete running - theirs legs, arms are blurred in the image.

Stabilising footage uses algorithms to detect and calculate the movement in the shot. It then moves and warps the image so the greatest amount possible doesn't move (I.e. The background - a "steady" shot). However, the motion blur captured in the image still exists - this creates the jitters and wobbles you are seeing.

Unfortunately, it's also (to my professional knowledge) impossible to remove - it's part of the original, recorded image.

If you have manual control over the camera, you can try changing the shutter speed to a faster value (1/big number or small number°) to reduce the amount of motion blur recorded. However, this can introduce other problems, including strobing, from a severe lack of motion blur, and making it more difficult to get an exposure in dark scenes.

Answer 4

The common name for the effect you're talking about is "Jello" and it's caused by rolling shutter in combination with camera shake.

Each frame is distorted by the rolling shutter because, for example, the left edge of your fence is at a different place left to right as the "shutter" selects the pixels as it rolls from top to bottom of the frame, resulting in a trapeziodal distortion of that frame. Next frame the shake may cause it to distort in the other direction. Over the course of several frames this results in the "Jello" effect with vertical lines apparently wobbling side to side. Running stabilising filters sometimes makes it look even worse because the wobble can be hidden a bit when there's a lot of other global motion going on.

Here's an OSS solution, it's a plugin for VirtualDub, it corrects shake, roll, and rolling shutter / Jello:

http://www.guthspot.se/video/deshaker.htm