How to get smooth motion when changing clip speed?

Question

Let's say I have a 30 fps video created by a camera panning at a constant speed while filming some static scenery.

When I watch the video, the motion in the video is perfectly smooth.
When I double the speed, the motion continues to be perfectly smooth.
When I halve the speed, the video looks choppy but again the motion continues to look smooth.

The problem comes when I try to change the speed by irregular numbers. Let's say I want to change the speed of the video to 120%. I then export the video at 30 FPS. The motion in the video then becomes visually choppy, jerky.

I quickly figured out what is happening:

By increasing the speed, my 30 FPS video becomes 30 * 1.2 = 36 FPS. But I am exporting at the standard 30 FPS so every 6th frame is dropped. This means that every 5 frames the camera moves at double speed. If I look at the motion frame by frame the camera movement is: slow, slow, slow, slow, FAST, slow, slow, slow, slow, FAST, slow, slow, slow, slow, FAST, etc.

So finally my question is what option do I have to get smooth motion or hide this problem when changing the speed of the video by small intervals?

Shooting at a higher shutter speed is not an option in my opinion, as it just blurs the motion until the problem is not visible.

I use Adobe Premiere and Adobe After Effects. Will the Time Warp plugin in After Effects be able to create smooth motion by using motion vectors? What if there is active movement in the scene? Will it still look good?

I have yet to see people truly understand this problem. How is this effect called? Does anyone have any link or article explaining what I described?

Answer 1

Shooting at a higher shutter speed is not an option in my opinion, as it just blurs the motion until the problem is not visible.

I don't get that part. Isn't hiding the problem until it is no longer visible what you want?
Also, a higher shutter speed results in less blurred frames, not the other way around.

I quickly figured out what is happening: by increasing the speed, my 30 FPS video becomes 30*1.2=36FPS. But I am exporting at the standard 30FPS so every 6th frame is dropped.

This is the default behaviour of the Media Encoder. Adobe calls it Frame Sampling. As you said, since frames will be dropped at regular intervals, it will produce curious results.

So finally my question is what option do I have to get smooth motion or hide this problem when changing the the speed of the video by small intervals?

Use one of the other Time Interpolation modes Premiere Pro has to offer. At the bottom of the Export Settings dialog, there's a dropdown menu containing three options: Frame Sampling, Frame Blending and Optical Flow. You will find the same dropdown menu in the Speed/Duration panel that can be accessed via the context-menu to change the interpolation mode of individual clips.

From the documentation:

The Optical Flow feature in Premiere Pro uses frame analyses and pixel motion estimation to create brand new video frames, resulting in significantly smoother speed changes, time-remapping, and frame-rate conversion. [...]

Frame Sampling repeats or removes frames as needed to reach the desired speed. Frame Blending repeats frames, and it also blends between them as needed to help smooth out the motion.

Here's another helpful article on using the Optical Flow interpolation.

I suggest you try all the interpolation modes and see what works best for your footage.

One more thing, you probably don't want to hear that, but 30 fps video is really bad for doing slow or fast motion effects (except maybe timelapses with 200% speed or more). As you explained yourself, there's not that many frames to go on so some frames will need to be dropped, synthesised or blended in some way which looks subpar at best. The interpolation algorithms do a decent job, but it will never look as good as it would with footage that was shot at a higher frame rate in the first place.

Answer 2

There is a substantial body of literature explaining how frame rate and shutter speed relate to achieve desirable perceptions of motion. In general, a shutter speed that is 1/2 of the frame rate (1/48th of a second for a 24fps frame rate) gives the best results. But there are reasons one would choose a (relatively) shorter or longer shutter speed, including the reason that the playback rate will be different than the capture rate. But first and foremost, one should understand how playback rate and shutter speed interact before worrying about interpolation strategies. The literature discusses not only this, but also acceptable rates of panning speed for a given field of view and shutter speed rate. Here is but one example of such a discussion.

There is also a substantial body of literature explaining how frame rate interpolations work or break down under various conditions of frame rate changes (some of which you have already noticed). Some frame rate interpolations work best with more motion blur. Some work better with less motion blur. And if you have not done the math ahead of time, you can find yourself with a frame rate and a shutter speed that will result in universally poor results regardless of the interpolation strategies available to you. In general, if you want to give optical flow a chance, you need sharp pixels, not blurry ones. Optical flow can both create "tweens" (synthetic frames in between actual frames captured by the sensor) and it can create motion blur for all frames (both real and synthetic). This is a computationally expensive process, and it is not uncommon for high-end production houses to actually customize their software to make it work best for the footage they have. But if you take care ahead of time, knowing what shutter speed will minimize the motion blur within your sequences, and knowing what types of subjects or objects to avoid so as not to break the quality of your optical flow engine, you can use software to realistically create in-between frames. If you don't prep ahead of time, you are likely to get results that are very artificial, calling attention to the algorithmic limitations of the software rather than the brilliance of your art-direction.