How are 3-dimensional overlays synchronized with videos?

Question

One popular feature on TV nowadays is to create 3D overlays on top of the recorded content. Also, as the camera changes direction or position, the 3D content view is subsequently adjusted. How is this implemented?

Answer 1

There are two different approaches that can be used. Motion/Camera tracking and Motion Control.

In Motion or Camera tracking, multiple non-coplanar fixed points in the scene are tracked and from that it is possible to extrapolate the position of the camera within the scene. This information can then be fed into a virtual camera used to generate the output from the 3d model such that when the 3d model output is superimposed over the video, the movements line up.

Motion Control is similar but uses a special camera rig to either track the camera movement exactly or to reproduce a movement plotted in the 3d software exactly in real life. This is the more precise but also much more sophisticated approach as it requires far more elaborate gear to do. It also allows for more accurate tracking of things like change in zoom verses actually moving the camera in (which are otherwise only evident by fairly subtle shifting of perspective and vanishing points.)

The result of both is effectively similar though. You end up with data on how the camera moves in the real life scene and it matches the movement done in the 3d scene. There are other factors such as lighting that also have to be matched for the best looking composition. Then both outputs are brought in to a composition package like After Effects where they can be overlayed and blended together. Sometimes various additional effects like depth of field and other virtual camera effects may be applied to help it blend.

Another thing is that if anything in the real scene needs to move in front of the 3d object, then they need something called a mask layer. This is normally produced by rotoscoping, which is a process of updating the image on every frame to follow the subject. This mask then tells the compositing software that the real video plate should be placed over the 3d plate on the areas where the mask is painted. Otherwise the 3d plate is used over the real plate. Alpha channel blending is used to convey information about the transparency of a 3d shape such that the image behind it can be visible.

If the 3d object has any refractive properties, then the real life plate may be fed into the 3d software as an environment. Then the 3d render engine can actually sample information from the environment in order to figure out refraction in things like a glass object or a 3d object with a mirror. Since it is mapping a 2d environment onto a 3d one, a lot of the selection of how the environment is applied would be done manually though.