One popular feature on TV nowadays is to create 3D overlays on top of the recorded content. Also, as the camera changes direction or position, the 3D content view is subsequently adjusted. How is this implemented?


There are two different approaches that can be used. Motion/Camera tracking and Motion Control.

In Motion or Camera tracking, multiple non-coplanar fixed points in the scene are tracked and from that it is possible to extrapolate the position of the camera within the scene. This information can then be fed into a virtual camera used to generate the output from the 3d model such that when the 3d model output is superimposed over the video, the movements line up.

Motion Control is similar but uses a special camera rig to either track the camera movement exactly or to reproduce a movement plotted in the 3d software exactly in real life. This is the more precise but also much more sophisticated approach as it requires far more elaborate gear to do. It also allows for more accurate tracking of things like change in zoom verses actually moving the camera in (which are otherwise only evident by fairly subtle shifting of perspective and vanishing points.)

The result of both is effectively similar though. You end up with data on how the camera moves in the real life scene and it matches the movement done in the 3d scene. There are other factors such as lighting that also have to be matched for the best looking composition. Then both outputs are brought in to a composition package like After Effects where they can be overlayed and blended together. Sometimes various additional effects like depth of field and other virtual camera effects may be applied to help it blend.

Another thing is that if anything in the real scene needs to move in front of the 3d object, then they need something called a mask layer. This is normally produced by rotoscoping, which is a process of updating the image on every frame to follow the subject. This mask then tells the compositing software that the real video plate should be placed over the 3d plate on the areas where the mask is painted. Otherwise the 3d plate is used over the real plate. Alpha channel blending is used to convey information about the transparency of a 3d shape such that the image behind it can be visible.

If the 3d object has any refractive properties, then the real life plate may be fed into the 3d software as an environment. Then the 3d render engine can actually sample information from the environment in order to figure out refraction in things like a glass object or a 3d object with a mirror. Since it is mapping a 2d environment onto a 3d one, a lot of the selection of how the environment is applied would be done manually though.

  • Thanks! I take it there is no consumer-grade solution for this? Jun 3 '13 at 14:42
  • 1
    Using After Effects and the 3d motion tracking it can provide is the closest you are going to get to consumer-grade. It's still a pretty intense process though. The most expensive part would be the software for actually making the 3d models. Typical 3d animation software runs for several thousand dollars a seat unless you do something lower end with Blender (which is free). After Effects is only a few hundred dollars for a seat or can be obtained for $20 a month through Adobe's Creative Cloud. It would still be labor intensive to rotoscope though.
    – AJ Henderson
    Jun 3 '13 at 16:50
  • I guess the question is - can the software infer movement and rotation of the camera just by analyzing the image? Jun 6 '13 at 15:11
  • @DmitriNesteruk - yes, though not as accurately as direct collection. Any plane in the video can be tracked and the corners of that plane will reveal information about the rotation, position and focal length of the camera in question.
    – AJ Henderson
    Jun 6 '13 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.