What program does this kind of tracking and how hard is it to achieve good results? I've seen this done in a lot of vfx breakdowns.

enter image description here

  • Basically without any mocap studio (or mocap hardware at least) it's hard work (or impossible) to reconstruct the exact motion. In order to deform (match) the head geometry some rigging skills and extra markers on the body are required to drive the bones by the markers. Might be interesting: fxguide.com/featured/…. Simple face feformation test via blenders object tracker: youtube.com/watch?v=C_oNv0uLxMA
    – p2or
    Feb 16 '16 at 20:21

I come from a Hollywood visual effects background, and I was a matchmove lead for a couple of shows. For the above process mentioned, Vfx facilities do it all the time.

Like Poor mentioned, it is very labour, skill and time intensive. Unfortunately, in almost all cases if you see the actor in the shot without a tracking suit, and if we need to track / match animate him, we will ALWAYS have to manually match-animate his motions by hand (and by eye).

When we think of matching character motion we tend to immediately associate that process with the imagery of behind the scenes publicity shots for Ironman, Pirates of the Carribean, the Golem in Lord of the Rings, The Avengers, Avatar, etc. In these behind the scenes shots we see men in motion capture suits, and face cameras.

You need to know that in these cases, these men in suits are meant to be entirely replaced with computer generated characters in the final shot. Golem, Ironman, The Hulk in Avengers, blue aliens in Avatar, were all replaced with CGI.

The fully replaced CG characters always get the limelight. For the tracking team, a huge amount of effort also needs to be put into manually tracking and matching live actors that do not get replaced by CG at the end of the day.

Why do we need to track live actors with a painfully and accurately matched up model in 3D? It is for interactions with the CGI character that never existed on the set when the footage was shot. For example, when Frodo Baggins was holding a dagger wrestling the Golem, at the time of shooting, the dagger (and all metallic parts of his costume) never have the Golem reflected in them. Similarly, there was no Golem to cast an accurate shadow on Frodo too, in the areas where they contact.

Thus in all the instances where you see any CGI character interact with footage elements that is shot live (physical location, physical props and physical actors), we have to create a digital counterpart in 3D and match them exactly, place them correctly in relation to the other objects in our 3D workspace, in order to capture shadows, reflections, colour diffusion, of the digital character.

Besides having to match and body-track digital counterparts of live actors for reflection and shadow-catching, we also need to have them match-animated for dynamics simulation purposes.

In John Carter I was in the digital costumes department, we have humans interacting with alien characters which are fully CGI. The characters all have flowing, draping costumes and accessories. We had to match-animate shots so that CG clothing items, hair and fur can collide, slide across and rest correctly on live human actors. If we do not create a digital counterpart to those actors and match them, those costumes, hair and fur will go straight through the live actors in the footage because those things never existed in the same 3D space where the simulation took place, for them to interact with. Finally to answer your question, lots of effort need to go into tracking a character. In the beginning we have the modelling stage, where we create a human model that need to have the same head, body and limb proportions as the actor. If these are wrong, you will fail to match up due to arms being too short, head being too large, and so on. Sexy industry methods like scanning of the actual actors are only reserved for films of very high budget, and where the effort and hours put in are justified. Most of the time, 3D artist will model them by eye, taking lots of references from the Web for proportions and such. Takes up much labour and time either way.

Then comes the rigging stage where the otherwise rigid model is put through the technically involved process of adding complicated controls that will later allow match-move animators to articulate the digital model to match the live actor.

Finally the match animation or body tracking artist will have to use their human understanding of perspective and how human actors move, to re-create the exact movement of the actor in the shot, from a 2d image to an accurate representation of movement in a 3D environment. Trust me, this is no easy feat and takes years of experience before an artist can become good at it.

Body tracking is several orders higher in difficulty than tracking inanimate, non deforming objects like a moving knife or a pole that was going to become a light-saber. Matching these rigid objects only require an animator to handle a set of translation (in X, Y, Z position), and a set of rotation (in X, Y, Z axes).

In body tracking, each and every joint of the human body has many different possible positions, many possible degrees of rotation. To control and manipulate one digital character is more difficult than tracking multiple non deforming objects.

Thus in a smaller budget or indie production, such accuracy is usually not required, and not viable given the time and resources needed. We would usually use cheats, hacks and shortcuts to get by.

Finally finally, here's the difference between the motion capture process that are featured behind the scenes, versus matchmoving a character (which never gets featured anywhere).

The purpose of capturing the motion for characters, is that they are ultimately going to be completely replaced by full CG characters. The reason for capturing motion of people in Mocap suits is so animators can acquire the mocap actirs' performance. Thereafter, the Mocap actor's performance is always modified. Due to jitter and inaccuracy in the captured data, Mocap data almost always needs cleaning up by artists. In most cases the captured movement will be modified, exaggerated more, acting and poses are pushed for greater impact. Some poses are changed because they do not work when viewed through the camera in the shot, or when composited in with real actors' acting, timing and reactions doesn't match or pace very well. Sometimes mocap data is used only for timing reference to the characters acting, as a guideline for the animator to start animating the digital character by hand from scratch. All these things are done so that we have a more engaging shot. Thus motion capture has no intent to match the action of an actor in shot. It serves a more artistic purpose - acting.

The purpose of matchmove tracking of an actor serves a more technical purpose, to get CG elements to "gel" with the live elements in the shot, to ground them firmly, getting them to interact and react with each other, therefore they need to match the live action as close as possible, if not exactly.

I hope this explanation is clear enough, and does not further confuse you in the end.

  • Thank you for such a great explanation, I was looking for some time for a good background explanation on human cg tracking. So in the end it all comes to manual matching everything, as I understand even the cloth folds.
    – Spider
    Apr 26 '17 at 1:42
  • 1
    Actually, a lot of things that were presented to us in Hollywood featurettes and behind the scenes, suggest that things were easy and with software/technological superiority, can be achieved at the push of a few buttons. Green screen is another process where the untrained viewer is led to associated green backgrounds with automatic colour keying processes where the artist just needs to click on a colour (like in photoshop) where the background would be removed cleanly & magically. That's not true! Check out Rotoscoping: goo.gl/yW2JgS. Apr 26 '17 at 2:30
  • 1
    Yep, I'm familiar with rotoscoping process, probably one of the most tedious movie editing process (of course not compared with manual 3D tracking).
    – Spider
    Apr 26 '17 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.