Track and replace a t-shirt design on a moving, distorted garment

Question

I'm wondering what would be some viable techniques to replace a t-shirt design some person is wearing in a footage with another design. Here's a simple example: https://placeit.net/c/mockups/stages/hipster-man-wearing-a-round-neck-tee-video-outside-a-store-at-night-a13565

This is online tool that allows anyone to upload an image and then puts in on a video footage. I'm interested how it might be reproduced in a program like After Effects or Premier.

It's important that image should follow any distortions/creases of the garment.

I've used Mocha planar tracking in After Effects, but I don't think that's sufficient because it only tracks one or few planes and it would be really hard to replicate distortions/movements of the garment. Is there some easier way?

I'm able to print any tracking markers, lines or anything else on the shirt that would help.

Answer 1

Nuke's Smart Vector toolset is built for this exact problem. I see you tagged After Effects, but you did not specify that After Effects is needed. I've done this a few times before and Mocha is your best bet for AE, but as you already know it has it's limitations. Nuke can actually track the creases and folds (though of course it isn't perfect). The biggest factor I've seen in my tests is shot duration - the longer the shot the more the vectors get all messed up. That said it can sometimes do magical things! I believe you can rent Nuke for a decent price if you just need to do one shot.

Here's a tutorial showing how to replace part of a shirt. It is a bit older, but because it addresses exactly what you are trying to do I thought I'd link it:

If you absolutely need to use AE I would combine a mocha track with a puppet warp which is hand-animated. Using Mocha you can stabilize the footage, overlay your logo and keyframe the puppet warp, and the re-track it back into the original footage.

Answer 2

My god, this is practically the nightmare of any vfx-artist out there. Yes, it is possible - but at what price? You need to:

• track the shirt using Nuke, pfTrack or any sort of high-end tracking software.
• erase the current design, either using the logo if you have it as a file to subtract it, or by painstakingly rotoscoping and patching up every frame of your footage.
• recreate the lighting, folds and creases at the places you patched out the old logo.
• attach the new logo to the tracked motion
• recreate parallax and layering of the fabric in creases, most likely by keyframing every frame of the footage once again.
• color correcting the replaced part so it fits in with the rest of the shot.

This may sound straight-forward and manageable, but it is a hell lot of work. Make sure to really think about this before attempting it - maybe go out with your t-shirt design and shoot the scene yourself. You will probably save yourself a lot of nerves and money.