Keying + Tracking onto Tablet/Smartphone with Green Screen Display in Adobe Premiere or After Effects

Question

1. I shot a commercial and in the commercial we have people holding an ipad or iphone and the screen of the devices were green like the pics below. Now I have to overlay some UI/UX from apps and web platforms onto the devices. I've never done this before and need a tutorial on how to do it. I know it involves keying, tracking and Adobe After Effects. There isn't much movement besides slight hand shaking and a tiny movement.

2. Also, I never know when to use tracking markers on the ipad/iphone green screen. We use an app that allows you to have a pure green screen or little tracking markers like the pics below. I've been told by some vfx people that they prefer just a green screen, but others say they need tracking markers because it will take super long if there are no tracking markers. They would have to do it frame by frame. So what I think I've gathered is that - if the green screen is completely still it can be a pure green screen, and if there is movement involved you need tracking markers?

http://bit.ly/1fHFnob

http://bit.ly/1IqDGlP

Answer 1

Trackers never hurt and there's almost no reason not to use them, but there's no substitute for pre-production and field testing. The only failure is failure to plan.

Ideally your markers would be high contrast, however you can place darker green markers on a green screen which can be entirely keyed out if the artist doesn't need them, but will still be useful if they are needed.

Since you've already shot this without markers, your best hope is a three-point tracker that can find and hold the screen edges. Since you say you have minimal motion, this shouldn't be too hard.

Answer 2

With these kinds of things, I suggest not even using a green screen in the first place, especially one that is in an app and casts a green glow on the actors hands or body and picks up on reflections. Its much easier to have, instead, either a green piece of paper (which honestly will not really work in daylight in terms of the key but will at least give you reference for the screen), and always use tracking markers. In my experience circular ones like the one you have in your app in the first picture work nicely.

I recently embarked on something similar:

Key takeaways were: - record in 50fps with a high shutter rate (to avoid motion blur which makes things difficult to track, can always add it back in) - use circular tracking markers with a clear differentiating color in the center (if you don't plan on green screening)

I used Mocha via AE and exported the corner pin data, and it was a lifesaver. As you said you don't have that much movement so I doubt you'll have to do much in the way of refining your track like I did (see green keyframes). But you will have to add an exclusion layer (basically rotoscoping a mask) around the actors hands if they enter the frame of the screen.

In my case I had to do a lot of manual tracking after exporting the corner pin data for sections where the actor walked across the screen and we didn't have all 4 tracking markers in the shot. But I doubt you will have to do anything like this.

Here are some good tutorials:

https://vimeo.com/73685360 - in depth, helped me a lot with the refining of the tracks

https://www.imagineersystems.com/videos/webinar-difficult-tracking-challenges-a-mocha-project-breakdown/ - good for learning about how mocha works

- dealing with fingers

- basics

Oh and here is my final result in case you were curious:

That little black difference between the monitor and the bevel really helps sell it in my opinion!