Add 3D visual effects to videos

Question

I'm trying to get some hands-on experience with 3D visual effects on videos. Similar to those AR apps, I want to add a 3D model with effects, say a dragon, to a video. Something like this famous demo:

But I don't really want to make it this fancy with camera tracking. Starting with simplest approach, probably just placing a dragon on the field (like its last scenes) with fixed camera.

What's the best way to do it? Do we have some tutorials, free or open source tools or SDK to do it for videos?

Answer 1

Yeah that was actually the best opening ceremony ever...

Doing something like this yourself is a hefty task, but first, here's what you would need:

• A software for tracking
• A software for modeling / animating
• (optionally something for compositing / color grading)

Luckily, Blender (which is free) can do all those things, but has quite a steep learning-curve. Due to its popularity, you will find many tutorials and guides though. Here's step by step how I would go about something like your example:

• Find footage to composite the elements on. It should be at least 1920x1080 and with as little compression as possible (it's best to record your own).
• Load the footage in a tracking-program. You can use Blender, After-Effects, Nuke or PfTrack - all are viable.
• After tracking, you can create a virtual camera that flies around in your scene according to the movement of the real camera. Export this or work with it from inside Blender if you're only using Blender.
• Now model out and animate your 3D-Assets and place them in your scene. Since the tracked camera matches the movement of the real camera, your 3D-Assets should stay in place and look like they're pinned to the ground.
• Apply your textures and shaders (and I can't stress how important this is) light your scene properly. Try to observe where the light is coming from in your original scene and place virtual lights accordingly.
• Render your scene, preferably as a .png or .exr-sequence, with only your virtual Assets visible.
• At last, composite the background and foreground together. You can do this in Blender as well, but programs such as After-Effects or Nuke give you more control over your look (they're not free programs though).

If your tracking looks off, make sure to delete any floating tracking-points that might have picked up false movements. Try matching the white-balance in compositing (the shift from teal to orange and purple to green). For added realism, you can create a "shadow-catcher" which will itself not be visible, but render out the shadows cast upon it. Those shadows should be included in a separate "pass", so you have your foreground, shadows and background. This gives you more control over the darkness and opacity of your shadows if you want to adjust them. And at last, if you want to go all in, you can capture an HDR environment from the place you were filming at - basically a 360° image of the surroundings. You can add this to the 3D-Software you're using to accurately display reflections on your foreground.

I hope this helped you out, have a great day!