Can you create a VR video that makes use of positional tracking?

Question

To my understanding there are basically 4 types of video:

1. Ordinary 2D videos - these can be viewed by pretty much any device that has a screen
2. 360 videos - these can be viewed by any smartphone that has a gyroscope
3. 3D videos - these can be viewed by 3D glasses along with the right kind of screen, or by using a VR headset
4. VR videos - these are a combination of types 2 and 3, and can be viewed by any VR headset, like a Samsung Gear VR headset or even a Google Cardboard headset

But I'm interested in whether there's a 5th type of video. My question is, is it possible to make a VR video that isn't just 360 + 3D, but also makes use of positional tracking? Such a video, if it existed, would only be viewable on a high-end VR headset like the Oculus Rift and the HTC Vive, which utilize both rotational tracking and positional tracking. They wouldn't be viewable on low-end headsets like Google Cardboard, or mid-tier headsets like Samsung Gear VR, since they only utilize rotational tracking.

If it is possible, how would you go about creating such a video? It's the equivalent of shooting an ordinary (360+3D) VR video not just from one point in a room, but from every point in the room. I imagine that it would be useful for creating virtual tours. In any case, even if you managed to create such a video, how exactly would you make that video by viewable by others? Are there any apps for Oculus Rift or HTC Vive that would allow you to play a positional-tracking VR video?

EDIT: I found out that there is such a type of video. It's called a "light field video" or "holographic video", and it can be captured with a special light field camera. But I still don't know how you would show that video to someone.

Answer 1

You can use photogrammetry if you had enough different viewpoints to build a 3d representation of the scene and apply all necessary texturing and lighting. The video would then play as an animated 3d scene rather than a traditional movie.

The amount of processing involved is high but it is possible. Microsoft's Hyperlapse video stabilization actually does some of this by determining a point cloud of the scene from some basic photogrammetry.

Software like Destinations on the Vive already use photogrammetry for reproducing landmarks in a fully explore-able 3D world. There are also several animated projects which are composed of scenes that you can fully explore. 8i also has done some early work in video like you describe. The quality level still leaves a lot to be desired, but work is being done in trying to accomplish what you are talking about.

As an early Vive owner who has tried all these things first hand, I'd say "volumetric video" is at the interesting experimental phase, but not really that great yet. I'm sure we'll see significant improvements in the future though as the technology is refined, particularly since it has implications outside of just VR and AR for things like computer vision.

Answer 2

For this you are going to have to build a 3d-world. This is because if you want to move position you are going to have to be moving in 3d. The easiest way to build up the 3d scene is with photogrammetry like Aj Henderson said.

I would load it into a game engine because a game engine is designed to render 3d scenes read time and already lots of them have vr features built in. For example Unreal Engine or Unity.