Achieve a video warping effect, where it appears like the camera was shifted

Question

I am writing this topic in regard to a very interesting effect I've observed while watching tennis games. An example can be found on the Facebook page called "Tennis Channel", e.g. the last posted video there with the shots of Grigor Dimitrov.

In the video, one can see that there is an effect of a "moving camera" that moves to the left/right when the footage is paused on a particular frame.

During this simulated movement of the camera the audience gets slightly blurred so this appears to be some video simulation or similar? I am pretty sure that there is no hawk-eye involved here. In addition to that there are other effects of zooming in and some "paint" alike effects on the relevant parts of the court.

How can I create such an effect?

Answer 1

This particular effect couldn't be done without some pretty advanced computer vision algorithms and a whole lot of processing. Based on the screen shot alone, it appears that it tries to identify flat surfaces and objects within that space so that it can alter the camera's perspective.

Note that this is quite imperfect as you have no material for the actual side of people's heads and such. Additionally, you can see that the system they were using doesn't understand transparent objects well. It saw the net as a wall and modified it differently from the court, which is why we end up seeing that disjointed line where the net is.

I'm not aware of any specific system that does this, but I wouldn't expect it to be cheap and I really, really wouldn't expect it to be free.

If you don't care about cost, the fundamental start to it would probably be something along the lines of SynthEyes, as long as the camera does move some. This can track where the camera actually moved and use that data to establish what depth things were at and what backgrounds looked like. From that, it may be possible to create a virtual scene which you can effectively move the position of the camera in.

If the shot isn't moving, then I couldn't find any tools that would be helpful and your best bet might be to try contacting the company that produced the video directly to ask them what they used. It appears to be something very specialized and not particularly well advertised as far as I can tell.

Answer 2

To achieve this effect, all you do is re-project the 2d video onto 3d geometry, move the 3d camera to create a parallax displacement, and re-render. I'm only familiar with the tools available in Maxon Cinema 4d, but I'm sure that Max, Maya, Modo, Blender, et al can do this too. It should be relatively easy with something as geometrically simple as a tennis court. Incidentally, the use of this technique is the reason you notice distortion in the paint on the tennis court, the audience, and the players. The 3d geometry isn't detailed enough to move the pixels correctly. Also, there's no foolproof way to automatically generate the textures of things behind things when the camera moves. Chances are, for sporting events, broadcasters like this use an automatic solution akin to Photoshop's content-aware fill, which will never produce results as good as a human matte painter. Google "3d projection mapping" to learn more about this technique.