1

I'm trying to overlay 2 videos which both have depth layers (e.g. how far a pixel is away from the camera) using ffmpeg filters. (I have a few hundred of those videos so I'd like to do batch job out of this)

the video files are perfectly in sync with each other and their depth layers. The Files are saved in the lagarith lossless encoding (which ffmpeg can read). The video files are also using a greenscreen which I already filter using this command:

ffmpeg -loglevel info -hide_banner -i "%fst%" -i "%snd%" -filter_complex "[1:v]chromakey=0x00ff00:0.5:0.0[ckout];[0:v][ckout]overlay[out]" -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result.mp4"

This command takes video %snd% makes the greenscreen transparent and puts the result on top of %fst% and writes it to result.mp4

The issue is, that I'd like also include the depth layers into the merge process, so that the nearer pixel is selected from both videos (the pixel with the brighter depth layer pixel). (the greenscreen can be rendered transparent in a later iteration, so we don't have to worry about that)

For the depth layers the variables %fst_depth% and %snd_depth% are defined.

Does anyone an idea at what filter I should take look at or even a full solution for this issue?

Thanks in advance.

UPDATE: I have partially a working solution:

@ffmpeg -i %fst% -i %snd% -i %fst_depth% -i %snd_depth% -filter_complex  "[2:v][3:v] blend=all_expr='if(lt(A,B),0,255)' [mask] ;[0:v][1:v][mask] maskedmerge [out]" -shortest -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result2.mp4"

The issue with this one is, that for some reason, the second video objects are missing most of the colors and are partially transparent for some reason. The weird blocky glitches got solved (depth map inaccurate), but the colors are still weird:

Example of a Video Frame: Video frame Example of the according depth Frame (black near, white far): Depth frame

2
  • 1
    About the glitches, check manually if the depth maps are accurate in those regions at those times.
    – Gyan
    Sep 14, 2018 at 15:56
  • @Gyan you were correct, the render truncated the values to 8bit which caused these artifacts ... thx Sep 15, 2018 at 19:14

1 Answer 1

0

The main issue was, that the Z-Layer was not accurate enough, as Gyan said. Addtionally the condition needed to be changed from lt(A,B) to lte(A,B).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.