I'm trying to overlay 2 videos which both have depth layers (e.g. how far a pixel is away from the camera) using ffmpeg filters. (I have a few hundred of those videos so I'd like to do batch job out of this)

the video files are perfectly in sync with each other and their depth layers. The Files are saved in the lagarith lossless encoding (which ffmpeg can read). The video files are also using a greenscreen which I already filter using this command:

ffmpeg -loglevel info -hide_banner -i "%fst%" -i "%snd%" -filter_complex "[1:v]chromakey=0x00ff00:0.5:0.0[ckout];[0:v][ckout]overlay[out]" -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result.mp4"

This command takes video %snd% makes the greenscreen transparent and puts the result on top of %fst% and writes it to result.mp4

The issue is, that I'd like also include the depth layers into the merge process, so that the nearer pixel is selected from both videos (the pixel with the brighter depth layer pixel). (the greenscreen can be rendered transparent in a later iteration, so we don't have to worry about that)

For the depth layers the variables %fst_depth% and %snd_depth% are defined.

Does anyone an idea at what filter I should take look at or even a full solution for this issue?

Thanks in advance.

UPDATE: I have partially a working solution:

@ffmpeg -i %fst% -i %snd% -i %fst_depth% -i %snd_depth% -filter_complex  "[2:v][3:v] blend=all_expr='if(lt(A,B),0,255)' [mask] ;[0:v][1:v][mask] maskedmerge [out]" -shortest -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result2.mp4"

The issue with this one is, that for some reason, the second video objects are missing most of the colors and are partially transparent for some reason. The weird blocky glitches got solved (depth map inaccurate), but the colors are still weird:

Example of a Video Frame: Video frame Example of the according depth Frame (black near, white far): Depth frame

  • About the glitches, check manually if the depth maps are accurate in those regions at those times. – Gyan Sep 14 at 15:56
  • @Gyan you were correct, the render truncated the values to 8bit which caused these artifacts ... thx – Pingger Shikkoken Sep 15 at 19:14

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.