I'm trying to overlay 2 videos which both have depth layers (e.g. how far a pixel is away from the camera) using ffmpeg filters. (I have a few hundred of those videos so I'd like to do batch job out of this)
the video files are perfectly in sync with each other and their depth layers. The Files are saved in the lagarith lossless encoding (which ffmpeg can read). The video files are also using a greenscreen which I already filter using this command:
ffmpeg -loglevel info -hide_banner -i "%fst%" -i "%snd%" -filter_complex "[1:v]chromakey=0x00ff00:0.5:0.0[ckout];[0:v][ckout]overlay[out]" -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result.mp4"
This command takes video %snd% makes the greenscreen transparent and puts the result on top of %fst% and writes it to result.mp4
The issue is, that I'd like also include the depth layers into the merge process, so that the nearer pixel is selected from both videos (the pixel with the brighter depth layer pixel). (the greenscreen can be rendered transparent in a later iteration, so we don't have to worry about that)
For the depth layers the variables %fst_depth% and %snd_depth% are defined.
Does anyone an idea at what filter I should take look at or even a full solution for this issue?
Thanks in advance.
UPDATE: I have partially a working solution:
@ffmpeg -i %fst% -i %snd% -i %fst_depth% -i %snd_depth% -filter_complex "[2:v][3:v] blend=all_expr='if(lt(A,B),0,255)' [mask] ;[0:v][1:v][mask] maskedmerge [out]" -shortest -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result2.mp4"
The issue with this one is, that for some reason, the second video objects are missing most of the colors and are partially transparent for some reason. The weird blocky glitches got solved (depth map inaccurate), but the colors are still weird:
Example of a Video Frame: 
Example of the according depth Frame (black near, white far): 
