Tool to find Duplicate Video files ignoring tags

Question

I am looking for an automated method to identify duplicated video files from my large collection of videos.

I have used fslint to do this in the past, but I started fixing the EXIF tags on my videos, which makes fslint identify them as different files. The only difference between the original file and the modified files are the EXIF tags, which in most cases have simply been time-shifted to correct for timezone or camera offset. The videos have not had any encoding or conversion of any type.

Is there a tool something that I can use to identify duplicate videos based only on the AV streams, i.e. ignoring the tags? I'm essentially looking to duplicate fslint-like functionality, but ignoring the EXIF tags. Does such a tool exist?

I have a suspicion that ffmpeg can be used to do this, but all I've been able to find so far is this answer, which shows how to blend two video streams together and display the difference. Doing so "works" in the sense that I get a solid green screen with my duplicate files (indicating no pixel differences), but I can't use that in a script to handle hundreds of comparisons because I still have to visually inspect the result.

Is there a way to get ffmpeg to give a yes/no answer?

I've seen this answer, but found the answer to be entirely too manual. I need an automated process that can go through large filesets in one run.

Answer 1

It might be overkill in your situation since the videos aren't actually different resolutions, frame rates and or codecs, but still :

As with another answer https://video.stackexchange.com/a/33521/32436 :

You could check out https://theophanemayaud.github.io/video-simili-duplicate-cleaner/ (macOS and Windows binaries available) It's open source, based on vidupe and QT and relies on ffmpeg.

The idea is to extract a few (eg 2, the first and the last) frames, and compare them. If the duration is close, and the frames look similar : you've got a duplicate.

It uses opencv for frame comparison, and ffmpeg for frame extraction (ffmpeg is also for video metadata which is very useful when trying to tell which video to keep !). Since it's open source, you can improve it to fit your needs !

Because it looks at the video data, you shouldn't have problems of tags or other metadata that has changed !

Answer 2

I had the same question and I managed to answer it by creating an open source program around ffmpeg and findimagedupes. Feel free to try it : https://github.com/PierreCrette/VideoDedup/blob/master/README.md