After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting.

To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.

How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually.

Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?

  • 1
    Very interesting question! – Dr Mayhem Nov 12 '12 at 10:40

I'd recommend a hybrid approach using both computers and people.

  1. Bucket the videos by their length (round to the nearest second)
  2. For each bucket, use ffmpeg to generate thumbnails at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)
  3. Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.

You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse. Good luck!

| improve this answer | |
  • Thumbnails are a great idea! The new videos came in as FLV, which I can't seem to thumbnail no matter what I try. This could work quite well – Ben Brocka Nov 13 '12 at 15:30
  • ffmpeg should be able to convert the flv files into something else (avi, mp4, etc). – jimbo Nov 13 '12 at 19:05
  • Yeah, I'll eventually convert them to MP4 but I want to make sure I lose as little quality as possible when I do that, so I've put it off until I can take the time – Ben Brocka Nov 13 '12 at 19:11
  • If you wanted to, you could find a program that would analyze the thumbnails to find duplicates. However, the time spent finding such a program then configuring it would not be worth it for a small project. – Cole Johnson Jun 18 '13 at 20:16
  • @BenBrocka Another solution is you make an unique hash string (exampley 8 digits character) from your file in uploading time and store it on your db, and add a condition on stored unique string for checking duplicate videos in upload time. – parsa Sep 14 '19 at 4:23

Video recognition software is I would say, a niche market which means you will either find very expensive software and/or software that targets government, movie industry and so forth.

It works by "finger printing" video so it can recognize the video despite different formats, compressions, cropping etc. This ability is not so common in the non-forensic world (at least not yet).

I did try to find something that is targeting consumers but was not able to find any. There do exists APIs (programming interfaces) that provides technology for doing this, and even free, but you will need to develop the software to use it yourself which I am fairly certain is beyond the scope of what you're asking for.

So in conclusion, unless you want to spend a noticeable amount of money on specialized software you will need to go through the stack manually. However, in my link above you will find a trial version of such software so you can see if that is what you're looking for and if it works for your purpose.

| improve this answer | |
  • the link is dead. – Fzs2 May 7 '16 at 14:47

You can try Video Comparer (shareware).


It quickly detects similar videos with image transformations and supports split videos into multiple CDs.

| improve this answer | |
  • This software did not detect similar videos (same scene/person taken a few minutes apart). – Clay Nichols Apr 25 '18 at 23:02

I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is http://duplicatevideosearch.com

| improve this answer | |

Similar images - https://tn123.org/simimages/ is a bit old, but still functional.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.