9

After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting.

To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.

How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually.

Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?

Improve this question
1
  • 1
    Very interesting question!
    – Dr Mayhem ♦
    Commented Nov 12, 2012 at 10:40

6 Answers 6

Reset to default
13

I'd recommend a hybrid approach using both computers and people.

  1. Bucket the videos by their length (round to the nearest second)
  2. For each bucket, use ffmpeg to generate thumbnails at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)
  3. Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.

You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse. Good luck!

Improve this answer
5
  • Thumbnails are a great idea! The new videos came in as FLV, which I can't seem to thumbnail no matter what I try. This could work quite well
    – Zelda
    Commented Nov 13, 2012 at 15:30
  • ffmpeg should be able to convert the flv files into something else (avi, mp4, etc).
    – jimbo
    Commented Nov 13, 2012 at 19:05
  • Yeah, I'll eventually convert them to MP4 but I want to make sure I lose as little quality as possible when I do that, so I've put it off until I can take the time
    – Zelda
    Commented Nov 13, 2012 at 19:11
  • If you wanted to, you could find a program that would analyze the thumbnails to find duplicates. However, the time spent finding such a program then configuring it would not be worth it for a small project.
    – Cole Tobin
    Commented Jun 18, 2013 at 20:16
  • @BenBrocka Another solution is you make an unique hash string (exampley 8 digits character) from your file in uploading time and store it on your db, and add a condition on stored unique string for checking duplicate videos in upload time.
    – Parsa Saei
    Commented Sep 14, 2019 at 4:23
4

Video recognition software is I would say, a niche market which means you will either find very expensive software and/or software that targets government, movie industry and so forth.

It works by "finger printing" video so it can recognize the video despite different formats, compressions, cropping etc. This ability is not so common in the non-forensic world (at least not yet).

I did try to find something that is targeting consumers but was not able to find any. There do exists APIs (programming interfaces) that provides technology for doing this, and even free, but you will need to develop the software to use it yourself which I am fairly certain is beyond the scope of what you're asking for.

So in conclusion, unless you want to spend a noticeable amount of money on specialized software you will need to go through the stack manually. However, in my link above you will find a trial version of such software so you can see if that is what you're looking for and if it works for your purpose.

Improve this answer
1
  • the link is dead.
    – john-jones
    Commented May 7, 2016 at 14:47
2

This seems related to : https://video.stackexchange.com/a/33521/32436

You could check out https://theophanemayaud.github.io/video-simili-duplicate-cleaner/ (macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.

The idea is to extract a few (eg 2, the first and the last) frames, and compare them. If the duration is close, and the frames look similar : you've got a duplicate.

You can "manually compare the extracted frames", or use thresholds to configure some automatic behaviors, and trash the duplicates.

It uses opencv for frame comparison, and ffmpeg for frame extraction (ffmpeg is also for video metadata which is very useful when trying to tell which video to keep !). Since it's open source, you can improve it to fit your needs !

I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !

Improve this answer
1

You can try Video Comparer (shareware).

http://www.video-comparer.com

It quickly detects similar videos with image transformations and supports split videos into multiple CDs.

Improve this answer
1
  • This software did not detect similar videos (same scene/person taken a few minutes apart).
    – Clay Nichols
    Commented Apr 25, 2018 at 23:02
1

I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is http://duplicatevideosearch.com

Improve this answer
-1

Similar images - https://tn123.org/simimages/ is a bit old, but still functional.

Improve this answer
0

Not the answer you're looking for? Browse other questions tagged or ask your own question.