1

I back up my data to pairs of external hard drives, and recently (in the process of transferring the contents to larger external hard drives) found that some of the data between them has diverged. Several of them are video files.

I used this answer to use ffmpeg's ssim and psnr filters to find the differences between the two videos. Once I had a location, I loaded the tracks into kdenlive, applied the "difference" transition, and seeked to the area where the difference was.

However, no difference was to be seen! I even extracted the audio, and used audacity to compare differences, and still nothing.

The files are clearly different; ffmpeg shows that. But if I can't see any differences, I don't know which file to copy over the other file.

Any idea what I could be doing wrong? This seems too simple to fail.

Improve this question
4
  • The files are clearly different; ffmpeg shows that. --> how?
    – Gyan
    Oct 7, 2020 at 6:29
  • @Gyan: The ssim.log and psnr.log files, generated by the method described above.
    – ulatekh
    Oct 7, 2020 at 16:22
  • Why use audacity to compare them? Isn't this a question about video? Use video vectorscope, waveform monitor, parade, etc. That's what they do. They're built to help you examine a video signal objectively, without the biases and limitations of your eyes and brain. Lossy compression is designed to exploit those limitations, expressly so that you DON'T perceive a difference.
    – Jason Conrad
    Oct 17, 2020 at 4:45
  • @JasonConrad: I was using audacity to compare the audio, to look for differences around the same area.
    – ulatekh
    Nov 20, 2020 at 17:53

3 Answers 3

Reset to default
1

Use a vector scope and waveform monitor to judge the difference, not your eyes. Even if you're doing an XOR comparison, there are cases where your eye isn't enough.

Here, for example, I've taken this sample footage from BMD's SDK (available here): enter image description here The original is 12bit RAW. I then transcoded it to H.265, using these settings: enter image description here If I then superimpose the transcoded version over the original, like this: enter image description here ... and set the blending mode to "difference," a.k.a., "XOR" enter image description here and then use the scopes to analyze the result, you can clearly see that the files are different. enter image description here The top half of the above image shows the timeline viewer, and the result of the "difference" comparison. To the naked eye, it looks black, as if there is no difference. The bottom half of the above image shows Resolve's waveform monitor. Here's a detail of that same image: enter image description here There's clearly noise at the bottom end of the signal. Other types of scopes will tell you different information about the comparison. For instance, you can use a parade to examine Y'CbCr, and see that the transcode operation has affected the image's chroma more than its luma. enter image description here

I know that you're trying to examine the difference between files that you've copied and not necessarily transcoded, but this example is meant to demonstrate that scopes can give you insight into the type of difference between the files, when that difference is invisible to the naked eye.

Improve this answer
4
  • Any options that don't cost thousands of dollars?
    – ulatekh
    Nov 11, 2020 at 17:11
  • DaVinci Resolve is free, and contains several different types of scopes. ScopeBox used to be a good standalone scope utility for under $99, but the price has gone up, and the features included in Resolve's free scopes have narrowed the performance gap.
    – Jason Conrad
    Nov 13, 2020 at 18:13
  • I just double-checked to see if Divergent Media have brought ScopeBox down to a fair price. Unfortunately, they've opted for a subscription pricing model which costs as much per year as their original software was for a lifetime. I can't see how their development costs could possibly justify the expense; video signal standards simply don't change that quickly, and that's kind of the point of having a standard!
    – Jason Conrad
    Nov 13, 2020 at 18:24
  • I don't understand why my answer has been downvoted twice, so I've updated it to be more clear. If somebody could please explain what I'm missing, I'll amend or withdraw. I do realize that OP is probably looking for a more programatic, bitwise comparison, but since this is video stack exchange, I believe standard video signal analysis tools are a fair answer. If this question is purely about file comparison, maybe video production stack exchange isn't the right place for it.
    – Jason Conrad
    Nov 21, 2020 at 1:26
0

Assuming ffmpeg's result is reliable and it does what you think (I don't know, don't use ffmpeg much), it seems that one of your drives, either the primary one or any of the backup drives is faulty. Use tools that calculate checksums, like md5 or sha or crc32 to verify that the files are indeed different. If the checksums differ you have to find out the faulty drive(s).

A few faulty bytes in a video are often not visible when watching, maybe the frame will be dropped where the faulty section is, maybe the way the video has been encoded masks the damage.

Improve this answer
1
  • I'm aware that one of the drives is faulty...the question is which one. The differences were first revealed to me with a sha256sum, and verified by a direct binary compare. And if the faulty frame was dropped, it should have shown up when viewing the result of kdenlive's "difference' transition.
    – ulatekh
    Oct 7, 2020 at 16:24
0

You will see clearly difference, even very subtle ones, with ffmpeg and its blend filter and xor mode within it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.