Comparing PSNR/SSIM of two videos with different fps/bit-depth

Question

I have a 720p, 30 fps, 8-bit source video. I'm downscaling it to 480p, and I want to measure quality using PSNR/SSIM under two scenarios:

1. Keeping crf/speed settings same, I want to encode in 8-bit(yuv420p) and 10-bit(yuv420p10le). I want to measure SSIM/PSNR using ffmpeg's filter. Now normally I'd do something like

   ffmpeg -i reference.mkv -i distorted.8bit.mkv/distorted.10bit.mkv -lavfi "ssim" -f null -
   

   where reference.mkv is 480p lossless ffv1 from the same source. Now the problem here is, that reference.mkv is 8-bit but one of the two distorted videos is 10-bit. Can we compare a 10-bit distorted video with an 8-bit reference?

2. I create two videos, with the same crf/speed/bit-depth settings. The first has the same fps as the source (30 fps). The second is downsampled to 18 fps. I want to measure the deterioration in quality (if any) caused purely by downsampling the fps.
   Again, the problem here will be that the reference lossless ffv1 is 30 fps, but one of the distorted videos is 18 fps. How do we compare objective metrics for two videos that differ in frame-rates?

Answer 1

PSNR and SSIM are image quality metrics. Their use on videos involves calculating the metric for each pair of frames and then applying some aggregating operation to the series of frame scores, typically the arithmetic mean. What this means is both inputs must have same frame count with each encoded frame paired with its source frame. So, deterioration of subjective perception caused by downsampling the fps isn't measured. For a valid and useful comparison, your best bet is to downsample the source to the same FPS using the same methods as used on the distorted video before carrying out the comparison.

As for the bit depth, both operands must have the same pixel format of which bit depth is one factor. I would recommend converting both inputs to the higher bit depth. This will allow you to preserve the details in the distorted input if you applied any filters to it after the bit depth increase.