Unless I'm bad at reading, this page states that colormatrix is only good for 8 bpc.
From Colorspace support in FFmpeg:
... colormatrix produces horrible quality for anything > 8bpc (8-bit per component)
Backstory. One of my typical workflows is to convert 16bit tiffs and 10bit DPX frames into 10bit prores. However, I think the input colorspace of the frames is assumed to be bt601 and I would always get encodes that looks just slightly off from my originals. I solved this by adding:
-vf colormatrix=bt601:bt709
And it looks fine visually. The difference between the source and encode I always just assumed was the lossy compression.
So can someone chime in on if the colormatrix=bt601:bt709
operation is 'supposed' to be bad when the source and encode are >10bpc? Or is the info on that page outdated? Or should I be doing it a different way?