How could an 8-bit encoding take more memory than a 10-bit encoding?

Question

I'll start off by saying that I know next to nothing about video encoding and I'm looking for a nice tutorial explaining the basics (I have a technical background and some knowledge of signal processing)

I've heard that encoding a video with 8-bit color can actually take more filesize than in 10-bit color. The explanation offered was that since the source file was in 10-bit color, it's somehow easier to stay in 10-bit color. I don't understand how that's possible, because whatever algorithm is used to convert between video formats can do what it needs to do, and when the time comes to convert to 8-bit quality, just round off the color. I don't see how rounding off a color could contribute to file size.

Answer 1

From Why does 10-bit save bandwidth (even when content is 8-bit)? (alt. URL)

So why does a AVC/H.264 10-bit encoder perform better than 8-bit?
When encoding with the 10-bit tool, the compression process is performed with at least 10-bit accuracy compared to only 8-bit otherwise. So there is less truncation errors, especially in the motion compensation stage, increasing the efficiency of compression tools. As a consequence, there is less need to quantize to achieve a given bit-rate. The net result is a better quality for the same bit-rate or conversely less bit-rate for the same quality: between 5% and 20% on typical sources.

See the link for more information.

Answer 2

Memory size

This is certainly not true for any encoding as reducing 10 bits to 8 bits is by definition reducing the size.

However, it is possible that some codecs adds a segment for alpha channel turning 24-bits (3 bytes (RGB) per pixel) into 32-bits (3 bytes for color RGB + 1 byte for alpha channel).

This way it may appear as if the video takes more memory, but it's the added alpha channel that adds to the data. In fact it would take the same space as the original 10-bits data (assuming there is only RGB) as it would use 32-bits to hold the 30 bits it represent. This is because in memory data needs to be aligned to a full 4/8 bytes segment to work efficiently so 30-bits data would take 32-bits (4x4 bytes) memory (this will become very technical and programming oriented so I'll leave it with that).

File size

The reduction of one number of bits to another can be performed in various ways such as shifting for example the 10 bits 2 bit places into 8 bits effectively cutting the two lower bits. This is the quick and dirty way of doing it leaving in general poor result but not affecting compression so much.

The other way is to do rounding and introducing noise to the data, what is called dithering. The quality is visually much better with this technique as a reduced representation, but it has one draw back: This noise will remove coherence in the signal and can increase the file size (but not memory size which would be the size of the raw image - the software will have to use uncompressed data for processing) as it's harder to compress as compression always bases it self on coherency. Noise is basically random numbers without much structure (I'm not gonna go into chaos theory and such :-) ).

For this reason the file size may become larger.