This is a complex topic, so best to link and excerpt
H.264 contains a number of new features that allow it to compress
images much more efficiently than JPG.
New transform design
Differently from JPG, an exact-match integer 4×4 spatial block
transform is used instead of the well known 8×8 DCT. It is
conceptually similar to DCT but with less ringing artifacts. There is
also a 8×8 spatial block transform for less detailed areas and chroma.
A secondary Hadamard Transform (2×2 on chroma and 4×4 on luma) can be
usually performed on “DC” coefficients to obtain even more compression
in smooth regions.
There is also an optimized quantization and two possible zig-zag
pattern for Run Length Encoding of transformed coefficients.
H.264 introduces complex spatial prediction for intra-frame
compression. Rather than the “DC”-only prediction found in MPEG2 and
the transform coefficient prediction found in H.263+, H.264 defines 6
prediction directions (modes) to predict spatial information from
neighbouring blocks when encoded using 4×4 transform. The encoder
tries to predict the block interpolating the color value of adiacent
blocks. Only the delta signal is therefore transmitted.
There are also 4 prediction modes for smooth color zones (16×16
blocks). Residual data are coded with 4×4 trasforms and a further 4×4
Hadamard trasform is used for DC coefficients.
A new logarithmic quantization step is used (compound rate 12%). It’s
also possible to use Frequency-customized quantization scaling
matrices selected by the encoder for perceptual-based quantization
Inloop deblocking filter
An adaptive deblocking filter is applied to reduce eventual blocking
artifacts at high compression ratio.
Advanced Entropy Coding
H.264 can use the state of the art in entropy coding: Context Adaptive
Binary Arithmetic Coding (CABAC) which is much more efficient than the
standard Huffman coding used in JPG.