I've been reading up on video compression. I understand that 'raw' media file sizes are to big for practical usage so they have to be compressed, but I'm wondering what happens when the video is being played back.

I mean is the whole video decoded (decompressed) and stored somewhere in memory (this doesn't seem possible) or are the individual frames decoded one at the time and then forgotten, so that when you want to replay a portion of the video you need to decode it again?

It's probably a very noob question and it's because I am one, so would appreciate any info.

  • 1
    Yes, the player only decodes frames as needed. The decoder's buffer stores all frames needed for the current queue of frames being played. When you replay a portion you played some time ago, it has be re-decoded. However, players may maintain a cache so a portion replayed very soon after its last play may still be in that cache.
    – Gyan
    Commented Oct 8, 2018 at 5:04

2 Answers 2


In most common compression schemes, there isn't a 1::1 correspondence between "video" frames and "file" (or stream) frames **. Any particular video frame may require the contents of several file frames to become a complete viewable image. So a player will read and buffer as many file frames as needed to reconstruct the video frame in question. The player may buffer much more, but it's not required to.

For example, if some video frame ends up as a P frame when encoded in the file, the player will need to look back to find the previous I frame, decode that, then apply the differences encoded in the intervening P frames to get the complete result.

** EDIT: There is of course a file frame for each video frame, but the point is that you can't usually read one frame from the file and get a complete video frame: that's only true for I frames.


The idea is to be able to stream the data and play it on a dumb device (such as a "DVD Player"). The dumb device is generally going to use as little hardware as possible, so for sure, you wouldn't have enough memory to decompress the entire movie then play it back. Plus that would take forever to get started. The streaming concept means you want the data to be played as it comes in.

My understanding of H.264 is that you have one frame called an I-Frame which you can decompress on its own. This is essentially a JPEG image within your movie file.

The following frames are going to be P-Frames which make use of the previous frame to generate the new frame. More or less, if only a character is moving on your screen, you just need to erase the character where it was and render it were it's now. The rest of the screen can remain unchanged. In that case, you can very much compress the image data of that one frame (i.e. an I-Frame may be 250Kb in a 4K movie whereas a P-Frame could be as small as 2Kb).

Further, the H.264 compression scheme supports B-Frames (Bidirectional). This is a bit more complicated. It means a frame can only be decompressed after some future frame gets decompressed. This means the device decompressing frames needs to be able to cache more decompressed frames than just the last I-Frame and the last P-Frame (and as far as I know, P-Frames may reference older P-Frames than the one just before).

Each decoder will have limits to what it can do. For Example, I'm using Jetsons to create an embedded system and they only support video compressed in 4:2:0 (2 chroma bytes for every 4 pixels). The number of buffers that a decoder has and whether B-Frames are supported are other features that may or may not work depending on the decoder.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.