What is the most basic standard raw video format?

Question

What I mean by the question is, for audio we have the WAV file format which stores a representation of audio where each word stores the intensity of one sample of audio. In addition to this there is a standard header format which is agreed on universally to make it easier to process these files. Similarly for image files we have the BMP file format where each word stores the RGB value for each pixel inside the image, along with a universally accepted header file that helps process the file. For text files we have TXT file format which just stores the ascii equivalent of each character on file. Non of these are compressed formats and one word of predefined size in each file represents one unit of the media type the file represents.

Is there anything similar for video files? Almost any video format I can think of (.wmv, .mp4 etc) are either proprietary or compressed formats. Does a video equivalent of BMP or WAV exist? Does such a format take into account both audio and video?

Answer 1

The reason there is no widely used uncompressed video format, as there is for audio is that the data rate for uncompressed video is so colossal that uncompressed video is kinda useless. While a relatively cheap device has been able to deal with a stream of PCM audio since the eighties, even today you need fast storage and a decent system to play a stream of uncompressed video. I'll illustrate by giving an example of the size of an uncompressed stream of video.

First it might be worth noting the difference between RAW and uncompressed codecs, and uncompressed and losslessly compressed codecs. Matt provided a good description of RAW codecs, but they are not prevalent in video, other than as camera codecs that get transcoded before use. But there are also uncompressed and lossless codecs that store the raster in a mathematically lossless way, so that each pixel in the decoded video will have the same colour information as the encoded video.

The difference between uncompressed and lossless is that uncompressed just store the colour channels as a stream of bytes, similar to a PCM stream in audio. At say, 10 bits per colour channel, each pixel will take 30 bits. Given that a single frame of 1080p video has around 2.1 million pixels, each frame is going to require a about 7.8mB, and at 24 fps thats ~187mB per second, or 671gB per hour. Even though storage is cheap, that's still going to pose a problem; just playing it back will be taxing. An example of an uncompressed codec is V210, developed by Apple, back when dinosaurs ruled the earth. It's a bit like wav, except that almost nobody uses it.

A solution to the storage and transport problem is to compress the data mathematically — as you would compress other files that you need to restore perfectly. An old lossless codec, Apple's Animation codec uses RLE encoding, which given the highly redundant nature of video streams gives fairly significant size reduction, at the overhead of more processing.

Other lossless encoding use more efficient algorithms, and the more efficient they are the more processing they generally require. So lossless codecs will allow you to retrieve the pixel data as it was originally, with no generation loss, but do not store the pixels as a simple stream of bytes. An example of an open source, mathematically lossless compressed codec is FFV1, developed by the makers of the ffmpeg software.

Something else worth noting is that many video streams are not full resolution. Almost all of the video you see on the web is encoded as YUV-420. This is what is know as chroma subsampling. YUV refers to the way colours are encoded: rather than use the RGB values of the pixels it stores the colour as luma (≈brightness), and two difference components called U and V (≈red - brightness and ≈ blue - brightness, the green component is ≈ the difference between the Y and the sum of the UV channels. There are perceptual weightings given to each component, hence the ≈s but the idea is the same).

Almost all video codecs use YUV encoding, and many sample the colour components at a lower resolution than the luma component. In the case of 420, or 4:2:0 as it is commonly written, for every 4 pixels of luma information there will be 2 pixels of colour information. Some codecs only allow one sort of subsampling (e.g. Apple's Prores 422, which is, not surprisingly 4:2:2 or DVCPro which was weirdly 4:1:1), others let you encode in a variety of subsampling pattern (h.264 has variants that let you encode 4:2:0, 4:2:2 or 4:4:4).

If you wanted a codec that was similar in ubiquity to wav, then probably Apple's ProRes might fit the bill. While not lossless, it manages a good tradeoff between file size and quality, and was one of the first lossy codecs to allow 10-bit sampling. Avid also make a similar codec called DNxHR, and there is an originally proprietary, but now open source codec called Cineform that fills a similar niche — these codecs are often called intermediate codecs, they're good for moving video from one stage of the post-production chain with minimal generation loss.

However the most commonly used video codec by far today is MPEGLA's h.264. This is a delivery codec, as it is suitable for the final delivery of content, especially via the web, and usually lossy, though it can be used to encode losslessly. The main reason that this is so common is that it does an excellent job of maintaining perceived quality at far, far lower data rates than uncompressed video.

Answer 2

This topic is really complex and you have confused some things.

WAV and mp4 are container formats, they can contain compressed or uncompressed data, while WMV is a series of codecs and formats. TXT is not a something like a RAW format - far from it. A txt file is stored as binary data. So if I programmatically open a Streamreader I will get a byte array and can interpret this as text but I have to apply the correct character encoding (like ASCII or UTF8). When you see text it has already been processed. It cannot be considered RAW by definition.

RAW data can contain lossless compressed or uncompressed data but not all uncompressed formats contain RAW data. (I simplify here, because there is also mathametically and visually lossless compression).

What is RAW?

I consider a file to be raw if it contains all necessary unchanged, original data to represent some entity. In order to store it, a format needs to be defined. This format sometimes contains additional data. Example: Canon's CR2 RAW format also stores a JPG preview inside. The RAW file can store its information in a compressed way, however this is always at least visually lossless, the only exception being additionaly data like a preview JPG which is stored lossy.

How to use RAW (super simplified)

The RAW format is never intended for consuming. It is a format to work on the data.

You develop or interpret a RAW. That's the step when you apply some logic to generate a viewable output. E.g. you see text. Or your screen shows an image or video. The high amount of information that a RAW provides, enables enough headroom for you to work much better. When done, you export it in another, consumable file format. This is when you generate for example a JPG from a photo RAW or a WMV from a video RAW. With video you use codecs at this steps, specifically interframe codecs or distribution codecs, because all work is done now. Afterwards or at the same time, you often put them in widely recognised containers that help to bundle video and multiple audio tracks, like mp4 or mkv.

RAW Video

Both in photography and videography you use a camera and it needs to read out its sensor in order to record something. The sensor is of course manufacturer-specific. It's only natural, that every camera works a bit different, has different needs and offers different features. Think of how the Bayer filter needs to be interpreted. Consequently manufacturers have their own RAW format. Last but not least, the camera manufacturers would lose their IP if they shared all those secrets. All those reasons stand against a common RAW format.

Yet you can convert a RAW to a common, open format with some drawbacks. For photography that is DNG and for videos it's CinemaDNG. Sometimes this conversion is built into the camera.

Specifying a common format is hard work as you need to account for everyone's needs. Adobe tried to establish the DNG format as a generic format in the photography sector, but they didn't take every need into account. Many years later it still is only used by few manufacturers. DNG can also be regarded as a derrived format. But your first step is still to use a propriety RAW format, then convert it to DNG. Probably those cameras which output DNG still use their own RAW formats internally.

But Adobe's other open format initiative, called CinemaDNG, seems to be accepted more widely. It's a container that can store video as a series of images each encoded as DNG. However if you look at Blackmagic, not every manufacturer is happy with it. According to Blackmagic their own RAW format performs much better and they regard CinemaDNG as obsolete.

You will be able to get proprietary RAW formats from many modern cameras. Arri and Sony offer uncompressed RAW, while RED and Blackmagic give you compressed RAW.

An alternative during post-production is video encoded with an intraframe codec. Something like ProRes. It still delivers very high quality.

A note on audio

You record audio in an umcompressed format. Professionally you often record on a separate device and later sync that in post mostly automatically. That's why you need a clapperboard/timecode. There's also music, ADR, SFX that will be added later, not when shooting of course. That will all come together at some point of editing. Before video and audio will be exported for distribution, there is a mixdown/mastering performed with the audio (with the goal to reduce or mix down many tracks to simple stereo or surround sound).

In essence, video and audio is kept seperately at the beginning, kept in sync but still separated during editing and bundled together during final export for distribution. Each workflow might vary a bit. That and for technical reasons is why all video RAW only record video, I think. Only CinemaDNG can include audio.

To sum up

There is no standard video RAW format. The closest thing to that is CinemaDNG which is actually a bit different than a true RAW format, and it comes with some drawbacks.

Keep in mind that RAW video is a topic for producing films for cinema or streaming platforms. It's a bit over the top for Youtube videos, hobbyist projects and the like.

Answer 3

Each manufacturer uses proprietary way to read data from the matrix and encode it to some popular format. As far as I understand the question, (and I am also interested in the answer) was how to decode/convert a movie, for example from mp4 format to the format where each frame would form a two dimensional matrix, with each element corresponding to particular pixel storing an information about the shade of grey in case of black and white movie, or three dimensional one where for each pixel three RGB values would be stored as an integer array frame(1024,640,3) for the resolution 1024x640. Of course the file containing the whole movie would be huge, and impractical for a typical storing/displaying of a movie, but this type of conversion always needs to be a first step for further mathematical processing of data for let say 2D Fourier transformation of each frame or something else making more sense.