Decoding H.264/AVC Frames in MP4: What's the Output Format?

Question

I am currently working on a personal programming challenge to develop an MP4 decoder using VB.NET and the .NET Framework 4.8.1 as a WinForms application. I have obtained the ISO 14496-12 documentation, which serves as the basic structure for MP4 files.

At this stage, I have successfully prepared various lists containing frame sizes, chunk positions, and information from the stsc and stts boxes. Additionally, I have been able to identify keyframes within the MP4 files.

However, I have come to realize that the ISO 14496-12 documentation is not sufficient for decoding the frame bytes, which are encoded in H.264/AVC. To proceed with decoding, I understand that I need to refer to the ISO/IEC 14496-10 documentation specifically for H.264/AVC.

My question is twofold:

• When decoding the frame bytes according to the documentation, would the output data be in a specific format, such as JPEG (identified by the FF D8 marker), or would it be raw bytes representing pixel data? This clarification would help me. In the former case, I could simply save an image using .NET functions.
• Considering my current progress and the information I have mentioned, do you foresee any need for me to purchase a third documentation source?

Any guidance or suggestions would be appreciated.

Answer 1

"would the output data be in a specific format, such as JPEG (identified by the FF D8 marker), or would it be raw bytes representing pixel data?"

An H.264 decoder should decode to raw YUV by default, although it's generally appreciated for standalone use to have a YUV4MPEG2 mode as well, for easier piping, and to have basic format metadata preserved.

Usually the encoding is in YU420P format. There are many examples of converting (via a For-loop) any YUV420p data into RGB data. You can finally encode the newly converted RGB into JPEG.

"Do you foresee any need for me to purchase a third documentation source?"

No, unless you want to also handle audio then you'll need the AAC documentation.
That is iso/iec 13818-7. You can also learn by analysing AAC bytes whilst following the source code of some open-source projects that are decoding AAC data. Articles online also explain structure of AAC. You can still get a result without the ISO document itself (eg: read articles from those who repeat its words as they teach about AAC audio's bytes structure).