How to encode scenes with different CRF into a H.264 MP4 video with a variable frame rate?

Question

Source Material

• Having many scenes which essentially are still frames (for many seconds)
• Commonly found in: Slideshows, Screen Recording or Software Mockup/Prototype

✅ This kind of source material wonderfully shrinks in file size without loosing any extra quality simply by using ffmpeg with -vf -mpdecimate -vsync vfr which decimates duplicate (=still) frames and in consequence makes a variable frame rate video from it.

But what if I have a few scenes which need more/less CFR ?

• The H.264 video throughout the majority of its runtime looks totally fine at CRF 28 (lowest recommended quality)
• But in the intro I slide in/out a portrait photo, which requires significantly more intra and inter frame compression as in the rest of the video (prototype of a flat design software, compresses really well).

🤔 How do I encode my intro at CR28, and the rest at CR18, and still get a variable frame rate in my video?

Answer 1

TL;DR

Correct Approach

1. Pick up the necessary scenes of videos with ffmpeg 's time related arguments -ss in-point, -t duration and -to out-point before the -i input argument and encode each such scene with your desired CRF and with VFR (see question above) as a standalone intermediary clip.
2. Then -f concat your scenes (same dimension, same codec, variable framerate (inherently different!), different CRF) together and use -movflags faststart to be streaming compatible.

• Full ffmpeg command lines for steps 1+2 in detail section below.

Playback of video file

• From ffmpeg 5.1.2:** on macOS 11 Big Sur

• ✅ QuickLook, QuickTime Player 10.5, mpv 0.35.0 — Play back fine.

• ❌ VLC: The variant with intro and main part both having VFR does not play back fine! Audio remains in sync throughout the video. But during the intro the portrait photo slide in gets stuck along the way and then from that position slides out later, and then when the main part starts, the video stream recovers and resumes correctly.

• 👉 Maximum compatibility solution: Intro part with fixed frame rate and start with variable frame rates only in the main part.

To visually debug/inspect your resulting video files

ℹ️ mpv can display frame numbers in its On Screen Display

• To your ~/.config/mpv/mpv.conf add the following line: osd-status-msg=${playback-time/full} / ${duration} (${percent-pos}%)\nframe: ${estimated-frame-number} / ${estimated-frame-count}
• Play back your video file and press hotkey ⇧-O to toggle OSD on/off.

Wrong Approach

1. Encode full video with VFR once as CR18, once as CR28.
2. Concat intro with inpoint/outpoint from CR18, followed by main part as CR28 with inpoint being outro's outpoint.

❌ Fail: Concat's inpoint/outpoint/duration functions seem to not work correctly with VFR sources. Result:

• Video track: Intro played twice. That by itself is already inacceptable.
• Worse: Audio plays continously, so after the outro is out of sync.

Details

Correct Approach

Creating the intro

ffmpeg \
-to 00:06.300 \ # The intro lasts til 6secs and .300 fractions
-i VID.m4v \
\
-movflags faststart \
-c:v libx264 -preset veryslow -crf 18 \
-vf mpdecimate -vsync vfr \
\
VID-intro-CR18-vfr.mp4

Creating the main part

ffmpeg \
-ss 00:06.300 \  # The main part starts when the intro ends.
-t 30s \         # Duration limited to -t 30s (seconds) during testing. Omit for real run.
-i VID.m4v \
\
-movflags faststart \
-c:v libx264 -preset veryslow -crf 28 \
-vf mpdecimate -vsync vfr \
\
VID-main-from-intro-endpoint-for-30secs-CR28-vfr.mp4

Creating concatenation instructions

assembly.txt

file VID-intro-CR18-vfr.mp4
file VID-main-from-intro-endpoint-CR28-vfr.mp4

Running concatenation

ffmpeg -f concat -i assembly.txt -c copy -movflags faststart \
VID--intro-CR18-vfr--main-CR28-vfr.mp4

Resulting file plays back fine.

Processing throws only two "Non-monotonous DTS in output stream" warning messages. That seems to be the timecode correction before/after the stitching point between intro and main part:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fd44a005580] Auto-inserting h264_mp4toannexb bitstream filter
[mp4 @ 0x7fd44a007b00] Non-monotonous DTS in output stream 0:0; previous: 95589, current: 25445; changing to 95590. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fd44a007b00] Non-monotonous DTS in output stream 0:1; previous: 278528, current: 277830; changing to 278529. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fd44a007b00] Starting second pass: moving the moov atom to the beginning of the file
frame=  467 fps=0.0 q=-1.0 Lsize=    1898kB time=00:00:36.32 bitrate= 428.0kbits/s speed=1.04e+03x
video:1311kB audio:566kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.112216%

With a high probability your timecode will get some auto correction at the stitching point(s), as the cuts most likely will not occur at a keyframes but at interframes. I guess the warning message just told us that.

Wrong Approach

Concatenation instructions by copying out the scenes from full length VFR videos encoded at different CRFs

assembly.txt

file VID-CR18-vfr.mp4
inpoint 00:00
outpoint 00:06.300

file VID-CR28-vfr.mp4
inpoint 00:06.300

The resulting file has issues (see above).

The warning message "Non-monotonous DTS in output stream" occurs ~450 times for a 49min video. Exceprt (with "..." omissions):

[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441344, current: 0; changing to 441345. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441345, current: 1024; changing to 441346. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441346, current: 2048; changing to 441347. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441347, current: 3072; changing to 441348. This may result in incorrect timestamps in the output file.

...

[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441364, current: 20480; changing to 441365. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441365, current: 21504; changing to 441366. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:0; previous: 95077, current: -7835; changing to 95078. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441366, current: 22528; changing to 441367. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441367, current: 23552; changing to 441368. This may result in incorrect timestamps in the output file.

...

[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441385, current: 41984; changing to 441386. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441386, current: 43008; changing to 441387. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:0; previous: 95078, current: -155; changing to 95079. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441387, current: 44032; changing to 441388. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441388, current: 45056; changing to 441389. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:0; previous: 95079, current: 357; changing to 95080. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441389, current: 46080; changing to 441390. This may result in incorrect timestamps in the output file.

...

[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441773, current: 439296; changing to 441774. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441774, current: 440320; changing to 441775. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7fecd573b440] Non-monotonous DTS in output stream 0:1; previous: 441775, current: 441344; changing to 441776. This may result in incorrect timestamps in the output file.
frame=38540 fps=38539 q=-1.0 size=   56832kB time=00:42:38.23 bitrate= 182.0kbits/s speed=2.56e+03x  frame=44130 fps=38101 q=-1.0 Lsize=   68207kB time=00:49:27.67 bitrate= 188.3kbits/s speed=2.56e+03x    
video:20486kB audio:45963kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.645229%