1

I am creating MPEG-TS segments for HLS playback from multiple ffmpeg processes (it will be used for parallel encoding at a later stage). The commands are as follows — they can be run in sequence for testing purposes:

#!/usr/bin/env bash
rm seg*.ts master.m3u8

ffmpeg \
  -f lavfi -i testsrc=s=320x240:r=30 -f lavfi -i sine=440 \
  -vf 'drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000099' \
  -pix_fmt yuv420p \
  -preset ultrafast -c:v:0 libx264 -x264-params "nal-hrd=cbr:force-cfr=1" -b:v:0 4M -maxrate:v:0 4M -minrate:v:0 4M -bufsize:v:0 8M -g 60 -sc_threshold 0 -keyint_min 60 \
  -f hls -hls_time 2 -hls_playlist_type event \
  -hls_flags independent_segments+append_list+omit_endlist \
  -hls_segment_type mpegts -hls_list_size 0 \
  -hls_segment_filename seg_01_%02d.ts -master_pl_name master.m3u8 -start_number 0 \
  -muxdelay 0 \
  -muxpreload 0 \
  -output_ts_offset 0 \
  -t 10 \
  master.m3u8

ffmpeg \
  -f lavfi -i testsrc=s=320x240:r=30 -f lavfi -i sine=440 \
  -vf 'drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000099' \
  -pix_fmt yuv420p \
  -preset ultrafast -c:v:0 libx264 -x264-params "nal-hrd=cbr:force-cfr=1" -b:v:0 4M -maxrate:v:0 4M -minrate:v:0 4M -bufsize:v:0 8M -g 60 -sc_threshold 0 -keyint_min 60 \
  -f hls -hls_time 2 -hls_playlist_type event \
  -hls_flags independent_segments+append_list \
  -hls_segment_type mpegts -hls_list_size 0 \
  -hls_segment_filename seg_02_%02d.ts -master_pl_name master.m3u8 -start_number 0 \
  -muxdelay 0 \
  -muxpreload 0 \
  -output_ts_offset 10 \
  -ss 10 \
  -to 20 \
  master.m3u8

You can load this in your browser if you simply run a web server like python3 -m http.server with:

<html>
  <head></head>
  <body>
    <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
    <video id="video" controls autoplay></video>
    <script>
      var video = document.getElementById("video");
      var videoSrc = "master.m3u8";
      var hls = new Hls({
        debug: true,
      });
      hls.loadSource(videoSrc);
      hls.attachMedia(video);
    </script>
  </body>
</html>

Now, the video seems to play fine. Looking at the frame counter, I see that no frames are lost (although it seems like the very last frame of segment set 1 is displayed a little bit shorter).

The audio, however, has a small dropout at the switch from segment set 1 to 2.

The generated M3U8 has an #EXT-X-DISCONTINUITY tag before switching to the second segment set.

I noted that the start times of the segments don't quite correspond when comparing the audio and video streams. To test this, I run ffprobe and look at the start_time and start_pts values:

for s in *.ts; do ffprobe -loglevel error -hide_banner -select_streams v -show_streams -of json "$s" | jq --arg input "$s" '.streams[0] | {$input, duration, start_time, start_pts}'; done | jq -s . 

This is the output:

[
  {
    "input": "seg_01_00.ts",
    "duration": "2.000000",
    "start_time": "0.023222",
    "start_pts": 2090
  },
  // ...
  {
    "input": "seg_01_04.ts",
    "duration": "2.000000",
    "start_time": "8.023222",
    "start_pts": 722090
  },
  // ...
  {
    "input": "seg_02_05.ts",
    "duration": "2.000000",
    "start_time": "10.000000",
    "start_pts": 900000
  },
  // ...
]

The second set is forced to start at 10 seconds, while the last segment of the first set actually extends to 8.023222 + 2 = 10.023222 seconds. This is due to the non-negative offset of the first segment of the first set.

For audio, the timestamps and durations are completely different:

[
  {
    "input": "seg_01_00.ts",
    "duration": "2.043344",
    "start_time": "0.000000",
    "start_pts": 0
  },
  // ...
  {
    "input": "seg_01_04.ts",
    "duration": "1.996900",
    "start_time": "8.034111",
    "start_pts": 723070
  },
  {
    "input": "seg_02_05.ts",
    "duration": "2.043344",
    "start_time": "9.976778",
    "start_pts": 897910
  },
  // ...
]

Is there any way to fix this, so that the segments generated by the second ffmpeg command can be stitched during playback, without choppy audio?


I have tried another approach, which consists of generating the audio stream separately before starting to encode the video. The audio and video playlists are written into dedicated .m3u8 files which I am simply referencing from a master.m3u8 playlist. This one I have to manually generate, because otherwise ffmpeg would overwrite it. The whole approach is shown here.

The problem is that the video won't start in hls.js:

[log] > Unknown video PTS for cc 0, waiting for video PTS before demuxing audio frag 1 of [0 ,10],track 0

This can be demoed here.

Once I remux everything into new sets of segments, as suggested by @Gyan, it works again. Demo is here.

However, this is not a workable solution here, since I need to be able to generate the HLS segments on the fly, and append to the playlists later.

0

You have to render the audio in one command.

MDCT-based codecs like AAC or MP3 carry out their transforms over multiple adjoining frames. When doing a segmented encode of a single stream, the encoder 'supplies' silent frames at the top and tail of each segment.

From Apple:

AAC requires data beyond the source PCM audio samples in order to correctly encode and decode audio samples due to the nature of the encoding algorithm. AAC encoding uses a transform over consecutive sets of 2048 audio samples, applied every 1024 audio samples (overlapped). For correct audio to be decoded, both transforms for any period of 1024 audio samples are needed. For this reason, encoders add at least 1024 samples of silence before the first ‘true’ audio sample, and often add more. This is called variously “priming”, “priming samples”, or “encoder delay”.

During muxing, these priming samples can either be discarded or kept. If kept, there will be a gap at the seams. If discarded, the decoder will use the tail frame of the previous segment for overlap purposes and the result will be a glitch.

6
  • So, in essence, there could be a dedicated process rendering the .ts chunks for audio completely separately — but how would you then combine them back into a single .m3u8 master playlist?
    – slhck
    Nov 17 at 8:43
  • See option var_stream_map in the hls muxer docs. You will have two inputs in ffmpeg: -f concat -i video_segs_list -i audio_stream -map 0 -map 1 .... You can also concat the video segs into one TS file and use that.
    – Gyan
    Nov 17 at 10:52
  • That indeed produces a playable stream. Thanks! But due to the live requirement and wanting to be able to append video segments to the playlist as they are being generated, I can't do a single remux command, because that will rewrite all the segments and playlist files. I can generate the audio in a separate thread before, and only then start outputting the video segments, but that approach yields a stream that never starts playing in HLS.js. It seems to wait for an initial video PTS but doesn't get any: pastebin.com/raw/MAYJyHs5 — any idea?
    – slhck
    Nov 17 at 13:35
  • I'll check out the details of your process tomorrow.
    – Gyan
    Nov 17 at 18:41
  • Much appreciated. In case this becomes more complex, if you are interested, we can discuss consulting offline.
    – slhck
    Nov 18 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.