0

I've been experimenting with adding WebVTT subtitles to an HLS playlist, but I'm starting to ask myself if what I'm trying to do is actually possible.

  • My main file is containing a single video track and 4 audio tracks, both in different languages.
  • I also have 4 subtitle tracks in WebVTT format, one per language.

I'm resizing the file and turning it into transport segments and the HLS playlists using the HLS muxer of FFmpeg 4.2 (Version from the Fedora 32 repos). This works fine, including different variant streams for the languages. Mapping the separate vtt files into the mix doesn't seem to work. Here's what I've been trying to do:

ffmpeg -i LFS_000420_heidelberg4sprachen.mp4 \ 
-i LFS_000420_heidelberg4sprachen_ger.vtt \ 
-i LFS_000420_heidelberg4sprachen_eng.vtt \ 
-i LFS_000420_heidelberg4sprachen_fra.vtt \ 
-i LFS_000420_heidelberg4sprachen_spa.vtt \
-map v -c:v libx264 -b:v 2000k -flags +cgop -g 75 -vf scale="-2:720" \ 
-preset veryfast -hls_time 3 \ 
-map 0:a:0 -ac 2 -map 0:a:1 -ac 2 -map 0:a:2 -ac 2 -map 0:a:3 -ac 2 -map 1:s:0 -map 2:s:0 -map 3:s:0 -map 4:s:0 \ 
-cc_stream_map "ccgroup:cc,instreamid:CC1,language:ger ccgroup:cc,instreamid:CC2,language:eng ccgroup:cc,instreamid:CC3,language:fra ccgroup:cc,instreamid:CC4,language:spa" \ 
-var_stream_map "v:0,agroup:audio,ccgroup:cc a:0,agroup:audio,language:ger,ccgroup:cc a:1,agroup:audio,language:eng,ccgroup:cc a:2,agroup:audio,language:fra,ccgroup:cc a:3,agroup:audio,language:spa,ccgroup:cc" \ 
-hls_flags append_list -hls_playlist_type event -hls_start_number_source generic -start_number 0 \ 
-master_pl_name test.m3u8 -hls_segment_filename ./heidelberg/segment_%v_%03d.ts heidelberg_720_%v.m3u8 

I also tried using -map 0:v and -c:0:v instead of usinst just -v, but this threw an "Invalid stream specifier" error.

The full output of ffmpeg is this:

ffmpeg version 4.2.4 Copyright (c) 2000-2020 the FFmpeg developers
 built with gcc 10 (GCC)
 configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' --extra-cflags=' ' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librsvg --enable-libsrt --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzimg --enable-libzvbi --enable-avfilter --enable-avresample --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-libmfx --enable-runtime-cpudetect
 libavutil      56. 31.100 / 56. 31.100
 libavcodec     58. 54.100 / 58. 54.100
 libavformat    58. 29.100 / 58. 29.100
 libavdevice    58.  8.100 / 58.  8.100
 libavfilter     7. 57.100 /  7. 57.100
 libavresample   4.  0.  0 /  4.  0.  0
 libswscale      5.  5.100 /  5.  5.100
 libswresample   3.  5.100 /  3.  5.100
 libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'LFS_000420_heidelberg4sprachen.mp4':
 Metadata:
   major_brand     : isom
   minor_version   : 512
   compatible_brands: isomiso2avc1mp41
   encoder         : Lavf58.29.100
 Duration: 00:05:00.03, start: 0.000000, bitrate: 6671 kb/s
   Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 6143 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
   Metadata:
     handler_name    : Core Media Video
   Stream #0:1(ger): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
   Stream #0:2(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
   Stream #0:3(fra): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
   Stream #0:4(spa): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
Input #1, webvtt, from 'LFS_000420_heidelberg4sprachen_ger.vtt':
 Duration: N/A, bitrate: N/A
   Stream #1:0: Subtitle: webvtt
Input #2, webvtt, from 'LFS_000420_heidelberg4sprachen_eng.vtt':
 Duration: N/A, bitrate: N/A
   Stream #2:0: Subtitle: webvtt
Input #3, webvtt, from 'LFS_000420_heidelberg4sprachen_fra.vtt':
 Duration: N/A, bitrate: N/A
   Stream #3:0: Subtitle: webvtt
Input #4, webvtt, from 'LFS_000420_heidelberg4sprachen_spa.vtt':
 Duration: N/A, bitrate: N/A
   Stream #4:0: Subtitle: webvtt
Stream mapping:
 Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
 Stream #0:1 -> #0:1 (aac (native) -> aac (native))
 Stream #0:2 -> #0:2 (aac (native) -> aac (native))
 Stream #0:3 -> #0:3 (aac (native) -> aac (native))
 Stream #0:4 -> #0:4 (aac (native) -> aac (native))
 Stream #1:0 -> #0:5 (webvtt (native) -> webvtt (native))
 Stream #2:0 -> #0:6 (webvtt (native) -> webvtt (native))
 Stream #3:0 -> #0:7 (webvtt (native) -> webvtt (native))
 Stream #4:0 -> #0:8 (webvtt (native) -> webvtt (native))
Press [q] to stop, [?] for help
[libx264 @ 0x560770c15700] using SAR=1/1
[libx264 @ 0x560770c15700] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x560770c15700] profile High, level 3.1, 4:2:0, 8-bit
[libx264 @ 0x560770c15700] 264 - core 159 r2999 296494a - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=6 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=75 keyint_min=7 scenecut=40 intra_refresh=0 rc_lookahead=10 rc=abr mbtree=1 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[hls @ 0x560770dc7240] Opening './heidelberg/segment_0_001.ts' for writing
[hls @ 0x560770dc7240] Opening './heidelberg/segment_1_001.ts' for writing
[hls @ 0x560770dc7240] Opening './heidelberg/segment_2_001.ts' for writing
[hls @ 0x560770dc7240] Opening './heidelberg/segment_3_001.ts' for writing
[hls @ 0x560770dc7240] Opening './heidelberg/segment_4_001.ts' for writing
[mpegts @ 0x560772504dc0] frame size not set
[mpegts @ 0x560772509940] frame size not set
[mpegts @ 0x56077250b680] frame size not set
[mpegts @ 0x5607725937c0] frame size not set
Output #0, hls, to 'heidelberg_720_%v.m3u8':
 Metadata:
   major_brand     : isom
   minor_version   : 512
   compatible_brands: isomiso2avc1mp41
   encoder         : Lavf58.29.100
   Stream #0:0(und): Video: h264 (libx264), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 25 fps, 90k tbn, 25 tbc (default)
   Metadata:
     handler_name    : Core Media Video
     encoder         : Lavc58.54.100 libx264
   Side data:
     cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1
   Stream #0:1(ger): Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
     encoder         : Lavc58.54.100 aac
   Stream #0:2(eng): Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
     encoder         : Lavc58.54.100 aac
   Stream #0:3(fra): Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
     encoder         : Lavc58.54.100 aac
   Stream #0:4(spa): Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s (default)
   Metadata:
     handler_name    : Core Media Audio
     encoder         : Lavc58.54.100 aac
   Stream #0:5: Subtitle: webvtt
   Metadata:
     encoder         : Lavc58.54.100 webvtt
   Stream #0:6: Subtitle: webvtt
   Metadata:
     encoder         : Lavc58.54.100 webvtt
   Stream #0:7: Subtitle: webvtt
   Metadata:
     encoder         : Lavc58.54.100 webvtt
   Stream #0:8: Subtitle: webvtt
   Metadata:
     encoder         : Lavc58.54.100 webvtt
[hls @ 0x560770dc7240] Unable to find mapping variant stream
av_interleaved_write_frame(): Cannot allocate memory
[hls @ 0x560770dc7240] Unable to find mapping variant stream
[hls @ 0x560770dc7240] Opening 'heidelberg_720_0.m3u8.tmp' for writing
[hls @ 0x560770dc7240] Opening 'heidelberg_720_1.m3u8.tmp' for writing
[hls @ 0x560770dc7240] Opening 'heidelberg_720_2.m3u8.tmp' for writing
[hls @ 0x560770dc7240] Opening 'heidelberg_720_3.m3u8.tmp' for writing
[hls @ 0x560770dc7240] Opening 'heidelberg_720_4.m3u8.tmp' for writing
[hls @ 0x560770dc7240] Opening 'test.m3u8' for writing
Error writing trailer of heidelberg_720_%v.m3u8: Cannot allocate memory
frame=   21 fps=0.0 q=-1.0 Lsize=N/A time=00:00:21.51 bitrate=N/A speed=54.4x    
video:140kB audio:65kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[libx264 @ 0x560770c15700] frame I:1     Avg QP:22.34  size: 95581
[libx264 @ 0x560770c15700] frame P:5     Avg QP:27.63  size:  6597
[libx264 @ 0x560770c15700] frame B:15    Avg QP:29.47  size:   919
[libx264 @ 0x560770c15700] consecutive B-frames:  4.8%  0.0%  0.0% 95.2%
[libx264 @ 0x560770c15700] mb I  I16..4:  8.0% 28.5% 63.5%
[libx264 @ 0x560770c15700] mb P  I16..4:  0.8%  1.0%  0.1%  P16..4: 26.5% 12.6%  5.2%  0.0%  0.0%    skip:53.8%
[libx264 @ 0x560770c15700] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  7.2%  1.8%  0.1%  direct: 1.2%  skip:89.7%  L0:35.4% L1:41.8% BI:22.8%
[libx264 @ 0x560770c15700] final ratefactor: 20.21
[libx264 @ 0x560770c15700] 8x8 transform intra:30.3% inter:21.5%
[libx264 @ 0x560770c15700] coded y,uvDC,uvAC intra: 77.6% 63.8% 17.4% inter: 2.4% 1.3% 0.0%
[libx264 @ 0x560770c15700] i16 v,h,dc,p:  8% 57% 19% 16%
[libx264 @ 0x560770c15700] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  8% 48% 15%  3%  5%  2% 10%  3%  6%
[libx264 @ 0x560770c15700] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 25% 10%  7% 12%  7% 13%  5% 11%
[libx264 @ 0x560770c15700] i8c dc,h,v,p: 48% 32% 14%  6%
[libx264 @ 0x560770c15700] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x560770c15700] kb/s:1355.78
[aac @ 0x560770c2df00] Qavg: 11115.382
[aac @ 0x560770b7b440] Qavg: 19112.230
[aac @ 0x560770b7cbc0] Qavg: 44664.793
[aac @ 0x560770c34e80] Qavg: 33372.531
Conversion failed!

Three things stand out for me here:

  • [mpegts @ 0x560772504dc0] frame size not set - This might just be an error that's always been there but I didn't notice it until now.
  • [hls @ 0x560770dc7240] Unable to find mapping variant stream - This would indicate that there's something wrong with either the -var_stream_map command or that %v causes problems.
  • av_interleaved_write_frame(): Cannot allocate memory - I'm pretty sure that RAM is not the issue here, the system has got 16GB. I didn't look into the sources, but I presume it's an error that occured due to the previous ones.

Adding subtitles should technically work, the documentation shows examples of adding subtitles (however, these are subtitles embedded in te video file itself, which is not an option for me).

Has anyone already done what I'm trying to do or can anyone see what's wrong here? The last resort would be just writing a script that creates a .m3u8 file for the subtitles and editing the main manifest file, but I'd be more than happy if FFmpeg could do that work for me.

Any help with this would be greatly appreciated.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.