0

I am using ffmpeg to combine two videos in one side-by-side. This is a recorded 1:1 video-call (WebRTC); each stream recorded separately by the media server.

I've come up with this command, which fits the needs of our project:

ffmpeg \
  -ss 1054403ms -c:a libopus -i str_CAM_Fs1t_con_CjOpssSpAe.webm \
  -i -c:a libopus str_CAM_GQcn_con_RzVjNmZY6i.webm \
  -filter_complex "[0][1]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[0s][1s]; \
                   [0s][1s]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s]; \
                   [0s][1s]hstack=inputs=2:shortest=1,setsar=1[v]; \
                   [0:a][1:a]amix[a]" \
  -map "[v]" -map "[a]" -ac 2 -r 30 -crf 23 out.mp4

The problem

Audio/video gets out of sync significantly in the resulting file, most noticeably on the right-side video. As far as I can tell, the desync is progressive - it's getting obvious in the middle, and by the end of the video it's off by 1-2 seconds. The videos are 50-60 minutes in average.

ffprobe output from source files:

Input #0, matroska,webm, from 'str_CAM_Immw_con_YmU3u1Tcmv.webm':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2020-06-01T10:05:48.000000Z
  Duration: 00:44:24.20, start: 0.000000, bitrate: 335 kb/s
    Stream #0:0(eng): Video: vp8, yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 1k tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio

Input #0, matroska,webm, from 'str_CAM_JK8k_con_FMwTtVVBqL.webm':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2020-06-01T09:56:15.000000Z
  Duration: 00:53:49.77, start: 0.000000, bitrate: 676 kb/s
    Stream #0:0(eng): Video: vp8, yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio

What I've tried

  • pre-converting source files separately to .mp4. Audio stays in sync in both files after this step, but out of sync after hstack

  • [0:a][1:a]amerge,aresample=async=1000[a] instead of amix (in fact, amerge was my first approach, but I found amix doing the same with the shorter syntax)

  • -vsync 0 instead of constant frame rate

  • ffmpeg -i source.mp4 -time_base 1/30 -c copy source1.mp4 (source.mp4 is the converted separately file from Step 1)

  • ffmpeg -i source.webm -c copy -map 0 -fflags +shortest -max_interleave_delta 0 source1.webm

Now, after the last command I get 'DURATION' in the metadata, and it looks like audio and video tracks lengths don't match:

Input #0, matroska,webm, from 'ed.webm':
  Metadata:
    ENCODER         : Lavf58.44.100
  Duration: 00:53:49.77, start: 0.000000, bitrate: 671 kb/s
    Stream #0:0(eng): Video: vp8, yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
      DURATION        : 00:53:49.732000000
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio
      DURATION        : 00:53:49.766000000

Honestly, I don't know where to go from this point, my (very little) understanding about video encoding has come to a brick wall, and I'm unable to get any new results in Google.

Please help me understand what's happening and what to do.

Update

debug output from the command:

Splitting the commandline.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'.
Reading option '-hide_banner' ... matched as option 'hide_banner' (do not show program banner) with argument '1'.
Reading option '-ss' ... matched as option 'ss' (set the start time offset) with argument '85526ms'.
Reading option '-c:a' ... matched as option 'c' (codec name) with argument 'libopus'.
Reading option '-i' ... matched as input url with argument './str_CAM_N0PW_con_Gz4hFc86ts.webm'.
Reading option '-c:a' ... matched as option 'c' (codec name) with argument 'libopus'.
Reading option '-i' ... matched as input url with argument './str_CAM_GPjV_con_NxaIMlsISO.webm'.
Reading option '-filter_complex' ... matched as option 'filter_complex' (create a complex filtergraph) with argument '[1][0]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[0s][1s];[0s][1s]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s];[1s][0s]hstack=inputs=2:shortest=1,setsar=1[v];[0][1]amerge,aresample=async=1[a]'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[v]'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[a]'.
Reading option '-r' ... matched as option 'r' (set frame rate (Hz value, fraction or abbreviation)) with argument '30'.
Reading option '-crf' ... matched as AVOption 'crf' with argument '23'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
Reading option '-t' ... matched as option 't' (record or transcode "duration" seconds of audio/video) with argument '300s'.
Reading option './out.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option v (set logging level) with argument debug.
Applying option hide_banner (do not show program banner) with argument 1.
Applying option filter_complex (create a complex filtergraph) with argument [1][0]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[0s][1s];[0s][1s]scale2ref='oh*mdar':'if(lt(main_h,ih),ih,main_h)'[1s][0s];[1s][0s]hstack=inputs=2:shortest=1,setsar=1[v];[0][1]amerge,aresample=async=1[a].
Successfully parsed a group of options.
Parsing a group of options: input url ./str_CAM_N0PW_con_Gz4hFc86ts.webm.
Applying option ss (set the start time offset) with argument 85526ms.
Applying option c:a (codec name) with argument libopus.
Successfully parsed a group of options.
Opening an input file: ./str_CAM_N0PW_con_Gz4hFc86ts.webm.
[NULL @ 0x7fffc338fbc0] Opening './str_CAM_N0PW_con_Gz4hFc86ts.webm' for reading
[file @ 0x7fffc3390680] Setting default whitelist 'file,crypto,data'
[matroska,webm @ 0x7fffc338fbc0] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
st:1 removing common factor 1000000 from timebase
[matroska,webm @ 0x7fffc338fbc0] Before avformat_find_stream_info() pos: 473 bytes read:32768 seeks:0 nb_streams:2
[matroska,webm @ 0x7fffc338fbc0] All info found
[matroska,webm @ 0x7fffc338fbc0] rfps: 34.000000 0.019089
[matroska,webm @ 0x7fffc338fbc0] After avformat_find_stream_info() pos: 44966 bytes read:65536 seeks:0 frames:100
Input #0, matroska,webm, from './str_CAM_N0PW_con_Gz4hFc86ts.webm':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2020-06-13T14:02:03.000000Z
  Duration: 00:55:22.57, start: 0.000000, bitrate: 1034 kb/s
    Stream #0:0(eng), 41, 1/1000: Video: vp8, 1 reference frame, yuv420p(progressive), 640x480, 0/1, SAR 1:1 DAR 4:3, 1k tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
    Stream #0:1(eng), 59, 1/1000: Audio: opus, 48000 Hz, stereo, s16 (default)
    Metadata:
      title           : Audio
Successfully opened the file.
Parsing a group of options: input url ./str_CAM_GPjV_con_NxaIMlsISO.webm.
Applying option c:a (codec name) with argument libopus.
Successfully parsed a group of options.
Opening an input file: ./str_CAM_GPjV_con_NxaIMlsISO.webm.
[NULL @ 0x7fffc33c5f80] Opening './str_CAM_GPjV_con_NxaIMlsISO.webm' for reading
[file @ 0x7fffc33bb2c0] Setting default whitelist 'file,crypto,data'
[matroska,webm @ 0x7fffc33c5f80] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
st:1 removing common factor 1000000 from timebase
[matroska,webm @ 0x7fffc33c5f80] Before avformat_find_stream_info() pos: 473 bytes read:32768 seeks:0 nb_streams:2
[matroska,webm @ 0x7fffc33c5f80] All info found
[matroska,webm @ 0x7fffc33c5f80] After avformat_find_stream_info() pos: 52734 bytes read:65536 seeks:0 frames:114
Input #1, matroska,webm, from './str_CAM_GPjV_con_NxaIMlsISO.webm':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2020-06-13T14:03:29.000000Z
  Duration: 00:53:59.94, start: 0.000000, bitrate: 1018 kb/s
    Stream #1:0(eng), 41, 1/1000: Video: vp8, 1 reference frame, yuv420p(progressive), 640x480, 0/1, SAR 1:1 DAR 4:3, 1k tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
    Stream #1:1(eng), 73, 1/1000: Audio: opus, 48000 Hz, stereo, s16 (default)
    Metadata:
      title           : Audio
Successfully opened the file.
[Parsed_scale2ref_0 @ 0x7fffc33b2d40] Setting 'w' to value 'oh*mdar'
[Parsed_scale2ref_0 @ 0x7fffc33b2d40] Setting 'h' to value 'if(lt(main_h,ih),ih,main_h)'
[Parsed_scale2ref_0 @ 0x7fffc33b2d40] w:oh*mdar h:if(lt(main_h,ih),ih,main_h) flags:'bilinear' interl:0
[Parsed_scale2ref_1 @ 0x7fffc33c19c0] Setting 'w' to value 'oh*mdar'
[Parsed_scale2ref_1 @ 0x7fffc33c19c0] Setting 'h' to value 'if(lt(main_h,ih),ih,main_h)'
[Parsed_scale2ref_1 @ 0x7fffc33c19c0] w:oh*mdar h:if(lt(main_h,ih),ih,main_h) flags:'bilinear' interl:0
[Parsed_hstack_2 @ 0x7fffc33e5840] Setting 'inputs' to value '2'
[Parsed_hstack_2 @ 0x7fffc33e5840] Setting 'shortest' to value '1'
[Parsed_setsar_3 @ 0x7fffc33caac0] Setting 'sar' to value '1'
[Parsed_aresample_5 @ 0x7fffc33cc000] Setting 'async' to value '1'
Parsing a group of options: output url ./out.mp4.
Applying option map (set input stream mapping) with argument [v].
Applying option map (set input stream mapping) with argument [a].
Applying option r (set frame rate (Hz value, fraction or abbreviation)) with argument 30.
Applying option ac (set number of audio channels) with argument 2.
Applying option t (record or transcode "duration" seconds of audio/video) with argument 300s.
Successfully parsed a group of options.
Opening an output file: ./out.mp4.
[file @ 0x7fffc33cb300] Setting default whitelist 'file,crypto,data'
Successfully opened the file.
detected 4 logical cores
Stream mapping:
  Stream #0:0 (vp8) -> scale2ref:ref
  Stream #0:1 (libopus) -> amerge:in0
  Stream #1:0 (vp8) -> scale2ref:default
  Stream #1:1 (libopus) -> amerge:in1
  setsar -> Stream #0:0 (libx264)
  aresample -> Stream #0:1 (aac)
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 1 times
[vp8 @ 0x7fffc33fb440] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 2 times
[vp8 @ 0x7fffc33fe580] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 3 times
[vp8 @ 0x7fffc34016c0] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 2 times
[vp8 @ 0x7fffc3404800] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 2 times
[vp8 @ 0x7fffc3407940] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[vp8 @ 0x7fffc33a5740] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 2 times
[vp8 @ 0x7fffc33a7d80] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 2 times
[vp8 @ 0x7fffc33e9a00] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 1 times
[vp8 @ 0x7fffc33ec800] Format yuv420p chosen by get_format().
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 1 times
[vp8 @ 0x7fffc33ef800] Format yuv420p chosen by get_format().
[Parsed_scale2ref_0 @ 0x7fffc3473a00] Setting 'w' to value 'oh*mdar'
[Parsed_scale2ref_0 @ 0x7fffc3473a00] Setting 'h' to value 'if(lt(main_h,ih),ih,main_h)'
[Parsed_scale2ref_0 @ 0x7fffc3473a00] w:oh*mdar h:if(lt(main_h,ih),ih,main_h) flags:'bilinear' interl:0
[Parsed_scale2ref_1 @ 0x7fffc3474080] Setting 'w' to value 'oh*mdar'
[Parsed_scale2ref_1 @ 0x7fffc3474080] Setting 'h' to value 'if(lt(main_h,ih),ih,main_h)'
[Parsed_scale2ref_1 @ 0x7fffc3474080] w:oh*mdar h:if(lt(main_h,ih),ih,main_h) flags:'bilinear' interl:0
[Parsed_hstack_2 @ 0x7fffc34762c0] Setting 'inputs' to value '2'
[Parsed_hstack_2 @ 0x7fffc34762c0] Setting 'shortest' to value '1'
[Parsed_setsar_3 @ 0x7fffc3477480] Setting 'sar' to value '1'
[Parsed_aresample_5 @ 0x7fffc3478a40] Setting 'async' to value '1'
[graph 0 input from stream 1:0 @ 0x7fffc3477a40] Setting 'video_size' to value '640x480'
[graph 0 input from stream 1:0 @ 0x7fffc3477a40] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 1:0 @ 0x7fffc3477a40] Setting 'time_base' to value '1/1000'
[graph 0 input from stream 1:0 @ 0x7fffc3477a40] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 1:0 @ 0x7fffc3477a40] Setting 'frame_rate' to value '1000/1'
[graph 0 input from stream 1:0 @ 0x7fffc3477a40] w:640 h:480 pixfmt:yuv420p tb:1/1000 fr:1000/1 sar:1/1
[graph 0 input from stream 0:0 @ 0x7fffc3474280] Setting 'video_size' to value '640x480'
[graph 0 input from stream 0:0 @ 0x7fffc3474280] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 0x7fffc3474280] Setting 'time_base' to value '1/1000'
[graph 0 input from stream 0:0 @ 0x7fffc3474280] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0x7fffc3474280] Setting 'frame_rate' to value '1000/1'
[graph 0 input from stream 0:0 @ 0x7fffc3474280] w:640 h:480 pixfmt:yuv420p tb:1/1000 fr:1000/1 sar:1/1
[graph_0_in_0_1 @ 0x7fffc348eb00] Setting 'time_base' to value '1/48000'
[graph_0_in_0_1 @ 0x7fffc348eb00] Setting 'sample_rate' to value '48000'
[graph_0_in_0_1 @ 0x7fffc348eb00] Setting 'sample_fmt' to value 's16'
[graph_0_in_0_1 @ 0x7fffc348eb00] Setting 'channel_layout' to value '0x3'
[graph_0_in_0_1 @ 0x7fffc348eb00] tb:1/48000 samplefmt:s16 samplerate:48000 chlayout:0x3
[graph_0_in_1_1 @ 0x7fffc3490240] Setting 'time_base' to value '1/48000'
[graph_0_in_1_1 @ 0x7fffc3490240] Setting 'sample_rate' to value '48000'
[graph_0_in_1_1 @ 0x7fffc3490240] Setting 'sample_fmt' to value 's16'
[graph_0_in_1_1 @ 0x7fffc3490240] Setting 'channel_layout' to value '0x3'
[graph_0_in_1_1 @ 0x7fffc3490240] tb:1/48000 samplefmt:s16 samplerate:48000 chlayout:0x3
[format_out_0_1 @ 0x7fffc3474680] Setting 'sample_fmts' to value 'fltp'
[format_out_0_1 @ 0x7fffc3474680] Setting 'sample_rates' to value '96000|88200|64000|48000|44100|32000|24000|22050|16000|12000|11025|8000|7350'
[format_out_0_1 @ 0x7fffc3474680] Setting 'channel_layouts' to value '0x3'
[format @ 0x7fffc34769c0] Setting 'pix_fmts' to value 'yuv420p|yuvj420p|yuv422p|yuvj422p|yuv444p|yuvj444p|nv12|nv16|nv21'
[Parsed_amerge_4 @ 0x7fffc3478180] No channel layout for input 1
[AVFilterGraph @ 0x7fffc3472880] query_formats: 17 queried, 23 merged, 0 already done, 9 delayed
[AVFilterGraph @ 0x7fffc3472880] query_formats not finished
[Parsed_amerge_4 @ 0x7fffc3478180] Input channel layouts overlap: output layout will be determined by the number of distinct input channels
[AVFilterGraph @ 0x7fffc3472880] query_formats: 1 queried, 9 merged, 23 already done, 0 delayed
[Parsed_amerge_4 @ 0x7fffc3478180] in0:stereo + in1:stereo -> out:4.0
[Parsed_aresample_5 @ 0x7fffc3478a40] [SWR @ 0x7fffc34791c0] Using fltp internally between filters
[Parsed_aresample_5 @ 0x7fffc3478a40] [SWR @ 0x7fffc34791c0] Matrix coefficients:
[Parsed_aresample_5 @ 0x7fffc3478a40] [SWR @ 0x7fffc34791c0] FL: FL:1.000000 FR:0.000000 FC:0.707107 BC:0.500000
[Parsed_aresample_5 @ 0x7fffc3478a40] [SWR @ 0x7fffc34791c0] FR: FL:0.000000 FR:1.000000 FC:0.707107 BC:0.500000
[Parsed_aresample_5 @ 0x7fffc3478a40] ch:4 chl:4.0 fmt:s16 r:48000Hz -> ch:2 chl:stereo fmt:fltp r:48000Hz
[Parsed_scale2ref_0 @ 0x7fffc3473a00] w:640 h:480 fmt:yuv420p sar:1/1 -> w:640 h:480 fmt:yuv420p sar:1/1 flags:0x2
[Parsed_scale2ref_1 @ 0x7fffc3474080] w:640 h:480 fmt:yuv420p sar:1/1 -> w:640 h:480 fmt:yuv420p sar:1/1 flags:0x2
[Parsed_hstack_2 @ 0x7fffc34762c0] [framesync @ 0x7fffc3476498] Selected 1/1000 time base
[Parsed_hstack_2 @ 0x7fffc34762c0] [framesync @ 0x7fffc3476498] Sync level 1
[Parsed_setsar_3 @ 0x7fffc3477480] w:1280 h:480 sar:1/1 dar:8/3 -> sar:1/1 dar:8/3
[libx264 @ 0x7fffc33a27c0] using mv_range_thread = 40
[libx264 @ 0x7fffc33a27c0] using SAR=1/1
[libx264 @ 0x7fffc33a27c0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX XOP FMA3 BMI1
[libx264 @ 0x7fffc33a27c0] profile High, level 3.1
[libx264 @ 0x7fffc33a27c0] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to './out.mp4':
  Metadata:
    encoder         : Lavf58.44.100
    Stream #0:0, 0, 1/15360: Video: h264 (libx264), 1 reference frame (avc1 / 0x31637661), yuv420p, 1280x480 [SAR 1:1 DAR 8:3], 0/1, q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc58.90.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
    Stream #0:1, 0, 1/48000: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, delay 1024, 128 kb/s (default)
    Metadata:
      encoder         : Lavc58.90.100 aac
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:1 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 32 times
*** dropping frame 6 from stream 0 at ts 4
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 3 times
*** dropping frame 7 from stream 0 at ts 5
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 5 times
*** dropping frame 8 from stream 0 at ts 6
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 4 times
*** dropping frame 9 from stream 0 at ts 7
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 5 times
*** dropping frame 10 from stream 0 at ts 9
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 2 times
*** dropping frame 10 from stream 0 at ts 9
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 3 times
*** dropping frame 11 from stream 0 at ts 10
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
    Last message repeated 6 times

...

*** dropping frame 919 from stream 0 at ts 917
[libx264 @ 0x7fffc33a27c0] frame= 869 QP=24.99 NAL=0 Slice:B Poc:234 I:0    P:951  SKIP:1448 size=989 bytes
*** dropping frame 920 from stream 0 at ts 918
*** dropping frame 920 from stream 0 at ts 919
[libx264 @ 0x7fffc33a27c0] frame= 870 QP=25.62 NAL=0 Slice:B Poc:238 I:9    P:1022 SKIP:1366 size=1462 bytes
*** dropping frame 921 from stream 0 at ts 920
[libx264 @ 0x7fffc33a27c0] frame= 871 QP=20.97 NAL=2 Slice:P Poc:248 I:170  P:1812 SKIP:418  size=11296 bytes
[libx264 @ 0x7fffc33a27c0] frame= 872 QP=24.64 NAL=2 Slice:B Poc:244 I:31   P:1274 SKIP:1080 size=3317 bytes
*** dropping frame 923 from stream 0 at ts 921
...
[libx264 @ 0x7fffc33a27c0] frame= 888 QP=21.04 NAL=2 Slice:P Poc:282 I:135  P:1533 SKIP:732  size=7943 bytes
[libx264 @ 0x7fffc33a27c0] frame= 889 QP=24.72 NAL=2 Slice:B Poc:278 I:10   P:1027 SKIP:1355 size=1865 bytes
[libx264 @ 0x7fffc33a27c0] frame= 890 QP=24.85 NAL=0 Slice:B Poc:276 I:0    P:804  SKIP:1596 size=716 bytes

...

Not writing 'clli' atom. No content light level info.
Not writing 'mdcv' atom. Missing mastering metadata.
frame=  923 fps= 49 q=-1.0 Lsize=    4233kB time=00:00:30.86 bitrate=1123.2kbits/s dup=0 drop=847 speed=1.64x
video:3714kB audio:484kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.818652%
Input file #0 (./str_CAM_N0PW_con_Gz4hFc86ts.webm):
  Input stream #0:0 (video): 939 packets read (3924348 bytes); 936 frames decoded;
  Input stream #0:1 (audio): 1560 packets read (117307 bytes); 1560 frames decoded (1497600 samples);
  Total: 2499 packets (4041655 bytes) demuxed
Input file #1 (./str_CAM_GPjV_con_NxaIMlsISO.webm):
  Input stream #1:0 (video): 871 packets read (1365403 bytes); 868 frames decoded;
  Input stream #1:1 (audio): 1545 packets read (97667 bytes); 1545 frames decoded (1483200 samples);
  Total: 2416 packets (1463070 bytes) demuxed
Output file #0 (./out.mp4):
  Output stream #0:0 (video): 923 frames encoded; 923 packets muxed (3803227 bytes);
  Output stream #0:1 (audio): 1447 frames encoded (1481728 samples); 1448 packets muxed (495667 bytes);
  Total: 2371 packets (4298894 bytes) muxed
4909 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0x7fffc33bc040] Statistics: 2 seeks, 20 writeouts
[libx264 @ 0x7fffc33a27c0] frame I:4     Avg QP:19.28  size: 34312
[libx264 @ 0x7fffc33a27c0] frame P:251   Avg QP:21.50  size:  9620
[libx264 @ 0x7fffc33a27c0] frame B:668   Avg QP:25.66  size:  1872
[libx264 @ 0x7fffc33a27c0] consecutive B-frames:  2.7%  1.7%  2.0% 93.6%
[libx264 @ 0x7fffc33a27c0] mb I  I16..4: 16.3% 70.5% 13.2%
[libx264 @ 0x7fffc33a27c0] mb P  I16..4:  2.9%  5.9%  0.5%  P16..4: 44.1% 14.0%  6.0%  0.0%  0.0%    skip:26.6%
[libx264 @ 0x7fffc33a27c0] mb B  I16..4:  0.2%  0.3%  0.0%  B16..8: 37.7%  1.5%  0.2%  direct: 0.7%  skip:59.4%  L0:46.2% L1:52.0% BI: 1.8%
[libx264 @ 0x7fffc33a27c0] 8x8 transform intra:63.3% inter:79.9%
[libx264 @ 0x7fffc33a27c0] coded y,uvDC,uvAC intra: 47.6% 57.8% 14.3% inter: 11.6% 9.1% 0.1%
[libx264 @ 0x7fffc33a27c0] i16 v,h,dc,p: 21% 18% 32% 29%
[libx264 @ 0x7fffc33a27c0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 17% 28%  5%  4%  6%  5%  5%  5%
[libx264 @ 0x7fffc33a27c0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 18% 16%  5%  6%  7%  6%  5%  3%
[libx264 @ 0x7fffc33a27c0] i8c dc,h,v,p: 54% 21% 21%  5%
[libx264 @ 0x7fffc33a27c0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fffc33a27c0] ref P L0: 58.7% 18.4% 16.6%  6.3%
[libx264 @ 0x7fffc33a27c0] ref B L0: 89.8%  8.3%  1.9%
[libx264 @ 0x7fffc33a27c0] ref B L1: 95.4%  4.6%
[libx264 @ 0x7fffc33a27c0] kb/s:988.74
[aac @ 0x7fffc33a4a80] Qavg: 242.930
[AVIOContext @ 0x7fffc3398a80] Statistics: 4162659 bytes read, 3 seeks
[AVIOContext @ 0x7fffc33a1940] Statistics: 1507328 bytes read, 0 seeks
Exiting normally, received signal 2.
  • Inputs must have same video fps and timebase when used in filtergraph. Next time post ffmpeg -v debug output of run command. – Paul B. Mahol Jun 10 at 8:56
  • @PaulB.Mahol I updated my original post with the debug output – ameotoko Jun 13 at 18:48
  • I think inputs or one of inputs are of variable frame rate, thus use fps=30 filter before each of hstack inputs. – Paul B. Mahol Jun 14 at 17:11
  • didn't work. I decided to abandon ffmpeg, since whenever I encode anything to mp4/h264/aac, the result is always out of sync, regardless of the machine, system, or ffmpeg version. Very frustrating... – ameotoko Jun 15 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.