Video concatenation puts sound out of sync

Question

I have several MOV files from a DSLR camera. I concatenate them with directions from this thread:

ffmpeg -safe 0 -f concat -i files_to_combine -vcodec copy -acodec copy temp.MOV

where files_to_combine is:

file ./DSC_0013.MOV
...
file ./DSC_0019.MOV

The result has image and sound in sync for the first clip and is out of sync by fractions of a second in the second clip, and out of sync by around a second for the last clip. It is probably related to this error from the log:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dd802200] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dd802200] st: 0 edit list 1 Cannot find an index entry before timestamp: 1000.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dd802200] Auto-inserting h264_mp4toannexb bitstream filter

How can I trim the frames to the available sound stream, then concatenate the two videos?

The full log from the ffmpeg command is:

ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.3_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dc00e000] Auto-inserting h264_mp4toannexb bitstream filter
Input #0, concat, from 'files_to_combine':
  Duration: N/A, start: -0.592000, bitrate: 36888 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/bt709/bt470m), 1920x1080, 35352 kb/s, 50 fps, 50 tbr, 50k tbn, 100 tbc
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
      handler_name    : SoundHandler
Output #0, mov, to 'temp.MOV':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/bt709/bt470m), 1920x1080, q=2-31, 35352 kb/s, 50 fps, 50 tbr, 50k tbn, 50k tbc
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dd802200] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dd802200] st: 0 edit list 1 Cannot find an index entry before timestamp: 1000.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f82dd802200] Auto-inserting h264_mp4toannexb bitstream filter
frame=41886 fps=547 q=-1.0 Lsize= 3789826kB time=00:13:58.75 bitrate=37014.8kbits/s speed=10.9x    
video:3631879kB audio:157123kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.021759%

Update (1 July 2019)

I thought that the files had a problem at the beginning or at the end, so I trimmed one second from each end, but it still had the sound out of sync:

FILES=files_to_combine
OUTPUT=show2.MOV
rm $FILES
for i in 3 4 5 6 7 8 9; do
    rm ${i}.MOV
    duration=$(ffprobe -v 0 -show_entries format=duration -of compact=p=0:nk=1  DSC_001${i}.MOV)
    trimmed=$(echo $duration - 1 | bc)
    ffmpeg -ss 1 -t $trimmed -i DSC_001${i}.MOV -vcodec copy -acodec copy ${i}.MOV
    echo file ./${i}.MOV >> $FILES
done

rm $OUTPUT
ffmpeg -safe 0 -f concat -i $FILES -vcodec copy -acodec copy $OUTPUT

When I trim a single file near the end, the sound and video do not seem out of sync:

ffmpeg -ss 00:09:20 -t 20 -i DSC_0014.MOV -vcodec copy -acodec copy end.MOV

When I concatenate only 30 seconds from each video, the result seems OK:

FILES=files_to_combine
OUTPUT=show2.MOV
rm $FILES
for i in 3 4 5 6 7 8 9; do
    rm ${i}.MOV
    duration=$(ffprobe -v 0 -show_entries format=duration -of compact=p=0:nk=1  DSC_001${i}.MOV)
    start=$(echo $duration - 30 | bc)
    end=$(echo $duration - 1 | bc)
    ffmpeg -ss $start -t $end -i DSC_001${i}.MOV -vcodec copy -acodec copy ${i}.MOV
    echo file ./${i}.MOV >> $FILES
done

rm $OUTPUT
ffmpeg -safe 0 -f concat -i $FILES -vcodec copy -acodec copy $OUTPUT

This last concatenation gives this error multiple times:

[mov @ 0x7fc3c7837400] Non-monotonous DTS in output stream 0:0; previous: 9080205, current: 9080200; changing to 9080206. This may result in incorrect timestamps in the output file.

So I am guessing that the problem is small differences in timestamps that accumulate and become more noticeable with longer durations and the concatenation of multiple files.

For reference, the DSLR that shot these clips is a Nikon D3300 and the result of ffprobe on one of the files is:

$ ffprobe DSC_0017.MOV -hide_banner
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fab70003800] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fab70003800] st: 0 edit list 1 Cannot find an index entry before timestamp: 1000.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'DSC_0017.MOV':
  Metadata:
    major_brand     : qt  
    minor_version   : 537331968
    compatible_brands: qt  niko
    creation_time   : 2019-06-12T23:52:37.000000Z
  Duration: 00:09:53.58, start: 0.000000, bitrate: 36843 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/bt709/bt470m), 1920x1080, 35300 kb/s, 50 fps, 50 tbr, 50k tbn, 100 tbc (default)
    Metadata:
      creation_time   : 2019-06-12T23:52:37.000000Z
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, 2 channels, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2019-06-12T23:52:37.000000Z

Update (9 August 2019)

I concatenated the files in iMovie and the sound and image are not as out of sync as with FFMPEG. Maybe iMovie aligns the timestamps at the end of each clip instead of concatenating the audio and image streams separately.

I ran the concatenation again with the latest ffmpeg 4.1.4_1 on these files and others from the same camera. The audio and image are in sync in one case (the results lasts 46 minutes) out of sync in another (the result lasts 48 minutes).

Answer 1

Usually audio recording is just a continuous steam, with no concept of video frames. The audio is just being encoded/output via a continuous buffer, which will just dump out wherever it is up to you when you stop recording video frames - it is not uncommon for there to be more (or even less) audio in the file than is actually required for the corresponding video.

I would try a de-mux/re-mux of the video files before concatenating, to see if that cleans things up. With the de-muxed files as well, you could calculate how much audio is actually there in the file vs the video play time. I don't have enough experience with ffmpeg to know if this will work, just crossing my fingers that when you re-mux the files it might do a calculation of the required audio length, and trim it appropriately for you (there might also be parameters available to force the audio to trim at the video length, if that doesn't happen by default).

An editor like iMovie is going to properly cut up the audio to exactly match the video cuts, which is why that works. You could also try outputting a single unedited clip from iMovie, and then de-mux that, and see how the audio length differs from the original. Not sure if iMovie lets you output "audio only", but be careful trying that - I know with Adobe Premiere, if I output an audio only WAV, it will differ in length (longer file) than the same clip being output with a video stream as well.

Sorry I can't offer anything more definitive than some guesses/suggestions, I'm fairly positive the root cause is audio length not matching the video in the original file, but I'm not familiar enough with ffmpeg to know the best way to resolve the issue using that tool.

Update from original poster

Import the files in question into a program like iMovie and check the beginnings and endings of each file. Does the video stream have more length than the audio stream? Or try to play it in a program like VLC: do you hear a drop in the sound at the end or the beginning?

If so, then your camera, for whatever reason, records more video than sound. You can deal with this quickly in FFMPEG by trimming each clip to the shortest stream:

ffmpeg -i input.MOV -map 0 -c copy -fflags +shortest -max_interleave_delta 0 output.MOV

You may need to play with -max_interleave_delta depending on your files. From the documentation:

‘shortest’

Stop muxing at the end of the shortest stream. It may be needed to increase max_interleave_delta to avoid flushing the longer streams before EOF.

The command is from Trim audio stream to video stream length .