2

I am using FFmpeg's built-in gdigrab & dshow to record the display & audio of my system. I am encoding to uncompressed H.264/AVC video & uncompressed PCM/WAV audio.

I had been having two problems with audio/video synchronization:

  1. Video & audio not aligned (video & audio do not start at same time).
  2. Audio drift (audio progressively becomes out of sync with video).

Originally I was using a single command for capture/encode. Something like the following:

ffmpeg -hide_banner -rtbufsize 1000M -f gdigrab -framerate 60 -draw_mouse 0 \
  -i "title=<window_name>" -f dshow -i audio="<sys_audio/mic>" -c:v libx264 \
  -preset ultrafast -qp 0 -x264opts keyint=1 -c:a pcm_s16le -ac 1 "<out_file>.mkv"

But I was able to solve my second problem (drifting) by capturing video & audio in separate FFmpeg processes. Note that I am using a UNIX-like shell (MSYS2 with BASH). The following is an example of a shell script I am running:

# capture audio & get PID
ffmpeg -hide_banner -rtbufsize 500M -f dshow -i audio="<sys_audio/mic>" \
  -c:a pcm_s16le -ac 1 "<out_file>-audio.wav" & APID=$!

# capture video
ffmpeg -hide_banner -rtbufsize 1000M -f gdigrab -framerate 60 -draw_mouse 0 \
  -i "<window_name>" -c:v libx264 -preset ultrafast -qp 0 -x264opts keyint=1 \
  "<out_file>-video.mkv"

# get exit code of video process
VIDRET=$?

# send interrupt signal to audio process after video process exits
kill -s SIGINT ${APID}

# mux video & audio if video process exited okay
if [ "${VIDRET}" -eq "0" ]; then
    ffmpeg -hide_banner -i "${OUTVID}" -i "${OUTAUD}" -map 0:0 -map 1:0 \
      -c copy "<out_file>.mkv"

    # delete temp video & audio streams if muxing succeeded
    if [ "$?" -eq "0" ]; then
        rm "${OUTVID}" "${OUTAUD}"
    fi
fi

So the problem left is that video & audio data do not start aligned correctly. The audio is generally somewhere between 0ms-300ms ahead of the video.

It can be easily solved by running the output file through FFmpeg again, or a separate program such as Avidemux, to adjust the audio delay. This can be done without re-encoding in Avidemux (not sure about FFmpeg).

However, I would prefer to solve this issue within the capture process/script so I can avoid the extra step of aligning the data manually.

Recently I have done a system cleanup by removing junk files, making sure fragmentation on my drive is low, & turning off unnecessary background processes. But audio/video alignment is still often off.

So finally, to the plain-&-simple question: Is there a way to get the two FFmpeg processes to begin capture at the same time to get video & audio as close to being synchronized as possible? Can I use the system clock as a seed?

Or, is there a better method than I am using right now? E.g., a single FFmpeg process that will capture both while keeping data aligned & avoiding audio drift.

It appears to me that the problem is simply that the audio process is starting before the video process. I've heard about piping FFmpeg commands, but am not sure of the correct way to do it. I have found some information on piping that I am trying to figure out:

FFmpeg information: Official static 64-bit build from Zeranoe.

ffmpeg version N-92511-g0279cb4f69 Copyright (c) 2000-2018 the FFmpeg developers
    built with gcc 8.2.1 (GCC) 20181017
    configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig
        --enable-gnutls --enable-iconv --enable-libass --enable-libbluray
        --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb
        --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine
        --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame
        --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264
        --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib
        --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc
        --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom
        --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va
        --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth

These are my system specs:

-- Edit --

I did figure out a pipe command, but the result is the same. Audio is slightly ahead:

ffmpeg -hide_banner -rtbufsize 500M -f dshow -ac 1 -i audio="<sys_audio/mic>" \
  -c:a pcm_s16le -f s16le pipe: | ffmpeg -y -hide_banner -rtbufsize 1500M -f gdigrab \
  -thread_queue_size 128 -framerate 60 -draw_mouse 0 -i title="<window_name>" -f s16le \
  -thread_queue_size 128 -i pipe: -map 0:0 -map 1:0 -c:v libx264 -preset ultrafast -qp 0 \
  -x264opts keyint=1 -c:a copy "<out_file>.mkv"
Improve this question

2 Answers 2

Reset to default
0

After doing what you describe with separate files for a while, I recently started using embedded streams instead. It's late and I'm tired, but you should be able to see how it works, and I can elaborate later if you need me to.

using separate output files, just so you can see how I have that set up...for reference.

ffmpeg -hide_banner \
    -re \
    $videoDecodeMode \
    -f pulse \
        -i $audioOut \
    -f pulse \
        -i $audioMic \
    -f x11grab \
        -r $frameRate \
        -s $resolution \
        -i :0.0 \
        -g 60 \
    $videoEncodeMode \
    -map 2:0 \
        -metadata handler="video:1920x1080@60" \
        "${exportName}" \
    -map 0:0 \
        -metadata title="audio:out" \
        "${exportName%.*}_audioOut.wav" \
    -map 1:0 \
        -metadata title="audio:microphone" \
        "${exportName%.*}_audioMic.wav" </dev/null &

using embedded streams

ffmpeg -hide_banner \
    -re \
    $videoDecodeMode \
    -f pulse \
        -thread_queue_size 1024 \
        -i $audioOut \
    -f pulse \
        -thread_queue_size 1024 \
        -i $audioMic \
    -f x11grab \
        -r $frameRate \
        -s $resolution \
        -thread_queue_size 1024 \
        -i :0.0 \
    -g 60 \
    -map 2:v \
    -map 0:a \
    -map 1:a \
    -c:a aac -ac 2 -b:a 128k -r:a 48000 \
    -metadata:s:v:0 handler="video:1920x1080@60" \
    -metadata:s:a:2 handler="audio:out" \
    -metadata:s:a:3 handler="audio:microphone" \
    $videoEncodeMode \
    "${exportName}"  </dev/null &

embedded streams, companded mic to clean up my line-noise problem a bit, and some stream mixing to put output and mic together on their own stream

ffmpeg -hide_banner \
    -re \
    $videoDecodeMode \
    -f pulse \
        -thread_queue_size 1024 \
        -i $audioOut \
    -f pulse \
        -thread_queue_size 1024 \
        -i $audioMic \
    -f x11grab \
        -r $frameRate \
        -s $resolution \
        -thread_queue_size 1024 \
        -i :0.0 \
    -g 60 \
    -filter_complex \
        "[1:a]aformat=channel_layouts=stereo,asplit=2[micOrig][micNew]\
        ;[micNew]\
            compand=0:0.2:-26/-900|-16/-16|0/-10|10/-900:6:0:0:0\
        [micCleaned]\
        ;[0:a][micCleaned]amix=inputs=2:duration=first[allAudio]" \
    -map 2:v -map 0:a \
    -map [allAudio] -map [micOrig] \
    -c:a aac -ac 2 -b:a 128k -r:a 48000 \
    -metadata:s:v:0 handler="video:1920x1080@60" \
    -metadata:s:a:2 handler="audio:out" \
    -metadata:s:a:1 handler="audio:combined" \
    -metadata:s:a:3 handler="audio:microphone" \
    $videoEncodeMode \
    "${exportName}"  </dev/null &
Improve this answer
0

If it is Windows then why not to use dshow instead of gdigrab?

Please look at the https://github.com/rdp/screen-capture-recorder-to-video-windows-free

If both video and audio would be via dshow then you could use ffmpeg like this:

ffmpeg -f dshow -i video="VIDEO_DEVICE":audio="FIRST_AUDIO_DEVICE" -f dshow -i audio="ANOTHER_AUDIO_DEVICE" ...
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.