Syncing Multiple Audio Streams With One Video Stream (FFMPEG)

Question

I'll try and keep this as simple as possible, and thanks for any help.

I have two PC's one that I'm gaming on, one that I'm recording on. I also have two RME audio cards (one in each respective PC) hooked up to each other to share audio between the PC's. My main display is hooked up to a displayport splitter that splits to my recording PC's capture card and I'm also routing my game audio from the sound card in my gaming PC to the capture PC's audio card.

I'm running discord on the capture PC and have my game audio, discord audio, and my personal mic all on separate input channels. I have the capture card video and the game audio channel output to the same MP4 file and then have my mic audio and the discord audio output to two separate WAV files. This way I can choose to omit the voices from the video so only game audio can be heard and or I can mix the volumes independently after recording. This way I can make sure the discord, my voice, or the game audio isn't overwhelming in the final product.

Everything seemed to be working great, until my capture card dropped a few frames over the course of 3 hours. The resulting footage had out of sync audio and video, the discord audio and my mic were also now out of sync with the game footage. After searching around for a very long time it seemed the solution was -vsync 1 but it didn't seem to work, here was my code for that:

ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize 2147M -pixel_format nv12 -r 100.00 -vsync 1 ^
-i video="Video (00 Pro Capture HDMI 4K+)":audio="Analog (1+2) (RME Fireface UC)" -map 0:0 -map 0:1 ^
-codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate 250M -bufsize 250M -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\PC%02d.mp4 ^
-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (3+4) (RME Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Theirs\TheirsPC%02d.wav ^
-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (5+6) (RME Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Mine\MinePC%02d.wav

That's all one line of code, carrots are line breaks. But after playing for a while and dropping some frames the problems persisted. I was under the impression that -vsync 1 forced the desired frame rate, duplicating frames to make sure the audio stayed in-sync. So I thought maybe I had it in the wrong place and tried this:

ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize 2147M -pixel_format nv12 -r 100.00 ^
-i video="Video (00 Pro Capture HDMI 4K+)":audio="Analog (1+2) (RME Fireface UC)" -vsync 1 -map 0:0,0:1 -map 0:1 ^
-codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate 250M -bufsize 250M -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\PC%02d.mp4 ^
-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (3+4) (RME Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Theirs\TheirsPC%02d.wav ^
-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (5+6) (RME Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Mine\MinePC%02d.wav

I actually have yet to drop a frame since then... is this my code causing this? With vsync working correctly would it no-longer display dropped frames in the console? Am I mapping correctly in conjunction with vsync? It almost seems like with -map 0:0,0:1 I'd be syncing the audio to the video but what I would really like is if I drop frames just take the last successful frame and duplicate it. That way (theoretically) the other two audio channels would stay in-sync too... I'm just trying to make sure that I don't screw up an important recording before it happens.

Thanks for any help.

Answer 1

Just thought I'd let you guys know that I got this working.

I needed to use -framerate instead of -r to get it to duplicate dropped frames, the reason I wasn't using -framerate in the first place is because it would spam the message "past duration too large" and the video wouldn't play back after exiting the recording. I also used an audio filter to delay the other channels so they they would start at the the same time as the video. I had to use -af "adelay=200|200" instead of -filter_complex "[1:0] adelay=200|200 [1:0]" for some reason. Ended up being cleaner anyways and it worked but I don't understand why the -filter_complex wasn't working an the -af was. I went ahead and applied -vsync 1 and -async 1 but I'm not quite sure if that is default and therefore redundant. Lastly I used -preset llhp (low latency high performance) which seemed to decrease the amount of dropped frames. The result - everything stays in-sync even if I drop frames and all tracks "start" at the same time:

ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize 2147.48M -pixel_format nv12 -framerate 200 ^
-i video="Video (00 Pro Capture HDMI 4K+)":audio="SPDIF/ADAT (1+2) (RME Fireface UC)" -map 0:0,0:1 -map 0:1 ^
-preset: llhp -codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate:v 250M -minrate:v 250M -bufsize:v 250M -b:a 320k ^
-ac 2 -r 100 -async 1 -vsync 1 -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\PC%02d.mp4 ^
-guess_layout_max 0 -f dshow -rtbufsize 2000M -i audio="Analog (3+4) (RME Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^
-af "adelay=200|200" -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Theirs\TPC%02d.wav ^
-guess_layout_max 0 -f dshow -rtbufsize 2000M -i audio="Analog (5+6) (RME Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^
-af "adelay=825|825" -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Mine\MPC%02d.wav

I was able to get the rid of the "past duration too large" warning by increasing the input framerate to 200, I have absolutely no idea why this would fix that message. Put -r 100 on the output to make sure it came out as 100FPS which is what I actually wanted the framerate to be. I am confused as to why several parts work the way they do and can’t find answers but nonetheless it’s working.