I'm merging together a bunch of videos (30 so far) into a "Brady Bunch" style amalgam using ffmpeg, and I've got that working reasonably well. The last big problem is the background noise. Everybody recorded at different locations, and some of those locations had computer fans in the background and some had air conditioners, and when you add them all together, the summed background noise is really noticeable.
I'm doing everything with complex filters, here's a rough idea what I've got going on:
color=size=1920x1080:c=0x000080 [base]
[0:a]volume=0.90,aresample=async=1:first_pts=0[aud0]
[1:a]volume=2.08,aresample=async=1:first_pts=0[aud1]
[2:a]volume=1.48,aresample=async=1:first_pts=0[aud2]
[3:a]volume=2.53,aresample=async=1:first_pts=0[aud3]
[4:a]volume=1.34,aresample=async=1:first_pts=0[aud4]
[5:a]volume=1.81,aresample=async=1:first_pts=0[aud5]
(and a bunch more adjusting the volume of each input video)
[28:a]volume=1.06,aresample=async=1:first_pts=0[aud28]
(and then we start the videos, at first only a subset are visible and then the large group
joins in, so I "zoom in" on the small group and then expand to bring in everyone else
when it's their turn)
[0:v] scale=w=if(lte(n\,1200.00)\,576\,if(lte(n\,1410.00)\,576.0-288.00*(n-1200.00)/210.00\,288)):h=if(lte(n\,1200.00)\,324\,if(lte(n\,1410.00)\,324.0-162.00*(n-1200.00)/210.00\,162)):eval=frame [clip0]
[base][clip0] overlay=shortest=1:x=if(lte(n\,1000.00)\,-608\,if(lte(n\,1175.00)\,-608.0+624.00*(n-1000.00)/175.00\,16)):y=if(lte(n\,1000.00)\,-342\,if(lte(n\,1175.00)\,-342.0+369.00*(n-1000.00)/175.00\,27)) [over0]
[1:v] crop=870:917:553:163,scale=w=if(lte(n\,1198.80)\,307\,if(lte(n\,1408.59)\,307.0-123.00*(n-1198.80)/209.79\,184)):h=if(lte(n\,1198.80)\,324\,if(lte(n\,1408.59)\,324.0-130.00*(n-1198.80)/209.79\,194)):eval=frame [clip1]
[over0][clip1] overlay=shortest=1:x=if(lte(n\,1000.00)\,166\,if(lte(n\,1175.00)\,166.0+222.00*(n-1000.00)/175.00\,388)):y=if(lte(n\,1000.00)\,-342\,if(lte(n\,1175.00)\,-342.0+353.00*(n-1000.00)/175.00\,11)) [over1]
(you get the idea)
(and then I merge all the audio together)
[aud0][aud1][aud2][aud3][aud4][aud5][aud6][aud7][aud8][aud9][aud10][aud11][aud12][aud13][aud14][aud15][aud16][aud17][aud18][aud19][aud20][aud21][aud22][aud23][aud24][aud25][aud26][aud27][aud28]amix=inputs=29[a]
[a]adelay=delays=200:all=1[b]
(I've run a first pass of loudnorm to measure the output levels, and I send them back in)
[b]loudnorm=linear=true:I=-14:LRA=11:tp=-2:measured_I=-30.18:measured_LRA=17.80:measured_tp=-15.23:measured_thresh=-41.39:offset=-0.77:print_format=summary[level]
What I'm thinking, is that most everybody starts out with a few seconds of silence. It seems like I should be able to measure the audio in that segment, and then use it to filter out the background per stream.
It seems like afftdn will do that, but I cannot for the life of me figure out how to use it. Apparently I need to send commands to it, as I have read here, but it's not at all clear how I make that work inside my complex filter. Something to do with ZMQ? Is ZMQ what's happening with all those "@n"s?
EDIT
The answer that Gyan refers to is this:
ffmpeg -i in -i profile -filter_complex "[1][0]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@n sn start; 1.0 afftdn@n sn stop',afftdn@n,atrim=1,asetpts=PTS-STARTPTS" noiseout.wav
Let me see if I can break this down and understand it.
In that question, [1] is the noise profile and [0] is the input audio.
[1][0]concat=n=2:v=0:a=1
Means concatenate the two audio streams, noise profile first, and generate as an output only an audio stream.
Then we get ready to send commands to afftdn, using the ZMQ syntax:
asendcmd=c='0.0 afftdn@n sn start; 1.0 afftdn@n sn stop'
In ZMQ, you have to give a filter an id so it knows which filter you are sending the command to. In this case, we've assigned the one and only afftdn the id 'n', and we are telling it to start measuring the noise profile at time 0 (the default so not specified) and stop measuring at time 1.
So now we actually invoke the afftdn filter:
afftdn@n
Then we trim off that first second of the audio stream (which is just the noise profile):
atrim=1
And finally, we want to reset the timestamp of the audio stream so that 'zero' is the new position (after the noise profile):
asetpts=PTS-STARTPTS
SO
For now at least, I'm going to always use the first second of the audio as my noise profile. Since it's already at the start of the audio, I don't need to isolate it, reset its timestamp, and concatenate it at the start of the audio (right?). So taking audio stream [aud0], I can do this:
[aud0]asendcmd=c='0.0 afftdn@n sn start; 1.0 afftdn@n sn stop',afftdn@n[faud0]
and then send [faud0] into the 'amix' instead of [aud0]
Yes?
SECOND EDIT
I'm now duplicating the first second of each audio stream and appending it on the front, just like in Gyan's answer. Here is one such audio stream:
[19:a]volume=31.59,aresample=async=1:first_pts=0[aud19]
[aud19]asplit[aud19a][aud19b]
[aud19a]atrim=0:1[aud19c]
[aud19c][aud19b]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@t sn start; 1.0 afftdn@t sn stop',afftdn@t=nr=90,atrim=1,asetpts=PTS-STARTPTS[aud19d]
Notice that since I've got so many audio streams, and each one has its own afftdn filter, I've given them all separate ids.
And still, the output audio does not sound significantly different from when I started!
THIRD EDIT
So I grabbed one of the videos that has a lot of noise at the front, kind of sounds like lots of air moving, maybe some 60 Hz fan noise too. I extracted the first 9 seconds into a file I cleverly named "trim.wav". I then ran ffmpeg twice, once normally, and once to output just the noise. The "just noise" file is totally devoid of sound, as if it didn't find any noise to work with. And as expected, the (supposedly) processed file sounds identical to the input.
Here are my command lines:
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asplit[aud0a][aud0b];[aud0a]atrim=0:1[aud0c];[aud0c][aud0b]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa,atrim=1,asetpts=PTS-STARTPTS[aud0d]" -map "[aud0d]" processed.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asplit[aud0a][aud0b];[aud0a]atrim=0:1[aud0c];[aud0c][aud0b]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=om=n,atrim=1,asetpts=PTS-STARTPTS[aud0d]" -map "[aud0d]" noise.wav
Here's what ffmpeg had to say:
ffmpeg version git-2020-08-07-fab00b0 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 10.2.1 (GCC) 20200805
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --enable-librav1e --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
libavutil 56. 58.100 / 56. 58.100
libavcodec 58. 99.100 / 58. 99.100
libavformat 58. 49.100 / 58. 49.100
libavdevice 58. 11.101 / 58. 11.101
libavfilter 7. 87.100 / 7. 87.100
libswscale 5. 8.100 / 5. 8.100
libswresample 3. 8.100 / 3. 8.100
libpostproc 55. 8.100 / 55. 8.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'trim.wav':
Metadata:
encoder : Lavf58.49.100
Duration: 00:00:09.00, bitrate: 1411 kb/s
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Stream mapping:
Stream #0:0 (pcm_s16le) -> asplit
asetpts -> Stream #0:0 (pcm_s16le)
Output #0, wav, to 'processed.wav':
Metadata:
ISFT : Lavf58.49.100
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Metadata:
encoder : Lavc58.99.100 pcm_s16le
[afftdn @ 00000261c3b81400] bn=4 -6 -10 -12 -12 -14 -17 -23 -24 -24 -24 -24 -24 -24 -15
[afftdn @ 00000261c3b81400] bn=5 -4 -8 -10 -11 -14 -18 -23 -24 -24 -24 -24 -24 -24 -15
size= 1545kB time=00:00:08.97 bitrate=1411.3kbits/s speed= 150x
video:0kB audio:1545kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.004929%
FOURTH EDIT
Based on comments, I tried adjusting the noise floor and also using just a single audio track. I ran these commands:
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=90[aud0d]" -map "[aud0d]" output-nr90.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=1[aud0d]" -map "[aud0d]" output-nr1.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=90:nf=-20[aud0d]" -map "[aud0d]" output-nr90-nf20.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=1:nf=-20[aud0d]" -map "[aud0d]" output-nr1-nf20.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=90:nf=-80[aud0d]" -map "[aud0d]" output-nr90-nf80.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=1:nf=-80[aud0d]" -map "[aud0d]" output-nr1-nf80.wav
and for the most part I don't hear any noticeable difference. However, with the filter 'afftdn@aa=nr=90:nf=-20' I hear what sounds like a 1-second reprieve from the noise at the very beginning of the audio, as if that noise was being filtered out while leaving all of the noise in the rest of the track. Am I just using the wrong filter? Is my noise not the type of noise that afftdn is effective on? Or am I somehow misusing the filter so it's only filtering the first second? For reference, here is the input track.
