0

I'm merging together a bunch of videos (30 so far) into a "Brady Bunch" style amalgam using ffmpeg, and I've got that working reasonably well. The last big problem is the background noise. Everybody recorded at different locations, and some of those locations had computer fans in the background and some had air conditioners, and when you add them all together, the summed background noise is really noticeable.

I'm doing everything with complex filters, here's a rough idea what I've got going on:

color=size=1920x1080:c=0x000080 [base]
[0:a]volume=0.90,aresample=async=1:first_pts=0[aud0]
[1:a]volume=2.08,aresample=async=1:first_pts=0[aud1]
[2:a]volume=1.48,aresample=async=1:first_pts=0[aud2]
[3:a]volume=2.53,aresample=async=1:first_pts=0[aud3]
[4:a]volume=1.34,aresample=async=1:first_pts=0[aud4]
[5:a]volume=1.81,aresample=async=1:first_pts=0[aud5]
(and a bunch more adjusting the volume of each input video)
[28:a]volume=1.06,aresample=async=1:first_pts=0[aud28]
(and then we start the videos, at first only a subset are visible and then the large group
joins in, so I "zoom in" on the small group and then expand to bring in everyone else
when it's their turn)
[0:v] scale=w=if(lte(n\,1200.00)\,576\,if(lte(n\,1410.00)\,576.0-288.00*(n-1200.00)/210.00\,288)):h=if(lte(n\,1200.00)\,324\,if(lte(n\,1410.00)\,324.0-162.00*(n-1200.00)/210.00\,162)):eval=frame [clip0]
[base][clip0] overlay=shortest=1:x=if(lte(n\,1000.00)\,-608\,if(lte(n\,1175.00)\,-608.0+624.00*(n-1000.00)/175.00\,16)):y=if(lte(n\,1000.00)\,-342\,if(lte(n\,1175.00)\,-342.0+369.00*(n-1000.00)/175.00\,27)) [over0]
[1:v] crop=870:917:553:163,scale=w=if(lte(n\,1198.80)\,307\,if(lte(n\,1408.59)\,307.0-123.00*(n-1198.80)/209.79\,184)):h=if(lte(n\,1198.80)\,324\,if(lte(n\,1408.59)\,324.0-130.00*(n-1198.80)/209.79\,194)):eval=frame [clip1]
[over0][clip1] overlay=shortest=1:x=if(lte(n\,1000.00)\,166\,if(lte(n\,1175.00)\,166.0+222.00*(n-1000.00)/175.00\,388)):y=if(lte(n\,1000.00)\,-342\,if(lte(n\,1175.00)\,-342.0+353.00*(n-1000.00)/175.00\,11)) [over1]
(you get the idea)
(and then I merge all the audio together)
[aud0][aud1][aud2][aud3][aud4][aud5][aud6][aud7][aud8][aud9][aud10][aud11][aud12][aud13][aud14][aud15][aud16][aud17][aud18][aud19][aud20][aud21][aud22][aud23][aud24][aud25][aud26][aud27][aud28]amix=inputs=29[a]
[a]adelay=delays=200:all=1[b]
(I've run a first pass of loudnorm to measure the output levels, and I send them back in)
[b]loudnorm=linear=true:I=-14:LRA=11:tp=-2:measured_I=-30.18:measured_LRA=17.80:measured_tp=-15.23:measured_thresh=-41.39:offset=-0.77:print_format=summary[level]

What I'm thinking, is that most everybody starts out with a few seconds of silence. It seems like I should be able to measure the audio in that segment, and then use it to filter out the background per stream.

It seems like afftdn will do that, but I cannot for the life of me figure out how to use it. Apparently I need to send commands to it, as I have read here, but it's not at all clear how I make that work inside my complex filter. Something to do with ZMQ? Is ZMQ what's happening with all those "@n"s?

EDIT

The answer that Gyan refers to is this:

ffmpeg -i in -i profile -filter_complex "[1][0]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@n sn start; 1.0 afftdn@n sn stop',afftdn@n,atrim=1,asetpts=PTS-STARTPTS" noiseout.wav

Let me see if I can break this down and understand it.

In that question, [1] is the noise profile and [0] is the input audio.

[1][0]concat=n=2:v=0:a=1

Means concatenate the two audio streams, noise profile first, and generate as an output only an audio stream.

Then we get ready to send commands to afftdn, using the ZMQ syntax:

asendcmd=c='0.0 afftdn@n sn start; 1.0 afftdn@n sn stop'

In ZMQ, you have to give a filter an id so it knows which filter you are sending the command to. In this case, we've assigned the one and only afftdn the id 'n', and we are telling it to start measuring the noise profile at time 0 (the default so not specified) and stop measuring at time 1.

So now we actually invoke the afftdn filter:

afftdn@n

Then we trim off that first second of the audio stream (which is just the noise profile):

atrim=1

And finally, we want to reset the timestamp of the audio stream so that 'zero' is the new position (after the noise profile):

asetpts=PTS-STARTPTS

SO

For now at least, I'm going to always use the first second of the audio as my noise profile. Since it's already at the start of the audio, I don't need to isolate it, reset its timestamp, and concatenate it at the start of the audio (right?). So taking audio stream [aud0], I can do this:

[aud0]asendcmd=c='0.0 afftdn@n sn start; 1.0 afftdn@n sn stop',afftdn@n[faud0]

and then send [faud0] into the 'amix' instead of [aud0]

Yes?

SECOND EDIT

I'm now duplicating the first second of each audio stream and appending it on the front, just like in Gyan's answer. Here is one such audio stream:

[19:a]volume=31.59,aresample=async=1:first_pts=0[aud19]
[aud19]asplit[aud19a][aud19b]
[aud19a]atrim=0:1[aud19c]
[aud19c][aud19b]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@t sn start; 1.0 afftdn@t sn stop',afftdn@t=nr=90,atrim=1,asetpts=PTS-STARTPTS[aud19d]

Notice that since I've got so many audio streams, and each one has its own afftdn filter, I've given them all separate ids.

And still, the output audio does not sound significantly different from when I started!

THIRD EDIT

So I grabbed one of the videos that has a lot of noise at the front, kind of sounds like lots of air moving, maybe some 60 Hz fan noise too. I extracted the first 9 seconds into a file I cleverly named "trim.wav". I then ran ffmpeg twice, once normally, and once to output just the noise. The "just noise" file is totally devoid of sound, as if it didn't find any noise to work with. And as expected, the (supposedly) processed file sounds identical to the input.

Here are my command lines:

c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asplit[aud0a][aud0b];[aud0a]atrim=0:1[aud0c];[aud0c][aud0b]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa,atrim=1,asetpts=PTS-STARTPTS[aud0d]" -map "[aud0d]" processed.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asplit[aud0a][aud0b];[aud0a]atrim=0:1[aud0c];[aud0c][aud0b]concat=n=2:v=0:a=1,asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=om=n,atrim=1,asetpts=PTS-STARTPTS[aud0d]" -map "[aud0d]" noise.wav

Here's what ffmpeg had to say:

ffmpeg version git-2020-08-07-fab00b0 Copyright (c) 2000-2020 the FFmpeg developers
 built with gcc 10.2.1 (GCC) 20200805
 configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --enable-librav1e --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
 libavutil      56. 58.100 / 56. 58.100
 libavcodec     58. 99.100 / 58. 99.100
 libavformat    58. 49.100 / 58. 49.100
 libavdevice    58. 11.101 / 58. 11.101
 libavfilter     7. 87.100 /  7. 87.100
 libswscale      5.  8.100 /  5.  8.100
 libswresample   3.  8.100 /  3.  8.100
 libpostproc    55.  8.100 / 55.  8.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'trim.wav':
 Metadata:
 encoder         : Lavf58.49.100
 Duration: 00:00:09.00, bitrate: 1411 kb/s
 Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Stream mapping:
Stream #0:0 (pcm_s16le) -> asplit
 asetpts -> Stream #0:0 (pcm_s16le)
Output #0, wav, to 'processed.wav':
 Metadata:
  ISFT            : Lavf58.49.100
  Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
  Metadata:
   encoder         : Lavc58.99.100 pcm_s16le
     [afftdn @ 00000261c3b81400] bn=4 -6 -10 -12 -12 -14 -17 -23 -24 -24 -24 -24 -24 -24 -15
     [afftdn @ 00000261c3b81400] bn=5 -4 -8 -10 -11 -14 -18 -23 -24 -24 -24 -24 -24 -24 -15
  size=    1545kB time=00:00:08.97 bitrate=1411.3kbits/s speed= 150x
video:0kB audio:1545kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.004929%

FOURTH EDIT

Based on comments, I tried adjusting the noise floor and also using just a single audio track. I ran these commands:

c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=90[aud0d]" -map "[aud0d]" output-nr90.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=1[aud0d]" -map "[aud0d]" output-nr1.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=90:nf=-20[aud0d]" -map "[aud0d]" output-nr90-nf20.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=1:nf=-20[aud0d]" -map "[aud0d]" output-nr1-nf20.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=90:nf=-80[aud0d]" -map "[aud0d]" output-nr90-nf80.wav
c:ffmpeg -y -i trim.wav -filter_complex "[0:a]asendcmd=c='0.0 afftdn@aa sn start; 1.0 afftdn@aa sn stop',afftdn@aa=nr=1:nf=-80[aud0d]" -map "[aud0d]" output-nr1-nf80.wav

and for the most part I don't hear any noticeable difference. However, with the filter 'afftdn@aa=nr=90:nf=-20' I hear what sounds like a 1-second reprieve from the noise at the very beginning of the audio, as if that noise was being filtered out while leaving all of the noise in the rest of the track. Am I just using the wrong filter? Is my noise not the type of noise that afftdn is effective on? Or am I somehow misusing the filter so it's only filtering the first second? For reference, here is the input track.

  • You would use the 2nd method in my answer. Instead of a separate profile input, you would split each audio stream into two. From one copy, use atrim to isolate a portion useful for profiling, then reset its timestamps using asetpts. Then concat it with the 2nd full copy. Execute commands using asendcmd as shown in the answer, then afftdn. Then use atrim to discard that segment and restore timestamps. – Gyan Dec 1 '20 at 13:03
  • Something doesn't seem quite right. If I change the final afftdn filter to afftdn@n=om=n (to just hear the noise) it works exactly as expected (gives me just the noise) but I'm not noticing a big difference in audio output even when I use afftdn@n=nr=30 – Betty Crokker Dec 1 '20 at 20:10
  • No difference in output even with afftdn@n=nr=80 – Betty Crokker Dec 1 '20 at 20:25
  • Try with a minimal command (say, 2 audio streams) and check. Add full command and full logs if it doesn't work. – Gyan Dec 2 '20 at 5:14
  • @Gyan Thanks for your help with this! I've edited my original question, and uploaded some smaller WAV files to my Google Drive so you have everything that I'm working with. – Betty Crokker Dec 3 '20 at 1:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.