1

How do I select the audio corresponding to a set of filter-selected videoframes?

E.g. at the moment I am sampling 1 in every 100 frames of a video (running at 15fps) using the FFMpeg select filter. When I try to select the audio of those frames with the aselect filter, the total audio duration is not equal to the video duration (2 or 3 seconds difference depending on which frames I selected). The audio sampling rate is listed as 22050Hz.

I tried using various -vsync and -async combinations but it made no difference. Is there a more accurate way of using aselect?

A quick example command:

ffmpeg -i %INPUT% -vf select='between(n\,200\,399)',setpts='PTS-STARTPTS' -af aselect='between(n\,200\,399)',asetpts='PTS-STARTPTS'-y test.mkv

where 'INPUT' is the name of some file containing compressed video and audio. I am testing with the file AV36_1.avi found on this site hosting video samples. I get a video duration of 13.3 seconds and audio duration of 12 seconds.

output:

ffmpeg version N-77045-ga16243a Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --
enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-l
ibilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enab
le-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --en
able-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --ena
ble-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc
 --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enabl
e-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --
enable-lzma --enable-decklink --enable-zlib
  libavutil      55.  9.100 / 55.  9.100
  libavcodec     57. 16.101 / 57. 16.101
  libavformat    57. 19.100 / 57. 19.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 20.100 /  6. 20.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Guessed Channel Layout for  Input Stream #0.1 : stereo
Input #0, avi, from 'AV36_1.avi':
  Duration: 00:00:32.93, start: 0.000000, bitrate: 2372 kb/s
    Stream #0:0: Video: indeo5 (IV50 / 0x30355649), yuv410p, 320x240, 2058 kb/s,
 15 fps, 15 tbr, 15 tbn, 15 tbc
    Metadata:
      title           : Steyr.avi ┬ΦΣσε #1
    Stream #0:1: Audio: adpcm_ms ([2][0][0][0] / 0x0002), 22050 Hz, 2 channels,
s16, 176 kb/s
    Metadata:
      title           : Sound Forge 4.0 Audio
Output #0, avi, to 'test.avi':
  Metadata:
    ISFT            : Lavf57.19.100
    Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 320x240, q=2-31, 200
 kb/s, 15 fps, 15 tbn, 15 tbc
    Metadata:
      title           : Steyr.avi ┬ΦΣσε #1
      encoder         : Lavc57.16.101 mpeg4
    Stream #0:1: Audio: mp3 (libmp3lame) (U[0][0][0] / 0x0055), 22050 Hz, stereo
, s16p
    Metadata:
      title           : Sound Forge 4.0 Audio
      encoder         : Lavc57.16.101 libmp3lame
Stream mapping:
  Stream #0:0 -> #0:0 (indeo5 (native) -> mpeg4 (native))
  Stream #0:1 -> #0:1 (adpcm_ms (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
frame=   98 fps=0.0 q=9.6 size=     294kB time=00:00:09.03 bitrate= 266.1kbits/s
frame=  200 fps=0.0 q=11.7 Lsize=     525kB time=00:00:13.33 bitrate= 322.5kbits
/s
video:429kB audio:72kB subtitle:0kB other streams:0kB global headers:0kB muxing
overhead: 4.625268%

I think I'm using the aselect filter wrong...

  • 1
    Paste your command line used and console output. – Gyan Dec 16 '15 at 4:43
  • @Mulvya done :) – jiggunjer Dec 16 '15 at 5:29
  • Please reformat the console properly, like first box here. – Gyan Dec 16 '15 at 5:58
  • Your command is selecting all frames from #200 to #399, not 1 of 100, and your sample file is indicated as "Very strange avi sample containing some stray LIST fixed by r20118" Maybe try with a sane sample. I'll test myself in the evening and get back – Gyan Dec 16 '15 at 6:03
  • The exact selection doesn't matter, the point is that the corresponding audio selection has a large duration discrepancy. This trivial example should involve less possible complications than a more involved selection of frames (like not(mod(n,100)) -vsync 0). It would also be harder to select the corresponding audio frames. – jiggunjer Dec 16 '15 at 6:36
2

This command, which used t reference for aselect, worked for me:

ffmpeg -i AV36_1.avi -vf "select=between(n\,200\,399),setpts=PTS-STARTPTS" -af "aselect=between(t\,(200/15)\,(399/15)),asetpts=PTS-STARTPTS" -y test.mkv

If you want to use n (frames), then you will need to know the frame size of the audio codec e.g. 1024 samples for AAC, variable for MP3..etc

  • Cool, but this conversion won't work if the video has variable framerate. The info about audio frame sample sizes is interesting, I thought the sampling rate (22050Hz in this case) was always constant over time. – jiggunjer Dec 16 '15 at 10:08
  • 1
    Sampling rate is constant, but compressed audio streams chunk n samples into frames which don't correspond to video frame boundaries. Your method won't work for VFR either, as the audio frame rate you're getting from ffprobe is CFR. A path to a general solution is to obtain the PTS of the first and last selected video frames and then supply the equivalent times to the aselect filter. – Gyan Dec 16 '15 at 10:14
  • Then it seems timestamps are the safest way. So would an mp3 frame with 100 samples last 10 times longer than a frame with 10 samples? If aselect can only select entire audio frames this could limit accuracy. But my gut is telling me that aselect selects samples instead of frames (when using timestamps at least). – jiggunjer Dec 16 '15 at 10:28
  • No. Depends on the sampling rate. If you have a 48000 Hz rate, then a typical 1152 samples MP3 frame is 0.04166 seconds of audio. If 24000 Hz, then 0.0833 s. – Gyan Dec 16 '15 at 10:38
  • but a typical MP3 frame does not exist, as they have a variable frame size...so some frames will have 1500 samples, others 900, etc. Ah but the timestamps might not be as linear as with video. – jiggunjer Dec 16 '15 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.