Source material

I have to hold 90 days of dashcam videos that average 2-5 hours a session. Each session is made up of 61 second videos. All told it is 2-2.5TB of storage. (5 hours * 60 minutes per hour * 90 days * 80MB). Diskspace is cheap but it takes time and money to put the sessions older than 30 days into an cloud based backup service like S3 as well as upload if needed for legal reasons.

hardware + setup

This is on a laptop with a 1.9Ghz i7 cpu, 16GB of ram, and /tmp /var/tmp are on SSD. A constant rate of 27 or 28 + vorbis rencode provides a major space savings from ~24GB to ~11GB with no apparent loss in quality. I already have a pipeline in place that copies out the telemetry data (g-forces, GPS, compass, and some other field), makes a manifest with relative path (ex the file per line text doc), and organizes the minutes into continuous driving sessions (eg is there more than one hour from one drive to next or does a session continue into the early next day).

Problem and investigation I've done

Right now the process takes +24 hours to finish for the worst case day of recordings. The console speed reports becomes progressively slower as it progress through each file.

As expected ffmpeg is mostly CPU bound, saturating the processor, but it only uses 4GB at maximum of resident memory.

Swap doesn't appear to increase and I could not find a temp file for the video but I do assume there is one (eg perhaps in /var/tmp).

Moving the audio stream to ogg doesn't have a significant time difference from just -acodec copy.

-sn has zero affect but I do want to keep them together with the concatenated file product

Guess on why and thoughts on a speedup

I don't know how far back ffmpeg goes in the video to find opportunities to optimize out parts of or whole frames that are mostly the same: stop lights or large blocks of black while driving at night. I am guessing it must go back a couple minutes or perhaps even the entire length of the concatenated video?

Previously I had tried the slow preset but artifacts & overall quality were greatly diminished.

  1. I have not tried it yet but one thought I had was to reencode the videos individually with the higher CRF (from I think 25 to 27) and then concat them
  2. Do the above but after concat'ing everything together, try a second pass for optimization.

Two ideals is a smaller concatenated video + the process to finish long before my next work shift.

CMD

REDACTED@LAPTOP-TVNAFRTD:/mnt/d/carvid/20180923$ ffmpeg -f concat -i manifest.txt -acodec libvorbis -scodec copy -crf 26 video2.mp4

Banner + configuration



ffmpeg version 3.2.12-1~deb9u1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --prefix=/usr --extra-version='1~deb9u1' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutl
--enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --
able-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwol
e --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec6188
--enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclc
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, concat, from 'manifest.txt':
  Duration: N/A, start: 0.000000, bitrate: 10806 kb/s
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], 10036 kb/s, 30 fps, 30 tbr, 240k tbn, 60 tbc
    Metadata:
      creation_time   : 2018-09-23T04:14:01.000000Z
      handler_name    :  Ambarella AVC
      encoder         : Ambarella AVC encoder
    Stream #0:1(eng): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, mono, s16, 768 kb/s
    Metadata:
      creation_time   : 2018-09-23T04:14:01.000000Z
      handler_name    :  Ambarella AAC
    Stream #0:2(eng): Subtitle: mov_text (text / 0x74786574), 2 kb/s
    Metadata:
      creation_time   : 2018-09-23T04:14:01.000000Z
      handler_name    :  Ambarella EXT
File 'video2.mp4' already exists. Overwrite ? [y/N] y
No pixel format specified, yuvj420p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x7fffe2947800] using SAR=1/1
[libx264 @ 0x7fffe2947800] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fffe2947800] profile High, level 4.0
[libx264 @ 0x7fffe2947800] 264 - core 148 r2748 97eaef2 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed
ef=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b
dapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=26.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'video2.mp4':
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0(eng): Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 30 fps, 15360 tbn, 30 tbc
    Metadata:
      creation_time   : 2018-09-23T04:14:01.000000Z
      handler_name    :  Ambarella AVC
      encoder         : Lavc57.64.101 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1(eng): Audio: vorbis (libvorbis) ([221][0][0][0] / 0x00DD), 48000 Hz, mono, fltp
    Metadata:
      creation_time   : 2018-09-23T04:14:01.000000Z
      handler_name    :  Ambarella AAC
      encoder         : Lavc57.64.101 libvorbis
    Stream #0:2(eng): Subtitle: mov_text ([8][0][0][0] / 0x0008), 2 kb/s
    Metadata:
      creation_time   : 2018-09-23T04:14:01.000000Z
      handler_name    :  Ambarella EXT
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (pcm_s16be (native) -> vorbis (libvorbis))
  Stream #0:2 -> #0:2 (copy)
Press [q] to stop, [?] for help

partial output while processing

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=1.54x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=1.26x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe29129e0] unsupported color_parameter_type aclcs speed=1.14x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe29129e0] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e78c0] unsupported color_parameter_type aclcs speed=1.04x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e78c0] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe374c080] unsupported color_parameter_type aclcs speed=0.991x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe374c080] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe291f9a0] unsupported color_parameter_type aclcs speed=0.967x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe291f9a0] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.949x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.922x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.919x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.911x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.904x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.899x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.894x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.895x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] unsupported color_parameter_type aclcs speed=0.897x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe2910780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8d00] unsupported color_parameter_type aclcs speed=0.892x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8d00] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8d00] unsupported color_parameter_type aclcs speed=0.887x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8d00] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe383b5a0] unsupported color_parameter_type aclcs speed=0.886x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe383b5a0] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe383b5a0] unsupported color_parameter_type aclcs speed=0.882x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe383b5a0] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe30e1780] unsupported color_parameter_type aclcs speed=0.882x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe30e1780] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8b20] unsupported color_parameter_type aclcs speed=0.882x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8b20] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8b20] unsupported color_parameter_type aclcs speed=0.876x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8b20] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8b20] unsupported color_parameter_type aclcs speed=0.878x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe36e8b20] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe37439e0] unsupported color_parameter_type aclcs speed=0.877x [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe37439e0] Auto-inserting h264_mp4toannexb bitstream filter [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fffe37439e0] unsupported color_parameter_type aclcs speed=0.875x

  • 1
    Right now the process takes +24 hours to finish for the worst case day of recordings. --> how long is the source? – Gyan Oct 7 at 5:56
  • @Gyan Last major concat was 5.62 hours or 367 individual video files. I did a test with just 8 files and found it slows down four videos into the process, going from 4 to 7 minutes per file. Working backwards to see if a read-ahead cache in the OS or on the SSD itself is being saturated/exhausted. – David Oct 8 at 0:54
  • FFmpeg does not create temp files for a command such as yours. Converting each segment individually and then copying during concat is a better method. – Gyan Oct 8 at 4:56

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.