6

We have developed an application to transcode source .mov files into .ogg, .mp4 and .webm output. It is currently running on AWS EC2 instance g2.8xlarge. It is working (yay!).

My question: Even though I am passing in -threads 0 to the ffmpeg command (actually setting ffmpeg.threads configuration in php-ffmpeg), the running process is sometimes only being executed on a single core. Why is this happening? See below output from htop command:

htop output

As you can see, Core #21 is maxed out. In a few seconds, it's going to switch to another one, rather than max out all of them like I would like and greatly speed up my encoding process. The situation is transient; during some runs all of the processors are maxed out, but during others they are not and we only get use of the one processor. A colleague mentioned that perhaps the codec we're using for some of the formats don't support multi-threaded execution during encoding, though I can't verify that is the behavior I'm observing yet.

Is this the case? If so, what codecs for the formats above will allow us to transcode into these target formats while taking advantage of all of our available hardware? The default codecs set for php-ffmpeg are below;

         Video        Audio
Ogg    libtheora    libvorbis
WebM   libvpx       libvorbis
X264   libx264      libfaac

Update

Looking at the running processes, below is what winds up being the ffmpeg command that is run for an MP4 (currently saturating all 32 cores):

ffmpeg -y -i my-video.mov -async 1 -metadata:s:v:0 start_time=0 -async 1 -metadata:s:v:0 -async 1 -metadata:s:v:0 start_time=0 -async 1 -metadata:s:v:0 start_time=0  -s 1920x1080 -s 1920x1080 -s 1280x720 -threads 0 -vcodec libx264 -acodec libfaac -b:v 1200k -refs 6 -coder 1 -sc_threshold 40 -flags +loop -me_range 16 -subq 7 -i_qfactor 0.71 -qcomp 0.6 -qdiff 4 -trellis 1 -b:a 128k -pass 2 -passlogfile /tmp-ffmpeg-passes55ad0d0233f711zdrg/pass-44ad0d02340a8 my-video-1200.mp4

I'm not actually building this command directly, php-ffmpeg is, though I do believe I have at least a modest amount of control over what goes in to it (for instance, I have no idea why there's multiple -metadata:s:v:0 entries at the beginning)

  • There's a lot of yuck-factor in that command line, apart from the duplicated options (-s three times, the final one with a different size). Explicitly setting a bunch of args to their current default values (e.g. -i_qfactor, -subq, -qcomp) is weird, and could give bad results with future libx264. (Probably not, but only because libx264 is pretty much done, and stable, not under heavy development. If it did stuff like this for x265, that would be bad.) Anyway, 2-pass 1200k is fine, but you might prefer target-quality crf. It doesn't specify a -preset. :( – Peter Cordes Jul 20 '15 at 15:43
  • libfaac isn't as good as libfdk_aac. If you're using this in a for-pay service, you'd need to check on the licensing of libfdk_aac, though. Also, this cmdline is missing -movflags +faststart – Peter Cordes Jul 20 '15 at 15:52
  • It's also possible to have ffmpeg produce multiple outputs from the same input. Just have multiple sequences of output-options output-filename on the commandline. So all in all, I'm not super impressed with php-ffmpeg, if that's the kind of cmdline it comes up with. Maybe you could use it differently to get it to generate multiple outputs at once, so there wouldn't be a single-threaded theora step. Anyway, if it works, then great, but beware of changes to encoder defaults, and the meaning of x264 subme levels changing, in ways that mean your cmdline hurts quality. – Peter Cordes Jul 20 '15 at 15:56
  • @Peter thanks so much. I think the answer here really is I need to debug how that cmd is being built. If I really can stuff multiple outputs into that command, I think that would probably give me the better shot of maximizing the load on the hardware – Jeff Lambert Jul 20 '15 at 16:00
  • trac.ffmpeg.org/wiki/Creating%20multiple%20outputs. And yeah, I agree that's probably best. Otherwise you have task that's single-threaded for some of its time, and loading all your cores for some other part of the time. Hard to schedule jobs that behave that way. – Peter Cordes Jul 20 '15 at 16:07
7

BTW, this question might be better on stackoverflow, or maybe unix.stackexchange, or maybe serverfault. This site is I think less focused on questions that don't involve decisions based on creative merit or at least perceptual video / audio quality. However, I'm all about the tech details, so I'll answer.

FFmpeg uses multi-threading by default, so you prob. don't need -threads 0. If your encode is bottlenecked on a single-threaded filter or decoder, you'll see full load on one core, and light load on many other cores.

One thing you can do is check mediainfo of your output video. x264 leaves its settings in an ASCII string in the h.264 header. So either strings -n20 or mediainfo to get:

...
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.051
Stream size                              : 455 MiB (89%)
Writing library                          : x264 core 146 r2538+1 d48ec67
Encoding settings                        : cabac=1 / ref=6 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=0.70:0.10 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=4 / lookahead_threads=1 / sliced_threads=0 / nr=50 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=5 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=22.5 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=3:0.60
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Note the "threads=4" in there. I think I manually set that on my quad core i5 2500k, instead of letting x264 use the default CPUs * 1.5, since I had a CPU-intensive filters (hqdn3d and lanczos-downscale) running.

Anyway, libx264 with a preset like slower should have no trouble keeping a lot of cores busy. There are some parts of encoding that are inherently serial (e.g. the CABAC encode of the final bitstream), so a high-bitrate video that doesn't spend a lot of CPU-time refining references (high subme) to multiple frames (high ref) might show a load pattern like yours (one thread using 100% CPU, others not).

I'm not 100% sure about faster presets being less parallel, but I know CABAC is serial.

To get massively parallel, libx264 could use a boatload of RAM to keep frames around, and keep doing lookahead for 2 or more GOPs, and encode those independently. It doesn't have an option to operate that way, though.

One way to make use of MANY cores is to run multiple separate encodes in parallel, instead of just a series of single encode using all cores. This only works if you have multiple input files that you want encoded separately. You're trading off threading overhead vs. more memory capacity and bandwidth (with an impact on caching, unless this is on a multi-socket system with separate L3 and DRAM for each cluster of CPUs, and you have the processes pinned to cores so one encode is using the cores in one socket, and the other the other).

  • Thanks for the insight. I asked here mainly because I don't have any questions at all regarding "code" and more about whats going on behind the scenes and thought this would be a better fit. I'll flag for moderator attention to migrate and then they can make the call. – Jeff Lambert Jul 20 '15 at 13:56
  • Right now we're sticking a message on an AWS SQS queue that has a link to each file. This instance has a job running that listens for those messages, downloads the file, transcodes it, and uploads the output files out once each completes. If I am reading this correctly, you're saying that it would probably make more sense for us to go ahead and launch a few of these worker processes and transcode multiple files in parallel rather than attempt to focus all cores on a single process? – Jeff Lambert Jul 20 '15 at 13:56
  • Yeah, if you're having problems saturating more cores, it's fine to run an encode or 3 in parallel. I think x264 should be able to saturate most of your 32 cores, but maybe only with a slower preset. Post your ffmpeg cmdline options, and console output, in your question. IDK if you're using something silly and low-quality like -preset veryfast. If so, then decode of the input might be the single-thread bottleneck. Or like I said, maybe a slow filter. – Peter Cordes Jul 20 '15 at 14:07
  • You're certainly going to want to overlap the download/upload of one xcode with the CPU usage of another xcode, if you're not planning to stream to/from ffmpeg on the fly for production use. (It may be possible to get the equivalent of -movflags +faststart on the fly, with a different muxer. I think I read something about that. Otherwise, if you're outputting mp4, you kinda need to output to a file so ffmpeg can put the moov atom at the front and shuffle the data over when the encode is done.) – Peter Cordes Jul 20 '15 at 14:10
  • Oh, I just read your Q in more detail. If you're outputting all 3 formats in one go (with the same ffmpeg command line, so the decode of the input only has to happen once), then if one of the 3 encoders is single-threaded, it will bottleneck the whole process. I think libtheora is not multi-threaded. wiki.xiph.org/TheoraEncoders says there was a multi-thread fork, but it died. (maybe never worked well, or wasn't compatible with other encoder improvements? Could be many reasons it didn't get merged.) lists.xiph.org/pipermail//theora-dev/2015-February/004374.html – Peter Cordes Jul 20 '15 at 14:21
4

libtheora is single threaded. There is a multithreaded experimental build, but is not maintained. I would suggest running it in parallel with the other encodes. Also if possible use libfdk-aac over libfaac. Much higher audio fidelity at the same bitrate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.