Yes, you have the right idea. If stib's suggestion of making multiple outputs in parallel with the same ffmpeg commandline doesn't quite do the trick, then use a temp file to hold a lossless copy of the output of any slow filtering:
ffmpeg -i src.mkv -vf yadif=3:1,mcdeint=2:1 -c:a copy -c:v libx264 -preset ultrafast -qp 0 deinterlaced.mkv
ultrafast except with lossless, it's WAY worse than
superfast -x264-params cabac=0, and not much faster.) CABAC takes most of the CPU time at lossless bitrates, but it does save 15% bitrate vs. CAVLC. If you have a ton of scratch space to work with, you could use an uncompressed output file. Lossless x264 is about as fast to decode as
ffvhuff, if not faster, but half the file size. I think it's your best bet for scratch files (pretty good CPU usage, ASM-optimized decoders, and one of the best as far as compression ratio). Simpler codecs like ffvhuff might have a place if you need to frame-step backwards without taking seconds per frame, or you could just lower
keyint in x264.
Then do whatever you normally do, but with
deinterlaced.mkv as your source.
I think I remember reading about someone having added support to x264 for making multiple encodes at different bitrates at the same time. So motion-search, frame-type decision, and a bunch of other analysis only has to be done once. I'm not sure if that was just for google's internal use, and it never got merged, or what. Neither the vanilla x264 cli frontend, nor the ffmpeg libx264 wrapper, have options to activate a feature like this. I'm pretty sure I would have noticed that in
x264 --fullhelp output at some point.
If you have to do a lot of encoding, it'd be worth searching around to see if there's any way to get your hands on a publicly available implementation of this.
This is different from having ffmpeg feed the decoded/filtered video to multiple instances of libx264 in parallel, since each instance would still need to do all the analysis independently.
If you do use stib's suggestion of multiple outputs from the same ffmpeg, reduce x264's number of threads. By default, each separate output file will auto-detect the number of CPUs in the whole machine, so you'll have 6 times the number of worker threads needed to saturate your CPU cores. Maybe use 2 or 3 threads for the HD outputs, and single-threaded for the lower-rez outputs, since you want to balance things so all your available CPU time gets used, without having more threads than needed causing extra context-switching overhead.
For HD video with a lot of reference frames, each libx264 instance would take a good chunk of RAM, too. Reducing thread count doesn't help with this much, since all the threads are reading the same data.
6 instances of libx264 at the same time, instead of multiple threads from the same libx264, will be competing for memory bandwidth / cache space. They'll each have their own copy of the frames, instead of getting cache hits from each other. This might or might not be an issue on your hardware. If this or the RAM-size is an issue, use a temp file and do 3 outputs from one run, and 3 outputs from the next.
yadif (and mcdeint) are not multithreaded.
Or you could deinterlace and output to lossless + some of your desired output rezzes, then generate the rest of your rezzes in another pass of the lossless file. If you don't have multiple videos queued up to process, this would let you pipeline things to get max use out of all your cores while still not doing all the encodes at once.
If you do have multiple videos to process, you could have a video deinterlacing while you encode another, if you think setting that up is simpler than multiple outputs + lossless from the ffmpeg that runs the deinterlace. Run the encodes with
nice -19 or something, so they don't take CPU away from the
ffmpeg process doing the deinterlace.