I make short films which I distribute through YouTube. I know YouTube uses VP9 for streaming, but YouTube still recommends uploading with h.264. It makes more sense to me to upload in as similar a format as will be streamed by YouTube, but I might be missing something.
Also, would uploading in a higher quality codec like DNxHR or Prores (I'm not concerned about bandwidth or processing time) better preserve video quality after YouTube's encoding?
tl;dr: Since Youtube reencodes all videos regardless of the upload format, it really isn't that important. Just export your video with a high bitrate to preserve quality. Also see my answer here regarding quality loss caused by Youtube.
Long answer: Each reencoding of a video to a compressed format lowers the quality. Usually, that means you'll lose quality at two points: When you export the edited video from your editing software and when you upload the exported video to Youtube, at which point it is reencoded to a highly compressed, streaming-compatible format. You have no control over that second step, so what you can do to achieve the maximum quality possible is make sure you lose as little quality as possible during the first encoding.
In theory, that would mean export to a perceptually (even though technically not) uncompressed format like Apple ProRes or DNxHD as you suggested. However, unfortunately, Youtube doesn't support those formats, so you'll have to use a compressed format. To minimize quality loss, set a high bitrate in your export settings (assuming rendering and upload time aren't an issue. If they are, you'll have to find some middle ground; exporting with a bitrate that is higher than the bitrate of the source material won't yield any more quality, so that is as high as I would go). If you do that, it doesn't really matter which codec you use, both are highly efficient regarding file-size/quality ratio (VP9 probably a bit more so, but that's more important when you're dealing with low bitrates). Youtube recommends H264, so that's what I would use. However, the best advice I can give you is to try out both, i.e. export the same video as both H264 and VP9 with identical bitrates/other settings, upload both to youtube and check which one looks better to you.
YouTube will re-encode whatever you give it. VP9 is currently too slow to encode. So the best choice is to either:
Give it the original footage, or if it is not accepted by YouTube or is too big to upload, then...
Re-encode to H.264
ffmpegffmpeg will accept just about any input and will provide a great quality output. Development is very active, so it is best to use a recent build which you can download for Windows, OS X, or Linux.
ffmpeg -i input.avi -c:v libx264 -preset medium -crf 18 -c:a copy output.mkv
ffmpeg -i input.avi -c:v libx264 -preset medium -crf 18 -c:a libvorbis -q:a 5 output.mkv
-crf 18 is considered to be roughly visually lossless. The range is a log scale from 0-51. 0 is lossless (huge files), 23 is default.
-preset controls the encoding efficiency, and therefore affects the encoding speed. Generally it is recommended to use the slowest preset you have patience for, or if in doubt just use medium (the default). Choices are: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow.
Your player may not like the output, but YouTube will. If that is the case then just use a sane player such as VLC.
Most of these answers are outdated. Youtube actually re-encodes our videos into VP9 making HD videos look rather better in terms of quality. Its only when you upload the video in its original encoding that looks degraded based on compression. It can take a day or a week for youtube to re-encode our HD videos to Vp9. I'm a youtuber myself and can really see the difference
