All anyone might be able to say is that there are no known holes in the various players. (I don't know if that's true, just that it's impossible to know that there are no undiscovered bugs in a complex piece of code.) H.264 streams are complex enough to have lots of corner cases. They're parsed with speed-optimized code written in C and assembly.
You're quite correct that this is a HUGE attack surface, esp. when you allow arbitrary input formats, including ones with crufty old demuxers / decoders written by some random guy a long time ago, who might or might not have had security in mind. This article about ffmpeg security fixes is quite interesting.
re: ascii strings inside video files. That's not uncommon at all. x264 output embeds a string like
x264 - core 142 r2455 021c0dc - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=6 deblock=1:2:2 analyse=0x3:0x133 me=umh subme=10 psy=1 psy_rd=1.00:0.50 mixed_ref=1 me_range=24 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-4 threads=6 lookahead_threads=1 sliced_threads=0 nr=200 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=6 b_pyramid=2 b_adapt=2 b_bias=0 direct=3 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=60 rc=crf mbtree=1 crf=25.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:0.50
And of course strings
will find lots of sequences of characters in the ascii range, just by random chance. Some containers use sequences of ascii characters as the magic values to indicate what type of object something is. e.g. mp4 has "moov", "mdat", and other atoms. If you can pick any sequence of 4 bytes for your format, you might as well pick ASCII characters to make things easier to debug.
I think transcoding all incoming video would incur a HUGE quality / CPU / storage overhead. Transcoding with nightly builds of ffmpeg would potentially shield you from known bugs in the typically old version of ffmpeg in VLC stable releases, and whatever bugs are in closed source stuff from Microsoft and Apple.
Remuxing is not a bad compromise.
There's also a balance between destroying metadata vs. passing through more potentially harmful stuff. (e.g. chapters, subtitles, ...)
If I were actually doing this, I'd probably store my output in mkv files. Using a different container is maybe better insurance against any weirdness from an input mp4 getting passed on to an output mp4.
Also, if you're actually doing this, I guess you'd run ffmpeg (or MP4Box for just mp4->mp4 remuxing) in a sandbox chroot or vm, or even just as an un-privileged user, so successful exploits are contained.