Can there be a reliable answer?
As with almost everything is video. The real answer is: It depends.
With mp4, the moov must be at the beginning, AND the mdat must properly interleave tracks. For example, If the there more than one megabyte of audio before the video, Then it won't work even with the moov at the start.
For TS, there must be a SPS, a PPS ...
No ffmpeg code can identify interesting frames. At most, there are filters which detect representative frames and filters which identify frames with a scene change. So, at best, there's a thumbnail filter for this, sort of. It can't detect "interesting" frames but it detects representative frames. So, frames from the middle of fades should be out.