1) Each software has its own advantages and disadvantages, there isn't a right answer. If you have ShadowPlay support, the hardware leverage is nice. FRAPS is good at dumping a really high quality video quickly, but it needs lots of memory and high speed disks to deal with the amount of data it dumps. I've not personally worked with DxTory, so I can't speak to that, though I believe it offers a slightly more balanced level of data rate (can use slower hdd) but at a cost of CPU performance. I could be wrong on that last part though.
2) There are two things you can do to help reduce the size. You can edit the video in a video editor such as Premiere or Windows Movie Maker in order to remove portions of the video that you do not need. This trims file size by reducing the amount of time the video covers.
You can also use what is known as a video encoder, such as Quicktime, Adobe Media Encoder or ffMPEG. A video encoder stores the video in a more compressed way that allows for smaller files size while minimizing the quality loss. Using a format like 2-pass VBR h.264 will produce very high quality files with a minimal size, however it is time consuming and CPU intensive to produce such files, so the live encoding tools can not use that level of compression. Running the slower encoding will produce smaller files as it can take the time to find more ways to save space.
The Youtube recommendations that you talk about are the preferred encoder settings that Youtube wants a source file to have, however Youtube itself will still transcode (make additional encodings) the file for streaming at different resolutions and speeds.
3) Your preference. Video standards tend to be either 30fps (NTSC TV (America/Japan)), 25fps (PAL TV (most of the rest of the world)) or 24fps (Cinema, used frequently by everyone). Choose based on what your system can handle and what you like the most. If you can't manage 24fps, then an even multiple tends to work best since frames will match up at the higher rate. (For example, 15fps on a 30fps video will result in each original frame being used twice, so the computer doesn't have to guess at any frames [called interpolation].)
