For projects which are strictly digital, ie, using pure digital synthesis and not recording any material which would be converted from analog -> digital, is there any advantage to using sampling rates above 44.1kHz?
Yes. Some examples:
Creation of frequencies you don't want
Aliasing from digital synthesis
Many square/sawtooth/triangle wave generators are naively-written, in that they produce an infinite number of harmonics, which are aliased and sound clearly bad. (
..., +1, +1, +1, +1, −1, −1, −1, −1, ... is not a correct square wave, and the aliased harmonics will produce radio tuning sounds in the background during portamento.)
If the sampling frequency is higher, this effect is reduced, because the aliasing frequency is farther away from the audio band.
Of course it would be better if the generator were written in a way that completely eliminates aliasing, but you can't always control that as a user. Even well-written ones are usually compromises, with "reduced" aliasing, not completely bandlimited, so higher sampling rate still helps.
Aliasing from digital distortion
Likewise, when you use any kind of digital non-linear distortion, it produces an infinite number of harmonics or intermodulation products. The ones that would be produced above the Nyquist frequency are actually aliased back into the audible range.
Although it's theoretically possible to distort in a bandlimited way, it isn't common for plugin coders to actually do this. Every guitar distortion plugin I've tested has aliasing, even processing at 96 kHz.
I'm not sure how much of a problem this is practically. Lots of things cause small amounts of distortion, like a compressor or volume fade, but the amount is already negligible, so the aliased amount is even more negligible. For heavy distortion, the aliased frequencies may also be not noticeable because they're buried in the noise. Regardless, higher sampling rate will help to minimize any harmful effects.
Lack of frequencies you do want
Another possible concern is that synthesized ultrasonic frequencies might become useful later in processing, even though you can't hear them directly in the recording:
Frequency shift from time changes
If you resample a wave to slow it down, such as in a soundfont player, those ultrasonic frequencies will become audible frequencies. If you had filtered them out to avoid aliasing at the lower sampling rate, the slowed-down sound would be missing the high end.
As said before, distortion will create new intermodulation frequencies at sum and difference locations from the frequencies in the original recording. This time, we're concerned about desirable audible frequencies being produced by distortion/modulation of ultrasonic frequencies (not related to aliasing). If those ultrasonic frequencies aren't in the recording before distortion, the output will be missing the audible frequencies they produce, and it won't exactly emulate an equivalent analog effect.
Again, I'm not sure if this is practically a problem, but it's at least plausible, and higher sampling rates that include ultrasound will improve it.
In general, working at higher sampling rates gives "headroom" to prevent problems with effects and stuff that may not be implemented correctly. Like photocopying a photocopy, the better the quality of each copy, the less degradation there will be in the final product.
Useless for playback
This is not to say that higher sampling rates are a good idea for playback of the finished mix. They're not. As described above, distortion of ultrasound can produce audible sound, and loudspeakers are the least linear thing in the audio chain, so you want to eliminate any ultrasound from the final mix to prevent it from being distorted by the speaker.
There's no benefit to higher sampling rates for music playback; they should only be used in the recording and processing stages. See 24/192 Music Downloads ...and why they make no sense.