I think one of the biggest problems is low quality source files.
This is a more likely crux of the issue. If the source file is crummy, say a 128k mp3 file, then yes, the sound on aac, mp3 or WMA will sound crummy on decent speakers, though may still sound fine on the standard computer speakers.
I'll be honest, I have some mp3 files (though at 192K or better) that don't sound anywhere as good as the more recent WAV files (last year or two for me-as I upgraded my drives for the capacity). M4A files sound better than mp3 at any bit rate, or even WMA. My opinion, and your mileage may vary. I even encode with some music to an FLAC file, which sounds almost as good as a WAV file. Source does matter is what I am saying.
When I first started my station, I ran only a 24K aac+ stream. It sounded good, but as mentioned, some people look at 128k mp3 to be the standard (or above). A 64K aac+ stream sounds good, and at 96K, is every bit as good as a 128K mp3 stream. I still think that aac+ sounds a bit bright overall, and I know the codecs, so realize the issue.
If you use decent headphones, you will hear the difference. Most Internet radio users just use crummy speakers at the office to listen, so at this time, may not be an issue for anyone but the audiophiles like you and me (and any others). This will change over time, and Internet radio operators will need to make the necessary adjustments.
I like aac+, but will not encode to it, until/unless it becomes universal, which it isn't now.