The only downside I've ever run across is the lag when loading Wygwam on the publish pages, especially if there are more than one on the page. Wygwam is still using CK Editor 3 so it's possible that when they eventually update this to CK Editor 4 (like Wyvern has) the performance issues will be reduced or eliminated altogether. The upside for me, however, far outweighs the downside. I have FULL CONTROL over the toolset I can give clients and pretty much anywhere I would use a text area, I use a Wygwam field.
The UX benefits are important to me so when clients see a field where they can enter a paragraph they see the WYSWIG field. The tools they are given are applicable to the content they are entering. If it makes sense to allow them to add bulleted lists they get the bulleted list menu. If I want to only allow them to bold or italicize text, pretty much everything is removed from the toolbar.
The only buttons that are consistent across ALL Wygwam fields for me are the source code button, the show outline boxes, and some form of the format drop down menu. The format drop down ALWAYS contains the normal style so if a paragraph somehow is removed they can easily fix it without having to deal with the source code. The show outlines is only there so they can turn off the default state of the outlines being on and the source code button is there for emergencies.
A lot of this applies to using Wyvern but I prefer the control panel Pixel and Tonic has provided for working with CK Editor and the advanced features much better then Wyvern. While I like the effort put into Redactor, I am not all that impressed with the results in use compared to Wygwam or Wyvern so I avoid Redactee and Editor. I stopped using Tiny MCE after figuring out what I can do with the advanced config settings in Wygwam.