There's a lot of controversy with Smash in general because of the "lol it's a party game" mentality, 'cause like.. as true as it may be that it was intended that way, it grew out of that because of the players.
Yes, the casual audience is the biggest factor, but giving a competitive side or some more advanced features would not hurt the game - casual players wouldn't really discover them in the first place.
I loved Brawl, but once I dipped back into Melee, I practically had to step into the territory of Project M, which takes the cake as my favourite Smash game, even if it isn't official or anything. It's Brawl, except with Melee's weight and mechanics and speed. A few things in it I disagree with (such as unnecessary punishment for accidental air dodging/facing away from the edge/ledge hogging; all of those don't really contribute to the game in a meaningful way save for PURE competitive perfectionism which should not exist) but at the end of the day, having a high ceiling for player growth only extends the lifespan of a game.
There's a reason Melee survived so damn long, and the fact that it was possible to do all this crazy stuff in it did not change its viability to be enjoyed as a party game. The original Brawl, however, can only really be enjoyed as a party game because of how awkward, defensive and clunky it is, and this is coming from someone who played Brawl for a LONG time, every day, for a solid year.
Ultimate is a very good step in the right direction, regardless! It's one that I can go to and play for the fun of it and have a decent time. It's good in a lot of its own ways.
Finally, my main is probably Samus; was the case in Melee, Project M and the likes. Trying to get her to work in Ultimate is harder, but I still like to try.