I don't think you're the only one. At least, I feel similar!
Mainly because I feel the bounce from Slipslide is also a slight too punishing as well-- no matter how much you hold, you'll always have that one momentum-weighted bounce that's just gonna make it harder to get onto smaller platforms (see: Egg Rock Act 2's disappearing platform room).
I know turning off the skates and just jumping normally works too, but... Doesn't feel as ideal to turn it off, but kind of feel like I have to because it's too much of a hassle even if I feel like I'm competent enough outside of that specific context-- I love doing Time Attack runs and being on the skates is, of course, the faster option... But knowing just the smallest of slip ups will cost me the run (Bottomless pits) influences things too.
If the player was punished for holding it too long, then sure! Leave it intact, but if you let go early enough, it'd be appreciated to implement the changes that Charybdizs mentioned, along with the removal of that one landing bounce if you use the ability right.
