Sounds like my usage is just different to yours. I can’t remember why but I got accustomed to listening to audio at increased speed around a decade ago and slowly cranked it up to the point that now I can follow certain people’s conversations slightly higher than 2x. Only with voices and cadence I’m familiar with though. Any guests on a show can really throw me off.

The silence trimming aspect is a bit absurd honestly. It makes laughter sound almost all the same and robotic; you have to infer where comedic, dramatic, or thoughtful pauses in the speech are; and if there’s a more rapid fire back and forth in the conversation it can be tricky to follow. Although that last point doesn’t happen with podcasts where all the speakers record separately and it’s edited together to be coherent.

If you listen to a lot of shows, with hundreds of hours of episodes, it’s worth dialing up as much as you can stand. Then again, if I didn’t have two dozen podcasts with decades of backlog, I sure wouldn’t be listening at auctioneer pace.