Typically a single millisecond won’t matter, but the general wisdom is that performance matters… that if a website takes more than three seconds to load, there is a percentage that is lost for each additional second. Huge sites like Amazon, Facebook, Walmart can convert this even down to milliseconds in terms of how it impacts conversations.
https://www.kickfire.com/blog/hold-up-wait-a-millisecond-how-response-time-impacts-your-website?hs_amp=true
However most websites powered by IPB are not to be operating at that scale. In addition, most websites powered by IPB also are not primarily e-commerce applications. (Many may have products/subscriptions for purchase, but generally if e-commerce was the primary goal… a site would be using applications designed for that regard.
Performance does matter and is relevant to a point. However there reaches a point of diminishing returns. A human won’t distinguish the difference between 20ms and 30ms. They absolutely would tell the difference between 300ms and 3000ms.
In terms of SEO, again… important, but according to Google content is KING. A site with good content would do better than a crappy site with no content but that was 30ms faster.
Ultimately there reaches a point where you are not really “gaining” anything except bragging rights. And there’s nothing wrong with that. However in reality in the “real world” you don’t have to have the fastest speed on the block to do well in Google. (It can’t suck, but it does not have to be the absolute fastest either.)
What’s funny for me is that I’ve typically seen third party ad networks (especially Adsense) be among the worst offenders in terms of those core web vital stats people focus so heavily on. So while having good performance matters, it’s not the end all/be all. (Which I THINK is one of the ideas you’re saying!) 🙂