As Matt mentioned, topics aren't going to be blazing fast, just due to the content that gets placed on them by users who are not really concerned with website optimization speed as much as getting their content posted (which is how it should be). Then there are things like your ads in there, reCAPTCHAs (through Post Before Registration), Google Analytics, Google Tag, custom blocks, etc... that you have added on top of that. Every integration you add has a consequence these days of adding more JavaScript.
There are some core elements to our Javascript and CSS it references that we are taking a look at for V5 but the large items there are integrations. If you're looking for a way to improve those, you would need to contact the party who made them.