I believe that the "Top Downloads" plugin is an Invision first party plugin, correct?  I've noticed that any page that has the "Top Downloads" plugin/block causes the page to load very slow.  I have it limited to 5 files, but still loads very slow.  My users love the plugin...  Is there anything that could speed up the loading of this plugin?

Would suggest disabling all third party applications/plugins to see if anything is interacting with it. Then if you're still seeing a slow down, you will need to have your hosting provider enable slow query logging to see what is happening there.