There isnt much I can really add to what has been said by Randy there, other than to say when loading a page, if there is more media, it will take longer to load that page with that media on it. In most cases, on a decent set up, that would not be an unusual amount.
If you are asking these kinds of questions, are you sure you want to host yourself? Have you considered our cloud offering?