If someone attaches files to a description of a file (some users do this by mistake), then those attachments are downloadable by guests. There seems to be no way to prevent this.

Check with @Nathan Explosion.  He has something to hide links and attachments from guests.  

Don't check with me as I don't use Downloads, and therefore don't develop for it.