This is incorrect or something is not quite right with your permissions you've set. If a member group does not have access to a forum, they cannot access attachments or the topics in that forum. (NOTE: embedded images will be available, however, as these are handled differently but sounds like this is not your case.)
If you've previously had the forum open to the public, bots may have stored those URLs and be attempting to access them. However, they will receive permission denied messages.
If you would like to submit a ticket, we're happy to give your permissions a once over for you.