If the number of followers is below a certain number, we issue the follow notification email immediately in the post process. Otherwise, if it is a larger number, that a mail system wouldn't be able to send reasonably immediately, it is scheduled in the background. This is by design.
If your SMTP server delays sending, you may wish to use another service or avenue. This is abnormal activity really for an SMTP provider to just delay the injection of your request, during the request causing slowdown of the application/software you're using. Typically, there is a max number of emails able to be sent in a duration and you are stopped afterwards.