SMTP is slow, we see it time and time again.
You could then use the SES app on the Marketplace to send your email via HTTP requests instead, which is much much faster.
You could, at your own risk - create a backup etc... remove the queue items with the following query (you may need to add your table prefix if you use one)
DELETE FROM `core_queue` WHERE `app`='core' AND `key`='Follow';
But please try and get upgraded as soon as you can, we no longer support the version you're using.