Guest posting is, unfortunately, always going to have a spam counterpart to it to a degree as you will have human and bot spammers. You could ensure that hCaptcha is enabled to it's hardest degree in your settings. However, that still might not be enough and will let in human spammers or sophisticated bots.
As @Miss_B stated, using something like Post Before Register would be advised if you want to add in additional validation steps for posters/users but want to have them start creating their post before registering (an illusion of guest posting).