You can have multiple servers to send mail. You can only have one solution that receives mail per domain level. You need to take care to setup your domain's SPF and DKIM records, however you can have one service used by IPS to send outbound mail and an entirely different service to receive mail.
Your MX records (mail exchange) will point to the configuration used by your inbound service. This will be supplied by your web host, a mail provider (Google Workspace, Amazon Workmail, Office 365, etc), and your IPS instance will point to whatever provider you want to be using for outbound mail generated by the community software. example.com can have one set of MX records, while forum.example.com can have another set, but you likely do not have a need for more MX records than whatever you'd like to use to receive e-mails to support@example.com, etc.
You will configure IPS to use the mail provider for your bulk mail activities, and separately setup e-mail for inbound mail.
The only slight complication is if you use the Commerce Support tool to process inbound e-mail. If so, you will need to provide for a way for IPS to receive that inbound mail. You can setup IPS to check via POP3/IMAP in that instance, or use forwarding to route to another domain/subdomain for processing.
I would direct your hosting provider to that folder and see that the memory_limit is set to -1. That often does not translate well with PHP and can actually place the memory limit lower than is expected here. I would suggest ensuring that any modifications made to your server also impact PHP in your "RDBforum" folder.
