Resending email via Email Error Log through SendGrid fails

[CoffeeCake]

Currently clicking "resend e-mail" 2,000 times after a short outage with our e-mail provider. Each click I think less positive thoughts about whoever is responsible for this design.

Please add an option for an administrator to requeue e-mails using a task. Something like "Reprocess E-mail Queue" and perhaps a setting for how many to attempt to send in each batch.

Edited March 3, 2021 by Paul E.
This title's for you, future time traveler.

[Matt]

[gamecore]

+1

[CoffeeCake]

[Chris Anderson]

 @Paul E.Have you considered utilizing a browser automation tool like imacros? 

[CoffeeCake]

Thanks for the tip, Chris. I've used that software in automation projects in the past where the source was out of our control.

For this task, I made various voodoo doll out of my mouse using popsicle sticks and cut out printed cartoon heads.

Edited March 1, 2021 by Paul E.

[Chris Anderson]

 

8 minutes ago, Paul E. said:

For this task, I made various voodoo doll out of my mouse using popsicle sticks and cut out printed cartoon heads.

I will leave you with this:

My condolences with your task at hand.   Hopefully no one else has to go through your pain.

[CoffeeCake]

And then, the OVERWHELMING majority of them failed to send, as it sent the e-mail addresses in the format:

["email@example.com"]

What in the actual hell?

[Jordan Miller]

18 hours ago, Paul E. said:

Currently clicking "resend e-mail" 2,000 times after a short outage with our e-mail provider. Each click I think less positive thoughts about whoever is responsible for this design.

Please add an option for an administrator to requeue e-mails using a task. Something like "Reprocess E-mail Queue" and perhaps a setting for how many to attempt to send in each batch.

I'm a bit confused. You began to send a bulk email, but your email provider had an outage, so you needed to resend. But had to do so one at a time versus resending the entire thing again (as to avoid double emailing someone?) 

[CoffeeCake]

No. Our email provider is Sendgrid. We hit our maximum number of e-mails for the month and Sendgrid stopped sending out e-mails. These backed up into a queue in the ACP. Not a bulk mail, but rather all the sorts of e-mails that go out (registration validation, notifications, digests, etc.). We do not send out bulk mail.

We reached out to Sendgrid to increase our limit, but that took a while.

Once our quota was increased, the only way to resend those e-mails was clicking on a "resend" button next to each one. Over and over again.

To add insult to injury, whatever resends the e-mails is broken so all that clicking was for naught. The e-mail address provided to the Sendgrid API was completely mangled, surrounded in brackets and quotes, instead of an actual e-mail address.

I opened a support ticket reporting the bug for the mangled data to the API, but that you'd need to click over and over again one at a time is absurd.

I'd consider looking into some sort of third party solution instead of ever doing that again (maybe something like this: https://sendgrid.com/docs/for-developers/sending-email/postfix/).

In the past, if we had an MTA issue before IPS, we'd simply run through our mailq.

[Jordan Miller]

Just now, Paul E. said:

No. Our email provider is Sendgrid. We hit our maximum number of e-mails for the month and Sendgrid stopped sending out e-mails. These backed up into a queue in the ACP. Not a bulk mail, but rather all the sorts of e-mails that go out (registration validation, notifications, digests, etc.). We do not send out bulk mail.

We reached out to Sendgrid to increase our limit, but that took a while.

Once our quota was increased, the only way to resend those e-mails was clicking on a "resend" button next to each one. Over and over again.

To add insult to injury, whatever resends the e-mails is broken so all that clicking was for naught. The e-mail address provided to the Sendgrid API was completely mangled, surrounded in brackets and quotes, instead of an actual e-mail address.

I opened a support ticket reporting the bug for the mangled data to the API, but that you'd need to click over and over again one at a time is absurd.

I'd consider looking into some sort of third party solution instead of ever doing that again (maybe something like this: https://sendgrid.com/docs/for-developers/sending-email/postfix/).

In the past, if we had an MTA issue before IPS, we'd simply run through our mailq.

Ohhhhh. I use SendGrid too. But if I have an overage they just charge me for it so the emails still go out. I haven't ever had to manually send one by one lol that's so tedious! I feel your pain. 

Do you have a plan that disallows that? 

[CoffeeCake]

9 minutes ago, Jordan Invision said:

Do you have a plan that disallows that? 

Our account was all sorts of messed up. We had to engage with support to resolve it. We couldn't even see or adjust the plan settings when logged into their system. I think it has to do with GCP integration, as we signed up through GCP.

[Jordan Miller]

On 3/1/2021 at 11:08 AM, Paul E. said:

Our account was all sorts of messed up. We had to engage with support to resolve it. We couldn't even see or adjust the plan settings when logged into their system. I think it has to do with GCP integration, as we signed up through GCP.

Oh how weird. I wonder if they can make some adjustments for you so moving forward this can't happen again. Thoughts? Manually sending emails one at a time seems no bueno 😅

[CoffeeCake]

We're stuck at IPS wanting to log into our ACP, and not quite on board with entertaining our elevated security practices for ACP access (we have a separate login before the ACP will even prompt for IPS login, and that separate login uses a timed two-factor code). We'll try and sort a way to work around that for IPS. Then we'll probably hear about the add-ons that all need to be disabled. We can reproduce it on our clean dev environment.

I get the support script and processes, yet absent tinkering around in our production environment, maybe we could skip to trying to replicate this issue by using a dev copy as follows:

1. Enabling Sendgrid, however provide invalid API key.
2. Generating emails from things like notifications, private messages, etc.
3. Confirming that the ACP notes failed e-mail delivery
4. Update Sendgrid API key by providing a valid key
5. Attempting to resend the e-mails and observe what happens (look for invalid e-mails in Sendgrid's activity reporting)

Maybe something as an advocate you can assist with?

[Charles]

For our support to help you even a little we will need to login. As @Lindy always says: if you take your car to get serviced, they are going to have to open the hood.

[CoffeeCake]

Understood, @Charles. I'll continue finding a way to get IPS onto our parking lot.

As far as car analogies go, I think this is more along the lines of telling you there's a manufacturing defect in all of the cars, and you can stare at the defect on the ones sitting neatly on your dealership and factory floor. You wanting to look under the hood of the one parked in my garage, and not wanting to push the garage door button to open up the building is a weird way to go about it.

I suppose there's a possible chance I've got a live snake wrapped all around the make-it-go-vroom-inator, but we are able to reproduce this behavior every time in a dev environment with nothing installed but a fresh out of the oven 4.5.4.2:

For now the part where we tell you that we encountered an issue with outbound e-mail + Sendgrid when there are API failures, and that attempts to resend things using the interface provided by IPS one e-mail at a time mangles the "to" address (and who knows what else) is an item we're checking off the list as "did what we could, barrier to get this recognized and fixed might be too high."

Future time travelers can find this when they are trying to figure out why a part of the engine fell out while driving down the highway and then it can be their problem.

[CoffeeCake]

Here's what it looks like on the Sendgrid end of things:

If it's not immediately clear what the problem is, SendGrid expects that you won't surround the address in brackets and quotation marks.

Edited March 3, 2021 by Paul E.

[Charles]

If you submit a ticket to support they can have a look at it. We use SendGrid on this site without issue though.

[CoffeeCake]

1 hour ago, Charles said:

If you submit a ticket to support they can have a look at it. We use SendGrid on this site without issue though.

SendGrid works wonderfully when there's not a problem. We've been using it before IPS and with vBulletin.

When there is a problem (in our case, it was an issue with used credits--but it can also happen if you invalidate an API key, or if SendGrid has an outage), and you attempt to manually resend e-mails after fixing that problem, IPS sends a mangled API request on the resend click that you do for each failed e-mail from the E-mail Error Log. In our case, I clicked that resend button thousands of times and then still had members complaining that they hadn't received an e-mail. We logged into SendGrid and saw that they were all received with an invalid e-mail address, where the address is surrounded in quotes and brackets. It looks like bad array conversion to JSON.

Tickets don't have any perceivable number any more that I see, but I've opened one. If one follows the steps to reproduce above, they will be able to duplicate the issue and we don't have to wait for us to get an edge account to make it into our firewall to look at our production environment. The screenshots above are from a fresh dev environment that I spun up just to test this, with no applications or plugins installed.

Edited March 3, 2021 by Paul E.

[Charles]

Great, thanks for opening a support request. We will have a look!