Jump to content

Community

Resending email via Email Error Log through SendGrid fails


CoffeeCake
 Share

Recommended Posts

Currently clicking "resend e-mail" 2,000 times after a short outage with our e-mail provider. Each click I think less positive thoughts about whoever is responsible for this design.

Please add an option for an administrator to requeue e-mails using a task. Something like "Reprocess E-mail Queue" and perhaps a setting for how many to attempt to send in each batch.

Diana Prince Dog GIF

Edited by Paul E.
This title's for you, future time traveler.
Link to comment
Share on other sites

Posted (edited)

Thanks for the tip, Chris. I've used that software in automation projects in the past where the source was out of our control.

For this task, I made various voodoo doll out of my mouse using popsicle sticks and cut out printed cartoon heads.

Edited by Paul E.
Link to comment
Share on other sites

 

8 minutes ago, Paul E. said:

For this task, I made various voodoo doll out of my mouse using popsicle sticks and cut out printed cartoon heads.

I will leave you with this:

go blazing saddles GIF

My condolences with your task at hand.   Hopefully no one else has to go through your pain.

Link to comment
Share on other sites

18 hours ago, Paul E. said:

Currently clicking "resend e-mail" 2,000 times after a short outage with our e-mail provider. Each click I think less positive thoughts about whoever is responsible for this design.

Please add an option for an administrator to requeue e-mails using a task. Something like "Reprocess E-mail Queue" and perhaps a setting for how many to attempt to send in each batch.

Diana Prince Dog GIF

giphy.gif


I'm a bit confused. You began to send a bulk email, but your email provider had an outage, so you needed to resend. But had to do so one at a time versus resending the entire thing again (as to avoid double emailing someone?) 

Link to comment
Share on other sites

No. Our email provider is Sendgrid. We hit our maximum number of e-mails for the month and Sendgrid stopped sending out e-mails. These backed up into a queue in the ACP. Not a bulk mail, but rather all the sorts of e-mails that go out (registration validation, notifications, digests, etc.). We do not send out bulk mail.

We reached out to Sendgrid to increase our limit, but that took a while.

Once our quota was increased, the only way to resend those e-mails was clicking on a "resend" button next to each one. Over and over again.

To add insult to injury, whatever resends the e-mails is broken so all that clicking was for naught. The e-mail address provided to the Sendgrid API was completely mangled, surrounded in brackets and quotes, instead of an actual e-mail address.

I opened a support ticket reporting the bug for the mangled data to the API, but that you'd need to click over and over again one at a time is absurd.

I'd consider looking into some sort of third party solution instead of ever doing that again (maybe something like this: https://sendgrid.com/docs/for-developers/sending-email/postfix/).

In the past, if we had an MTA issue before IPS, we'd simply run through our mailq.

Link to comment
Share on other sites

Just now, Paul E. said:

No. Our email provider is Sendgrid. We hit our maximum number of e-mails for the month and Sendgrid stopped sending out e-mails. These backed up into a queue in the ACP. Not a bulk mail, but rather all the sorts of e-mails that go out (registration validation, notifications, digests, etc.). We do not send out bulk mail.

We reached out to Sendgrid to increase our limit, but that took a while.

Once our quota was increased, the only way to resend those e-mails was clicking on a "resend" button next to each one. Over and over again.

To add insult to injury, whatever resends the e-mails is broken so all that clicking was for naught. The e-mail address provided to the Sendgrid API was completely mangled, surrounded in brackets and quotes, instead of an actual e-mail address.

I opened a support ticket reporting the bug for the mangled data to the API, but that you'd need to click over and over again one at a time is absurd.

I'd consider looking into some sort of third party solution instead of ever doing that again (maybe something like this: https://sendgrid.com/docs/for-developers/sending-email/postfix/).

In the past, if we had an MTA issue before IPS, we'd simply run through our mailq.

Ohhhhh. I use SendGrid too. But if I have an overage they just charge me for it so the emails still go out. I haven't ever had to manually send one by one lol that's so tedious! I feel your pain. 

Do you have a plan that disallows that? 

Link to comment
Share on other sites

9 minutes ago, Jordan Invision said:

Do you have a plan that disallows that? 

Our account was all sorts of messed up. We had to engage with support to resolve it. We couldn't even see or adjust the plan settings when logged into their system. I think it has to do with GCP integration, as we signed up through GCP.

Link to comment
Share on other sites

On 3/1/2021 at 11:08 AM, Paul E. said:

Our account was all sorts of messed up. We had to engage with support to resolve it. We couldn't even see or adjust the plan settings when logged into their system. I think it has to do with GCP integration, as we signed up through GCP.

Oh how weird. I wonder if they can make some adjustments for you so moving forward this can't happen again. Thoughts? Manually sending emails one at a time seems no bueno 😅

Link to comment
Share on other sites

We're stuck at IPS wanting to log into our ACP, and not quite on board with entertaining our elevated security practices for ACP access (we have a separate login before the ACP will even prompt for IPS login, and that separate login uses a timed two-factor code). We'll try and sort a way to work around that for IPS. Then we'll probably hear about the add-ons that all need to be disabled. We can reproduce it on our clean dev environment.

I get the support script and processes, yet absent tinkering around in our production environment, maybe we could skip to trying to replicate this issue by using a dev copy as follows:

  1. Enabling Sendgrid, however provide invalid API key.
  2. Generating emails from things like notifications, private messages, etc.
  3. Confirming that the ACP notes failed e-mail delivery
  4. Update Sendgrid API key by providing a valid key
  5. Attempting to resend the e-mails and observe what happens (look for invalid e-mails in Sendgrid's activity reporting)

Maybe something as an advocate you can assist with?

Link to comment
Share on other sites

Understood, @Charles. I'll continue finding a way to get IPS onto our parking lot.

As far as car analogies go, I think this is more along the lines of telling you there's a manufacturing defect in all of the cars, and you can stare at the defect on the ones sitting neatly on your dealership and factory floor. You wanting to look under the hood of the one parked in my garage, and not wanting to push the garage door button to open up the building is a weird way to go about it.

I suppose there's a possible chance I've got a live snake wrapped all around the make-it-go-vroom-inator, but we are able to reproduce this behavior every time in a dev environment with nothing installed but a fresh out of the oven 4.5.4.2:

screenshot.thumb.jpg.37f04ca89d34bf8c13d68a46ec64ecae.jpg

For now the part where we tell you that we encountered an issue with outbound e-mail + Sendgrid when there are API failures, and that attempts to resend things using the interface provided by IPS one e-mail at a time mangles the "to" address (and who knows what else) is an item we're checking off the list as "did what we could, barrier to get this recognized and fixed might be too high."

Future time travelers can find this when they are trying to figure out why a part of the engine fell out while driving down the highway and then it can be their problem.

Link to comment
Share on other sites

Posted (edited)

Here's what it looks like on the Sendgrid end of things:

sendgrid.thumb.jpg.6c50209b1faac6ea0b90c3f38e2aa924.jpg

If it's not immediately clear what the problem is, SendGrid expects that you won't surround the address in brackets and quotation marks.

Edited by Paul E.
Link to comment
Share on other sites

Posted (edited)
1 hour ago, Charles said:

If you submit a ticket to support they can have a look at it. We use SendGrid on this site without issue though.

SendGrid works wonderfully when there's not a problem. We've been using it before IPS and with vBulletin.

When there is a problem (in our case, it was an issue with used credits--but it can also happen if you invalidate an API key, or if SendGrid has an outage), and you attempt to manually resend e-mails after fixing that problem, IPS sends a mangled API request on the resend click that you do for each failed e-mail from the E-mail Error Log. In our case, I clicked that resend button thousands of times and then still had members complaining that they hadn't received an e-mail. We logged into SendGrid and saw that they were all received with an invalid e-mail address, where the address is surrounded in quotes and brackets. It looks like bad array conversion to JSON.

Tickets don't have any perceivable number any more that I see, but I've opened one. If one follows the steps to reproduce above, they will be able to duplicate the issue and we don't have to wait for us to get an edge account to make it into our firewall to look at our production environment. The screenshots above are from a fresh dev environment that I spun up just to test this, with no applications or plugins installed.

Edited by Paul E.
Link to comment
Share on other sites

  • CoffeeCake changed the title to Resending email via Email Error Log through SendGrid fails
 Share

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

We use technologies, such as cookies, to customise content and advertising, to provide social media features and to analyse traffic to the site. We also share information about your use of our site with our trusted social media, advertising and analytics partners. See more about cookies and our Privacy Policy