Webhooks First Looks

[Ocean West]

Just playing with receiving a web hook to another software/service, it appears the data is sent as content-type: application/x-www-form-urlencoded   I think it would be helpful to have a way to choose how to send the data because the receiving side it's impossible to unencoded it as the JSON object is entirely escaped. 

For testing to see what comes back in the payload I am playing with https://pipedream.com/ pretty quick way to build a receiver to see what the forums is sending. 

[Ocean West]

I find that the content for the web hook are rather verbose. I would be nice if we could filter and shape the content of the output. 

I can foresee a wizard where we choose the "event"  and see a model of what is going to be sent and then pair down the objects / arrays being sent for particular web hook target url. 

One main concern is when I sent an example it included my IP address and email that typically is a GDPR issue that a third party receiver should NOT need this data. 

Edited January 26, 2022 by Ocean West

[Ocean West]

if Zapier / Integromat  is using this same web hooks technology then anyone governed by GDPR may wish to suspend this as ip / emails are being sent.

 

[Ocean West]

it seems web hooks runs as root / admin access should we have a way to run under a particular group? 

[Ocean West]

another observation it seems that each event sends a single json object that triggers the event. I would suspect in the future that may be a limiting factor for larger or very busy sites.

In other web hooks I've seen they would typically cache multiple events into one pay load by making it an array and once the system passes a threshold would send an array of one item or as many as a dozen items. 

 

[Ocean West]

playing with this web hook it contains invalid JSON

IPS\forums\Topic

Something with the tags and missing the firstPost object key

 

Update: it may be my receiver that is borking the code. - trying to convert from form-url-encoded to  clean json.

Edited January 26, 2022 by Ocean West

[Ocean West]

I can confirm this issue spent some time with postman and reqestbin and send several tests:

Ideally the method you send should just be application/json and not form-url-encoded if you must add a key to the that so that it can be properly parsed. But I would rather just receive JSON and not have to do extra decoding.

 

[Ocean West]

Would love to hear feedback 😉  @Daniel F

[Daniel F]

On 1/26/2022 at 5:01 AM, Ocean West said:

it seems web hooks runs as root / admin access should we have a way to run under a particular group? 

Could you clarify what you mean here?They don't run under a specific account, they're permission independent.

On 1/26/2022 at 4:38 PM, Ocean West said:

another observation it seems that each event sends a single json object that triggers the event. I would suspect in the future that may be a limiting factor for larger or very busy sites.

 

That's how Webhooks work:)

Quote

In other web hooks I've seen they would typically cache multiple events into one pay load by making it an array and once the system passes a threshold would send an array of one item or as many as a dozen items. 

TBH I've never seen this before, do you have any examples?

The whole point from Webhooks is to notify other services/apps/integration ASAP about one specific event.

[Ocean West]

I use Mandrill / And SparkPost and when they send web hook for read/clicks etc the send an array 

{
 "mandrill_events" :
  [ ]
}

This could be one object or many objects based on the cash they have in x minutes your receiver would iterate thru the array to handle each action. https://mailchimp.com/developer/transactional/docs/webhooks/

(yes it is easier to handle one at a time instead of bulk actions)

-

There are no permissions applied to the webhook so everything is visible including email / ip / personal information;  most of the data is entirely not necessary in the payload when sending off to a third party service. In addition to being a glaring breach of privacy policy and terms of services, GDPR. I my current use case I am sending it to myself but if ever wanted to employ some third party service this would be a non-starter.

-

While this is being actively developed sending proper payload is critical sending as JSON would be really helpful I can't move forward with my project because the payload I receive is escaped and become invalid JSON.

- 

It would be nice to see some more configurations to selected the web hook perhaps being able to have a web hook log to see the data being sent and if you can visualize the payload with a JSON editor that would allow you to disable keys arrays that don't need to be sent to a given web hook. 

[Ocean West]

The more I play with this I can see my hook receiver is going to be much more complex than it needs to be.

Each even that triggers needs to have more 'filters' in place:

forumsTopic_create for example blasts out anytime a post is made anywhere in any forum. I would have to receive all this noise (which could exceed a web hook limits) and then test to see if it happens to match something I want. Seems like an awful waste of bandwidth if I only wish to focus on one parent forum and its sub-forum.

I would envision to for each of these hooks some wizard of options to restrict the content (pre-filter) before it is considered a viable action to send off to the url.

Adds even more complexity if you send multiple events to same url as the payloads are vastly different and would first need to process logic based on the event being sent. I would almost have specific web hook for each event. 

 

Edited January 28, 2022 by Ocean West

[Ocean West]

*bump*

are we any further along to fix the payload to application/json

 

 

[Ocean West]

Ok asking again when will this be addressed? I cannot parse the JSON object, as it is being sent. 

[Matt]

Tagging @Daniel F who is the lead engineer on our APIs.

[Daniel F]

On 1/26/2022 at 2:55 AM, Ocean West said:

it appears the data is sent as content-type: application/x-www-form-urlencoded   I think it would be helpful to have a way to choose how to send the data because the receiving side it's impossible to unencoded it as the JSON object is entirely escaped. 

I have added this to our internal roadmap.

 

We're definitely not going to make the payload configurable, that's literally going to increase the support requests because the webhook doesn't work as expected because of missing data.

A platforms webhook payload is standardized, so that any other consumer knows exactly what he's going to get, so renaming e.g. the member_id key to "user_id" will break any consumer.

That said, I can see why you want to have such feature, but it's really better suited as a 3rd party app or plugin.

 

[Ocean West]

Really? you have no intention to make it pessimistic by nature and keep it wide open with the current entire payload?

There is PII that is provisionally not allowed to be sent to any third party and which they have no business receiving?

So once you fix the json encoding I will have to send it to myself weed out the data elements I do not wish to send if I ever wanted to send data to a third party for processing. I will always have to be the man in the middle as I could never just send a web hook to a third party due to the content and privacy policies in place.

I know we're only talking about, a forum software but as programmers it behooves us to be accountable and self regulate otherwise the powers to be will impose regulation on the industry. - Watch any of Uncle Bobs videos to understand my points.

 

[Adam Goodrich]

When will the json encoding be fixed? Makes consuming this way more difficult than it should be.

[Daniel F]

We’ve listed to the feedback:)

In an upcoming release( not 4.7.0) the content type header will be configurable. 

[Ocean West]

Round of applause for adding this to the web hooks. 👏

But let's now talk about a way to configure what is in the payload.

It seems that the new DataLayer feature can toggle elements off we need this sort of thing in the web-hooks so as not not to send PII data to a third party. 

[Daniel F]

12 minutes ago, Ocean West said:

But let's now talk about a way to configure what is in the payload.

No, that's not going to happen!

A webhooks payload is standardized and it's going to introduce all kind of weirdness once external services receive manipulated payloads with customized or removed data.

I have raised an internal suggestion to encrypt PII data within the payload, so we may bring this with an upcoming release, but I can't share any further information about this yet.

[Adam Goodrich]

What / when can we expect to see commerce data in the web hooks? 

Also, I would like to make Invision a core component in something bigger by integrating it into our solution.

Would it be possible to authenticate users via a web service call ?

Additionally, I would also like to do commerce via web services... although perhaps this could be initiated by doing a popup, and then monitoring what happens by getting call backs via web hooks.