Time out errors indexnow

[AlexJ]

Indexnow issue:

Seeing this errors for indexnow. Is anyone experiencing same OR do they publish their IP's? I don't have much outbound firewall on server but inbound is restricted. 

 

Same happens randomly for giphy:

IPS\Http\Request\CurlException: https://api.giphy.com/v1/gifs/search?
Connection timed out after 10001 milliseconds (28)

Edited January 13, 2023 by AlexJ

[Mark H]

I believe the above topic is relevant. It's an issue between your server and indexnow. This is something your Host would need to investigate as a first step.

[AlexJ]

@Mark H - Tks it helps a bit. Can you please share one thing:

Timing from IPS to IndexNow i.e. how often IPS calls the request and interval. 

----

For giphy, does the request original from webserver? I thought that's more of a client side? Can you please confirm. 

[Jim M]

49 minutes ago, AlexJ said:

Timing from IPS to IndexNow i.e. how often IPS calls the request and interval. 

This would be as frequent as content is posted.

49 minutes ago, AlexJ said:

For giphy, does the request original from webserver? I thought that's more of a client side? Can you please confirm. 

This is from the server, yes.

[AlexJ]

47 minutes ago, Jim M said:

This would be as frequent as content is posted.

1 hour ago, AlexJ said:

Ok so if it's continuous problem, i should see more. So I am guessing indexnow is rate limiting our site? I don't have popular site.. so i am curious... 

48 minutes ago, Jim M said:

This is from the server, yes.

Ok, tks. How the connection works? User -> Server -> Giphy API -> Cloudflare -> Server -> User see Giphy - Is this correct? Can you please share details. Thank you. 

[Randy Calvert]

1 hour ago, AlexJ said:

Ok, tks. How the connection works? User -> Server -> Giphy API -> Cloudflare -> Server -> User see Giphy - Is this correct? Can you please share details. Thank you. 

It’s a little more nuanced than that. A user clicks the giphy button. That triggers an api call from your server to giphy to get a list of results. Those results are retuned from your server to the user. If a user clicks an image it is directly downloaded from giphy to the user. 
 

1 hour ago, AlexJ said:

Ok so if it's continuous problem, i should see more. So I am guessing indexnow is rate limiting our site? I don't have popular site.. so i am curious... 

Not really. It’s more likely a network issue between your server and IndexNow.  It could be any one of a dozen things happening in the middle mile. Ultimately IPB only can say they submitted a request to IndexNow and did not get a response back in 10 seconds. It could have processed it in 11 seconds and was fine. 

Also you mentioned using Cloudflare. If you’re using their Early Hints, this is duplicated effort. You could just turn it off in IPB as it can be triggered from CF. 

[Jim M]

15 hours ago, AlexJ said:

Ok so if it's continuous problem, i should see more. So I am guessing indexnow is rate limiting our site? I don't have popular site.. so i am curious... 

If this is the first it's happened, it's more likely a network error. 

15 hours ago, AlexJ said:

Ok, tks. How the connection works? User -> Server -> Giphy API -> Cloudflare -> Server -> User see Giphy - Is this correct? Can you please share details. Thank you. 

As Randy mentioned, there's a bit more to that going on but in a nutshell that is how the data transformation works.

[AlexJ]

13 hours ago, Randy Calvert said:

Also you mentioned using Cloudflare. If you’re using their Early Hints, this is duplicated effort. You could just turn it off in IPB as it can be triggered from CF. 

Thanks for the tip! Appreciate it. 

13 hours ago, Randy Calvert said:

It’s a little more nuanced than that. A user clicks the giphy button. That triggers an api call from your server to giphy to get a list of results. Those results are retuned from your server to the user. If a user clicks an image it is directly downloaded from giphy to the user. 

Need to find a way to whitelist Giphy on Cloudflare. 

5 minutes ago, Jim M said:

If this is the first it's happened, it's more likely a network error. 

15 hours ago, AlexJ said:

I had more errors before but i have started whitelisting their ASN blocks. I am guessing some IP's might be under different ASN. Initially, I wasn't sure if it's Indexnow issue rate limiting websites or just firewall. Looks like later. 

Thank you both once again. 

[AlexJ]

Ok, I think need some additional help. 

Giphy ASN is - AS54113 when I trigger the API call from my browser. When I try to check through other VPN's. I get this IP. 77.111.246.126 which is Opera Mini Proxy. 

So how the connection from Giphy API -> Cloudflare works? I need to find a way to whitelist Giphy for my site, in cloudflare. ASN doesn't look like solution to me. 

Can I whitelist using path.. /v1/gifs/search  ? But that would fall under Server -> Cloudflare -> Gifhy connection, no? And we need solution from Giphy -> Cloudflare -> Server. 

All our firewall rules are in Cloudflare server. 

Edited January 14, 2023 by AlexJ

[Jim M]

Sorry, instructions and investigation into a service to whitelist would be outside our scope of support. 

[AlexJ]

Just now, Jim M said:

Sorry, instructions and investigation into a service to whitelist would be outside our scope of support. 

No. I am asking about how IPS software communicates with Giphy and how Giphy communicates back to IPS. 

If I am unclear about how IPS software works, I can't find a solution. 

 

[Jim M]

19 hours ago, AlexJ said:

IPS\Http\Request\CurlException: https://api.giphy.com/v1/gifs/search?
Connection timed out after 10001 milliseconds (28

This would be the URL you mentioned here. How your server processes that outbound request would be something only you know. 

[AlexJ]

1 minute ago, Jim M said:

This would be the URL you mentioned here. How your server processes that outbound request would be something only you know. 

Let's take specific example, so I can explain better. 

https://api.giphy.com/v1/gifs/search?q=Wake%2Bup&api_key=xxxx&limit=30&offset=30&lang=en&rating=pg-13

When I trigger that from browser, I get remote address of 199.232.38.2. 

When I trigger that from opera browser, with proxy, I get remote address of  77.111.246.126

Now both IP's are in different ASN and later is under Opera VPN. 

So question is - when IPS sends above API call to Giphy, forget which server for a second here.. -> What's expected response from Giphy to IPS?  

OR May be above is more technical question to Giphy team rather then IPS team? Since question revolves around Giphy response. I am trying to figure out what's expectation on IPS side, so I can whitelist it and call it a day. 

[Jim M]

Sorry, We can only tell you which URL we reach out to for giphys API. What result that is would be up to your network and Giphy. 

[Randy Calvert]

Giphy will return a JSON string with a HTTP status code of 200.  It does not have one fixed IP....  it has different IPs based on where you are around the world.  You can't just whitelist a single IP and it solve your problem.  IF you do have outbound HTTP restrictions, you would need to allow all outbound requests for the hostname....  not the IP address.  

Just FYI...  I can almost guarantee that CF is not your problem.  CF is involved on the INBOUND request (because its a reverse proxy).  Meaning someone types in www.yourdomain.com and it routes to CF which in turn sends it to your server.  

The call to Giphy is essentially the opposite.  Your server directly initiates a request to the Giphy server which is not routed through CF.  So you're not going to be whitelisting anything related to Giphy at Cloudflare.  

Also if it works sometimes, but other times it does not...  you're not running into a whitelist issue.  If it was blocked by the firewall, it would always fail until whitelisted.  If it's intermittent, I would highly suspect something within the network stack OR a high load on the server, or something similar that is causing your problem.  

 

Edited January 14, 2023 by Randy Calvert

[AlexJ]

53 minutes ago, Randy Calvert said:

The call to Giphy is essentially the opposite.  Your server directly initiates a request to the Giphy server which is not routed through CF.  So you're not going to be whitelisting anything related to Giphy at Cloudflare.  

 

Wouldn't it be Server -> Giphy -> Cloudflare -> Server? OR Are you saying, since server is making a call -> Giphy will bypass CF and then direct send response to server? So it would be something like below:

Server Request -> Giphy

Giphy Response -> Server

In both cases, CF is bypassed? I don't have outbound firewall. Only inbound for http and https.

However, you are right i.e. if it's firewall issue, I should always get that but I don't get it all the time, so it's very hard to figure it out. 

FYI: On CF, i whitelisted hostname. Few months, got DDoS's, with avg 12-50M Request/second. So I set up all the stuff through CF and put restrictions on it. Now server IP is not exposed.. so CF takes care of majority of DDoS. No issues anymore except this annoying indexnow and Giphy errors - randomly. 

Thanks for your insight. 

[Randy Calvert]

1 minute ago, AlexJ said:

Wouldn't it be Server -> Giphy -> Cloudflare -> Server? OR Are you saying, since server is making a call -> Giphy will bypass CF and then direct send response to server? So it would be something like below:

Because the server is making the request, Cloudflare is never called.  Cloudflare is only called when someone types in "youdomain.com" because your domain is pointing DNS resolution to them first.  Your server in this case is making a call to "api.giphy.com".  As a result, the DNS resolution would never trigger a call to Cloudflare.