IndexNow: 403 invalid_api_key Array

[Gabriel Torres]

Hi,

Since March 1st, 2022, we are seeing lots of errors in the System Logs regarding IndexNow.

E.g.:

403 invalid_api_key Array
(
    [host] => www.clubedohardware.com.br/
    [key] => df4011de0e4203ba2b937bd97f3c1746
    [keyLocation] => https://www.clubedohardware.com.br/df4011de0e4203ba2b937bd97f3c1746.txt
    [urlList] => Array
        (
            [0] => https://www.clubedohardware.com.br/forums/topic/1590646-depois-de-trocar-o-processador-%C3%A9-necess%C3%A1rio-formatar/
        )

)

Please advise.

Cheers,

Gabe.

[Daniel F]

Have you disabled the setting recently? Was anything else changed on march 1?

[Gabriel Torres]

Hi Daniel F,

Nothing has changed here. I enabled IndexNow soon after upgrading to 4.6.10. The txt file that the error message is complaining simply doesn't exist.

Cheers.

[Daniel F]

While looking at the issue, I've noticed that you have thousands of logged errors in your system log.

Have you tried to disable all 3rd party apps and plugins?

Few of them like the Rules App are definitely not compatible with the current IPS release! 

[Gabriel Torres]

@Daniel F I will take a deeper look into this and get back to you here.

[Daniel F]

There's a similar (but not same:D ) issue reported here 

 

I'm investigating both right now.

[Daniel F]

Your issue is not related to the other topic.

So what happens on your community, is that any not existing page redirects back to your index page instead of letting IPS handle the response, which also results in that the IndexNow Key File logic is never called!

There must be some 3rd party application or plugin changing this behaviour.  

[Gabriel Torres]

@Daniel F Thanks for the investigation, I will fix this over the weekend and will let you know the results.

[Gabriel Torres]

@Daniel F Many thanks for the pointer. This was an old setting we had in our nginx configuration file for our domain. I removed the line that was causing this behavior. However, I am still seeing these messages, however, all messages I am seeing have the same key name (6b6de56e4ca9b6fb281a2792dc18de36).

One thing, though. If I try opening www.clubedohardware.com.br/test (for example), now I get the custom 404 page generated by the IPS software. But if I try www.cubedohardware.com.br/test.txt (for example), I get a default 404 page from nginx. If the key file (6b6de56e4ca9b6fb281a2792dc18de36.txt) is a "virtual" file handled by the routine for handling 404 errors, the issue might be this (nginx 404 page vs IPS handler). Maybe I need to configure my nginx conf file so 404 errors are directed to a specific URL/PHP file?

[Gabriel Torres]

The reason I am seeing the behavior I described in my previous post is because I have cache headers assigned to static files, and a 404 page was not configured for this case.

I reconfigured the conf file as per the tutorial below:

The tutorial tells us to use the 404error.php file. This file only prints a basic "not found" message instead of using the default, fancy, not found routine from IPS. See the difference below:

So, right now I wonder how to configure our nginx setup so that the "fancy" 404 error handler is also used for not found static files. This should fix our original issue.

Thanks.

[Gabriel Torres]

The reason I was seeing the behavior I described in my previous posts was because I have cache headers assigned to static files, and a parameter was missing (we use nginx here). I will check the log files again after a few hours to see if this change solved the issue.

Before:

#cache for static files
        location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|htm|html|txt|swf|cur|ttf|woff|woff2)$ {
                access_log off;
                expires    365d;
        }

After:

#cache for static files
        location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|htm|html|txt|swf|cur|ttf|woff|woff2)$ {
                try_files $uri /index.php?args;
                access_log off;
                expires    365d;
        }

[Gabriel Torres]

Typo: try_files $uri /index.php?$args;

[Gabriel Torres]

@Daniel F With the configuration I did, the .txt URL now opens and displays a key. So, this part is now solved.

E.g. https://www.clubedohardware.com.br/6b6de56e4ca9b6fb281a2792dc18de36.txt

However, I still see errors regarding this specific key I pasted above in the System log. Maybe I need to delete/reset it in a table in the database?

Thanks for all your pointers.

[Gabriel Torres]

I have finally solved this issue, by disabling and re-enabling IndexNow in the Integrations menu. In summary, here is what was causing the issue:

1. 404 error pages were not being directed to the IPS software, so the software wasn't being able to create the "virtual" txt files containing the API key. In our install, this was being caused by a rule in the nginx conf file that set caching for static files.

2. After fixing that, the error was being caused by a mismatch between the API key that was currently being used by the IndexNow module and the actual request. This was solved by simply disabling and re-enabling IndexNow in the Integrations menu.

Thanks!