Google AdSense issues flagged: 'Crawler: Page not found' AND 'Crawler: Unknown error'

[Darren Lyons]

It doesnt matter what we do, we just can't seem to get the AdSense policy centre to clear these alerts. We've given AdSense admin access so it should be able to see the hidden pages that are causing the errors and they're all from the same part of the forum.

Just wondering if anyone had come across anything like this before? I can't see from the previous posts on here. 

[Marc Stridgen]

It may actually be worth reaching out to their support on this to see if they can answer the question, or at least tell you what they are seeing there when they try to access, if you are sure you have provided access

[Randy Calvert]

When I’ve seen page not found errors it has been because the page being requested was not visible to guests. Remember Adsense crawls your site as a guest. If it is posted on a page that requires being logged in to view, the crawler can’t access it. 

This is part of why I made ads visible to guests not logged in on my site when I did Adsense. 

Edited July 10, 2023 by Randy Calvert

[Darren Lyons]

2 hours ago, Randy Calvert said:

When I’ve seen page not found errors it has been because the page being requested was not visible to guests. Remember Adsense crawls your site as a guest. If it is posted on a page that requires being logged in to view, the crawler can’t access it. 

This is part of why I made ads visible to guests not logged in on my site when I did Adsense. 

Yes, all the pages are not available to guests, which is why we gave AdSense admin access through the Crawler Access section of AdSense. And it shows it's crawling, but the errors remain.

So is that an option in AdSense, where we can disable ads on pages that aren't accessible to guests? The revenue is so small from logged in people that it doesnt matter.

[Randy Calvert]

Adsense is not aware of guest vs member, BUT… IPB is. What I personally did was put the following in my default theme in the global template:

{{if \IPS\Member::loggedIn()->member_id === NULL}}

         ADSENSE CODE

{{endif}}

This basically says to add the Adsense code if a user is not logged in. This would prevent issues where ads are shown to people who can’t access something as a guest.  Once you add it, rebuild them cache from the support section of the ACP to pick up the changes more quickly. 

Edited July 11, 2023 by Randy Calvert

[DawPi]

{{if ! \IPS\Member::loggedIn()->member_id}}

 

[Darren Lyons]

Thanks @Randy Calvert and @DawPi, we've added Randy's code and appears to work, now just need to wait for those pages to be crawled again.

Cheers!