rayzir Posted July 15, 2023 Share Posted July 15, 2023 Here's a silly question that I'm trying to find the answer to, but seem to be coming up short. I'm revamping my Google Adsense and I get crawler errors from time to time. Usually, it's when they're trying to access stuff from behind a login account. The ads I'm getting don't seem to be all that targeted. So I'm thinking maybe they're just serving generic ads because they can't read the content? I'm wondering if I have to do something about adding permissions for these crawlers. I have a member group called BOTS. Is that a default group that IPBoard created for google bots and the like? Am I supposed to create accounts for them? I moved over to IP Board from PHPBB many years ago. So I'm not sure if this group name is a remnant from that old system and unrelated to IPBoard. I guess the question is, do I need to do anything to allow these crawlers? Is the BOTS group something I need? Or is the ad serving more based on the user and what the user can see? Thanks for your insight! Link to comment Share on other sites More sharing options...
Jim M Posted July 15, 2023 Share Posted July 15, 2023 Google Adsense would view your community as a guest (non logged in user) so if a guest doesn’t have access to the page, neither does Google Adsense. I’m afraid, there are no means in Invision Community to get them a logged in view. rayzir 1 Link to comment Share on other sites More sharing options...
rayzir Posted July 15, 2023 Author Share Posted July 15, 2023 Thanks for the info. Is the BOTS member group a default group with IP Board? Or do you suppose that come over when I migrated from my old board? Link to comment Share on other sites More sharing options...
Randy Calvert Posted July 16, 2023 Share Posted July 16, 2023 It is not part of IPB. rayzir 1 Link to comment Share on other sites More sharing options...
Solution Nathan Explosion Posted July 16, 2023 Solution Share Posted July 16, 2023 This post was recognized by Marc Stridgen! Nathan Explosion was awarded the badge 'Helpful' and 5 points. 12 hours ago, rayzir said: I'm wondering if I have to do something about adding permissions for these crawlers. https://support.google.com/adsense/answer/161351?hl=en-GB Change the following items to suit... "Restricted directory or URL" - provide the full URL of the area they can't access "Login URL" - change domain.com to your own site "auth" - change user@emailaddress.com to the username or email address (I'd advise email address) for the account they are to use "password" - change thepassword to the password for the account Click 'Add login' rayzir 1 Link to comment Share on other sites More sharing options...
rayzir Posted July 17, 2023 Author Share Posted July 17, 2023 On 7/16/2023 at 1:09 AM, Nathan Explosion said: https://support.google.com/adsense/answer/161351?hl=en-GB Change the following items to suit... "Restricted directory or URL" - provide the full URL of the area they can't access "Login URL" - change domain.com to your own site "auth" - change user@emailaddress.com to the username or email address (I'd advise email address) for the account they are to use "password" - change thepassword to the password for the account Click 'Add login' Thank you. I'm following your instructions. A couple of questions, if you don't mind. For the restricted directory or URL section, can I put just the root, and that will take care of any sections that the bot can't read? For instance, I have guests be able to read topic titles, but not the content unless they're logged in. Or do I have to supply the full path for each specific area that would require a login? Lastly, for _processLogin, is the parameter value "usernamepassword", or is it the auth parameter followed by the password parameter? (so using your example "user@emailaddress.comthepassword" Thanks for your help! Link to comment Share on other sites More sharing options...
Nathan Explosion Posted July 17, 2023 Share Posted July 17, 2023 10 minutes ago, rayzir said: Lastly, for _processLogin, is the parameter value "usernamepassword", or is it the auth parameter followed by the password parameter? (so using your example "user@emailaddress.comthepassword" It's as indicated - usernamepassword The other question...you'll have to try it out to know, or ask Google themselves. I don't use AdSense rayzir 1 Link to comment Share on other sites More sharing options...
rayzir Posted July 17, 2023 Author Share Posted July 17, 2023 Thank you. You've been very helpful. I really appreciate it! Link to comment Share on other sites More sharing options...
LemonGrenade Posted August 8, 2023 Share Posted August 8, 2023 On 7/15/2023 at 8:49 PM, rayzir said: Thanks for your insight! I've been trying to fix crawling on that /discover/unread/?id=1 for literally years! Obviously, it's a very popular page so would be losing a big part of the revenue without ad serving on that. Remember to edit your robots.txt in admin as it is listed not to crawl. On 7/15/2023 at 9:34 PM, Jim M said: Google Adsense would view your community as a guest (non logged in user) so if a guest doesn’t have access to the page, neither does Google Adsense. I’m afraid, there are no means in Invision Community to get them a logged in view. So Google has no problem with /discover/ as it's displayed for guests but how to workaround /discover/unread/?id=1 I've added a Google login as mentioned above but still get the warnings and a 'Must Fix'. If someone could create a Bot Group Plugin (like the phpBB Bot Group), they would be sitting on a goldmine. Link to comment Share on other sites More sharing options...
Jim M Posted August 8, 2023 Share Posted August 8, 2023 The unread stream is exclusively available to logged in members only so you cannot provide this to a guest. Link to comment Share on other sites More sharing options...
LemonGrenade Posted August 8, 2023 Share Posted August 8, 2023 Google can crawl 'logged-in' pages and display ads, just not /discover/unread/?id=1 Link to comment Share on other sites More sharing options...
Jim M Posted August 8, 2023 Share Posted August 8, 2023 33 minutes ago, LemonGrenade said: Google can crawl 'logged-in' pages and display ads, just not /discover/unread/?id=1 Believe there is confusion here. In the case of the unread activity stream, it cannot be made available to guests. Whereas, some other pages, like the All Activity Stream for instance can be visible to both members and guests. Link to comment Share on other sites More sharing options...
LemonGrenade Posted August 8, 2023 Share Posted August 8, 2023 Yep understood @Jim M The activity stream for members shows the new streams for them individually, but how to get at least that page displayed for Google in some capacity? There must be some sort of workaround so they can crawl that page and in turn, display ads easier. The media company displaying ads did appeal to Google (somehow) and now ads get displayed, but it still shows 'unknown crawl error' with restricted ads. Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 9, 2023 Share Posted August 9, 2023 There is no way in which to get that displayed to guests. The page itself simply wouldnt make sense in that context as guest cannot have unread content as they arent a member georgebkk 1 Link to comment Share on other sites More sharing options...
LemonGrenade Posted August 9, 2023 Share Posted August 9, 2023 Understood as mentioned on the last post. But, is there a way to display 'a' page with that url to a bot group somehow? It's not that it needs to show to guests, there is already /discover/ but a way to show /discover/unread/?id=1 page to Googlebot. Link to comment Share on other sites More sharing options...
Nathan Explosion Posted August 9, 2023 Share Posted August 9, 2023 Posted previously in this topic - if you want Google Adsense to login to your site, then set up an account for it and configure away... SeNioR- and LemonGrenade 1 1 Link to comment Share on other sites More sharing options...
LemonGrenade Posted August 10, 2023 Share Posted August 10, 2023 22 hours ago, Nathan Explosion said: Posted previously in this topic - if you want Google Adsense to login to your site, then set up an account for it and configure away... Thanks Nathan, I'll give that another go! I looked at the details and was still using the username so have set it up again! All the best! Link to comment Share on other sites More sharing options...
Recommended Posts