Jump to content

AdSense Crawler


Donkerrood

Recommended Posts

  • 2 weeks later...
On 6/2/2018 at 8:28 AM, Morgin said:

It would be ideal given Adsense has to be one of the most popular revenue generators for forums. 

Absolutely correct. Adsense produces way more profit than Viglink does. To me, Viglink is a joke.

I know it would help my income a ton load if I could get Google Adsense logged into my site. 

Link to comment
Share on other sites

As far as I can tell it's working on our forum.

360074987_adsensecrawler.thumb.png.5a73642d4ad6eac8aa00ce1fc092f1c9.png

http:// is our domain (domain.com) Might not be necessary, stems from before we used https. But as long as it's working I do not dare to remove it...
https:// is our forum folder (domain.com/forums)

Link to comment
Share on other sites

8 hours ago, bpn said:

As far as I can tell it's working on our forum.

360074987_adsensecrawler.thumb.png.5a73642d4ad6eac8aa00ce1fc092f1c9.png

http:// is our domain (domain.com) Might not be necessary, stems from before we used https. But as long as it's working I do not dare to remove it...
https:// is our forum folder (domain.com/forums)

Unfortunately this "crawling" sign is not guarantiing that crawler can access the site. I got this green mark too, but still crawler not accessing the site.

Can you see the crawler in your login logs?

Link to comment
Share on other sites

You're sure right Sergey. :huh:

Tried to build a new login, copied from the one that appears working.  This gets status "Not crawling".
So that green mark is misleading.

I tried to build logins to SMS and VB-forums, same problem there.

Quote

Unfortuantely this will mean that you cannot use the AdSense login functionality unless it can read the CSRF from the page and supply it (I'm not familiar enough with AdSense to know if it can do this).

Kind Regards, Stuart Silvester
Invision Power Services, Inc.

This is such an important issue that I think IPS should investigate it further.

Edited by bpn
Link to comment
Share on other sites

  • 3 weeks later...
  • 3 weeks later...

Since there is no update on this topic from IPS, recently I found a small workaround, unfortunatelly it is not bringing the Adsensecrawler to locked forums, but allowing not to loose this part of the traffic completely and serve the static/other ads instead

But I sill hop IPS will resolve this issue with Google. ))

Link to comment
Share on other sites

  • 3 weeks later...
On 3/19/2014 at 2:02 PM, Joriz said:


As far I understand the auth_key is always 880ea6a14ea49e853634fbdc5015a024 for guests.

Seems its csrfKey now and its changes everytime?

<input type="hidden" name="csrfKey" value="c5044f27fd3b835d14ab3f0bfe91fe90">

 

Link to comment
Share on other sites

  • 5 weeks later...

I'm seeing a lot of crawler errors as well, in relation to the pages that require a login.  Features like /discover/unread and private forums mostly. 

Anyway, I don't want crawlers accessing our data behind logins, so, it looks like I need a directive to do this.  Here's what Google says about it  https://support.google.com/webmasters/answer/6062608?visit_id=636773196575520711-1843243888&amp;rd=1:

"You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. If you want to block your page from search results, use another method such as password protection or a noindex directive."

I then click on the noindex directive and get this page:  https://developers.google.com/search/reference/robots_meta_tag

It looks like the best way to hide pages is through a HTTP noindex directive.  Do we have more information on how to add this 'noindex directive' to the areas we want to hide from crawlers?

PS: I already have these pages in my robots.txt file, but, it seems Google is showing them anyway.

 

Edited by Mike Gholson
Link to comment
Share on other sites

13 hours ago, Mike Gholson said:

I'm seeing a lot of crawler errors as well, in relation to the pages that require a login.  Features like /discover/unread and private forums mostly. 

Anyway, I don't want crawlers accessing our data behind logins, so, it looks like I need a directive to do this.  Here's what Google says about it  https://support.google.com/webmasters/answer/6062608?visit_id=636773196575520711-1843243888&amp;rd=1:

"You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. If you want to block your page from search results, use another method such as password protection or a noindex directive."

I then click on the noindex directive and get this page:  https://developers.google.com/search/reference/robots_meta_tag

It looks like the best way to hide pages is through a HTTP noindex directive.  Do we have more information on how to add this 'noindex directive' to the areas we want to hide from crawlers?

PS: I already have these pages in my robots.txt file, but, it seems Google is showing them anyway.

 

You can use the live meta tag editor to do this. From the AdminCP visit System > Site Promotion > Search Engine Optimization > Meta Tags (tab)

Link to comment
Share on other sites

  • 2 weeks later...
On 11/9/2018 at 6:21 AM, bfarber said:

You can use the live meta tag editor to do this. From the AdminCP visit System > Site Promotion > Search Engine Optimization > Meta Tags (tab)

Thanks @bfarber, I found that page.  Do you know what I need to do so it doesn't show anything in /discover ?  I do have an entry for it, so do I just remove it?

Link to comment
Share on other sites

  • 2 weeks later...
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...