Jump to content

Community

Donkerrood

AdSense Crawler

Recommended Posts

On 6/2/2018 at 8:28 AM, Morgin said:

It would be ideal given Adsense has to be one of the most popular revenue generators for forums. 

Absolutely correct. Adsense produces way more profit than Viglink does. To me, Viglink is a joke.

I know it would help my income a ton load if I could get Google Adsense logged into my site. 

Share this post


Link to post
Share on other sites

As far as I can tell it's working on our forum.

360074987_adsensecrawler.thumb.png.5a73642d4ad6eac8aa00ce1fc092f1c9.png

http:// is our domain (domain.com) Might not be necessary, stems from before we used https. But as long as it's working I do not dare to remove it...
https:// is our forum folder (domain.com/forums)

Share this post


Link to post
Share on other sites
8 hours ago, bpn said:

As far as I can tell it's working on our forum.

360074987_adsensecrawler.thumb.png.5a73642d4ad6eac8aa00ce1fc092f1c9.png

http:// is our domain (domain.com) Might not be necessary, stems from before we used https. But as long as it's working I do not dare to remove it...
https:// is our forum folder (domain.com/forums)

Unfortunately this "crawling" sign is not guarantiing that crawler can access the site. I got this green mark too, but still crawler not accessing the site.

Can you see the crawler in your login logs?

Share this post


Link to post
Share on other sites

You're sure right Sergey. :huh:

Tried to build a new login, copied from the one that appears working.  This gets status "Not crawling".
So that green mark is misleading.

I tried to build logins to SMS and VB-forums, same problem there.

Quote

Unfortuantely this will mean that you cannot use the AdSense login functionality unless it can read the CSRF from the page and supply it (I'm not familiar enough with AdSense to know if it can do this).

Kind Regards, Stuart Silvester
Invision Power Services, Inc.

This is such an important issue that I think IPS should investigate it further.

Edited by bpn

Share this post


Link to post
Share on other sites

Since there is no update on this topic from IPS, recently I found a small workaround, unfortunatelly it is not bringing the Adsensecrawler to locked forums, but allowing not to loose this part of the traffic completely and serve the static/other ads instead

But I sill hop IPS will resolve this issue with Google. ))

Share this post


Link to post
Share on other sites
9 hours ago, Mopar1973Man said:

I'd rather have a true fix instead of a band-aid that might fail...

Fully agree, but I with given absence of ture fix and no promisses of it, I would rather not loose money)))

Share this post


Link to post
Share on other sites
On 3/19/2014 at 2:02 PM, Joriz said:


As far I understand the auth_key is always 880ea6a14ea49e853634fbdc5015a024 for guests.

Seems its csrfKey now and its changes everytime?

<input type="hidden" name="csrfKey" value="c5044f27fd3b835d14ab3f0bfe91fe90">

 

Share this post


Link to post
Share on other sites

We use a CSFR key to prevent bots attempting to log in as you.

Which is specifically what AdSense are trying to do.

I'll speak to the team to see if there's something we can do, but we'd be looking to remove or mitigate protection designed to keep your accounts safe.

Share this post


Link to post
Share on other sites

I'm seeing a lot of crawler errors as well, in relation to the pages that require a login.  Features like /discover/unread and private forums mostly. 

Anyway, I don't want crawlers accessing our data behind logins, so, it looks like I need a directive to do this.  Here's what Google says about it  https://support.google.com/webmasters/answer/6062608?visit_id=636773196575520711-1843243888&amp;rd=1:

"You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. If you want to block your page from search results, use another method such as password protection or a noindex directive."

I then click on the noindex directive and get this page:  https://developers.google.com/search/reference/robots_meta_tag

It looks like the best way to hide pages is through a HTTP noindex directive.  Do we have more information on how to add this 'noindex directive' to the areas we want to hide from crawlers?

PS: I already have these pages in my robots.txt file, but, it seems Google is showing them anyway.

 

Edited by Mike Gholson

Share this post


Link to post
Share on other sites
13 hours ago, Mike Gholson said:

I'm seeing a lot of crawler errors as well, in relation to the pages that require a login.  Features like /discover/unread and private forums mostly. 

Anyway, I don't want crawlers accessing our data behind logins, so, it looks like I need a directive to do this.  Here's what Google says about it  https://support.google.com/webmasters/answer/6062608?visit_id=636773196575520711-1843243888&amp;rd=1:

"You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. If you want to block your page from search results, use another method such as password protection or a noindex directive."

I then click on the noindex directive and get this page:  https://developers.google.com/search/reference/robots_meta_tag

It looks like the best way to hide pages is through a HTTP noindex directive.  Do we have more information on how to add this 'noindex directive' to the areas we want to hide from crawlers?

PS: I already have these pages in my robots.txt file, but, it seems Google is showing them anyway.

 

You can use the live meta tag editor to do this. From the AdminCP visit System > Site Promotion > Search Engine Optimization > Meta Tags (tab)

Share this post


Link to post
Share on other sites
On 11/9/2018 at 6:21 AM, bfarber said:

You can use the live meta tag editor to do this. From the AdminCP visit System > Site Promotion > Search Engine Optimization > Meta Tags (tab)

Thanks @bfarber, I found that page.  Do you know what I need to do so it doesn't show anything in /discover ?  I do have an entry for it, so do I just remove it?

Share this post


Link to post
Share on other sites

If you require support or assistance, I would encourage you to post in the appropriate peer feedback forum or submit a ticket. This area is more intended for suggestions for future changes in the software. 🙂 

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...