Jump to content

Community

Donkerrood

AdSense Crawler

Recommended Posts

1 hour ago, Unlucky said:

Coming to this topic late.

Has anyone got the solution to get this working?

We would really like to add adverts on the messenger pages.

Thanks

hmm.. this is a good topic, I just found out that Google have no problem display the Ads in the messenger and profile pages on my site. Only in forums locked for guests it is not... So it is somehow able to crawl there...

Share this post


Link to post
Share on other sites

I've done a "dirty" hack to allow a crawler to login into my community. Attention: This changes core login functionality for one user and will be overwritten when you update:

  1. Create a user with name "Crawler" and give him access to the areas the crawler should have access to.
  2. Open file system/Login/Login.php and edit a method authenticate():
    public function authenticate( \IPS\Login\Handler $onlyCheck = NULL )
    {
    	try
    	{
    		if ( isset( \IPS\Request::i()->_processLogin ) )
    		{
    			// Allow access for the AdSense bot
    			if (\IPS\Request::i()->auth != 'Crawler') {
    				\IPS\Session::i()->csrfCheck();
    			}
    	.....
    }

     

  3. Go to your Adsense account and add credentials:
    Untitled-1.png.19624a788f5fab9f7c261b6e1dd7f898.png
    Be aware of the additional field _processLogin to be added to login credentials.

Again, it works, but this is a very dirty solution that works till next update. I hope that IPS can add a setting so that we can exclude bot user from checking csrf in the future, so that we do not need so dirty hacks.

Edited by Sonya*

Share this post


Link to post
Share on other sites

I've put together a plugin which handles this in a different location, making it hookable - have sent a new copy of it to @Mopar1973Man to try it out. If anyone would like a copy then please PM me directly.

I won't be releasing it to the MarketPlace, as it does circumvent the security which IPS has built in to the product (albeit for only one configured account) - all I ask is you chuck me a beer 😉

Share this post


Link to post
Share on other sites

In Adsense settings, you give restrictrect url, like discover/unread.

How about other urls? I have member areas, where are tons of topics etc. How crawler know it needs to be logged in in those areas?

Share this post


Link to post
Share on other sites
1 minute ago, Nathan Explosion said:

In the same way - give the url that they can access (you can add more, not just one) with the credentials they are allowed to use (and make sure that the account has the access to view those areas)

Impossible, there are about 1000-2000 topics.

Share this post


Link to post
Share on other sites
14 minutes ago, Owdy said:

Impossible, there are about 1000-2000 topics.

And those topics are in a container called a forum......unless things have changed since I used Adsense years ago, it will access anything in the same 'directory'. So, in theory.....

Add:

https://invisioncommunity.com/forums/

And it should be able to then crawl.....

https://invisioncommunity.com/forums/topic/368323-adsense-crawler

If it doesn't work in that way now, then you'll have to ask Google.

 

 

Share this post


Link to post
Share on other sites
4 hours ago, Owdy said:

Google logs in, but i never see ads in /discover/unread/

Hi, is working well for us so double check all steps again in case something has been missed.

Or maybe it's the type of add you are trying to show?

We are only using Text & display ads in the member logged in sections

Share this post


Link to post
Share on other sites
6 hours ago, Owdy said:

Google logs in, but i never see ads in /discover/unread/

What YOU see on the page is not what BOT sees on the page as it is dependant on user. It's possible that you do not see the ads but your users do.

Share this post


Link to post
Share on other sites
3 hours ago, Owdy said:

Now Adsense says they got Unknown error in page  /discover/unread/

 

Same here. I assume that the hack I have made does only work for login. But when the crawler tries to get to other pages, it still fails due to the csrf check. Probably the hack should go further, so that not only login but the session itself after login does not make csrf check. I am investigating further.

 

Share this post


Link to post
Share on other sites
On 2/1/2019 at 1:03 PM, Sonya* said:

Same here. I assume that the hack I have made does only work for login. But when the crawler tries to get to other pages, it still fails due to the csrf check. Probably the hack should go further, so that not only login but the session itself after login does not make csrf check. I am investigating further.

 

can you help me for this?

Share this post


Link to post
Share on other sites
52 minutes ago, greek_parea said:

can you help me for this?

I have given up. I have tried different ways to bypass csrf check for one user. But any attempt failed. I can successfully login as Crawler in browser however the crawler itself fails. The account is blocked after three failed attempts and there are no logs in IPS of what is wrong with the login. Without logs in IPS and any logs from AdSense it is like pocking in the dark. Sorry!

Share this post


Link to post
Share on other sites

I know this is an old thread but I wish I seen it sooner, the answer is in your permission settings, the problem I find is that modules and menus in IPS have separate permission tabs, you might allow a crawler to crawl the "Classified" section, but at the same time restrict the menu link to be seen to guests or visa versa etc, you must check that you allow the robot to see everything you've linked from public menues also to avoid landing page link following errors, if menu isn't visible to a crawler it can't error on a path it can't see to attempt travel on etc. Also, avoid static ads, always use auto sizing ads for ips4 :)

Share this post


Link to post
Share on other sites

It's hard to explain in a sentence but if crawler accesses though guest, the bot follows links only avail to guest so if you use a crawler app to crawl your website as guest and see where it's able to go, youll see where your permissions could be at fault? Hope it helps 

Share this post


Link to post
Share on other sites

So just to confirm, as of today there is not a simple or effective way to enable Adsense ads to display to logged in members?

Is this correct IPS staff?

Share this post


Link to post
Share on other sites
8 hours ago, Stormlilly said:

So just to confirm, as of today there is not a simple or effective way to enable Adsense ads to display to logged in members?

Is this correct IPS staff?

You can display ads to logged in members, however if Adsense requires that a page be guest-viewable in order for ads to display there, then any pages which guests are unable to view would not be able to display ads.

Some sites lock everything down and only logged in users can access basically anything. Some sites are pretty much wide open, guests just can't post. There's a lot of variation in how a site can be configured, so it's hard to give a one size fits all answer.

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...