AdSense Crawler

[Sergey_SV]

1 hour ago, Unlucky said:

Coming to this topic late.

Has anyone got the solution to get this working?

We would really like to add adverts on the messenger pages.

Thanks

hmm.. this is a good topic, I just found out that Google have no problem display the Ads in the messenger and profile pages on my site. Only in forums locked for guests it is not... So it is somehow able to crawl there...

[Sonya*]

I've done a "dirty" hack to allow a crawler to login into my community. Attention: This changes core login functionality for one user and will be overwritten when you update:

1. Create a user with name "Crawler" and give him access to the areas the crawler should have access to.
2. Open file system/Login/Login.php and edit a method authenticate():

   public function authenticate( \IPS\Login\Handler $onlyCheck = NULL )
   {
   	try
   	{
   		if ( isset( \IPS\Request::i()->_processLogin ) )
   		{
   			// Allow access for the AdSense bot
   			if (\IPS\Request::i()->auth != 'Crawler') {
   				\IPS\Session::i()->csrfCheck();
   			}
   	.....
   }

    

3. Go to your Adsense account and add credentials:
   
   Be aware of the additional field _processLogin to be added to login credentials.

Again, it works, but this is a very dirty solution that works till next update. I hope that IPS can add a setting so that we can exclude bot user from checking csrf in the future, so that we do not need so dirty hacks.

Edited December 22, 2018 by Sonya*

[Owdy]

I get parse errors with that. Can you show exatly how to edit Login.php? @Sonya*

 

edit: got it.

Edited December 22, 2018 by Owdy

[Mopar1973Man]

Could some write this up a bit clearer it creating a mess on my end. I need to know the edit on the login.php file. What to look for and what to insert where.

Edited December 23, 2018 by Mopar1973Man

[Nathan Explosion]

I've put together a plugin which handles this in a different location, making it hookable - have sent a new copy of it to @Mopar1973Man to try it out. If anyone would like a copy then please PM me directly.

I won't be releasing it to the MarketPlace, as it does circumvent the security which IPS has built in to the product (albeit for only one configured account) - all I ask is you chuck me a beer 😉

[lordi]

just sent you PM @Nathan Explosion, thank you

[Owdy]

In Adsense settings, you give restrictrect url, like discover/unread.

How about other urls? I have member areas, where are tons of topics etc. How crawler know it needs to be logged in in those areas?

[Nathan Explosion]

In the same way - give the url that they can access (you can add more, not just one) with the credentials they are allowed to use (and make sure that the account has the access to view those areas)

[Owdy]

1 minute ago, Nathan Explosion said:

In the same way - give the url that they can access (you can add more, not just one) with the credentials they are allowed to use (and make sure that the account has the access to view those areas)

Impossible, there are about 1000-2000 topics.

[Nathan Explosion]

14 minutes ago, Owdy said:

Impossible, there are about 1000-2000 topics.

And those topics are in a container called a forum......unless things have changed since I used Adsense years ago, it will access anything in the same 'directory'. So, in theory.....

Add:

https://invisioncommunity.com/forums/

And it should be able to then crawl.....

https://invisioncommunity.com/forums/topic/368323-adsense-crawler

If it doesn't work in that way now, then you'll have to ask Google.

 

 

[Owdy]

Google logs in, but i never see ads in /discover/unread/

[Unlucky]

4 hours ago, Owdy said:

Google logs in, but i never see ads in /discover/unread/

Hi, is working well for us so double check all steps again in case something has been missed.

Or maybe it's the type of add you are trying to show?

We are only using Text & display ads in the member logged in sections

[Sonya*]

6 hours ago, Owdy said:

Google logs in, but i never see ads in /discover/unread/

What YOU see on the page is not what BOT sees on the page as it is dependant on user. It's possible that you do not see the ads but your users do.

[Owdy]

Yeah, maybe. 

[Owdy]

In /online/ seems it doesnt crawl anyting

 

 

[Mopar1973Man]

Thank you @Nathan Explosion .

He jumped in and fixed my typo. I'm now on the working list. I'll continue to monitor the status over the next few days. 

[Owdy]

Now Adsense says they got Unknown error in page  /discover/unread/

 

Edited February 1, 2019 by Owdy

[Sonya*]

3 hours ago, Owdy said:

Now Adsense says they got Unknown error in page  /discover/unread/

 

Same here. I assume that the hack I have made does only work for login. But when the crawler tries to get to other pages, it still fails due to the csrf check. Probably the hack should go further, so that not only login but the session itself after login does not make csrf check. I am investigating further.

 

[greek_parea]

On 2/1/2019 at 1:03 PM, Sonya* said:

Same here. I assume that the hack I have made does only work for login. But when the crawler tries to get to other pages, it still fails due to the csrf check. Probably the hack should go further, so that not only login but the session itself after login does not make csrf check. I am investigating further.

 

can you help me for this?

[Sonya*]

52 minutes ago, greek_parea said:

can you help me for this?

I have given up. I have tried different ways to bypass csrf check for one user. But any attempt failed. I can successfully login as Crawler in browser however the crawler itself fails. The account is blocked after three failed attempts and there are no logs in IPS of what is wrong with the login. Without logs in IPS and any logs from AdSense it is like pocking in the dark. Sorry!

[VladTheGreat]

I know this is an old thread but I wish I seen it sooner, the answer is in your permission settings, the problem I find is that modules and menus in IPS have separate permission tabs, you might allow a crawler to crawl the "Classified" section, but at the same time restrict the menu link to be seen to guests or visa versa etc, you must check that you allow the robot to see everything you've linked from public menues also to avoid landing page link following errors, if menu isn't visible to a crawler it can't error on a path it can't see to attempt travel on etc. Also, avoid static ads, always use auto sizing ads for ips4 :)

[LemonGrenade]

How are you making it work @VladTheGreat as guests just see /discover (which displays all ads) but members see discover/unread/ (which hardly displays any)

[VladTheGreat]

It's hard to explain in a sentence but if crawler accesses though guest, the bot follows links only avail to guest so if you use a crawler app to crawl your website as guest and see where it's able to go, youll see where your permissions could be at fault? Hope it helps 

[Stormlilly]

So just to confirm, as of today there is not a simple or effective way to enable Adsense ads to display to logged in members?

Is this correct IPS staff?

[bfarber]

8 hours ago, Stormlilly said:

So just to confirm, as of today there is not a simple or effective way to enable Adsense ads to display to logged in members?

Is this correct IPS staff?

You can display ads to logged in members, however if Adsense requires that a page be guest-viewable in order for ads to display there, then any pages which guests are unable to view would not be able to display ads.

Some sites lock everything down and only logged in users can access basically anything. Some sites are pretty much wide open, guests just can't post. There's a lot of variation in how a site can be configured, so it's hard to give a one size fits all answer.