CheersnGears Posted March 10, 2015 Share Posted March 10, 2015 Should we use robots.txt to block crawlers of user profiles? Yes/No/Discuss?Edit: I don't mean if we should use robots.txt or not... I mean should we allow it or not. Link to comment Share on other sites More sharing options...
Grumpy Posted March 11, 2015 Share Posted March 11, 2015 Why would you block crawling unless your stuff was secret? ...in which case they shouldn't be able to crawl, instead of being told not to crawl. Link to comment Share on other sites More sharing options...
CheersnGears Posted March 11, 2015 Author Share Posted March 11, 2015 That's why I'm asking the question. Link to comment Share on other sites More sharing options...
Makoto Posted March 11, 2015 Share Posted March 11, 2015 I don't. More so for privacy reasons though. Most of my members prefer not having their profile information publicly available to be indexed by search engines (and in the case of contact information being posted, crawled by spam bots).I don't think member profiles offer that much of a benefit personally, nor do I think they bring any real harm, but it depends. It's something you'd need to research yourself. If you're using good analytics software, you should be able to easily measure how many of your visitors actually come from links to profile pages. Link to comment Share on other sites More sharing options...
opentype Posted March 11, 2015 Share Posted March 11, 2015 It’s also a privacy issue to me. But even more so to the individual user. A per-user setting would be the best. The advantage of making the profiles fully public: Traffic when people search for your user’s names.The disadvantage: Making your users angry when they appear in search engines without actually wanting it. Link to comment Share on other sites More sharing options...
CheersnGears Posted March 11, 2015 Author Share Posted March 11, 2015 It dawned on me this afternoon why I opened it up for crawling in the first place. I use the Google Authorship tags - http://community.invisionpower.com/topic/396635-google-authorship-and-googleplus-tags/It sucks, my original post in there has been erased by an errant 4.0 upgrade process.... but basically, if you intend to impliment Authorship, then you must allow user profiles to be crawled. Link to comment Share on other sites More sharing options...
Ricw Posted March 12, 2015 Share Posted March 12, 2015 Should we use robots.txt to block crawlers of user profiles? Yes/No/Discuss?Edit: I don't mean if we should use robots.txt or not... I mean should we allow it or not.Thanks for the question. I never gave this much thought until, after reading this topic. Link to comment Share on other sites More sharing options...
Grumpy Posted March 12, 2015 Share Posted March 12, 2015 I understand the privacy concerns to block search engines.But isn't the question whether you should use robots.txt to block search engines instead of other means? Link to comment Share on other sites More sharing options...
CheersnGears Posted March 12, 2015 Author Share Posted March 12, 2015 I understand the privacy concerns to block search engines.But isn't the question whether you should use robots.txt to block search engines instead of other means?No, that isn't the question. If you don't want your user profiles to be crawled by Google, then you should do both blocking in Robots, blocking via meta tags, and blocking via ACP. The question I was asking is what would be the reasons for or against allowing google to crawl the user profiles. Link to comment Share on other sites More sharing options...
David.. Posted March 19, 2015 Share Posted March 19, 2015 No, that isn't the question. If you don't want your user profiles to be crawled by Google, then you should do both blocking in Robots, blocking via meta tags, and blocking via ACP. The question I was asking is what would be the reasons for or against allowing google to crawl the user profiles. Well, did you choose to block them? What line did you add on robots.txt?And what can Spam Bots get from user profiles on a forum? Link to comment Share on other sites More sharing options...
CheersnGears Posted March 19, 2015 Author Share Posted March 19, 2015 Well, did you choose to block them? What line did you add on robots.txt?And what can Spam Bots get from user profiles on a forum?No, I left them open because I use the google authorship tags.If you wish to block you would add a Disallow:/user/* to your Robots.txt and then prevent guests from viewing user profiles. Link to comment Share on other sites More sharing options...
opentype Posted March 19, 2015 Share Posted March 19, 2015 No, I left them open because I use the google authorship tags.Why?http://searchengineland.com/goodbye-google-authorship-201975 Link to comment Share on other sites More sharing options...
CheersnGears Posted March 20, 2015 Author Share Posted March 20, 2015 Why?http://searchengineland.com/goodbye-google-authorship-201975I know about that, but what is the point of shutting off crawl access at this point? Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.