Invision Community 4: SEO, prepare for v5 and dormant account notifications Matt November 11, 2024Nov 11
Posted March 10, 20159 yr Should we use robots.txt to block crawlers of user profiles? Yes/No/Discuss?Edit: I don't mean if we should use robots.txt or not... I mean should we allow it or not.
March 11, 20159 yr Why would you block crawling unless your stuff was secret? ...in which case they shouldn't be able to crawl, instead of being told not to crawl.
March 11, 20159 yr I don't. More so for privacy reasons though. Most of my members prefer not having their profile information publicly available to be indexed by search engines (and in the case of contact information being posted, crawled by spam bots).I don't think member profiles offer that much of a benefit personally, nor do I think they bring any real harm, but it depends. It's something you'd need to research yourself. If you're using good analytics software, you should be able to easily measure how many of your visitors actually come from links to profile pages.
March 11, 20159 yr It’s also a privacy issue to me. But even more so to the individual user. A per-user setting would be the best. The advantage of making the profiles fully public: Traffic when people search for your user’s names.The disadvantage: Making your users angry when they appear in search engines without actually wanting it.
March 11, 20159 yr Author It dawned on me this afternoon why I opened it up for crawling in the first place. I use the Google Authorship tags - http://community.invisionpower.com/topic/396635-google-authorship-and-googleplus-tags/It sucks, my original post in there has been erased by an errant 4.0 upgrade process.... but basically, if you intend to impliment Authorship, then you must allow user profiles to be crawled.
March 12, 20159 yr Should we use robots.txt to block crawlers of user profiles? Yes/No/Discuss?Edit: I don't mean if we should use robots.txt or not... I mean should we allow it or not.Thanks for the question. I never gave this much thought until, after reading this topic.
March 12, 20159 yr I understand the privacy concerns to block search engines.But isn't the question whether you should use robots.txt to block search engines instead of other means?
March 12, 20159 yr Author I understand the privacy concerns to block search engines.But isn't the question whether you should use robots.txt to block search engines instead of other means?No, that isn't the question. If you don't want your user profiles to be crawled by Google, then you should do both blocking in Robots, blocking via meta tags, and blocking via ACP. The question I was asking is what would be the reasons for or against allowing google to crawl the user profiles.
March 19, 20159 yr No, that isn't the question. If you don't want your user profiles to be crawled by Google, then you should do both blocking in Robots, blocking via meta tags, and blocking via ACP. The question I was asking is what would be the reasons for or against allowing google to crawl the user profiles. Well, did you choose to block them? What line did you add on robots.txt?And what can Spam Bots get from user profiles on a forum?
March 19, 20159 yr Author Well, did you choose to block them? What line did you add on robots.txt?And what can Spam Bots get from user profiles on a forum?No, I left them open because I use the google authorship tags.If you wish to block you would add a Disallow:/user/* to your Robots.txt and then prevent guests from viewing user profiles.
March 19, 20159 yr No, I left them open because I use the google authorship tags.Why?http://searchengineland.com/goodbye-google-authorship-201975
March 20, 20159 yr Author Why?http://searchengineland.com/goodbye-google-authorship-201975I know about that, but what is the point of shutting off crawl access at this point?
Archived
This topic is now archived and is closed to further replies.