JLS Posted December 27, 2015 Share Posted December 27, 2015 I need some direction with a robots.txt. I need all the standard recommended stuff blocked and then I'd also like to block user profiles unless there is good reason not too. I understand they can be viewed by the public but seems a waste... Thank you in advance for help with this Link to comment Share on other sites More sharing options...
tnn Posted December 27, 2015 Share Posted December 27, 2015 This was graciously provided by @laltroweb.it (https://laltroweb.it/robots.txt). Further reading in this thread -> https://community.invisionpower.com/topic/422077-my-robotstxt-fpr-ipb4/ Something to get you started Link to comment Share on other sites More sharing options...
JLS Posted December 27, 2015 Author Share Posted December 27, 2015 Cool that helped thanks. Now, I kinda think I need to allow bots in the profiles but no-index them to get those few thousand pages removed from google. Then once removed do a no-follow. Does that sound right to you or any others reading this? Thanks Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.