Birgit Schultz Posted March 28, 2022 Share Posted March 28, 2022 Hi, I have a paid membership and do not want Google or any other search index to index any content of my forum. I thought I had done so but members now show me screenshots of Google results they found when searching for their names. This is not wanted by them nor me either. I have copy pasted the robots.txt file from this page to the System -> Site Promotion -> Search Engine Optimization -> Crawl Management section I am unsure whether I have to edit this file somehow and whether I will also have to upload it via ftp to my server since my forum resides in a directory. Thank you! Birgit Link to comment Share on other sites More sharing options...
Randy Calvert Posted March 28, 2022 Share Posted March 28, 2022 A couple of notes... Google crawls your site as a guest. That means if Google can see/access it, so can the rest of your non-paid membership. You might want to review your forum guest permissions. The Robots.txt in that link is for SEO purposes. So that Google crawls the areas of your site that contains content and not things like profiles which don't add a lot of value. If you want to ask all search engines to not crawl your site at all, you need to use something like the following in your robots.txt. User-agent: * Disallow: / Birgit Schultz 1 Link to comment Share on other sites More sharing options...
Dean_ Posted March 28, 2022 Share Posted March 28, 2022 (edited) Is your site paid only? If so, why not close the whole site from guests, so you must be logged in order to see the forum? Edit; I can see from your site, if it's the correct one, that all the forums are in fact hidden from guests. If, the above was done after Google indexed your site, these pages would of course now be archived in Google. You'd need to contact them to have the site removed. Edited March 28, 2022 by Dean_ Birgit Schultz 1 Link to comment Share on other sites More sharing options...
Birgit Schultz Posted March 28, 2022 Author Share Posted March 28, 2022 (edited) @Dean_, the site is zirkel.marketing-zauber.de - and I closed the whole site from guests right from the beginning. The user profiles are partially from a later date (members who came recently) and I don't know how to exclude them. The content of the profile is not visible, but the name and photo as well as the title picture and their birthday as can be seen here: the left side shows the view via Google when not logged in - the right side the logged in view. The forum is completely paid only so that I do not need anything to be indexed by google. Edited March 28, 2022 by Birgit Schultz Link to comment Share on other sites More sharing options...
Randy Calvert Posted March 28, 2022 Share Posted March 28, 2022 If you don't want search engines crawling your site at all, use the robots.txt code I gave you above. It says regardless of user agent (meaning Google, Bing, or anything at all that references robots.txt), disallow the root. If it disallows the root, it won't crawl the rest of the site. And remember... that code does not remove anything that is already crawled. It stops new crawls. So you might need to use Google webmaster tools to request anything already there be removed. Birgit Schultz 1 Link to comment Share on other sites More sharing options...
Birgit Schultz Posted March 28, 2022 Author Share Posted March 28, 2022 Thank you very much, @Randy Calvert! I still don't understand where I have to "install" the robots.txt - in the forum admin at System -> Site Promotion -> Search Engine Optimization -> Crawl Management section or as a robots.txt file on my server or both? Link to comment Share on other sites More sharing options...
opentype Posted March 28, 2022 Share Posted March 28, 2022 (edited) 52 minutes ago, Birgit Schultz said: The content of the profile is not visible, but the name and photo as well as the title picture and their birthday as can be seen here: You need to change the permissions here in the Applications section: This is the most important entry. You can completely hide profiles from guests and search engines this way. You might want to go through the other entries as well, as the user names can appear in widgets, in search and other places. Keep in mind that images remain publicly accessible after these changes and it might take weeks or months until they disappear from search results. You can request a removal for specific images from Google if necessary. Edited March 28, 2022 by opentype Dean_, Birgit Schultz and Daniel F 1 1 1 Link to comment Share on other sites More sharing options...
Solution Randy Calvert Posted March 28, 2022 Solution Share Posted March 28, 2022 14 minutes ago, Birgit Schultz said: Thank you very much, @Randy Calvert! I still don't understand where I have to "install" the robots.txt - in the forum admin at System -> Site Promotion -> Search Engine Optimization -> Crawl Management section or as a robots.txt file on my server or both? That's the correct area. Change the select option to "Custom" and paste the two lines that I gave you there. As bots re-crawl the site, they will see that entry and receive your instruction to not crawl it. Birgit Schultz and Daniel F 1 1 Link to comment Share on other sites More sharing options...
Birgit Schultz Posted March 28, 2022 Author Share Posted March 28, 2022 Okay, I did as you explained, @Randy Calvert, @opentype and @Dean_ - thank you all so much for your support! Dean_ 1 Link to comment Share on other sites More sharing options...
Recommended Posts