How to prevent search engines from indexing any content of the forum?

[Birgit Schultz]

Hi, 

I have a paid membership and do not want Google or any other search index to index any content of my forum. 

I thought I had done so but members now show me screenshots of Google results they found when searching for their names. This is not wanted by them nor me either. 

I have copy pasted the robots.txt file from this page 

 to the System -> Site Promotion -> Search Engine Optimization -> Crawl Management section 

I am unsure whether I have to edit this file somehow and whether I will also have to upload it via ftp to my server since my forum resides in a directory. 

Thank you!

Birgit 

[Randy Calvert]

A couple of notes...

• Google crawls your site as a guest.  That means if Google can see/access it, so can the rest of your non-paid membership. You might want to review your forum guest permissions.  
• The Robots.txt in that link is for SEO purposes.  So that Google crawls the areas of your site that contains content and not things like profiles which don't add a lot of value.  

If you want to ask all search engines to not crawl your site at all, you need to use something like the following in your robots.txt.

User-agent: *

Disallow: /

[Dean_]

Is your site paid only? If so, why not close the whole site from guests, so you must be logged in order to see the forum?

Edit; I can see from your site, if it's the correct one, that all the forums are in fact hidden from guests.

If, the above was done after Google indexed your site, these pages would of course now be archived in Google. You'd need to contact them to have the site removed.

Edited March 28, 2022 by Dean_

[Birgit Schultz]

@Dean_, the site is zirkel.marketing-zauber.de - and I closed the whole site from guests right from the beginning. The user profiles are partially from a later date (members who came recently) and I don't know how to exclude them. The content of the profile is not visible, but the name and photo as well as the title picture and their birthday as can be seen here:

the left side shows the view via Google when not logged in - the right side the logged in view.

The forum is completely paid only so that I do not need anything to be indexed by google. 

 

Edited March 28, 2022 by Birgit Schultz

[Randy Calvert]

If you don't want search engines crawling your site at all, use the robots.txt code I gave you above.  It says regardless of user agent (meaning Google, Bing, or anything at all that references robots.txt), disallow the root.  If it disallows the root, it won't crawl the rest of the site. 

And remember...  that code does not remove anything that is already crawled.  It stops new crawls.  So you might need to use Google webmaster tools to request anything already there be removed.  

[Birgit Schultz]

Thank you very much, @Randy Calvert! 

I still don't understand where I have to "install" the robots.txt - in the forum admin at 
System -> Site Promotion -> Search Engine Optimization -> Crawl Management section

or as a robots.txt file on my server or both?

[opentype]

52 minutes ago, Birgit Schultz said:

 The content of the profile is not visible, but the name and photo as well as the title picture and their birthday as can be seen here:

You need to change the permissions here in the Applications section:

This is the most important entry. You can completely hide profiles from guests and search engines this way. You might want to go through the other entries as well, as the user names can appear in widgets, in search and other places. 

Keep in mind that images remain publicly accessible after these changes and it might take weeks or months until they disappear from search results. You can request a removal for specific images from Google if necessary. 

Edited March 28, 2022 by opentype

[Randy Calvert]

14 minutes ago, Birgit Schultz said:

Thank you very much, @Randy Calvert! 

I still don't understand where I have to "install" the robots.txt - in the forum admin at 
System -> Site Promotion -> Search Engine Optimization -> Crawl Management section

or as a robots.txt file on my server or both?

That's the correct area.  Change the select option to "Custom" and paste the two lines that I gave you there.  

As bots re-crawl the site, they will see that entry and receive your instruction to not crawl it.  

[Birgit Schultz]

Okay, I did as you explained, @Randy Calvert, @opentype and @Dean_ - thank you all so much for your support!