Understanding sitemap generation

[Chinna444]

Hello,

I am trying to understand how sitemap option in ipb4 works and below are my questions..

My forum has 1000's of posts everyday and i am using recommended settings for sitemap in IPB admin area. Does the site map generate every day when the content is updated? Does it only update titles or even the content? 

 

[Martin A.]

Your sitemap will only serve a link to where the search engines can find your new content. Title and post content is not stored in those files. It's updated periodically by a task.

You can see what the sitemap index looks like here: https://invisionpower.com/sitemap.php
And this is what's inside a sitemap: https://invisionpower.com/sitemap.php?file=sitemap_content_forums_Topic_1

[Chinna444]

6 hours ago, Martin A. said:

Your sitemap will only serve a link to where the search engines can find your new content. Title and post content is not stored in those files. It's updated periodically by a task.

You can see what the sitemap index looks like here: https://invisionpower.com/sitemap.php
And this is what's inside a sitemap: https://invisionpower.com/sitemap.php?file=sitemap_content_forums_Topic_1

got it... so the task will update the sitemap every day?...can you please let me know where can i find the setting for the task?

[Martin A.]

Or several times per day, depending on how many topics you have. The task runs every 15 minutes, and if I read the code correctly, it updates the oldest of the files each run.

Do a search in the ACP for "task". You'll find the "sitemapgenerator" task there.

[KentT]

A couple of related questions from an IPS noob:

1.  How do I set up a ROBOTS.TXT file to point to that sitemap, and where do I put it?  I assume it goes in the HTML docs (where IPS resides), but I'm not sure that is correct.

2.  How can I prevent these bots from indexing Pages records that are not publicly viewable?  I've set permissions on some categories of records, but it seems the bots are indexing it, just the same....

[Chinna444]

On 1/1/2017 at 3:50 AM, Martin A. said:

Or several times per day, depending on how many topics you have. The task runs every 15 minutes, and if I read the code correctly, it updates the oldest of the files each run.

Do a search in the ACP for "task". You'll find the "sitemapgenerator" task there.

Thank you.