IPS Sitemap generator using special database table source for refreshing - core_sitemap.
Primary search engine source of sitemap is url https://example.com/sitemap.php which is list of sub-sitemap files. You can see list of that files proceed for this link.
Each of that file contain no more than 1000 urls to specail pages (profile status, topic (without number of pages or comment) and other elements, with supported sitemap as core extension).
One of our case is forum with more than 100k topics, more than 4.2kk posts and more than 6kk users. So with simply math we have 5214 sitemap files (you can simply count number of that files with command
select count(*) from core_sitemap; // 5214
Sitemap generator task run by default once per 15 minuts and update only one oldest element from that big list. With simple math we can try to answer question 'how many time we need for update everything?' (because users can post not only in newest and may post in some old topics... but.. new created topic will add to sitemap file only when ALL older files will newer than current file with new topic inside). So, how much time we need for update?
5214*15 = 78210 minuts = 1303 hours = 54 days! 54! days! Search engine will add your newest content after 54 days after them posted. Incredible thing. Not believe? Or want to know this lag for your community? You can simple know your lag time with that sql:
select FROM_UNIXTIME(updated,'%a %b %d %H:%i:%s UTC %Y') from core_sitemap order by updated asc limit 1; // Wed Nov 01 14:13:49 UTC 2017
Yep.. In our case oldest file last updated in 1 November...
What we should do for fix it? Very fast solution - create a temp file, like a 'mycustomsitemapupdater.php' with this content:
<?php
require 'init.php';
$generator = new \IPS\Sitemap;
$generator->buildNextSitemap();
$last = \IPS\Db::i()->select('FROM_UNIXTIME(updated, "%a %b %d %H:%i:%s UTC %Y")', 'core_sitemap', null, 'updated asc', 1)->first();
print_r('Oldest time now: ' . $last . PHP_EOL);
And run it via web or cli so times, what you want (before oldest time not be so old).
Solution for a longer time - add this script to the cron and run it every minute or, which better - change task 'sitemap generator' run time from 15 mins to one minute (but it may be not solve you special problem, if you need to update it faster - do it with smart).
Better solution - wait for IPS updating of that system.
Thanks for attension!
P.S. If you read my text with negative speach - it's wrong. I love IPS and just want to make attension for that problem and help others with their large communities.
@DawPi Any chance of adding an option in your plugin to disable all other search options available (as we have Pages installed) and leaving only "Google" (which we would rename to "All") and "Advanced?"
On Pages we only have the option to enable/disable the search for each database, if we disable it there we also lose the option to search each database individually using the advanced search, which we want to keep.
Many thanks!
Two requests for this in a couple of day … Interesting! I have a look at creating Pages block templates for displaying forum posts the SuperGrid way. Look out for a “SuperTopics” product on the Marketplace in a couple of days. ;-)
Silnei L Andrade reacted to Numbered in Large community? You have a problems with sitemap!
Silnei L Andrade got a reaction from crmarks in Pages SuperTopics (Support Topic)
