Download: XML Sitemap Generator Task v1.0.5

[stoo2000]

File Name: XML Sitemap Generator Task v1.0.5
File Submitter: stoo2000
File Submitted: 04 Aug 2009
File Updated: 23 Jul 2010
File Category: Miscellaneous XML Files

IP.Board 3 Sitemap Generator Task v1.0.5
About the Sitemap Generator Task
This task was originally updated to accommodate my own forums, due to the demand and the fact that no one has made a new task to address the demand I have decided to enhance this task and release it for use. This task uses built in IP.Board methods for generating FURLS, this means that this task is compatible with custom FURL templates.

The task is Improved and built on Google Sitemap Generator for IPB 2.2 by CheetahShrk.

Version 1.0.5
Fixes

Version 1.0.4
Enhancements

[*]Corrected minor typo in gzip process.

Version 1.0.3
Fixes

[*]Added option to ping ASK! with sitemaps.

Version 1.0.2
Fixes

[*]Localisation bug introduced in 1.0.2 - affects locales that use , as a number seperator.

Version 1.0.1
Fixes

[*]Removed use of old class_XML.php to use the new IP.Board 3, classXML.php method.

Version 1
Fixes

[*]Fixed an issue where URLS with no SEO Title generated would put old style URLS in sitemap, and not use FURLS.

[*]PING function altered to work properly. [*]Added option to choose if GZIP should be used [*]Upgraded to work with IP.Board 3.0.x

Enhancements

[*]Now uses the database class correctly – uses prefixes correctly.

[*]Easier to use settings, with forum multi-select [*]Supports IP.Board 3.0.x FURL’s & legacy urls

IMPORTANT!
There is a bug in IPB 3.0.4 that will prevent FURLS being generated for your sitemap, as such any sitemaps generated will include legacy urls, if you would like to continue to use FURLS you need to run the task manually, then disable it. This bug is fixed in 3.0.5.

Click here to download this file

[*]Bing Ping added

[Mercury_]

Hello.

Thank you for this great modification.

I've got it installed, but It generates a timeout and locks the task.
My forum has around 20.000 topics.

Is it possible to improve the performance of this mod?

Or a better way in my opinion:
- Add a possibility to create multiple sitemaps.
Lets say .. I want an sitemap(X) for all topics in the forum 1, 2 and 3. This task runs at 1am. Then I set up an other sitemap(Y) for all topics in forums 4, 5 and 6 and this task runs at 1.20am ... and so on.
You can add easily multiple sitemaps in Google. Or add all created sitemaps to the indexsitemap.xml.

In this way you can split al the load.


One more question:
Is it possible for you to add all pages in a topic to the sitemap?


Let me know what you are thinking about my opinion. :) Thanks.

[stoo2000+]

Hi Mercury.

I don't seem to have access to this thread from my usual acccount...

No matter how many sitemaps you create you are still working with that data in one request, so it will still timeout.

Could you possibly send me an excerpt of your error_log detailing the problem ?

The solution may be to extend the max execution time whilst processing the data, although I would need to look into how that affects shared hosts.

As you mention splitting it into multiple tasks would be an idea but an avenue I'd like to avoid at the moment.

[Mercury_]

Hi.

If I select all forums, I'll get a 504-error after 30 seconds.
There is no sql-error or a log entry in the Task-Manager.

It simply times out.


Now what I did ... instead of selecting all forums I selected only 3 of them.
Then I startet to run the task and it successfully finished the sitemap in less than 5 seconds. That is all good.

But how can I bring all the other forum in?

That is where I pointet at.

Give the possibility in your mod to create more tasks (propably only by import the xml with some ID-changes).
Now you have multiple tasksk which can run at diffrent times (leave 5 minutes time between).
Sitemap-Task 1 ... includes forums 1-5 and creates topicsitemap1.xml
Sitemap-Task 2 ... includes forums 6-11 and creates topicsitemap2.xml
Sitemap-Task 3 ... includes forums 11-15 and creates topicsitemap2.xml

And so on ...

By this sheme to split the work load you prevent a timeout.

I don't think that this is a lot of work.
You just need to use some more variables.

[stoo2000+]

It's really something that needs some looking into really, if processing a large amount of data is causing a time out, and 50,000 item limit per forum might mean that it still times out even if it is split up.

Give this a shot.

Find (Line 274):

 while ( $r = $this->DB->fetch() )

        {

            $content = array();

replace with:

      while ( $r = $this->DB->fetch() )

        {

			// lets see if this solves some timing out issues.

			set_time_limit(30);


            $content = array();

[Mercury_]

I still get an 504-error. :(

[stoo2000]

I'm not so sure that is related to the sitemap generator, Have a quick search for 504 error online, it seems it could be an issue with an upstream provider i.e your web hosting.

[InvisionHQ]

Many warnings on the topicsitemap, all related to "Tag: priority"

[stoo2000]

[quote name='InvisionHQ' date='04 November 2009 - 08:44 AM' timestamp='1257324255' post='1875351']
Many warnings on the topicsitemap, all related to "Tag: priority"

Could you PM me some examples please, and link me to your topicsitemap file, I've had this version live on one of my own forums for a few days, without any Google reported issues.

[InvisionHQ]

PM sent

[stoo2000]

Looks like it is a Bug/Feature in PHP that replaces . with , in integers depending on your locale, Will correct it and send you a copy to test.

[dyelton]

Community SEO (http://www.communityseo.com) handles sitemaps just fine, but I had to stop using their software as they were giving me extremely high server loads (not related to sitemaps).

I have a very large forum (1+ million posts) and this sitemap task also times out for me. I'm using version 1.0.3.

[stoo2000]

[quote name='dyelton' date='05 November 2009 - 02:17 PM' timestamp='1257430664' post='1875858']
Community SEO (http://www.communityseo.com) handles sitemaps just fine, but I had to stop using their software as they were giving me extremely high server loads (not related to sitemaps).

I have a very large forum (1+ million posts) and this sitemap task also times out for me. I'm using version 1.0.3.

That's one reason why it's also limited to 50,000 items, I do want to look in the future at indexing over time, and not at once. You could try adding set_time_limit(30) into the topic loop, that may prevent the time out, but it will still take a small while to construct the XML file.

[AlexJ]

In the docs it says CHMOD the above files to 0777. Does it also applies to task sitemap.php? More or less if I run task it only generates 2 or 3 links thats it.

[stoo2000]

You only need to chmod the xml/xml.gz files.

Make sure you select the forums you want to put in the sitemap (in the sitemap task settings)

[AlexJ]

Thanks got it working. :) For some reason it didn't worked for first time. I did run task again back and it worked well. We have only 3000 topics so it worked out pretty fast. Just to double check indexsitemap has only 2 links in it. forumsitemap.xml.gz and for topicsitemap.xml.gz. Is that how it should be?

Is their anyway to show sitemap on forums as a link?

[stoo2000]

Yep the indexsitemap.xml.gz is just a location to let search engines know about the different sitemaps that are generated. The XML Sitemaps are intended for Search engines, you won't get any benefits by linking to it, since it informs search engines when it has been generated :)

[AlexJ]

Cool thanks. And one more last question. Does it makes huge difference if my website uses www.domain.com extension and forum uses .domain.com extension?

Does it effects page ranking? or Google analytics tracking?

[stoo2000]

[quote name='AlexJ' date='11 November 2009 - 11:51 PM' timestamp='1257983516' post='1878343']
Cool thanks. And one more last question. Does it makes huge difference if my website uses www.domain.com extension and forum uses .domain.com extension?

Does it effects page ranking? or Google analytics tracking?

Nope that doesn't matter at all, obviously you would want two Google Analytics accounts so you can track forum and website traffic.

[AlexJ]

Oh i was using same. :( I thought all sub domains and folders are included in one main profile of Google analytics.

[TheMac]

Hello, I have this: Sitemap Generator v1.0.3 (LOCKED)
But when I unlock they lock other time...

[stoo2000+]

That means that there was a problem running the task, you should unlock the task and run it manually. If you see any errors please send them to me in a PM.

[TheMac]

Ok, now I think that runs ok :)
Thanks! ;)

[Enkidu]

I'm getting a pile of re-direct errors and they all have this format:



Now where did this showtopic come from?

I'm using FURL and never ever switched back to showtopic :blink:

where did this come from?

http://tabee3i.com/index.php?showtopic=729

[stoo2000]

[quote name='Enkidu' date='26 November 2009 - 12:47 AM' timestamp='1259196460' post='1882513']
I'm getting a pile of re-direct errors and they all have this format:



Now where did this showtopic come from?

I'm using FURL and never ever switched back to showtopic :blink:

where did this come from?

The task uses built in IP.B functions to come up with the FURL, do you have any other SEO mods installed ?

http://tabee3i.com/index.php?showtopic=729