Large community? You have a problems with sitemap!

[bfarber]

What errors is that link causing?

I wouldn't nofollow internal embeds (those are essentially links to other pages on your site which you absolutely do want to be followed and page rank passed to), however if you visit that link and view the page source, a canonical tag is already set to the real topic URL, so Google should be able to follow that link and index it, but point back to the canonical URL.

[sadams101]

For links that end with ?do=embed google is indexing them as:

Crawl anomaly

Status: Excluded

 

For the other link I was asking about, it appears that /members is no longer used at all, and all the old links in google's index need a 301 redirect, but again, I am still trying to verify this!

[Ocean West]

robots.txt   yes or no ?

[bfarber]

On 2/16/2018 at 7:31 PM, sadams101 said:

For the other link I was asking about, it appears that /members is no longer used at all, and all the old links in google's index need a 301 redirect, but again, I am still trying to verify this!

We do not natively use /members in 4.x, so you can redirect it.

[sadams101]

But you see...this is the issue. When you stop using these directories you MUST build in 301 redirects...you can't simply make it so that all of my indexed links are left broken. Rather than me pointing them to my home page, which is definitely not ideal, they should redirect to the proper members is possible.

Here are more issues with all of my blogs:

http://www.celiac.com/gluten-free/blog/164/entry-1580-get-over-yourself-rachael-ray/

is not properly redirecting to:

https://www.celiac.com/gluten-free/blogs/entry/1580-get-over-yourself-rachael-ray/

Again, it looks like you've made some changes to directories, without redirects. Does anyone know how to write a 301 redirect that will change /blog/*/ to /blogs/entry/  ??

I suspect these changes occurred around June of last year, no?

Also, you may notice the blog above, and many others include your markup code. Why is that? Lots of my site's older posts and blogs show this code.

Edited February 21, 2018 by sadams101

[bfarber]

We redirect most old URLs to their new location, however there is no 'member list' in 4.x so there is no proper location to redirect that old URL to. Subsequently, a 404 error response is the best response. It should be noted that not every link should be redirected somewhere when it is no longer valid. Many times, it is appropriate to return a 404 error and simply let Google shed that URL from its index.

I will take a look at the old blog URLs there. That may have been an oversight.

[Numbered]

On 21.02.2018 at 9:02 PM, sadams101 said:

Here are more issues with all of my blogs:

http://www.celiac.com/gluten-free/blog/164/entry-1580-get-over-yourself-rachael-ray/

is not properly redirecting to:

https://www.celiac.com/gluten-free/blogs/entry/1580-get-over-yourself-rachael-ray/

Again, it looks like you've made some changes to directories, without redirects. Does anyone know how to write a 301 redirect that will change /blog/*/ to /blogs/entry/  ??

This will show proper blog on your bad url address (better than get 404):

 

Of course, much better rewrite with 301. It can be done with your web server or some hook some file. If i need to do this i'll hook \IPS\Session\Front, read method. Because we need to redirect your ulr to correct place with 301 before something will start working in main engine. Something like that:

{i deleted it}

P.S. Need to rename this topic to "Every SEO Q/A"  

Edited February 22, 2018 by Upgradeovec

[Daniel F]

10 minutes ago, Upgradeovec said:

This will show proper blog on your bad url address (better than get 404):

 

Of course, much better rewrite with 301. It can be done with your web server or some hook some file. If i need to do this i'll hook \IPS\Session\Front, read method. Because we need to redirect your ulr to correct place with 301 before something will start working in main engine. Something like that:

public function read( $sessionId )
{
    $url = \IPS\Request::i()->url()->data['path'];
    if (mb_strpos($url, '/blog/') === 0) {
        preg_match("/\/blog\/(\d+)?\/.+/", $url, $res);
        \IPS\Output::i()->redirect(\IPS\Http\Url::internal('app=blog&module=blogs&controller=entry&id='.$res[1], 'front'));
    }
    return call_user_func_array( 'parent::read', func_get_args() );
}

or the same in ready plugin file Blogs Redirection.xml

 

P.S. Need to rename this topic to "Every SEO Q/A"  

NOPE, please not! For the record, I'm not talking about your use case, only about your hook... Just a minor suggestion, please use  IPS\Application: convertLegacyParameters() instead of overriding the session class. This would be the "official" place for such redirect, this definitely doesn't belong into the session object!

You can see some examples in the core or forums app:)

 

[sadams101]

Hi Daniel F,

Instead of explaining this in a way that only developers might understand, could you please put this out in a way that I can understand?

I just need a 301 redirect here. I also am curious about why so many of my older posts and blogs display your markup language? Example:

https://www.celiac.com/gluten-free/blogs/entry/1580-get-over-yourself-rachael-ray/

Edited February 22, 2018 by sadams101

[bfarber]

3 hours ago, sadams101 said:

Hi Daniel F,

Instead of explaining this in a way that only developers might understand, could you please put this out in a way that I can understand?

I just need a 301 redirect here. I also am curious about why so many of my older posts and blogs display your markup language? Example:

https://www.celiac.com/gluten-free/blogs/entry/1580-get-over-yourself-rachael-ray/

I logged a bug report about the issue you raised earlier and @Ryan Ashbrook couldn't reproduce. Please submit a ticket so we can take a look. Redirects for first-party applications (that have appropriate destinations to redirect to) should be handled out of the box without plugins.

[Numbered]

4 hours ago, Daniel F said:

NOPE, please not! For the record, I'm not talking about your use case, only about your hook... Just a minor suggestion, please use  IPS\Application: convertLegacyParameters() instead of overriding the session class. This would be the "official" place for such redirect, this definitely doesn't belong into the session object!

You can see some examples in the core or forums app:)

Sorry sorry -) I just want to make a main way for do needed stuff. And session class just as example . And i didn't know about convertLegacyParameters() method before. Thanks)

I hook session class read method before for working with sso stuff. Totally agree with bad place for that )

upd. deleted my code for prevent bad using it for others.. but it still saved in your quoting of my post.. 

Edited February 22, 2018 by Upgradeovec

[Ryan Ashbrook]

4 hours ago, sadams101 said:

Hi Daniel F,

Instead of explaining this in a way that only developers might understand, could you please put this out in a way that I can understand?

I just need a 301 redirect here. I also am curious about why so many of my older posts and blogs display your markup language? Example:

https://www.celiac.com/gluten-free/blogs/entry/1580-get-over-yourself-rachael-ray/

 

1 hour ago, bfarber said:

I logged a bug report about the issue you raised earlier and @Ryan Ashbrook couldn't reproduce. Please submit a ticket so we can take a look. Redirects for first-party applications (that have appropriate destinations to redirect to) should be handled out of the box without plugins.

Correct - we do have an alias for the previous URL structure to properly redirect, so we'll need to take a look at your site specifically to see why this is not working.

[sadams101]

I created a support ticket.

[sadams101]

I will post what I find here, but in my case, the redirects may not be working properly because I do not have IPB in my root directory.

Edited February 22, 2018 by sadams101
Wrong forum!

[sadams101]

Just in case anyone else is having the blog redirect issue, one work around if you have access to your .htaccess file is this 301 redirect created by Upgradeovec:

Quote

RedirectMatch 301 ^/gluten-free/blog/.*?/entry-(.+) https://www.yourdomain.com/blogs/entry/$1

I am still hoping that there is a better answer and will post it here if I find it.

[CSSlife]

Having issues with indexing also. Switched to IPS from XF little over a week ago, went from having pages indexed within minutes to having none indexed whatsoever in the last 24 hours+. Something is up.

[AlexWebsites]

14 hours ago, CSSlife said:

Having issues with indexing also. Switched to IPS from XF little over a week ago, went from having pages indexed within minutes to having none indexed whatsoever in the last 24 hours+. Something is up.

I think IPS regenerates the sitemap once daily and then you need to wait for indexing within 24 hours. Not sure about zenforo, but when I was using vbulletin on one of my sites I moved over to IPS it was more immediate as well, which I think was because it pinged google on new topic creation. 

[AlexWebsites]

On 2/2/2018 at 11:44 AM, Matt said:

10 days might be fine depending on how often Google visits your site.

Again, the frequency that Google visits your site has nothing to do with the sitemap.

In 4.3, we have added the lastmod timestamp, and added a button to rebuild your index from scratch.

Also, just double check your forum and topic permissions. Remember, if a guest cannot see the page, then Google cannot either.

Did you guys add the timestamp to 4.2.8 released today by chance?

[AlexWebsites]

On 1/9/2018 at 3:18 AM, Upgradeovec said:

Did it.

Before:

After:

No issues detected by several sitemap online checking tools:

I did it very ugly. Just for try and check. You can improve it by yourself (and share it with us, please):

/applications/core/extensions/core/Sitemap/Content.php

line 209: after $data line add that:

if (get_class($node) === 'IPS\forums\Forum' && isset($node->last_post)) {
    $data['lastmod'] = $node->last_post;
}

and line 259 (line 262 after add previous) add after $data line that:

if (get_class($item) === 'IPS\forums\Topic' && isset($item->last_post)) {
    $data['lastmod'] = $item->last_post;
}

After that the sitemap script should re-generate all sub-sitemaps for write new data to db.

And I haven't done changing correct lastmod in index sitemap, depended on newer date inside sub-sitemap.

Thanks.

@Upgradeovec its been about 2 months since this post, has adding this helped with indexing?

[CSSlife]

2 hours ago, AlexWebsites said:

@Upgradeovec its been about 2 months since this post, has adding this helped with indexing?

curious to know this also

[Numbered]

Google Search Console didn't show changes on graphs. But I copy some comment, posted 8 hours ago - it present in google and shows link to the source well. Ok for me. Before previous things i can't find content, posted a lot of months ago.

But one more interesting thing - i added robots.txt 2018-03-03 and graph 'index count' boost x3.. Just because i block some links, which google tried to get, but get error or get content-less page. This errors may worse indexing of correct links. May be not, who knows .

[Midnight Modding]

Off topic, but I was on a site a few days ago that has nearly 50 million posts! I would have been happy if I could ever get a forum with 100 posts per month. That's how pathetic mine were. lol.

Edited March 15, 2018 by Midnight Modding

[mark007]

There are a few problems in indexing pages:

https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/?page=0

is the same as

https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/?page=1

and the same as

https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/

 

[mark007]

There should be an redirect ...

Right now a robots.txt like this does make sense:

Disallow: /.../topic/*?do=findComment&comment=*
Disallow: /.../profile/
Disallow: /.../topic/*tab=comments*
Disallow: /.../*?page=1$
Disallow: /.../*?page=0$

 

[PPlanet]

I have a doubt regarding the sitemap and members profiles. I have blocked search engines from accessing members profiles.

Google sees this as an HTTP Error: 403 and keeps warning me about it.

In a way it makes sense, but I suppose that Google warns me because it sees them in the sitemap yet can't access them.

Now what it's blocking Google to access profiles is a third party app. It's not blocked under System > Profiles

I wonder if I deselect permission there if they'll stop trying to access and therefore they stop warning me too.