Submitted URL has crawl issue / Failed: Crawl anomaly

[sadams101]

@bfarber your description is correct, so the non-FURL is showing up, then redirecting to the FURL. Unfortunately for me this is happening on every article page in my Pages app, as I allow guests to comment. You can view source and search for:

?app=core&module=system&controller=login

For example here:

https://www.celiac.com/articles.html/17-of-celiac-disease-relatives-also-affected-r680/

and here:

https://www.celiac.com/articles.html/a-journey-through-the-nexvax2-celiac-disease-vaccine-clinical-trial-r4753/

Unfortunately this issue may be having a significant effect in google search, at least according to the forum post...crawl budget, etc..

[bfarber]

Thanks, looks like it might be a bug. I'll take a look for a future maintenance release.

[sadams101]

Another, possibly related issue, is that google apparently cannot crawl my ~130 categories in my Pages app because they are not in the sitemap. Is there a reason why Pages categories would not be in the sitemap?

[Sonya*]

Categories are included. If yours not, then it could be an issue with your website.

[sadams101]

I could not find my categories when doing a search in the sitemap via phpMyAdmin. Do you know if there is a separate setting somewhere for this? I did not see this in the sitemap settings in my ACP.

[Sonya*]

21 minutes ago, sadams101 said:

I could not find my categories when doing a search in the sitemap via phpMyAdmin. Do you know if there is a separate setting somewhere for this? I did not see this in the sitemap settings in my ACP.

No. There are no settings for categories. The only setting is to include databases (all of them) or not. I do not use recommended settings for the sitemap, so I can select what to include and what not. Do you use recommended setting for the sitemap?

[Adlago]

1 hour ago, sadams101 said:

I could not find my categories when doing a search in the sitemap via phpMyAdmin. Do you know if there is a separate setting somewhere for this? I did not see this in the sitemap settings in my ACP.

Do not search the database. Run your sitemap into the address bar ... https://your site /sitemap.php
and check what addresses are extracted in the xml file you will receivea

 

PS. See your database categories

https://www.celiac.com/sitemap.php?file=2_sitemap_database_categories

 

[sadams101]

Take a look here for that category...the root category is not there, only it's sub categories:

https://www.celiac.com/sitemap.php?file=2_sitemap_database_categories

and I mean this category:

https://www.celiac.com/articles.html/miscellaneous-information-on-celiac-disease/

and the same seems to be true for other categories...the root is not there if there is a sub category.

[Adlago]

9 minutes ago, sadams101 said:

https://www.celiac.com/articles.html/miscellaneous-information-on-celiac-disease/

From this category your site map retrieves 6 addresses

[Sonya*]

34 minutes ago, sadams101 said:

and I mean this category:

https://www.celiac.com/articles.html/miscellaneous-information-on-celiac-disease/

There is no need to index this URL. It does not contain any valuable content, it is just listing. Look into the source of the page:

<meta name="robots" content="noindex, follow">

It will not be indexed even if you add it manually.

[Adlago]

57 minutes ago, sadams101 said:

https://www.celiac.com/articles.html/miscellaneous-information-on-celiac-disease/

This address can index 167 addresses for you, see PM.
 

[sadams101]

@Sonya* this is being flagged by Semrush as an SEO issue for me, not something good. I have dozens of categories that used to rank very well in google which have now dropped out of the rankings completely. I don't need IPB to put such tags like noindex into my site for me, only the site owner should opt into that. I was never asked when I installed the IPB update that caused this, which was likely 4.4.4, whether I wanted to remove these from google's index. This should have been something where a choice was given.

[bfarber]

The category apparently only hosts subcategories, and not records, is that correct?

The problem appears to be that it is configured to allow records directly, but there are none there. When that happens, our table helper puts out a noindex, follow tag to tell search engines "there is no valuable content on this page, but please do follow the links, and please do return later to see if the status has changed" basically.

The easy solution would be to change the category setting "Show Records in Listing?" to no. This should resolve your concern.

[sadams101]

Perhaps provide us with a warning that you are de-indexing an entire category by selecting "yes" on a setting? Just a thought. Better yet, how about creating a setting where you have to check a box BEFORE a noindex tag is inserted into ANY page on my IPB site--so it would say "Check this box to de-index this category and insert a noindex tag"?

[sadams101]

But in any case, your solution did not work, I changed that setting as you recommended, deleted the site's cache using the support tool, and the tag is still there:

 <meta name="robots" content="noindex, follow">

 

 

[sadams101]

In the end I had to move an article under that category to make this tag go away. Now I will be paying to develop a plug in to stop such tag injections on my site, and allow me to control them. I would be grateful if you can share with us other pages where your software injects such tags...I am aware of the profiles without content, but where else is this happening?

[sadams101]

I just got this from IPB in response to a ticket on the noindex sitemap issue mentioned above:

Quote

I have submitted a change request for an upcoming maintenance release which will (1) ensure the parent categories are included in sitemaps and (2) stop the noindex <meta> robots tag from being output. Upon review by the rest of the development team, I anticipate at this time that the change should be incorporated into an upcoming 4.4.9 maintenance release.