sadams101

In the last few months I've noticed a sharp uptick in my spam reports from Comcast and other large email providers. When I view the spam reports, 90%+ of them contain this auto-response:
The number of these reports has been steadily increasing over time, so the risk is that my site/domain/server will be blocked from sending email to many large providers like Comcast.
What is clearly happening here is that bots are trying to post, and are inserting either random email addresses from a spam list, or simply generating fake email addresses. 
They are being reported as spam by those who receive them because they are spam...these emails were not generated by the users who are reporting them as spam, they are created by bots who are simply using their emails in an automated attempt to spam using my forum.
Due to this I've turned off the Post Before Registering feature in the admin CP, and recommend that everyone else do the same, unless they want to be blocked from sending email to large email providers.

I had @DawPi design a plugin that does exactly this, and it also handles forum posts that have been archived. So if you have archived posts, rather than all links to those old posts being gone and no longer available from the poster's profile, there is an "Archives" tab that includes links to all archived posts.
Here is how that looks:
https://www.celiac.com/profile/2-scott-adams/

I had @DawPi design a plugin that does exactly this, and it also handles forum posts that have been archived. So if you have archived posts, rather than all links to those old posts being gone and no longer available from the poster's profile, there is an "Archives" tab that includes links to all archived posts.
Here is how that looks:
https://www.celiac.com/profile/2-scott-adams/

I am having an issue where the "cache" reading in Redis is zero...see below:

 

It looks like I am allocating too much memory to Redis. Here is a medium busy time:
Redis 3.2.12    76.29M of 10.00G memory used
99309 keys stored in db0
Any ideas as to how much I should allocate? 1G ?

I think so, I am using a cPanel server with Centos 7.8. Unfortunately they still use Redis server v=3.2.12, which may explain why I'm not seeing data in the "Cache" part. It works, but it my not be working as well as it should. Upgrading my version of Redis is not advisable.

To flush the Redis cache the command line Apache server command is:
redis-cli FLUSHALL I have command line access to my server, I'm not sure if you do.
To restart on command line use one of these two:
systemctl restart redis service redis restart  
This was also very helpful:
https://redis.io/topics/admin

Some speed issues I've pointed out at more than 2 years ago, both on this forum and in support tickets, are still present. This is why I've gone off on my own path and have worked with people who have overcome most of them. I was a solid 80 on mobile before this new update. This new JS requirement will be harder to deal with, but I'm working with a programmer now to see what can be done. If I find anything out I'll post it here.

I am still running redis, but have turned off the "Cache page output for guests" feature (mine was set to 15 minutes) due to a bug with it that still exists. My site speed consultant @Adlago has also experienced the bug, and warned me not to use this feature. What happens is that the cache gets corrupted, for example it will cache a desktop version of the site with zero CSS, so the user experience is horrible, at least for that 15 minutes.
I was hoping the bug was fixed, but while I was running some speed tests I found proof that it still exists. In screen 1 is a normal speed test, and if you look at the test there is a screen shot of the page as it should look, and in screen 2 if you look at the screen shot it took of my site you will see a view of it without any CSS. Clearing my cache in the ACP: Support > Something isn't working right, made this go back to normal, and verify that it is indeed an issue.
Although it did speed my site up by ~10%, it is not worth the trade off. It's sad that a bug like this, one that I reported to IPB years ago, still makes this feature unusable.
Screen 1 speed test with normal screen shot view:

 
Screen 2 of a test on the exact same page, but in this one there is no CSS that got cached, and the screenshot reflects this:

PS - If google's spider is indexing a site without CSS, those pages are excluded from search results. Lot's of pages on your site could get indexed while this issue is occurring.

@AlexWebsites no worries, even if your cached pages don't get used much it won't hurt anything. I guess the question is what gets cached? My understanding is that the whole page is cached, including dynamic content like posts. I could be wrong. Perhaps only the stuff that is duplicated over and over like blocks, widgets, header and footer get cached. Either way 15 minutes is fine with me for guests.
Anyway, now I am trying to understand why my "cache" in Redis is zero. I see @jair101
 has content cached under "cache". Any ideas? Are there certain settings I need to enable to make full use of this?
My community has over 900K posts and over 500 active users during peak times.

I don't use Nginx or a cloud server, but why bother even caching it if you only cache it for 1 minute, or 30 seconds? What is the point? The cache output for guests is to reduce resources. I assume the cached pages are created, for example, when bots crawl pages, or user do. When other request a page that has been visited in the last 15 minutes, in my case, they will get a cached version. I this is set to only 1 minute, the likely hood of serving a cached page is pretty small I think.

So I've had a new application created by @DawPi which does the following (I've always thought that the profile page is set up wrong, especially because I have about 5,000+ articles, many by doctors and other authors, and you can't find their About Me without fishing around for it--the about me tab is nearly impossible to find in the mobile format):
1) There is an option to add "noindex, follow" meta tags on profiles where all of the poster's content has been archived;
2) If there is About Me content, there is an option to display that content above the user's Activity Stream, and NOT show the About Me tab. It also removes the About Me tab IF there is no About Me content. You can see this here:
https://www.celiac.com/profile/2-scott-adams/?tab=activity
3) If the user has posts that have been archived, a new "Achives" tab will appear on their profile, and when you click it all of their archived posts can be found (currently they are detached from the user forever, and cannot be linked to the user).

I also have Redis Cache = 0.
I made ticket support to my host company. They did an analysis and here's what they answered:
======
" We have reviewed your request and confirm that Redis is being used effectively. We cannot confirm the reason for the administrative part of the site to write "Cache: 0", but we assume that this module sends a request to one of the bases available in redis. The keys and their values are distributed in different bases, such as they are stored, as you can see in the screenshot - many sessions.
We checked the information directly in the redis port for your user, successfully accumulating objects, and we see that even the cache hit rate is high enough for MISSs:
keyspace_hits: 147352670
keyspace_misses: 3603491
In conclusion - we see that the system correctly uses Redis and from the point of view of the hosting environment there is no obstacle to its optimal use. "
====
I think these analyzes confirm what @bfarber said, ie. no issue with redis cache = 0.
 

My ACP view with Redis on looks like this...not sure why cache says 0

 
My settings are like this:

In my tests on a fairly busy live and non-private version of IPB Redis performs worse than memcached. I recently switched to Redis though, because I've been informed by IPB's tech support that they will soon be eliminating memcached in IPB, as they believe Redis outpeforms it, and all other options (again, I've not experienced this). Have you tried memcached?

I think it's a good idea to be more specific, and certainly adding "sponsored" is very important to be able to do, so that google knows a link is sponsored. Now it can be "sponsored" without "nofollow" and still be fine with google. 
For all my forum links and comment links I decided this was best because it is more specific: "external ugc nofollow."
Google indicates on that page that forums, for example, can use "ugc" to indicate that it is user generated, which is different than just nofollow, which doesn't tell you anything (it could be nofollow because it is sponsored, so now google cannot tell whether it is sponsored or flagged nofollow because it is UGC).
Interestingly google also says that for "trusted" users, for example old forum members who regularly post good links it is ok to remove the "nofollow" on them. So there really should be a setting where you can set the default for all users, but then go in and change them for selected users, or perhaps even make it tied to advancing to a certain group, for example those who have over 100 posts will have the "nofollow" removed.
Just some ideas, but what is clear is that google cannot now tell whether a link is nofollow because it is sponsored, or just UGC, and it is important to tell google this. By adding the UGC tag I've done that.

I've had constant TTFB issues with IPB4 which I've never been able to solve using any caching method, and I've tried them all. Although I am not a coder, I have suspected that there is a delay somewhere in how your code loads, and this delay can be replicated on other boards I've tested. I certainly hope you can look into this further, given the huge effect it has on everyone who uses the software.

I've had constant TTFB issues with IPB4 which I've never been able to solve using any caching method, and I've tried them all. Although I am not a coder, I have suspected that there is a delay somewhere in how your code loads, and this delay can be replicated on other boards I've tested. I certainly hope you can look into this further, given the huge effect it has on everyone who uses the software.

I wanted to post an updated on my custom posts sitemap, as there are various opinions here about whether or not it can help or hurt an IPS site. In my case I have nearly 1M posts, and none of their URL's were in the sitemap directly. Now, it appears that google has indeed indexed nearly all of them, and I do not see any issue regarding the canonical links for them on the forum category pages. Currently I see zero errors, and now only 71.5K are excluded, and zero are listed in the "Duplicate, submitted URL not selected as canonical" line.
Also, my site traffic has been slowly recovering...in the last 30 days I've seen a 17% increase over the prior 30 days. So far I have seen nothing negative about using it, and I've been running it for 6 months now. During the time I've been running it I've seen slow and steady improvement in my page rank, and more importantly, the number of key words in google's index for my site has increased dramatically (it hit an all-time low around 6 months ago--coincidence?).
 

PS - The forum's category pages, beyond the initial category page, are also not in the sitemap in both the forums and pages apps, and we've also created site maps to include all pages, for example /page/2/, /page/3/, /page/4/.

I wanted to follow up with something interesting. As you may recall I had some custom sitemaps developed which include a map to every post made, and a map for all comments.
After submitting the maps I saw the "Discovered - currently not indexed" shoot up to ~923K, and it just stayed there for months. Most of the URL's there were the individual post links.
Last month I saw a big change--a fast drop in Discovered - currently not indexed, as you can see below. It dropped to 320K. As this started happening I also saw my organic traffic, and the number of key words indexed increase.

We've fixed this issue, which also existed in the posts sitemap...now all is fine, thank you!

I've also added @DawPi's new article comment sitemap:
https://www.celiac.com/sitemap_comments.php
Unless I've missed something I believe I now have ALL content in my site mapped in agonizing detail! 😅
 

So the profile's do exist in the default sitemap, and there are various settings for the profiles in the system, so doing another sitemap for them was unnecessary (I did add the Profile Meta Tags plugin, because profiles did not include that data by default). Given how important EAT is in google's ranking now, the profiles in IPS seem to have, at least until recently, been sorely neglected...especially for sites like mine that use Pages and have thousands of articles by authors with high EAT authority. 
I am currently using the mentioned custom posts sitemap:
https://www.celiac.com/sitemap_posts.php
and custom extra pages sitemap, so that posts on pages like this one, #11 in this thread, will be in my sitemap (they are not currently for an unknown reason):
https://www.celiac.com/sitemap_pages.php
and I will soon be launching a new article comments custom sitemap, that will basically do the same as the posts sitemap, but for comments on my articles.
So far the improvement I am seeing has been across the board, so an increase in natural search traffic, total number of keywords for which my site ranks, etc. 
Of course this could all be just coincidence...has anyone else here who has complained about having a high rate of "Crawled but not in google's index" seen any improvement like I've demonstrated? If so, please share it here.

You guys do what you like...there is no sitemap for posts--there should be. If all sitemaps are set up correctly, Google only crawls everything in the maps once, then only crawls the new content when it sees the lastmod date change. There is no finite crawl budget for my site, this is nonsense.
PS - If there were such a think as a crawl budget, certainly not putting a lastmod date in the sitemap would cause google's spider to waste a ton of time trying to find new content...IPB currently has this issue if you're not running 4.4.