Clover13 Posted August 10, 2023 Posted August 10, 2023 4.7.12 I run two sites on the same server that are nearly identical installs, both of which I've had my host evaluate for errors/issues during this period of unresponsiveness. Neither site indicated any issues at the host level (i.e. errors, resource issues, etc). There were no recent updates to either site that I feel would have influenced this behavior. Only one site had issues with unresponsiveness described below. No alerts were received. The unresponsiveness period was nearly 12 hours (I was away from cell reception and had no idea the site was in this state until I returned to find no activity since users couldn't post). Symptoms observed: All users experienced the same symptoms, so it wasn't isolated to one device or one set of devices. It was definitively a site related issue. Main site would load and present the landing page (activity stream) Main site buttons wouldn't work Main site menu wouldn't open Admin CP would load and present the landing page (had to manually type in URL since main site menu wouldn't open) Admin CP menu would not open Admin CP Support page would open but all of the checks would just show the status check spinning indefinitely Fix: I was able to clear the cache with the link on the AdminCP Support page and that seemed to resolve the issue. Simple fix, but I still have no idea what happened or why. I don't see any errors in the IPS error or system logs pointing to any issues at this time. Anyone have any ideas or experienced something similar before? I've never seen this occur before, and I've been using IPS for 10+ years now.
Marc Posted August 10, 2023 Posted August 10, 2023 I've not seen that happen, but it something not really possible to investigate if there are no error logs and the issue is no longer present. I would advise if you see it happen again, let us know so we can investigate. The most likely cause in situations like this, is an inability to write something to disk at the time. It being with both sites, it seems very highly unlikely to be the software, and more likely to be environment. After all, the chances of the same thing happening on both sites at the very same time is very unlikely.
Jim M Posted August 10, 2023 Posted August 10, 2023 Building off of what Marc said, it sounds like the JavaScript may not have written correctly or retrieved correctly from your storage and broke most of these items which require it to function. Then once that temporary issue was resolved on your storage, the issue went away.
Clover13 Posted August 10, 2023 Author Posted August 10, 2023 6 minutes ago, Marc Stridgen said: I've not seen that happen, but it something not really possible to investigate if there are no error logs and the issue is no longer present. I would advise if you see it happen again, let us know so we can investigate. The most likely cause in situations like this, is an inability to write something to disk at the time. It being with both sites, it seems very highly unlikely to be the software, and more likely to be environment. After all, the chances of the same thing happening on both sites at the very same time is very unlikely. Just to be clear, it only happened on 1 site despite both being on the same host server. That is what lead me to believe it was software sided rather than server sided. If it had been both sites, I would have thought server related. However, it could have been something between the server and clients for that one site that was a problem. 1 minute ago, Jim M said: Building off of what Marc said, it sounds like the JavaScript may not have written correctly or retrieved correctly from your storage and broke most of these items which require it to function. Then once that temporary issue was resolved on your storage, the issue went away. Agree, my sense was something became corrupted and stuck as a result. Clearing the IPS cache seemed to resolve it, just no idea how it got in that state initially for all users and only for one of the sites. The other aspect just recalled that is unique to this site, is that it is CDNing the Theme Resources (File Storage). I wonder if something with CF got messed up that caused it, although I did not clear the CF cache at all, only the IPS cache. I am not sure what that operation does exactly, so I can't comment on what it clears and the corresponding repopulation of the cache. How exactly does the IPS cache clearing work with File Storage such as Theme Resources that are on a CDN (such as CF)? I will keep an eye on things and let you guys know if it happens again so you can review. Given the time of discovery (9PM EDT) and 12 hour outage, I was just working on getting the site back to operational at that point.
Jim M Posted August 10, 2023 Posted August 10, 2023 Each time you clear the Invision Community cache, it rebuilds dynamic theme items like CSS and JavaScript. If there was an issue with your CDN or writing to your storage, the clear cache would rewrite the files on your storage and attribute a new cache busting URL to the CSS and JavaScript so that you CDN would need to re-cache the files as well. Clover13 1
Clover13 Posted August 12, 2023 Author Posted August 12, 2023 So it looks like this came back up, but now on my other site. It's currently happening. Tried two different browsers and Incognito/private modes. I have not cleared the IPS cache yet. These theme files are local to the server, they are not CDNed. FTPing to the server shows these files are indeed where they should be. From Chrome: GET https://www.domainxyz.com/uploads/themes/javascript_tthumb/front_front_tthumb.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3224 GET https://www.domainxyz.com/uploads/themes/javascript_global/root_js_lang_1.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3226 GET https://www.domainxyz.com/uploads/themes/javascript_core/global_global_core.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3223 GET https://www.domainxyz.com/uploads/themes/javascript_global/root_library.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3228 GET https://www.domainxyz.com/uploads/themes/javascript_global/root_front.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3235 GET https://www.domainxyz.com/uploads/themes/javascript_core/front_app.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3225 GET https://www.domainxyz.com/uploads/themes/javascript_global/root_framework.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3232 GET https://www.domainxyz.com/uploads/themes/javascript_core/front_front_statuses.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3237 GET https://www.domainxyz.com/uploads/themes/javascript_rsvpevents/front_front_widget.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3231 GET https://www.domainxyz.com/uploads/themes/javascript_core/front_front_streams.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3230 GET https://www.domainxyz.com/uploads/themes/javascript_core/front_front_widgets.js?v=066306acec1691876423 net::ERR_ABORTED 404 www.domainxyz.com/:3229 GET https://www.domainxyz.com/uploads/themes/javascript_core/front_front_core.js?v=066306acec1691876423 net::ERR_ABORTED 404 jquery-ui.js?v=066306acec1691876423:4 Uncaught ReferenceError: jQuery is not defined at jquery-ui.js?v=066306acec1691876423:4:157 at jquery-ui.js?v=066306acec1691876423:4:165 (anonymous) @ jquery-ui.js?v=066306acec1691876423:4 (anonymous) @ jquery-ui.js?v=066306acec1691876423:4 jquery-touchpunch.js?v=066306acec1691876423:10 Uncaught ReferenceError: jQuery is not defined at jquery-touchpunch.js?v=066306acec1691876423:10:1155 (anonymous) @ jquery-touchpunch.js?v=066306acec1691876423:10 (index):3240 Uncaught ReferenceError: ips is not defined at (index):3240:4 (anonymous) @ (index):3240 From the Safari console: [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (root_library.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (front_front_core.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (front_front_tthumb.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (global_global_core.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (root_js_lang_1.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (root_front.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (front_app.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (front_front_statuses.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (root_framework.js, line 0) [Error] Failed to load resource: the server responded with a status of 404 (Not Found) (front_front_streams.js, line 0) [Error] ReferenceError: Can't find variable: jQuery (anonymous function) (jquery-ui.js:4:164) Global Code (jquery-ui.js:4:166) [Error] ReferenceError: Can't find variable: jQuery Global Code (jquery-touchpunch.js:10:1162) [Error] ReferenceError: Can't find variable: ips Global Code (www.domainxyz.com:2659)
Clover13 Posted August 12, 2023 Author Posted August 12, 2023 I'm not going to leave it like this for long if I can resolve it by clearing the cache. Hoping the above information is helpful enough to determine what may be going on or where to look.
Clover13 Posted August 12, 2023 Author Posted August 12, 2023 OK, nevermind, this was one is on me. I must have accidentally moved the files and my FTP client had cached versions showing, so they weren't where they should be. Working through resolving getting things where they need to be.
Recommended Posts