@CoffeeCakeOnce again, Cloudflare is not the problem. Downloads serve just fine over Cloudflare from the same webserver as long as IPS is not serving the download.
Additionally, Cloudflare does not cache file downloads like this. They cache small static resources not large zip archives. The file in question is in excess of 200mb and another is in excess of 1.2gb.
Either way, I did already confirm Cloudflare was serving the file just fine outside of IPS if you read into my original messages.
This is one of the first things I verified and should be the first thing anyone verifies when using Cloudflare or any similar CDN service.
I actually solved the problem with IPS being completely unhelpful. I received a follow-up to my ticket that I decided to ignore. They effectively told me, go ahead and provide us access to the site, and then we can escalate your issue for a chance that our higher tier support can investigate it. They couldn't even tell me that they absolutely would work with me on the issue, just that there was a chance that they may work with me, only after I provide all of my site details, because higher tier "needs to decide how to proceed". This was after me pressing for them to support me and their software without me being required to disable Cloudflare.
Here is the last response I received from IPS regarding this issue:
<Screenshot removed by Matt M: Let's not identify and single out single staff members for company policy, thanks>
Yes, IPS still absolutely refused to help unless I fully disabled Cloudflare. Fat chance as my site during this period was getting DDoS multiple times per day. It thankfully has since slowed.
These are just the ones left in my trash:
Sorry, but I shouldn't have to beg for proper support. Not wasting my time with exposing my site over a chance to be helped. IPS should support their product without giving customers the run-around, but I digress. I ignored the response and eventually solved it myself. IPS support is probably fine for basic questions but good luck on anything truly technical.
Solution
It is isolated to the IPS software as I expected. The cause of the problem is my own doing, but it is something IPS may want to address in future versions or at least be sure to clearly document to avoid others wasting their time.
Prior to v4.5.4.2 or possibly a few versions before, IPS did not care about file size as shown in the download system. IPS would serve the raw file from disk and the download would complete as expected.
Sometime after v4.2.x, IPS started serving download size headers according to the file size stored within IPS. The software no longer reads the size from disk and instead serves its last known size saved in the database, stored when the file is uploaded or updated via the IPS interface.
This results in a premature end of download if the size in IPS is smaller than the size of the file on disk. This now made sense as to why we were seeing an unexpected end of file without corrupted headers or data, but the browser would report as a successful download because received the full size as reported by the headers.
This was noticed because I am using dev pipeline automation to automatically update specific downloads we offer through our website. This allows us as developers to deploy changes to a repository, then automatically deploy our updates over to the production download link once we deem an update is ready, without having to mess with IPS and reupload the file each time. Pipelines handles all of this for us on a merge to master branch.
Of course over time as updates are added, file size will change, usually increasing. This caused a file size skew but we ignored it because we tested and verified downloads were working okay regardless of size in IPS versus size on disk.
Sometime between v4.2.x and v4.5.4.2, IPS internal logic was updated to force downloads to the file size stored in IPS. This triggered our sudden unexplained post-update download problems. I only noticed the issue after seeing the resulting download size matched what is reported in IPS. I manually adjusted the file size stored in IPS, and the exact same file, on the exact same webserver, over the exact same Cloudflare configuration, just now served by IPS instead of a raw link, completed the next download successfully with no other changes, no disabling of Cloudflare, and no clearing of any cache either within IPS or at Cloudflare.
Naturally I have still ditched the download system and am just serving direct links for these files now, as IPS gives no mechanism to easily update the size of a file. I would need to create a small API to update the size of the file in the database so that IPS knows the correct header size to send. I've adjusted our thought process on how we provide downloads and I no longer need the internal download system.
This will affect anyone using modern DevOps to deploy updated file downloads powered by IPS. Hopefully IPS can either revert this behavior or provide a mechanism for developers in our situation to more easily update the size of a file, such as a webhook we can call or something similar that we can trigger through command line calls. Absolute worst case scenario, at least give a visible warning somewhere that modification of files on disk can cause download problems and to avoid this in general.
This thread can be marked as resolved. I've also followed up on my ticket to provide IPS with the solution in hopes they will see and address in some way in future versions. I definitely do not look forward to my next interaction with them, and I do not have my hopes up considering we are still waiting for that updated public documentation they promised years ago.
TL;DR: IPS fixed a sensible bug that caused a new bug with my non-standard way of working with their download system. The problem is I could not find this documented in update notes, and IPS team was completely unhelpful without demanding me to expose my site, and could not give me any solid answers on possibility of assisting me without exposing my site. They passed the blame to CDN services without understanding the actual problem.
