Managing lots of app resources

[Aiwa]

I'm working with a 3rd party data set, bf4stats.com to be specific.  Their API returns paths to the appropriate files in their image pack.  The image pack is quite large.  I'm able to pair it down to about 17M, but I need all 17M of those images.  

The app I'm writing is starting with BF4, but more games will be added that will have image packs just as large.  So when the app supports 3 games, that'll be roughly 51M worth of image resources.  Servers simply aren't going to be able to handle importing that, and the practice of using the /resource folder is not going to be sustainable. 

CiC doesn't offer FTP, so as far as I'm concerned, that's off the table.  The only other option I see is to release the images as a separate resource pack to upload as theme resources for each game... EXCEPT that feature currently only accepts a single file at a time up to 1.2M, not even close to being helpful here.

What are my options?  Any other ideas to facilitate this? 

[Marcher Technologies]

This might sound a bit insane, but.... is there any chance you can get away with NOT importing them to the install and just referencing them? It's sounding like it's actually more wasteful to import them instead of reference them, on the surface, due to mass duplication of a large set of files, unless the source in question blocks hotlinking.

(correct me if incorrect, but it sounds like 17mb x theme count, per install, being duplicated, which seems a bit extreme if the images are freely linkable.)

[Aiwa]

With Steam, the images are freely linkable and supplied over the Steam CDN, and I make use of that.  I have no resources in my Steam app.  With Battlefield they are not freely linkable, hence the image pack.  

Like I said, necessary for my application, just not a sustainable practice.  I've debated moving them to the /interface folder, which should help me in the short term, but it won't help me in the long run, unless I write my own import tool to import image packs for games and put them in the interface folder.  But I may run into permissions/server issues there.

This was easy in 3.x, it was all FTP upload.  But this, everything must import via the ACP, is no where near as flexible for image heavy applications.

[newbie LAC]

http://files.bf4stats.com/img/bf4/%folder%/%image%

Examples

[Aiwa]

23 minutes ago, newbie LAC said:

http://files.bf4stats.com/img/bf4/%folder%/%image%

Examples

EDIT: never mind... Found it.  It does appear this is for public use.

[bfarber]

I was going to suggest, if the resources aren't public from the vendor, you could perhaps upload them to a CDN and link them in your app to your own CDN. The downside being, you will incur costs as a result, but you wouldn't have to distribute the files.

[Aiwa]

37 minutes ago, bfarber said:

I was going to suggest, if the resources aren't public from the vendor, you could perhaps upload them to a CDN and link them in your app to your own CDN. The downside being, you will incur costs as a result, but you wouldn't have to distribute the files.

Yeah, I looked into that. Data storage space wouldn't be much, but the # of requests and data transfer would incur a not insignificant cost. 

To make that cost effective if have to heavily leverage either browser caching or some other cache method. 

The public endpoint is only http, I'm adding a fallback to use local images if the client wants https, but FTP would be required.