4.1 - Ability to cache remote images

[chilihead]

27 minutes ago, Mike John said:

If the source is missing, it errors out like it would if the remote image itself didn't exist.

So wait this won't preserve images, say when users post using photobucket, and remove later, but you are indefinite proxies? Or did I misunderstand?

By source I meant remote, sorry I think I confused you.

[Michael.J]

4 minutes ago, chilihead said:

So wait this won't preserve images, say when users post using photobucket, and remove later, but you are indefinite proxies? Or did I misunderstand?

As long as the image exists at the time of posting it should always be available then locally and cached if you don't purge the cache.

[chilihead]

[sobrenome]

51 minutes ago, Mike John said:

Adjust the setting to "indefinitely" and that will disable the task that clears out the cache every X days.

Just a straight download of the remote image without any reductions or compression. It basically swaps out the remote image url when posting to point to the image proxy page and then load the images through there. On that page it checks if the url is present in the database and loads the cached version, if it's not, it caches it again and loads that image. If the source is missing, it errors out like it would if the remote image itself didn't exist.

That's bad. Really would like a feature not to proxy remote images, but to make them local, to avoid 404s in the future.

[chilihead]

From what I understand, that is essentially what it does.

[Michael.J]

10 minutes ago, sobrenome said:

That's bad. Really would like a feature not to proxy remote images, but to make them local, to avoid 404s in the future.

I think my use of "proxy" was confusing. It is caching remote images locally but the file that it serves it through is called imageproxy.php.

So for example you add a remote image in your post, lets say "remotedomain.com/image.png". That's swapped out to "yourdomain.com/applications/core/interface/imageproxy/imageproxy.php?img=remotedomain.com/image.png" and the image is now served through that link. Every time that link is accessed it checks for a local cached version and uses that instead. If no cached image exists it downloads a local copy again.

[sobrenome]

So if I set the cache not to expire it's the same as have a local image, right?

Is it a safe function?

Is there a chance to cached images to be deleted by IPS upgrade or AdminCP support function cache cleaner?

[chilihead]

On 12/7/2015 at 11:26 AM, Nathan Explosion said:

There is an 'indefinitely' option in the caching settings, so yes you can cache forever (the default is 7 days)

Additionally, the cache cleaning (which clears the 'datastore' folder) has no effect on these cached images (which are stored in 'uploads/imageproxy')

 

[Michael.J]

1 hour ago, sobrenome said:

So if I set the cache not to expire it's the same as have a local image, right?

Is it a safe function?

Is there a chance to cached images to be deleted by IPS upgrade or AdminCP support function cache cleaner?

Yes that's correct. When viewing the post it will appear as it's a local image and load locally from your server.

Well as far as I can see the image is checked internally that it's an image and not something else malicious. In theory an security updates wouldn't be retroactive to already cached and downloaded images but I you would have to confirm this with IPS.

I don't see how. The cached images are stored in a separate database table and folder. If your really paranoid, you could download your core_image_proxy database table and and uploads/imageproxy/ folder to ensure you have a backup before upgrading.

[sobrenome]

16 hours ago, Mike John said:

Yes that's correct. When viewing the post it will appear as it's a local image and load locally from your server.

Well as far as I can see the image is checked internally that it's an image and not something else malicious. In theory an security updates wouldn't be retroactive to already cached and downloaded images but I you would have to confirm this with IPS.

I don't see how. The cached images are stored in a separate database table and folder. If your really paranoid, you could download your core_image_proxy database table and and uploads/imageproxy/ folder to ensure you have a backup before upgrading.

Please, confirm with IPS. If you are right, this solution is perfect for my community!

[Carl Maltby]

Glad I came across this. Our site has over a decade of content also (same as yourself @sobrenome) which is mostly photo-heavy educational tutorials. Without the images, much of this very rich resource ends up instabroken. I agree with the above and understand the concerns raised about any intellectual property issues. My personal suggestions for image caching would be the same as that discussed, so we're going to move towards permanently caching also. It makes sense to allay link rot.

I genuinely think that the next minor release (4.2.x) should include some form of Moderation tool to allow site staff to do basic maintenance work on remote linking.

My proposal is for a tool that generates a report on content that includes remotely-linked content relying on locally-cached versions due to the remote source breaking.

That is, posts/threads/content that have become permanently (or at least, look like it) reliant on cached copies would benefit from an elective conversion tool. This tool would move the file out of the cache and into the post as a local attachment. This neatly sidesteps the issue of taking a member's control over (or responsibility for) attachments posted. They could easily go in and edit "their" content as easily as they could if it were a remote file.

A concern this raises is sites that return a default image in place of broken image URLs. ("we cannot find this photo, here's a cat!")
Without knowing more about it, I presume that the current caching system would happily replace the cached original versions of images with the default [photo of a cat] returned by the error handler?

------

Another very positive benefit of this idea is that the cache can be made more lightweight and efficient by permanently bringing files which hammer the cache "this side of the fence". Nobody wants a constantly-expanding cache, especially not IPS. The servers will just gip at it eventually.

Opinions?

[OctoDev]

I tried to read the replies here, is it storing the images in database or in actual files`? anyone know?

[Nathan Explosion]

Actual files, in uploads/imageproxy 

[Carl Maltby]

Interesting, however this might well be the subject of a separate thread....I turned caching to "indefinite" however I can't see the folder /public_html/uploads/imageproxy in our file system.

Are you guys cloud hosting with IPS or on your own servers? We don't seem to have any form of caching method available ("- no caching-") in our ACP.

[Nathan Explosion]

localhost and own server.

2 hours ago, Carl Maltby said:

Interesting, however this might well be the subject of a separate thread....I turned caching to "indefinite" however I can't see the folder /public_html/uploads/imageproxy in our file system.

Are you guys cloud hosting with IPS or on your own servers? We don't seem to have any form of caching method available ("- no caching-") in our ACP.

That last line - "-no caching-" is what you see when you look at 'caching method', which has nothing to do with the feature being discussed.

Show the settings you have for your "Allow remote images" etc section. And an important question: have you actually posted a remote image since enabling the settings? The folder is created on the first use of the functionality (ie the first time someone posts a remote image)

[Carl Maltby]

I thought as much. I decided to pop "cache" and "caching" into the ACP settings search to check all instances. I was in two minds as to whether it would have any bearing.

Yes, I altered the settings for remote images. No such folder has automagically appeared and tests so far on whether remote images are in fact cached are showing them not to be. So yes. I uploaded an image straight to the root folder of ProjectGuitar.com which didn't work (I suspected it might not, being on the same domain) and then one using Photobucket. That caused its own issues since I managed to break Photobucket. That of itself was a result I think. Still, no caching of remote images seems to be happening.

Just checked the FTP structure again. We have one! Seems to be working as expected, however there seemed to be some delay between activating it and caching happening. Not sure if that is indicative of anything. Most tests required, methinks.

[chilihead]

Anyone know...

• If you upgrade from 3.4 to 4 if the rebuild will proxy the current remote images? Or not because proxy is off by default? Is so, any way to turn that on before upgrading?
• If you convert from vB using the (upcoming) 4 converter will it proxy the current remote images with proxy on before upgrade process?

If not is there a tool to rebuild the current images so they proxy after these processes?

Thanks

[Nathan Explosion]

18 minutes ago, chilihead said:

Anyone know...

• If you upgrade from 3.4 to 4 if the rebuild will proxy the current remote images? Or not because proxy is off by default? Is so, any way to turn that on before upgrading?
• If you convert from vB using the (upcoming) 4 converter will it proxy the current remote images with proxy on before upgrade process?

If not is there a tool to rebuild the current images so they proxy after these processes?

Thanks

I'll be doing a test upgrade tomorrow and can check it out for you - not that difficult to test the rebuild results.

As for "is there a tool to rebuild...." - no, there isn't a tool to use in conjunction with this option.

Will update tomorrow/next day once I've done the upgrade and the rebuild (3 million posts) has completed.

 

[Carl Maltby]

Bloody hell. We're only at 458k....I think our forum manager pruned maybe 4-5 figures off that, but seriously....

[sobrenome]

Old posts images will be cached as the topics are visited or only new posts after the feature activation will take benefit off proxy caching?

[Nathan Explosion]

4 hours ago, sobrenome said:

Old posts images will be cached as the topics are visited or only new posts after the feature activation will take benefit off proxy caching?

Will let you know later - my personal feeling is that it's a 'from this point onwards' setting.

[Nathan Explosion]

Just in advance of that, I've done a quick test - posted a remote image with the image proxy functionality disabled. I then enable the option and posted the same image again - the first post continues to server from the remote location, and will not update to server from local. And the second post will server from the cached local version.

Code (for a serve from remote):

<p>
	<img alt="Kelly_Brook_at_2015_TCA_%28cropped%29.jp" class="ipsImage" height="750px" src="https://upload.wikimedia.org/wikipedia/commons/e/ef/Kelly_Brook_at_2015_TCA_%28cropped%29.jpg" width="566px">
</p>

Code (for a serve from local cache):

<p>
	<img alt="Kelly_Brook_at_2015_TCA_%28cropped%29.jp" class="ipsImage" height="750px" src="http://localhost/ips_41/applications/core/interface/imageproxy/imageproxy.php?img=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2Fe%2Fef%2FKelly_Brook_at_2015_TCA_%2528cropped%2529.jpg&amp;key=521db6674d0bfbbfaf76b58638a09ce518ed6f476fc97ec4d04be8904573da7b" width="566px">
</p>

If I go back and edit/save the first post then yes it updates to server from local cache.

So, on a live board toggling this option from off to on will not force older posts to update to use the new functionality. An option, or plugin, to rebuild those older posts would be required.

Also, toggling on to off doesn't affect items that are already cached unless you manually delete the content of the uploads/imageproxy folder. In this event (content deleted) the post would need to be edited to delete the existing broken image and put the original url in.....to guard against automatic deletion, I suggest enabling the 'indefinitely' option prior to toggling the function to off from on.

Also - if 2 people post the same image (ie the same exact url) then it will only cache one of them so no multiple cached versions of the same image.

I'll update later, or tomorrow, once I've tested out the upgrade/rebuild effect with this option enabled.

Note: for any eagle-eyed readers, I've logged the issue with the 'alt' attribute as a bug:

 

[Hexsplosions]

I have an open ticket with IPS as some images just will not embed when using the image proxy. I have two IPS forums hosted with the same company on separate VPSs which are configured exactly the same. One one forum the images cache fine, on the other they do not. I've been through everything with a fine tooth comb to try to find a difference between the two, but there is none.

It also appears images hosted on dropbox can't be shared very easily with image proxy enabled.

[Nathan Explosion]

Just now, Evil Edwina said:

I have an open ticket with IPS as some images just will not embed when using the image proxy. I have two IPS forums hosted with the same company on separate VPSs which are configured exactly the same. One one forum the images cache fine, on the other they do not. I've been through everything with a fine tooth comb to try to find a difference between the two, but there is none.

It also appears images hosted on dropbox can't be shared very easily with image proxy enabled.

What's the image url?

[Hexsplosions]

3 hours ago, Nathan Explosion said:

What's the image url?

There are many.

This one would not cache the other day, at all, but today it did:

http://cdn.meme.am/instances/400x/56387108.jpg

I initially thought it may be timing out, but it worked on one forum just fine and not the other, after repeated attempts.

The thing is, it loads in the editor just fine. You paste the link, it is parsed by the editor and the image is displayed. As an end user you think everything is fine. It's when the image is submitted that it falls over itself and it posts a broken image instead.

Here is another:

http://g.bf4stats.com/AQOvIwiO/pc/Evil-Edwina.png

This one is in my signature. It point blank refuses to load.

Yet, oddly, other users have an image in their signature which the image proxy has completely ignored:

http://rotasig.com/rotasig30/sigs/kadalfi.gif

http://g.bf4stats.com/RDq1vnu2/pc/SeriousRikk.png

And if you look at that second one, it's the same URL structure as the one I posted above from my signature, so why one has been ignored and the other refuses to load is beyond me.

It just doesn't seem to work.

I'm sticking with full TLS at the moment but that means my forum is riddled with broken images and mixed content security warnings, both of which are bad.

I'm hoping support can identify the cause and get a fix out, ASAP.