Invision Community 4: SEO, prepare for v5 and dormant account notifications By Matt Monday at 02:04 PM
.Nuno. Posted October 28, 2011 Posted October 28, 2011 Hi, I don't know your setup, but theses 3 posts have everything you need: http://community.invisionpower.com/topic/328398-unbelievable-results-with-varnish/page__view__findpost__p__2107407 http://community.invisionpower.com/topic/328398-unbelievable-results-with-varnish/page__view__findpost__p__2112855 http://community.invisionpower.com/topic/328398-unbelievable-results-with-varnish/page__view__findpost__p__2114476
altenerg Posted October 30, 2011 Posted October 30, 2011 Hi, I don't know your setup, but theses 3 posts have everything you need:http://community.inv...ost__p__2107407http://community.inv...ost__p__2112855http://community.inv...ost__p__2114476 Thanks. So I no longer need to make edits to the IPB files? You add RPAF proxy apache mod? thanks
altenerg Posted October 30, 2011 Posted October 30, 2011 Made changes. Images cache fine, but all forum, gallery, etc pages say MISS in headers. Is there file edits I need to make?
.Nuno. Posted October 30, 2011 Posted October 30, 2011 Thanks. So I no longer need to make edits to the IPB files? You add RPAF proxy apache mod? thanks That's not an edit ... that a file that increments the topic views using esi you need rapf to get the correct ip on logs otherwise your logs will be filled with 127.0.0.1
.Nuno. Posted October 30, 2011 Posted October 30, 2011 Made changes. Images cache fine, but all forum, gallery, etc pages say MISS in headers. Is there file edits I need to make? We don't know nothing about your setup, so it's hard to help you!
altenerg Posted November 13, 2011 Posted November 13, 2011 After some research, I get a good hit rate and absolutely no login or logout issues using this: backend apache { .host = "127.0.0.1"; .port = "8080"; } sub vcl_recv { if (req.http.Accept-Encoding) { if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unkown algorithm remove req.http.Accept-Encoding; } } if (req.url ~ ".(ico|png|gif|jpg|swf|css|js)$") { return(lookup); } } sub vcl_fetch { set beresp.ttl = 31556926s; if (req.url ~ ".(ico|png|gif|jpg|swf|css|js)$") { unset beresp.http.set-cookie; } remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = req.http.rlnclientipaddr; return(deliver); } sub vcl_deliver { remove resp.http.X-Varnish; remove resp.http.Via; remove resp.http.Age; remove resp.http.X-Powered-By; if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } I'm avg over 300 req/sec on hosting my site completely in Amazon Micro EC2 ebs.
.Nuno. Posted November 13, 2011 Posted November 13, 2011 I'm avg over 300 req/sec on hosting my site completely in Amazon Micro EC2 ebs. Only for this files: "ico|png|gif|jpg|swf|css|js"
altenerg Posted November 13, 2011 Posted November 13, 2011 Showed no performance gain (300 req/sec with either way) when trying to cache the pages. Wasn't worth it. imo
.Nuno. Posted November 13, 2011 Posted November 13, 2011 Something is not working for you: http://community.invisionpower.com/topic/328398-unbelievable-results-with-varnish/page__view__findpost__p__2075421 http://community.invisionpower.com/topic/328398-unbelievable-results-with-varnish/page__view__findpost__p__2077422 Static files using nginx with KeepAlive: 25200 req/s php files without Varnish: 25.70 req/s php files with Varnish: 3472.84 req/s I'm running a small 768MB vps.
mat206 Posted November 14, 2011 Author Posted November 14, 2011 After some research, I get a good hit rate and absolutely no login or logout issues using this: I'm avg over 300 req/sec on hosting my site completely in Amazon Micro EC2 ebs. Of course you do. Want to know something funny? Your config probably caches exactly zero PHP pages. Your Varnish config is all wrong for IPB. You left out about all the good bits that actually detect non-logged in members and cuts out their cookies before passing to apache so you can get a cacheable version of the page. On gamedev.net we've had as many as 16,000 users on over a 15 minute period without so much as a hiccup. Here's my current config for the newest Varnish: backend default { .host = "127.0.0.1"; .port = "8081"; } sub vcl_fetch { ## Remove the X-Forwarded-For header if it exists. remove req.http.X-Forwarded-For; ## insert the client IP address as X-Forwarded-For. This is the normal IP address of the user. set req.http.X-Forwarded-For = req.http.rlnclientipaddr; // Dont cache 302 redirects and anything else other than what should be cached if (beresp.status != 200 && beresp.status != 404) { set beresp.ttl = 0s; return (deliver); } unset beresp.http.Server; set beresp.http.Server = "IPB"; ## Deliver the content return(deliver); } ## Deliver sub vcl_deliver { ## We'll be hiding some headers added by Varnish. We want to make sure people are not seeing we're using Varnish. ## Since we're not caching (yet), why bother telling people we use it? remove resp.http.X-Varnish; remove resp.http.Via; remove resp.http.Age; ## We'd like to hide the X-Powered-By headers. Nobody has to know we can run PHP and have version xyz of it. remove resp.http.X-Powered-By; if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } sub vcl_recv { remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; ## We have to pipe file downloads or varnish may cut off long download times if (req.url ~ "files") { return (pipe); } set req.http.X-Device = "pc"; if (req.http.User-Agent ~ "iPad" || req.http.User-Agent ~ "iP(hone|od)" || req.http.User-Agent ~ "Android" || req.http.User-Agent ~ "SymbianOS" || req.http.User-Agent ~ "^BlackBerry" || req.http.User-Agent ~ "^SonyEricsson" || req.http.User-Agent ~ "^Nokia" || req.http.User-Agent ~ "^SAMSUNG" || req.http.User-Agent ~ "^LG") { set req.http.X-Device = "mobile"; } if (req.request == "POST") { return (pass); } if (req.url ~ "(section=markasread|section=login|register)") { return (pass); } else { if (req.http.cookie && (req.http.cookie ~ "member_id" && req.http.cookie !~ "member_id=(0|-1)")) { if (req.url ~ ".(png|gif|jpg|swf|css|js)$") { return(lookup); } else { return(pass); } } else { unset req.http.cookie; set req.grace = 15s; } } } sub vcl_hash { if (req.http.cookie && req.http.cookie ~ "member_id") { hash_data("auth"); } hash_data(req.http.host); hash_data(req.url); hash_data(req.http.X-Device); return (hash); }
altenerg Posted November 15, 2011 Posted November 15, 2011 Until my benchmarks show improvement in pages per second, I call all of that extra vcl code useless BLOAT. The static files are what cause slower loading so I'm fine with caching just those. Thanks though.
altenerg Posted November 15, 2011 Posted November 15, 2011 For the heck of it tried to test that code on clone EBS instace I start whenever I wanna test stuff. Simply replaced 8081 with 8008 in my case and it works but but can't log in lol. No matter how fast that is, its not worth it if members cant log in. I'm on varnish 2 so I had to revert the last few lines of code for it to work: sub vcl_hash { if (req.http.cookie && req.http.cookie ~ "member_id") { set req.hash += "auth"; } set req.hash +=req.http.host; set req.hash +=req.url; set req.hash +=req.http.X-Device; return (hash); } Even so it tops me out at around 312 pages per second -Test ec2 clone instance has no other load. Not much dif if any considering my 300 is was with a few users online. I've simply changed my code to pass non static so that hit rate "looks" better.
Ambar Posted November 16, 2011 Posted November 16, 2011 It's working fine for me and doesn't log me out. I'm using an AWS instance with varnish 3 in front of apache2, but no nginx.
altenerg Posted November 16, 2011 Posted November 16, 2011 What version of Varnish? And what is your .vcl config?
Ambar Posted November 16, 2011 Posted November 16, 2011 varnish 3.0.2, and the vcl.config that mat206 posted in #110, changing only the backend port. No modifications to IPB 3.2.3.
mat206 Posted November 16, 2011 Author Posted November 16, 2011 Until my benchmarks show improvement in pages per second, I call all of that extra vcl code useless BLOAT. The static files are what cause slower loading so I'm fine with caching just those. Thanks though. BLOAT? Do you even know what that code does? This code below from the recv function allows users to get to non-cached login and registration pages. IF they have cookies sent with their request (which they do) NO images, media, or PHP pages will be cached.. and since IPS always has users sending cookies, you can forget about Varnish caching a thing unless you do the config right. The second part of the code below checks to see if they are sending cookies, and if so checks to see if they have member_id set. This is what happens when they are logged in.. however, when member_id is 0 or -1 they are either logged in as guest or logged out entirely. For these requests you want to PASS them through to your Apache backend so they get fresh content. OTHERWISE, if they aren't logged in there is no point to the cookies.. the code below will get rid of the cookies and send the request to the backend server. At that point THOSE requests will be cached. if (req.url ~ "(section=markasread|section=login|register)") { return (pass); } else { if (req.http.cookie && (req.http.cookie ~ "member_id" && req.http.cookie !~ "member_id=(0|-1)")) { if (req.url ~ ".(png|gif|jpg|swf|css|js)$") { return(lookup); } else { return(pass); } } else { unset req.http.cookie; set req.grace = 15s; } }
Ambar Posted November 16, 2011 Posted November 16, 2011 Thanks for the explanation, mat. I really appreciate it. If I'm interested in caching static files for a bit longer, where would that go?
*Salvo* Posted November 16, 2011 Posted November 16, 2011 Is there someone able to install it on a dedi with cPanel for a "small" fee?
Ambar Posted November 18, 2011 Posted November 18, 2011 I made some edits to mat206's config above. First off, we use Google Analytics, so added a couple lines to clean those cookies before going to the server. Second, we don't need to cache image, js, etc files on a per-user basis, so remove those cookies. These two changes improved the cache hit rate markedly. backend default { .host = "127.0.0.1"; .port = "8080"; } sub vcl_fetch { # strip the cookie before the image is inserted into cache. if (req.url ~ ".(png|gif|jpg|swf|css|js)$") { unset beresp.http.set-cookie; } ## Remove the X-Forwarded-For header if it exists. remove req.http.X-Forwarded-For; ## insert the client IP address as X-Forwarded-For. This is the normal IP address of the user. set req.http.X-Forwarded-For = req.http.rlnclientipaddr; // Dont cache 302 redirects and anything else other than what should be cached if (beresp.status != 200 && beresp.status != 404) { set beresp.ttl = 0s; return (deliver); } unset beresp.http.Server; set beresp.http.Server = "IPB"; ## Deliver the content return(deliver); } ## Deliver sub vcl_deliver { ## We'll be hiding some headers added by Varnish. We want to make sure people are not seeing we're using Varnish. ## Since we're not caching (yet), why bother telling people we use it? remove resp.http.X-Varnish; remove resp.http.Via; remove resp.http.Age; ## We'd like to hide the X-Powered-By headers. Nobody has to know we can run PHP and have version xyz of it. remove resp.http.X-Powered-By; if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } sub vcl_recv { remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; ## We have to pipe file downloads or varnish may cut off long download times if (req.url ~ "files") { return (pipe); } set req.http.X-Device = "pc"; if (req.http.User-Agent ~ "iPad" || req.http.User-Agent ~ "iP(hone|od)" || req.http.User-Agent ~ "Android" || req.http.User-Agent ~ "SymbianOS" || req.http.User-Agent ~ "^BlackBerry" || req.http.User-Agent ~ "^SonyEricsson" || req.http.User-Agent ~ "^Nokia" || req.http.User-Agent ~ "^SAMSUNG" || req.http.User-Agent ~ "^LG") { set req.http.X-Device = "mobile"; } if (req.request == "POST") { return (pass); } if (req.url ~ "(section=markasread|section=login|register)") { return (pass); } else { // Remove has_js and Google Analytics __* cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;s*)(_[_a-z]+|has_js)=[^;]*", ""); // Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;s*", ""); if (req.http.cookie && (req.http.cookie ~ "member_id" && req.http.cookie !~ "member_id=(0|-1)")) { if (req.url ~ ".(png|gif|jpg|swf|css|js)$") { return(lookup); } else { return(pass); } } else { unset req.http.cookie; set req.grace = 15s; } } } sub vcl_hash { if (req.http.cookie && req.http.cookie ~ "member_id") { hash_data("auth"); } hash_data(req.http.host); hash_data(req.url); hash_data(req.http.X-Device); return (hash); }
altenerg Posted November 19, 2011 Posted November 19, 2011 I'm only interested in the static content. Images, javascript, css, account for most of pages data anyway. So not gonna kill myself trying to cache what I already have gzipped and then opcode cached with eaccelerator. I got rid of varnish. Now using G-WAN, its a lot faster loading of 90% of the pages content:http://nbonvin.wordp...-server-to-use/ Thanks for the tips though. I'm sure this chunk of code will prove useful for many.
San1ty Posted November 23, 2011 Posted November 23, 2011 Hi Everyone, Is it still required to modify one of the core IPB files in order for this to work? I don't mind adding some code to a skin template but I'd rather not touch IPB files. Is there an alternative way to get varnish to work for me?
San1ty Posted November 23, 2011 Posted November 23, 2011 I use lighttpd and would really like to use varnish, so I'm talking about an alternative way to use varnish.
mat206 Posted November 23, 2011 Author Posted November 23, 2011 Hi Everyone, Is it still required to modify one of the core IPB files in order for this to work? I don't mind adding some code to a skin template but I'd rather not touch IPB files. Is there an alternative way to get varnish to work for me? You don't have to edit any core files to make this work..
mat206 Posted November 23, 2011 Author Posted November 23, 2011 alternative... use nginx fastcgi_cache No offense, but you came here for help on a varnish config.. took absolutely nobody's advice on how to get it working properly (and thus have none of your own real-world benchmarks to compare to), and now are suggesting alternatives? Cached compiled php scripts (opcode caches) are not the same as what Varnish does. If you want to suggest alternatives start another thread please. Facebook, Twitter, The New York Times, WikiPedia, WikiMedia, Globo, The Hindu, NBC Universal Bravo TV, The Grammy Awards and Data.gov.uk are some of the major organizations using Varnish.. it might be possible that they don't all collectively have a bunch of idiot sysadmins running their show.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.