Best Practices for Scaling in AWS

[Randy Calvert]

Hi folks!  

I wanted to start a topic to discuss best practices around scaling a website in the cloud.  Today my IPB site runs on a VPS that I have full root level access to.  All of the system services needed run on the box.  (A full LAMP stack along with DNS and email.)

I’m looking at building out a new cloud-based design that I’m hoping will allow for

• Better availability
• Improved performance
• Increased security

My initial thought for the deployment would be the following:

• Initially deploy 2 cloud servers to serve as the web tier.  (This would be scaled up as needed)
  • These servers would only allow traffic on 80/443 to the internet. 
  • Server to run only Apache and PHP.  (Disable all other services)
  • Limit SSH to internal network which would be accessed via VPN.
  • Load balance requests between all of the web servers equally.  
• Deploy one master RDB instance with a read only shard.  
  • Create firewall rule to only allow MariaDB to communicate over MySQL port.  
• Deploy a cloud server to handle Redis and ElasticSearch. 
  • This server would not have public IP address and only be accessible from within the VPC.
  • Only Redis and ElasticSearch ports would be allowed to access web tier.
• Setup SES to handle outbound email sending.  
  • I would use a 3rd party service to handle inbound email.  

 

• Use Cloudflare as WAF and bot protection in front of the site.  
  • Limit LB to only accept requests from Cloudflare IP addresses to ensure someone does not bypass security controls. 

I’m struggling with how to handle storage.  I could attach an EBS volume and use it for the entire home directory.  This would simplify updates so that updating one instance updates all web instances.  However it would also mean if an update goes south, it takes down all of my instances at the same time.  I could also instead only use EBS for things like the uploads folder and other locations in the file system settings.  Then simply take a server out of production, update it… then add it back to the pool once complete.

I’m trying to think through the pros and cons of each approach.  One challenge I see is that even if I went with the second approach, the IPB updater may make changes to the DB.  This may more than likely break the site while the update is occurring on the server not in production.  

Does anyone have any thoughts on the best approach to handle this?  Are there other areas I should be considering as well?

[Randy Calvert]

Hey @Jordan Miller… a crazy idea here…

I loved the interview you did with Charles earlier.  As a follow up, what about a session around IPB’s CiC offering?  Maybe with Lindy or someone else who is familiar with the CiC architecture.

The idea would be to:

• Show the value of having a fully hosted solution and all of the work that goes into making it a premium offering.  (It would be a great response to the “well I can host it with X provider for only Y per month”)
• It would show the work that goes into developing an enterprise class solution and show the thought that went into various areas around those key areas such as redundancy and availability that others don’t have to think about with the hosted solution.
• It would provide thought leadership around best practices on things self hosted customers could take if they wanted to put in the time/effort to do things themselves.  The idea would be to promote better and more secure communities for everyone which is in all of our interests.  

I understand that a video might be difficult, but this could also be a series of blog posts as well.  Would love to collaborate on something if you’re interested.  

Edited December 31, 2021 by Randy Calvert

[Runar]

44 minutes ago, Randy Calvert said:

Does anyone have any thoughts on the best approach to handle this?  Are there other areas I should be considering as well?

I think it all depends on your requirements, budgets, and on how much traffic you need to prepare for.

My community used to be structured like in your example, with separate (virtual) servers for each service, but not too long ago I ditched this approach and moved to one larger and more powerful server. I haven’t noticed any decrease in performance, and no one has complained so far either. It has made my life easier in regard to maintenance, as I only need to care for one server instead of a handful. It’s also much easier to set up from a security perspective.

Another point is that I don’t think Invision Community works too well with multiple web servers without some extra manual work. As you said, you would need to update the web servers one by one, but due to changes to the database all of them would have to be offline during the update.

I'm sure it's possible, but my community, and I think this applies to most other communities as well, is simply not large enough to justify the extra work and money it would require.

[gigantor]

We have our own infrastructure, so I can't comment on AWS with the exception of their Simple Email Service (SES).

Its blacklisted by almost everyone because every website developer uses it for their web forms.

I wouldn't use it.

 

 

Edited January 5, 2022 by gigantor