Jump to content

How IPS uses its hosting services to grow and learn

Origins of IPS offering hosting services

IPS has always considered itself a services company above and beyond a traditional software company. Through the years we have always found ways of providing offerings to our clients from phone and ticket support all the way to our new spam monitoring service and upcoming chat service that extend our software beyond a standalone application.

Soon after IPS was founded we started offering hosting services. We wanted to bring a complete experience to our customers who were interested in both using our software and the advantages of a managed hosting environment that went along with it. The response was immediate and, to this day, IPS hosting provides these great managed hosting approaches to communities.

Over the years IPS grew and we decided it was necessary to stop renting servers from third-party providers. Not only would this reduce our operating costs but, by owning our own hardware, would offer us great flexibility for our service offerings. In 2006 we began leasing space in a quality data center in Reston, VA outside Washington DC and started the process of purchasing, configuring, and installing all the various hardware required. From servers and switches through our core routers and load balancers.

Growth brings us to today

It is now late-2009 and three years after we started, our data center facilities now include:

  • Fully redundant, multi-honed 10 Gigabit connections to providers such as Level 3, Global Crossing, Nlayer, Savvis, and Cogent.
  • Cisco 6509 core router
  • Cisco 3500 distribution switches
  • Foundry networks load balancing and failover equipment
  • Internal and external network separated for fast backups and restores

In addition to these capabilities, the facility in which we locate our equipment also features everything one would expect from backup power to high security entry requirements.

Here are a few actual pictures of our racks and servers:

(For those of you interested: The RES.S#### means "RESton" - with the servers at our backup facility starting with a different prefix plus - "S" for server (routers, switches, KVMs, etc. marked differently) and the number is the ID of that hardware. For example, if you're hosted on "server40" here at IPS, that's actually RES.S1048.)

Managing our own network connections, network hardware, and server hardware has allowed IPS to provide everything from shared hosting to high-traffic, load balanced solutions. We have experience hosting everything from fun, hobby sites all the way to highly visible communities owned by corporations with tens of thousands of online users. Today we provided managed hosting and services for everyone from national sports teams to world-famous music artists.

In addition to community hosting, operating our own facility allows IPS to offer additional services to enhance our software. Services like our spam monitoring service and upcoming chat service were made possible for free or low-cost to our customers by leveraging the resources available to us in our own facility.

Enhancing our software by using our software

Above all else the most important advantage of our own hosting facility is constant, first-hand knowledge of our software's operations. Because we host our own software on the order of thousands of installations the performance of the software is paramount. Our hosting staff and development staff constantly work together to target areas of the software needing optimization. We do not resort to the simple approach of telling customers to "add more server power" if they are experiencing performance issues as we know, first hand, how much traffic a server should be capable of handling using IPS software and can advise our customers accordingly.

We run our own company forums on a simple VPS setup without any fancy optimizations like Sphinx search, memcache, and other specialized server software. By keeping the setup simple we can emulate what a customer would have on the average hosting provider and learn how to optimize our software in a normal environment. We certainly could implement multi-server clusters for our own company sites but we think it's important to use our software as others do and learn from that experience.

Over the years, our hosting facilities have allowed us to grow our offerings and learn how to create a software platform that can scale with that growth. We will continue to leverage our unique position to develop software and host it all in one, managed environment. Those clients who choose not to host their community with IPS can always be assured that our constant monitoring and learning process will benefit them as well through our continued software updates.


Recommended Comments

[quote name='Jaggi' date='05 November 2009 - 08:02 PM'] i'm confused, i thought you made bfarber sleep in the server room, i didn't see his sleeping bag in any of the pics.

I heard they make him sleep standing up inside a rack!

Link to comment
Share on other sites

I congratulate IPS on their efforts; however, I would like to add a note of caution to anyone about using IPS hosting services.

We had been a customer of IPS dedicated hosting services for several years. In October of 2008 we had a minor outage, more like a hiccup which caused our hosted dedicated server to hang. This occurred over the weekend. Despite repeated requests for emergency support from IPS and to Charles, we had to wait until Monday to have someone troubleshoot the server. It was determined that perhaps a memory leak caused the server to hang. In November 2008 the server hung again and again despite efforts to get someone and Charles, to look at our server over a weekend, it took until Monday to look into it. At which point, IPS determined that the hard drive was bad and we decided to replace it and recover from backups. We were backing up to a second partition on the same drive. Although not IPS's direct fault, the system administrator in the data center decided to re-format the drive and place it back in the server. This impact caused the loss of our backups. The drive was replaced and we were force to recover one of our IP.Board's from a March 2008 backup causing us to lose 4,000 members and countless postings. Our other IP.Board was lost completely. That was 9 years of data destroyed. IPS's answer from the CEO was that we should have had better backups and believed it was no fault of theirs. Charles never replied to our request for assistance throughout this ordeal. In December 2008 we moved our hosting to a full service hosting company whose core business is hosting.

We continue to be an IPS customer for IP.Board and IP.Blog because they are great products. We raise a huge cautious flag to those who decide to use hosting services from a provider whose core business is not hosting.

Link to comment
Share on other sites

  • Management

Though I cannot recall the specifics you are presenting here they do appear to be a unique set of circumstances. We have two methods (emergency tickets and a special phone number) for contacting our hosting staff during after-hours and weekends. From the description, it also appears you were not being hosted on a managed server in our own data center at the time. Whereas our managed hosting operations feature backups performed automatically on behalf of clients, un-managed services do not.

Much has changed in our operations in the year since you experienced this unfortunate event through our hosting services. For example, the un-managed service you were on at the time is a service we do not offer anymore. We have learned that a managed environment for clients is always easier both on IPS and the client as our staff can manage all aspects from maintenance to backups.

None of this will correct the issue you faced back then of course, but we have changed our approaches over the years to continue to provide a better experience.

Link to comment
Share on other sites

To be transparent, emergency tickets and the special phone number were contacted during both outages and we did not receive a response from either. We had a managed environment - IPS was doing all the software upgrades and did the initial setup of the backups. At the time of our purchase of the dedicated hosting service, the IPS website spoke about the Reston data center and this is where we believed that it was being hosted. To our surprise, we later learned after the major incident, that IPS was using a 3rd party to host their servers through a provider in Texas.

My post is merely to caution those to ask lots of questions before deciding to use hosting services from a provider whose core business is not hosting. Our decision cost us thousands in lost revenue and almost our complete business.

Link to comment
Share on other sites

[quote name='Jeffrey Roberts' date='06 November 2009 - 09:48 PM'] My post is merely to caution those to ask lots of questions before deciding to use hosting services from a provider whose core business is not hosting. Our decision cost us thousands in lost revenue and almost our complete business.

No offence, but if your business site(s) were backed up on the same hard drive, (yes I know it was partitioned) as your live sites, then that was a huge fault in your own planning/running of the server.

I cannot understand why if you are hosting a business website, that you did not take regular backups both locally and remotely…

I know that the system administrator accidentally wiped the hard drive, but mistakes happen. After all you would be in the same boat if the server hard drive died and you couldn’t recover it.

Always plan for the worst and hope for the best :)

Link to comment
Share on other sites

Great blog post about your hosting. Detailed, explained and pictars!

As for Jeffrey's backup issue, I run backups of my site, then download them to my hard drive, then copy to my external. This way if we ever lose the site, I got a simple backup on file at the root of my server. If I lose the entire hard drive, I still have a backup, and if I lose my hard drive, I still have it on my external. If I happen to lose it on the external (at this point, it's kind of impossible to lose all 4 at once), there's nothin I can do. I'm very cautious when it comes to backing a server up, because downloading a 9GB backup once every 1-3 months is a drain on bandwidth.

Link to comment
Share on other sites

Point taken on the backups. Our assumption was that since IPS setup and was managing our dedicated server we believed that remote backups were occurring.

I think we all learned something from the experience and it seems from Charles note that IPS is in better control of their hosting offerings now.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Create New...