So I realize this could potentially be a pretty deep rabbit hole, but I thought I’d ask the question here. I’m in the process of trying to put together a simple web server configuration that is more robust than a single server configuration. Truthfully, I’ve never set something like this up, but for what I’m after I’m not looking for something amazingly complex or expensive.
The server needs to basically handle HTTP requests, and interface with a database. In my mind, this would require two servers in separate co-location centres (which would help if one centre went down and not the other). One machine would be the primary web/database server, and the second machine would be the slave. A cron job would synchronize the files on the slave every 5 minutes or so to make sure it stays up to date, and the database would be set up in a master/slave scenario.
All of that basically makes sense to me so far, but if you know of a better configuration let me know. I know some people purposefully put the database server on its own machine, but to keep redundancy I believe I’d need four machines in that scenario, and not two, which is more complicated than I want to start out.
The piece of the puzzle I don’t really understand is how to check for a failure and how to recover from one. In my mind this would require some kind of external DNS service that would ping the primary web server and alter the DNS record for when the primary stopped responding. As these machines are in separate co-location facilities, I don’t think I can do any type of IP takeover (at least, not that I’m aware of). Can anyone recommend a way to solve this piece of the puzzle, or suggest another way of doing this?
Reading through your article, you seem to have gone through the same thought chain as me – firstly a separate database server and then an external DNS service.
As, obviously an external DNS service would defeat the whole idea, would it be possible to set your first Nameserver on your DNS records to point to your master server, and then the second/ third Namserver to your slave server?
Hmm, why would an external DNS defeat the idea? I’m sort of operating under the assumption that a paid-for DNS service probably has its own redundancy built in.
The problem with DNS address switching is address propagation. It can take a long time because of caching.
I’ve heard of people using cloud computing hosts to get (relatively) cheap redundancy.
The problem with that approach is that you generally need everything in the same data centre.
I’m ok with a small DNS caching delay. With low TTLs, I’ve heard you can get 5 – 10 minute switch over, which is good enough for me.
In 2001 when I worked for a large shop we found a significant number of the big ISPs did their own DNS caching and didn’t honour TTLs. This may no longer be the case. Even if it is, it may not be an issue with your target audience.
Unfortunately I don’t think there’s a good way to handle this across multiple data centers (without spending a lot more money). And the setup really depends on the use case. For example, if you’re working mainly with anonymous traffic you can cache almost all your content on a CDN.
Also, I know with MySQL (this use to be the case at least), you actually need 3 servers running in order to do proper failover and recovery. And rsync’ing data every 5 minutes could make it difficult to recover once the master server comes back online, since they could be missing up to 5 minutes of data.
At the last company I worked for we had a load balancer in front of 4 identically configured webservers all returning the same data. Of course you need redundant load balancers in this setup too but I think it would also be possible to run the webservers in different data centers this way. I don’t have a solution for putting the load balancers in different data centers though :). And yeah, I think the IP takeover stuff is too slow and not really an option as you and Dale have both mentioned.
Thanks Scott. I’m just trying to avoid having to set up 4 machines just to obtain a slight bit of redundancy.
I’m not worried about the file system changing in the event of a failure. Only the database content would change in that scenario I would think. Given that I only want two servers, recovering from a failure would probably involve manually dumping the slave database and restoring it on the master.
Let’s say we removed the constraint that these are in different data centres – what would you recommend in that case?
If they’re both on the same subnet you could monitor the master server from the slave, and when the master crashes set the mac address (on a virtual interface) of the slave to match that of the master. That should allow you to takeover the IP address without any propagation delays.
i now set up a redundant webserver-“cluster” (2 Servers with httpd and bind9 as dns)
bind 9 on server1 refers to itself for a specific domain
bind 9 on server2 refers to itself too for that specific domain
it works – the problem is the client-dns-cache, or the dns-cache from other dns-caching-servers!
did you test it successfully by setting down the TTL values to a few minutes? i read about another option: adding two hierarchical A-entries for one domain-zone in bind…
quite complicated topic 🙂
Kim