Yesterday I was casually flipping around some apps on my iPhone and opened up my Adsense application which shows me how much I’ve made from Google Adsense each day. I noticed almost immediately that there was an anomaly – for the current day I had served almost 4,000 ad impressions, when normally it would have been more like 200 around that time of day. I then followed up by logging into Google Analytics to see if I had been mentioned somewhere, and sure enough I noticed that my number one referrer was Reddit.com at the time. Reddit gets around one billion page views a month I believe, so any mention on that website can be enough to completely cripple a website with new visitors.
Apparently someone had shared a recent post of mine on Reddit discussing how to cut a SIM into a Micro or Nano SIM. It’s not exactly rocket science, as many others pointed out in the comments. But it generated a lot of discussion from people who said it does indeed work (which it does) and people who didn’t think it would work at all.
But for me I wasn’t really interested in the comments on Reddit, but rather whether or not my website was going to be able to handle the load of all those new users. At the time Google Analytics was showing around 300 active users, and the CPU for my $20/mo VPS unit was sitting around 35%. I tried to log into the admin panel for WordPress, but I couldn’t get to it from Amsterdam due to my slow internet connection and the massive amount of traffic my website was serving.
My website is actually fairly well-tuned for a WordPress based website, but I’ve never had to face a traffic spike like this. I wrote a post a few weeks ago talking about how to speed up your WordPress website, and I thankfully completed all those same optimizations on my own website prior to the Reddit mention.
First, I had the APC PHP cache installed on my server which greatly reduces the load for PHP scripts. Second, I had W3 total cache installed which essentially removes PHP and WordPress from the equation for spikes like this, since Apache will simply serve static html files after they are cached. I also had Infinity Cache installed, which is a soon-to-be-released mobile caching solution for WPtouch. And lastly, I serve all of my static website assets from MaxCDN, which is a global content delivery system.
So in terms my site’s ability to actually be able to survive the Reddit Effect, I was in pretty good shape.
But since I couldn’t log into my admin panel, I decided to try and optimize Apache and PHP a bit further to try and make my server a bit more responsive.
httpd.conf Adjustments
One of the first adjustments I did was to increase the MaxServers for httpd up to 20 from 10. Since the VPS is memory limited, I left it at 10 simply because I didn’t see much point raising it higher. But I figured doubling the amount of processes available to handle the incoming requests would help, and it did.
Another quick adjustment I made was to disable KeepAlive connections. Normally KeepAlive connections are good because clients can request multiple items in essentially the same request. But often an Apache process will continue waiting around for more requests, even after the client is finish. That means that particular Apache process isn’t available to serve any more requests until it essentially times out. So I disabled KeepAlives and then rebooted Apache from the command line.
After my changes Apache stabilized at serving over 100 page views per minute, with approximately 500 visitors active at any time.
Reddit Traffic Spike
Final Results
The total traffic onslaught lasted about eight hours, during which time my website served close to 20,000 page views. During the most intensive hour, there were almost 6,000 page views.
Traffic Spike
I logged into MaxCDN after it was over to check the traffic stats, and they showed approximately 50 GB worth of data transferred during that time period, which is about 5% of the total amount I’m allocated for an entire year. That’s a substantial amount of traffic. I haven’t done the math, but I suspect my little VPS would have basically fallen over had I not used the CDN, simply because it wouldn’t have been able to keep up with the bandwidth requirements of all those requests.
MaxCDN During The Spike
Key Takeaways
For the most part my little $20 DigitalOcean VPS stayed active during the eight hour assault on it by Reddit users. It amounted to nearly 20,000 unique page views during that period of time, with almost 50GB worth of data transferred. I’m quite pleased the VPS didn’t fall over, and I owe that mostly to the various caching systems I had in place, as well as my choice for serving most of my static content from a CDN.
A few people on Reddit commented that my site was down during that period, which was my experience at the beginning as well. This was simply the result of having so many people try to view that particular webpage that Apache couldn’t keep up with the requests. But after making a few quick Apache adjustments, Apache’s ability to handle webpages nearly doubled, which allowed me to log back into my site and seemed to make the website in general more responsive for everyone.
So even if you have a small VPS, by installing W3 Total Cache and utilizing a CDN like MaxCDN, you should be able to handle the Reddit Effect if it happens to you too.
I’m in the process of setting up another small $5/mo VPS with a Varnish instance in case it ever happens again. With that addition to the mix I don’t think I’ll have any issues at all with being mentioned on Reddit again in the future.
Did you try to use AWS Cloudfront at all ?
@Hareem – We used CloudFront for a while over at BraveNewCode, but I didn’t really like it that much. I personally find MaxCDN’s interface much friendlier, and it integrates with W3 Total Cache so you basically can just set and forget it for WordPress.
Great write up!
I’m also hosting my blog @ DO’s VPs, but i’m using the $5/month plan (considering migrating to $10). I would suggest you to migrate from apache to nginx, which handles concurrent access much better, specially for slower servers. Varnish is also a great idea too!!! Keep up the great work! 😉
Hello,
Awesome post. Out of interest, do you have my DB on a different box? Or on the same box as Apache?
Scott
@Scott – Same box, it was all running on the $20/VPS. I have the MySQL query cache enabled as well though. For the most part though MySQL is taken out of the equation in this scenario by W3 total cache, which caches the content to disk once it is generated.
@Daniel – I’ve been thinking of it, but I do a lot of active WordPress development on my server, and Nginx doesn’t always play nice with it.
@Duane – Ahh that’s cool. W3 works quite well for the few WordPress sites i do run. One of the sites i run does have an app and db server which seems to help.
I agree with Daniel Andrade Nginx or lighttpd will greatly increase the speed of your website, and Nginx can serve MANY more static pages faster than Apache, here is a graph on a bench mark test I did a few months back with static pages (the numbers are in seconds):
Simultaneous requests: 1,000
Tests: 10
————+————
Nginx | Apache
————+————
3.56 | 33.42
3.46 | 33.49
3.42 | 33.46
3.72 | 33.35
3.38 | 33.59
3.93 | 33.28
3.77 | 33.49
3.56 | 33.50
4.20 | 33.28
3.41 | 33.38
Great write-up!
Varnish is definitely a good idea, we run 2 Varnish instances on a high-traffic website and it does miracles in both pagespeed and taking the load of the back-end servers.
Thanks everyone. I just made a new VPS and set up Nginx on it.
Way to go Duane! You will not regret 🙂
Hi,
I am just wondering, since you mentioned DO VPS, do you use virtualmin or similar or you do everything manual?
I thought about it for a while to host my own blog, but then I thought of all the problems I may get, with being hacked and all. Not only do you have to take care of all optimizations, you also have to take care of security and that got me off.
@Mason – I was using ISPConfig3 on the VPS at the time, which is like a free version of cPanel. I recently changed everything over to a new VPS where I am doing most of it manually, but using Postfix.Admin to handle the virtual emails.
Why did you fall back to manual?
How do you monitor security?
I mean, what does it take to have good security of a VPS?
@Mason – I went to manual because it’s too hard to set up Nginx when you are using cPanel or ISPconfig, and I wanted to try that.
I don’t really monitor security to be honest. I make sure all my passwords are secure, that Nginx/Apache runs in a restricted environment, and that I keep all the apps on my server up to date. Outside of that I don’t really worry too much about it. cPanel is deployed everywhere and it’s fairly insecure, since most people run all their scripts as Apache. So I really don’t think running your own server is much worse than using something like cPanel.
I’ve had a website online for 7 years and never been hacked, or at least never noticed it. I don’t keep credit card information on my server, so really in the odd event I ever had a security breach, it would be only an inconvenience. You can stress all you want about trying to keep secure, but even Adobe and all their teams can’t prevent a breach, so at the end of the day I recognize I can only do my best in that regard.
Makes sense. Thanks.
What about updating your system, for restart, do you migrate your site somewhere else?
Or just wait with 10 min downtime?
@Mason – Rebooting my VPS only takes about 60s, so it’s not an amount of downtime I’m really concerned with.
What about backup?
Do you use wordpress plugin or do you have something more advanced set up?
How is Nginx compared to Apache?
Is it really that faster?
I will soon move my blog to VPS, but 5$ one, on apache for a start. Almost ready.
The most easy for me to set up, but it would interesting to know if I should start looking at nginx for the future.
@Mason – There are various plugins for WordPress that do a good job. But I am just using a little nightly shell script to back all the files and databases up to Amazon S3.
I haven’t noticed a big difference to be honest, mainly due to using W3 Total Cache on my site. Without that I suspect Nginx would outperform Apache, but they seem pretty close in terms of speed with it enabled based on my benchmarks.
Hi Duane,
I couldn’t help noticing how you optimised your website.
Your website is really pice of art.
Would you know how you managed to (google speed test):
– Leverage browser caching
as of now I didnt found a plugin that would do that, I can guess you made optimisations in the website code?
– Eliminate render blocking Javascript and CSS
This is another thing I cant get sorted.
Thanks!
Hey Mason,
1) If you use W3 Total Cache, you can simply enable the Browser Caching support
2) You probably don’t have much control over render blocking CSS and Js as often it’s just part of the theme.
Good post Duane!
Duane your post gave me good directions to get me fixing my page