About two years ago I decided I would test out Media Temple’s grid service package, mainly because it was highly recommended by WordPress users, and seemed to perform quite well. In fact, for the first few weeks I was a Grid Service user, I thought things ran actually pretty smoothly. Unfortunately though, over the next few months, things would degrade rather rapidly. My site was plagued by ever increasing latency, and would often go down for 4- 6 hours at a time due to MySQL outages or random, seemingly unannounced maintenance periods.
It got so bad at one point that I actually came to the conclusion I could do a better job myself by hosting my website at home, and transitioned this website to a little server in the corner of my living room. With the exception of a blown power supply that took my site down for a few hours, it ran for the most part without any hiccups for a year and a half or so.
Obviously running a full blown hosting company is a lot harder than running a single machine in the comfort of your own home. But I would like to think that we use hosting companies to host our websites because these companies represent the experts, and that they have the technical know-how to keep our sites running smoothly. As recent events have proven, that is not always the case.
Shortly after noon on Saturday, Media Temple posted this on their blog:
We are currently working on restoring availability to all of Cluster 2 at this time. We will have an update shortly with more details and an estimated time of resolution.
Since then, Media Temple has posted various updates regarding the issue, most of which firmly place the blame on their storage vendor:
According to Bluearc engineers our recent firmware update introduced a new bug that we have never seen. Their developers are currently determining our best course of action. We are considering a rollback of the recent change we made or working with Bluearc to patch our current running version.
At this very moment we are trying to restore all services on the Cluster by isolating out Storage Segment 2*, which appears to be the problematic segment. This will ensure that a great majority of those affected will have their service restored. Those on Storage Segment 2 might have to wait a while longer pending the actions we take with Bluearc.
Thinking back over the last few years, I am fairly certain that Media Temple has routinely blamed their storage vendor for all their problems. And while it very well may be true that 100% of their problems are related to the storage, ultimately Media Temple has to take responsibility for these problems and to make corrective actions to fix them. Given that these issues have plagued Media Temple ever since I first signed on (which has to be at least two years ago at this point), you have to wonder why they haven’t been able to fix things. In fact, if this current situation is any indication, things at Media Temple are getting worse, not better.
Most shared hosting companies in the same league as Media Temple (and I use that term loosely, since in my opinion, most of those companies outperform Media Temple) are priced around $7/month. Media Temple charges $20. In the old days, that price reflected the promise that Media Temple’s Grid Service would be the first incarnation of shared hosting that didn’t completely suck. But if time has proven anything, it’s that the Grid Service performance really hasn’t matched the hype. Also tragic in this is that the cluster that is affected the most by this appears to have some of the oldest (and subsequently most loyal) Media Temple customers on it.
We pretty much take for granted these days that websites should be always on. When I punch a URL into a browser, rarely do I ever think that I won’t get a response from the server. Had BraveNewCode been affected by this (and it is actually located on Media Temple), I would be livid at this point, primarily because I know that having a dead website ultimately translates into losing real money when you are a business. Thankfully none of our Media Temple websites went offline, but it’s not lost on me that possibly thousands of businesses were affected by this, and ours might have been too.
At current count, we are approaching 48 hours of downtime on Media Temple’s servers, something that I don’t think I’ve ever seen happen to this extent. Given that most hosting companies guarantee 99.9% uptime (at least in the brochures), that basically means that Media Temple has exhausted their downtime for the next 5.5 years.
I’m really interested to see what Media Temple has to say tomorrow, and what they are willing to offer their customers for this outage. I’ve seen what Media Temple has Twittered about over the past 36 hours, and have also been reading up on their blog. But so far, while they have been apologetic, I haven’t seen any mention of compensation, which is something I think they are going to need to address soon.
I really don’t know what the fallout will be from this. I suspect a lot of people are extremely angry and disappointed with Media Temple now. I for one don’t plan to be caught with my Media Temple pants down in the future, and started the process of moving a few of my sites off of their service tonight. I’ve given Media Temple countless chances over the years to show me that they can improve, but at each turn they continue to disappoint me, and it’s not something I plan to let happen again. And I suspect that tomorrow, when Media Temple finally gets those websites back online, that you’re going to see an unprecedented amount of anger and disappointment in the blogosphere directed at Media Temple.
If you’re on Media Temple and looking for some other web companies, here’s a small list:
That will be why my FTP was down yesterday then… had to upload files via rsync – not exactly the most straightforward way of getting files to a server! Hope that they can sort out their Grid Service once and for all – but so far all we are seeing is empty promises…
Media Temple’s grid service (gs) is a unique environment indeed – and it is very storage intensive. MT is a great company and it is exciting to see them push the envelope, but outages are hard for all involved, for the customers absolutely, but MT and our team have been at highest escalation mode through the weekend to reduce the impact, and begin the work of bringing customers up and getting to root cause to prevent any recurrence. We look forward to some good news, and this becoming a blip in the rear-view mirror.
Thanks for chiming in Louis. I’m sure everyone affected appreciates you guys working through the weekend to fix things up.
That being said, I sincerely hope that Media Temple is completely transparent with this incident and works to compensate people due to the outage. As I mentioned above, two days of outage represents the same downtime you would expect over the course of 2000 days at 99.9% uptime. I am all for pushing the boundaries and experimenting a bit with the technology behind a service — but in that case, Media Temple really shouldn’t be charging $20/month for a service that, based on it’s performance, still appears to be in R & D mode.
Duane, thanks for your comment. Personally speaking (not in a spokesperson role), I respect MT a great deal, and have considered moving my blog from Register.com their way and may soon. They have a good team and a strong array of services. Once this is resolved, I’ll talk to Alex and see if I can accelerate that process.
[…] Duane Storey – The Day That Media Temple Stood Still […]
i actually really love mediatemple’s DV service, it’s fast, actually reliable, and is thankfully in no way connected to the horrible grid-service. it seems like GS is more of a “beta” and shouldn’t have ever been used in production. seems MT are pretty trendy though, but maybe with such a huge downtime, that’ll change.
I sincerely hope Media Temple improves — and by “improves” I mean doesn’t try to upsell me another service that will go down for 3 hours at a time sporadically at least 10 times a year.