At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.
Hey folks; as many of you may have noticed, we were down since somewhere around 10pm last night (06/29, Eastern Daylight Time) until about 1:20 or so PM today (06/30). Our hosting provider, Amazon's Web Services, experienced a fairly widespread power failure and a bunch of resulting system failures due to a massive electrical storm that parked itself in the general Virginia/DC area last night (many reported this was the worst storm they'd ever seen). While things were nominally functional at about 3:30AM today, our main disks weren't quite ready to rock. Once they were ready to use again (~9-10AM today), there was a decent amount of corruption in the database. We wanted to make sure we didn't lose any data if at all possible, so we worked for a few hours to repair and restore the database.
Some of you may have noticed the site being up and down the last hour or two; we were checking to see if there was still any lingering corruption in the database (there was, which is why we brought the site back down a couple of times). Things seem like they should be back to normal now, however. As usual, we'll be keeping an eye out on everything to see what's up. Our sincerest apologies for the down time!
No worries, thanks for the massive effort and bringing it back! Admins thou totally rock! \m/
Amazon.com was still up somehow. I've never seen that site crash ever.
Amazon is distributed across their entire network. We are, to my knowledge, only located in that one datacenter in Virginia.
Only the N.Virginia server was showing 'power issues', so may be that's why.
Well, maybe you guys should consider distributing to more than one data center?
Sure, if you want to pay for it.
Sure, how much does it cost?
The 'power issues' are very common in India -_-
Heh. Oh, it'll be some combination of many hours of development effort, server space, and bandwidth costs. In short, quite a bit more than we are able to do for the time being.
A problem for another day though, as our current server is back at full strength. :D
Definitely appreciate the offer to help pay for the distribution costs, but yeah, it's a pretty intricate process. There are very few companies that distribute across data centers, because at minimum, the transfer of actual data to keep all the data centers in sync gets extremely costly. That's ignoring the infrastructure changes on the engineering side that we'd need to be able to distribute that way. At our current size, I'm afraid it just doesn't really make sense to do cross-data-center distribution. In fact, it barely even makes sense to distribute across different machines in the same data center.
I live in Virginia :3 I lost and I mean, psh, I didn't notice OS down last night at all <.< >.> because, you know I have a social life...yeah, that's it... xD Thanks for getting it back up guys, you're awesome :D
Things are still a bit hairy at the moment folks. A few things we've tried to smooth the experience out haven't helped much, so we may be at Amazon's mercy for a little while longer.
We're doing what we can!
whenever i refresh it is showing : uh! oh ! lost the connection ... Trying to reconnect now . Is this the server problem or just because of my net ?
All right folks, we've just finished moving everything onto a fresh new server. Things should be running smoothly now; as usual, let us know if that is not the case, and we'll let you know whatever we know if further problems come up.