Jyosua Posted January 30, 2013 Share Posted January 30, 2013 I put a notice on the main site detailing this as well, but the events are worth detailing in full, here. Yesterday, shortly after arriving home around 5PM PST, I got the brainy idea of rebooting the server. I was hoping it would take care of some of the lag we've been seeing when we get large influxes of new users. But after sending the reboot command, the server failed to restart: 40 minutes of nothing. Uh-oh. I started getting worried, so I contacted the data center and one of their guys started taking a look at it. At first, he thought it just needed to do a filesystem check and would be starting up afterwards, so he notified me of such. That sounded like a reasonable reason for the server to refuse to finish booting, so I bought it. An hour later, it turned out that the server was getting hung up on boot still, even after passing the mandatory filesystem check. Well, as it turned out, the hard drive was failing. Thankfully (Sorta? XD), we've been down the road of horrible data loss before, and I had nightly backups. So, after verifying my backups were okay, the data center dude pulled the drive and gave us a fresh one (slightly bigger than before, too) with a clean OS install. Then came the backups. Unfortunately, the general support dude was much slower, and didn't finish restoring our backup files to the new disk until 3AM PST. As most of you know, I actually have a real engineering day job, so by this point I was already long asleep in bed. I could have probably done the file restore myself, much faster, but there were some connection issues that they had to fix that prevented me from doing that anyhow. Regardless, I saw the file restore was complete when I woke up this morning for work, but couldn't start truly fixing things until I got home tonight. Now, you're probably wondering why it took me nearly 4 hours to finish fixing the server after I got home tonight. As it turns out, our nightly backups aren't just a straight-up disk mirror. I actually have to reconfigure all the programs we use and had to restore the database from the raw physical database files mysql saves. This was a pain, and much more complicated than a fresh install is. Thankfully, I figured it out and we should be back at 100% operating capacity. We might have lost about a day's worth of posts, since the last backup was the same morning as the server went down, but it shouldn't be much. Finally, this has made me realize that I have no way of telling people what's going on when we have unanticipated outages like this. As a result, I will always post updates on things like this to my twitter account: @Jyosua from now on. If you want to know what's going on when stuff like this occurs, follow me there. As a final note, I'm still looking into possibly upgrading the server in the future, but it's still a tiny bit out of our price range at the moment. If ad revenue continues to improve as more people visit SF and proves to be stable, we should be able to afford it in the near future. Link to comment Share on other sites More sharing options...
shadowofchaos Posted January 30, 2013 Share Posted January 30, 2013 If ad revenue continues to improve as more people visit SF and proves to be stable, we should be able to afford it in the near future. Cash in on that FE13 release/early release hype Jyo. Thanks for the update and thanks for keeping this forum as good as you can. The lurker admin appears in the forum's time of need. Link to comment Share on other sites More sharing options...
Phoenix Posted January 30, 2013 Share Posted January 30, 2013 Now, you're probably wondering why it took me nearly 4 hours to finish fixing the server after I got home tonight. As it turns out, our nightly backups aren't just a straight-up disk mirror. I actually have to reconfigure all the programs we use and had to restore the database from the raw physical database files mysql saves. This was a pain, and much more complicated than a fresh install is. Thankfully, I figured it out and we should be back at 100% operating capacity. We might have lost about a day's worth of posts, since the last backup was the same morning as the server went down, but it shouldn't be much. You have saved us quite a few headaches by going through a few of your own. On behalf of the rp board ... THANK YOU. Link to comment Share on other sites More sharing options...
Iridium Posted January 30, 2013 Share Posted January 30, 2013 Oh, thank goodness you're back. Now I don't need to stalk Gamefaqs and /vg/ for FE13 updates anymore... shudder. You have my strongest gratitude. Link to comment Share on other sites More sharing options...
Knight Posted January 30, 2013 Share Posted January 30, 2013 Yay, we're back, I kept opening Google Chrome and pausing for many a minute wondering what was missing. Link to comment Share on other sites More sharing options...
Zanarkin Posted January 30, 2013 Share Posted January 30, 2013 I was feeling lonely and despair my internet home was ripped from my feet oh noes where i'm i going to -- oh wait its back up yay. Link to comment Share on other sites More sharing options...
zahlman Posted January 30, 2013 Share Posted January 30, 2013 Yay, we're back, I kept opening Google Chrome and pausing for many a minute wondering what was missing. Google Lissae, obviously. Link to comment Share on other sites More sharing options...
King Marth 64 Posted January 30, 2013 Share Posted January 30, 2013 Thank goodness that the site is back online. I wondered what was that about when it was crashed. Link to comment Share on other sites More sharing options...
Dai Posted January 30, 2013 Share Posted January 30, 2013 I was worried something was wrong with my account or my internet. Glad to see the whole thing's been resolved and everything's back up, tough. Link to comment Share on other sites More sharing options...
Ϲharlie Posted January 30, 2013 Share Posted January 30, 2013 In the immortal words of Superbus, "[Josh's] HD shit the bed yesterday. And [...] it shit the bed HARD." Link to comment Share on other sites More sharing options...
eclipse Posted January 30, 2013 Share Posted January 30, 2013 I'm glad it was all sorted out~! Thanks for helping us in our time of need~! Link to comment Share on other sites More sharing options...
Fateborn Posted January 30, 2013 Share Posted January 30, 2013 (edited) Jesus christ... That scared me. And it lost Sharpy's draft group of all of our draft picks. Good thing I have everything stored inside me if it involves me directly...more or less...) ...BETTER GO CHECK THAT SPRITE CONTEST! Edited January 30, 2013 by Ubel Engel Link to comment Share on other sites More sharing options...
Fruity Insanity Posted January 30, 2013 Share Posted January 30, 2013 Thank goodness we're back. That must've been a lot of work. Thanks. I was, like, refreshing the page every 10 seconds. @_@ Serenes Forest withdrawal. Link to comment Share on other sites More sharing options...
Kriemhild Posted January 30, 2013 Share Posted January 30, 2013 I'm pretty impressed that you worked so hard to get that site back up and did in so fast. You could have relaxed a bit, and have it revive on the 1st February, the Birth of SF. It only goes to show just how dedicated you are in maintaining this site. Thank you very much, Jyosua. For a Root Admin that almost never shows up, it's very reassuring to see that you still care immensely for the site to go through all that in such little time. We're very lucky to have someone like you in charge. Link to comment Share on other sites More sharing options...
cyron Posted January 30, 2013 Share Posted January 30, 2013 Thank you, its good to see the site up and running again Link to comment Share on other sites More sharing options...
Gold Vanguard Posted January 30, 2013 Share Posted January 30, 2013 (edited) I wondered why I couldn't get into the website yesterday. Edited January 30, 2013 by Gold Vanguard Link to comment Share on other sites More sharing options...
Raven Posted January 30, 2013 Share Posted January 30, 2013 You could have relaxed a bit, and have it revive on the 1st February, the Birth of SF. That is a silly suggestion. Link to comment Share on other sites More sharing options...
nocturnal YL Posted January 30, 2013 Share Posted January 30, 2013 Awesome. I was wondering where to ask about this site's downtime. I thought the server finally snapped when it couldn't handle too many visitors. Link to comment Share on other sites More sharing options...
Popo Posted January 30, 2013 Share Posted January 30, 2013 In the immortal words of Superbus, "[Josh's] HD shit the bed yesterday. And [...] it shit the bed HARD." Thankfully, Josh is a pro at cleaning up shit and making things awesome again. :D Link to comment Share on other sites More sharing options...
AstroknighticalFeats Posted January 30, 2013 Share Posted January 30, 2013 Thanks for the touch-up Jyosua! Hopefully things will run more smoothly. Link to comment Share on other sites More sharing options...
Scarlet Posted January 30, 2013 Share Posted January 30, 2013 Ah... I was thinking something like this had happened when I suddenly started getting 404'd trying to access the good ol' forest. Thanks for getting it back up~ Link to comment Share on other sites More sharing options...
Tiaramaki Posted January 30, 2013 Share Posted January 30, 2013 So -that's- what happened. I was worried that something happened with the site due to the whole Canada thing. But it's back up now, and my worry is dead. Good bright-ish spot on my work day, too; raaaiiiin~... Link to comment Share on other sites More sharing options...
Quintessence Posted January 30, 2013 Share Posted January 30, 2013 Thank goodness the Forest is safe! Thank goodness Grandmaster Jyosua avoided another massacre. SERENES IS STILL ALIVE!!!! Link to comment Share on other sites More sharing options...
Loki Laufeyson Posted January 31, 2013 Share Posted January 31, 2013 Thankfully (Sorta? XD), we've been down the road of horrible data loss before, and I had nightly backups. lol Peeps were kinda worried but it looks good now. Good thing there wasnt that data loss. I wont forget that witch hunt...hooo... Link to comment Share on other sites More sharing options...
Rhaegar Targaryen Posted January 31, 2013 Share Posted January 31, 2013 Outstanding work. Much appreciated. Link to comment Share on other sites More sharing options...
Recommended Posts