As many of you are well aware, the wiki is currently down. For those of you that haven't been around during my IRC explanations, I'll briefly explain the situation in a more public area:
===============================
<span style="color: #bbbbbb;">
In an effort to clean up our MySQL database and rid ourselves of antiquated tables from 2004 - I began what would soon be known as Suck Day For The DJB late on Tuesday night, soda in hand...sunflower seeds close by.
I began - as any good planner when attempted a mass deletion of about 70 database tables - with a database backup. With a gleam in my eye and 2 sunflower seeds in my mouth, I set off to make ANOTHER MySQL database to copy all the tables to in case something went awry. I soon found out that I was not able to add any additional databases to the DJB site due to the limits of our hosting plan. I swigged some soda and moved on to Plan B...generating my own backup file.
Navigating deftly around phpMyAdmin, I managed to generate a hefty backup file for the database I was working on. Life was good. I ate 12 sunflower seeds (spitting out the shells, of course).
With the backup safely on my desktop, my mouse danced across the screen as I checked boxes next to tables - sentencing each to their doom. My hands became slick with sweat from the shear effort of sealing the fate of so many tables. I took a breath. I took a drink. I clicked "Drop."
Knowing that my clicking skills were top notch - and if any foul up were to happen, I always had the backup - I didn't read over the list of tables (there were a lot) and promptly confirmed their death.
Work complete. I sighed in relief. I ate 34 more sunflower seeds. I drank from my seemingly endless glass of ice chilled soda.
My ears perked up at the sound of...what was that? A private message in IRC.
"Yay. I'm popular!" I thought to myself as I Apple+Tabbed to my IRC window (yes, I use a Mac).
A message from Shadow: <span style="color: cyan;">"the wiki is f'd."</span> (paraphrased)
My hands flew over my keyboard and mouse as I navigated to the database. I looked at the tables. Was something missing? I checked my own personal installs of MediaWiki (the software we use for DBWiki) and counted the tables and compared. The numbers didn't jive.
My eyes sprang open, sunflower seeds rocketed from my mouth and hit my wife, soda gushed from my nose wetting my pants.
Then I gathered my senses. "No worries," I thought, "I have a BACKUP!"
As if possessed by 12 coder ninja monkeys, I knew that those tables would be easier to recover with a full restore. I purged the tables and got to work on the restore!
ERROR
"WTF!"
I looked closer at the backup file. It was truncated partway through. Despite the successfull downloading of the backup file...phpMyAdmin had given me a partial one that cut out partway through an insert statement.
Not good. Not good at all.
</span>
===================================
So that's that.
This is where we are now. Jac is trying to get a backup from our server admins (who do a backup on a nightly basis). If that all works out - and there is a 99% chance that it will - we will lose anywhere from 5 to 24 hours worth of data. If a backup cannot be restored by them, we may be reverting to an older copy of the database which will set us back further than would be desirable.
This predicament grew from an effort to make our site better but has turned into something frustrating/worrisome for everyone. For that I am sorry, as I was the one that clicked the delete button.
Has everything been lost? Definitely not. But some data will be unrecoverable. Stay tuned and cross your fingers that we have only lost a day's worth of data.
You need to be logged in to post comments
Now why can't we have reports in this sort of layout...? =P
A very entertaining way to detail something catastrophic! lol
agrees with Prelate
It has a comical flavor that brought a smile to my face even though my favorite part of the DJB is borked. Though feel confident with you and Jac and Karel on it. So good luck!
so, today you learned to check if your backups were successful? :P
This incident, did, in fact reinforce the need for me to check backups. I had never had a problem before with backing up with phpMyAdmin...I just took all those other times for granted during an inappropriate time and place.
Hey it happens. I would like to say thanks for all your hard work even if it didn't turn out like you planned.
comment deleted
Hmm,
good way to report a bad thing/mistake.
is like when the doctor gives you a shot but also a lolly-pop when you're a kid.
(so what do we get for accepting the pain of wiki-less-ness?? You gonna hand out sabers and robes too?? (just teasing really))
Is there an IRC Log/Photo/videoclip of like 2 seconds after the realisation hit going around? Cos i'm close to becoming willing to be paying to see the realtime reaction of realising something like you described! Or will it be possible to buy it once the economy project gets put into action??
pop? you should have drank more Jager. It makes all your problems go away.
If everything else is lost, I have my wiki page still saved! we at least have one solid page of writing.
Suck day for the DJB....surely you could have come up with something better than that.
The Day of Suckage....no, too dirty; in fact, that's probably the cleanest thing I could come up with....
I read this and wanted to cry for you...... We all have done things like this. Spent hours working on something go to hit save and it despersts to art work lost because your computer crashs. It always hurts but this I am so sorry this happend. We all feel your pain!
Dont worry we dont blame you at least I dont. It was a mistake that any one could have made and I am sure we all make mistakes because NO ONE is prefect.
Good luck with the furuter work.
(I cant spell please dont fault me for that.)
LOL, awesome report, even though something bad happened.
No worries, we put our trust in ya. =)
AHHHHHHHH!!!!!!!!!!! NOT THE WIKI!!!!!!!!!!!!!!!!
it's ok.. in reality, it's no biggy.. cuz you work your butt off to keep stuff working right. We've all had times where we have done the exact same thing... case in point, i had finished my 50 page paper and i didnt save it, and then i had a power outage as i was printing it. :P
You do an amazing job, so if we have to re write stuff, it's kinda out faults too cuz we also should have backups of out work. :P
Keep up the good work!
<3
You Maniacs! You blew it up! Ah, damn you! God damn you all to hell!
Orv, you're a failure! I can't believe this disaster! Arghhh!
I'm joking, I still get those butterflies in my stomach every time I think of you. Nice report, it was enjoyable. I was so inspired, I went out and bought sunflower seeds and a book about programming. Now I'm just like you.
This is a sad, sad day.
And yes, your report was very inspiring. It made me want to post this comment.
Well, I dunno what was scarier, that Wiki going down, or the fact when I read your report like a good redneck I had a bag of sunflower seeds in hand. I laughed at the report mate, brilliant way of breaking the bad news.
Heck, a fresh start might even be a good idea, who knows.
On a side note, don't feel too bad, my HDD on my laptop corrupted completely, so I spent the last two days fixing that. I lost a lot of data, including a lot of stuff on a DB Project. It happens. Drink a beer, say a few curse words, back to the grind (As long as your commanding officer aint around, they dont like us drinking when on duty).
My favorite part was the Apple reference. Power to Macs :P
Crappy day for the Borker.
And in response to Aabs, maybe it's because you used an Apple :P
Well, it seems like their is a wave of borkage going around this week. My computer decided to randomly fubar as well, so I understand that pain. I've been defragging it for two days so far and it's still defragging. Let's hope your wiki problem is much easier to solve!
Give Orv a job like this, and look what happens.
Orv, you know what to do. Come back to me baby :(