[melnibone_ca]

some servers that I worked on ran some fairly generic backup scripts to tar up the filesystems, and dump them to scsi dds4 tapes.

The script also took car to verify that tar could read back the file index from the tape...at least in theory

tar -tvvf /dev/st0 would faithfully read back the index but, tar -xvvf failed. That sucked. Thankfully, we had other backups to fall back to.

Backup policy was Daily backups Mon, wed, thurs, fri.

A weekly backup was done tuesday night, and taken off site, round-robin fashion.

A monthly backup was done on the 28th of every month, and taken offsite round-robin.

We also had a hotspare system to rsync the entire disk image of 6-10 critical servers to a central machine (nightly cronjob) so we always had at least yesterdays data...

This was also handy if we wanted to take a machine offline for maintenance or repair. As the hotspare had the entire filesystem for each server on separate partitions over multiple disks, a little filesystem magic in /etc/fstab, and the bootloader conf in grub.conf, and we had a hotspare backup server online in no more than a reboot.

We werent quite big enough to worry about raid arrays, but we thought about it. Can't comment on that, sorry....

Anyway, I hope you get your stuff sorted out, and I feel for the poor tech who mistyped an rm -rf foo *

that hurts. I think every unix admin has had a similar experience at some point. :-/

[Skip]

Hey everyone, thanks for taking this in stride. I wish I could!

We're talking about doing something to help compensate for the loss of images, and of course your time and energy.

Teflon & I were discussing how to improve the image integration into the forums (we agree not to be like OG). Anyway, we'll make it up to you kind folks somehow.

There is an upside to this. In fixing this problem the techs got to swap out our RAID controller for a top o' the line new one. This evidently also solved the bottleneck with the two processors that caused a problem.

So we should be blisteringly fast now...(hopefully!)

[Skip]

melnibone_ca
Hey thanks for the empathy! They were supposed to back it up to a server they use just for backups, so it would've been on a hard drive & taken only 10 minutes or so to restore all the sites. Instead it's probably going to take a couple of days, with much stuff missing.

The tech told me he was installing an upgraded control panel. He'd installed a test copy and it worked fine, so he went to delete it and wrote out the wrong directory in the command line (one keystroke off I guess).

That's what I don't like about Unix. Pretty unforgiving and not interactive enough. It should've warned...

"You SURE you want to delete every domain on this server, IDIOT?"

But I guess that computers aren't quite there yet...

Although it WAS human error, not the computer.

And in this tech's defense, he did fess up immediately and was still up at 1am working on the server.

[Skip]

LOL!

Notice the time difference! The two posts immediately below this one were actually posted before (I had to figure out why mine was posting above theirs after I'd read them).

Turns out the tech's have just installed the new RAID system into our server (with the wrong time set on it).

[THC4SIM]

mmmmmm, i dont think the post counter's working right also...

THC4SIM

[Freedom Fido]

we all learn from mistake
"dont worry be happy"

[Einsteinguy]

It's only data.
We all know this is a new site so working out the bugs is part of it.
Hate when you find out backup isn't working is when you need it.

Seen it a few times and it's not pretty , it is always a good idea to test backup before you need it!


Einstein

[groo]

Originally posted by Skip
Why wouldn't you keep a copy of your images? How else you gonna post them on OG? [/b]

I keep all my images and can easily repost them.

My beef is with so-called tech companies who don't follow through on their SLA. I'm in the industry myself, and have had to deal with far too many disaster recoveries that cost long, long hours because of incompetent data center management.

Sure the company got all kinds of penalty fees paid and service discounts because of the screwups, but that didn't end up in my pocket and it wouldn't have given me back the lost nights anyhow.

Their incompetence makes the rest of the tech industry look bad.

[groo]

Originally posted by Skip
melnibone_ca
That's what I don't like about Unix. Pretty unforgiving and not interactive enough. It should've warned...

"You SURE you want to delete every domain on this server, IDIOT?"

But I guess that computers aren't quite there yet...

Although it WAS human error, not the computer.

And in this tech's defense, he did fess up immediately and was still up at 1am working on the server.

If you think Windows, AS/400, or any other system is more "secure" when the system admin reconfigs the server, you are seriously fooling yourself.

Even if the system did have such checks, you can bet that the first thing an admin does is disable the dialogs and warnings that keep them from doing the job.

It's one thing to have to click the warning once when you do something on a home PC. It's quite another when you have to do it 30-40 times per day, every day.

Yes, typos happen -- I don't blame the tech for a typo. I blame the organization for not verifying their backups. Most large sites I've worked do a full system restore from archives twice a year, using the hot failover servers. The sole purpose is to make sure that the backups are archiving the information needed to run the business.

I had to restore an entire project once because of a typo in a job script. Mistakes happen, which is why there is no excuse for a data center that does not follow their backup schedule or verify their process.

(I probably read more pissed off than I actually am. Comes from being one of the people who has to identify the cause of screwups like that and make sure they don't happen again. Among other high-stress aspects of the job. )

[Skip]

The backups were never setup (someone failed to do that job), so again that too, was human error.

I'm really surprised people lost posts or PMs cause they did recover the database, and there shouldn't be anything missing, unless you managed to post while the system was being restored (and I warned on the home page about not posting).