Surprise, surprise!!!

White House Says It Routinely Overwrote E-Mail Tapes From 2001 to 2003
http://www.washingtonpost.com/wp-dyn/content/article/2008/01/16/AR2008011602202.html?hpid=moreheadlines

E-mail messages sent and received by White House personnel during the first three years of the Bush administration were routinely recorded on tapes that were "recycled," the White House's chief information officer said in a court filing this week.

During the period in question, the Bush presidency faced some of its biggest controversies, including the Iraq war, the leak of former CIA officer Valerie Plame Wilson's name and the CIA's destruction of interrogation videotapes.

Man I'm so shocked and awed by this!

bullshit.

Can we be REAL?

What is this, 1992? Fucking TAPES? The people making this shit up are more ignorant than they think WE are.

.

And, yes, some parts of the federal government are waaaay behind the technology curve.

However, even in the face of that, losing the backup tapes for the email system quite likely was done on purpose, it will be hard to prove.

which makes the lie hard to "prove" except to those who really know.

For over 10 years I ran one of the federal government's LARGEST repositories of data.

As part of that mass storage system, I also managed the backups of many other systems, including a few large email servers, and maybe 400 or so desktops.

It was common, industry practice at that time (90's) to have backups of systems use what was called a tape cycle or tape ring. Basically a sequence of backup tapes would be assigned to a user or system. There might be 10 to 12 tapes in the cycle. Every time a backup happens (let's say nightly) an incremental backup would be taken to tape X in the cycle, the next night it would go to X+1... weeklies would go to Y, monthlies to Z and so on. usually, weeklies and monthlies would be full backups... this is so you could recover any given file by having the restore program mount the last full backup and all of the dailies up till the time of the last change to the file that you want to restore. This allows for incremental (less space is used since only files changed that day are backed up) and for minimizing the number of tapes needed to do a restore (1Y or Z plus average of 3.5 X), which speeds up the restore process (instead of dealing with 30 or more incremental).

The part that stinks is that we would, as a matter of policy, take every third or forth monthly dump OUT of the tape cycle and substitute another unused tape into the cycle pool. Those tapes would be kept forever. (well, at least until the oxide starts falling off or 7 years). So the email system (if it followed standard procedures) might well have cycled the daily captures of email, but if no one deletes (and I don't mean just remove it from your mail folder, but actually deletes it) the email near continuously, one of the weekly or monthly full dumps should contain a copy... and one of those full dump tapes was probably saved... at least until the guys with the degausser showed up! And standard practice would be to have 4 to 12 or those full dumps per year for that email server.

Of course, modern practice would be nothing like this. You would simply never delete anything and never, ever write it to tape. Much cheaper and faster to simply record every file change to "continuous backup system" on it's set of RAID disk arrays, probably many TBs to 100s of TB s in size. I know, I patented the technology, started a company to do this, and sold it for fairly big bucks to a large computer backup vendor. Back in 2003.

When I ran a public library, we used to tape everything, too.

We were in the process of putting our card catalog and collection onto a machine readable system. We had a large number of interlibrary loan requests on our computers too, all sorts of intra-office information, and loads of e-mail. We did not want to lose any of our data, so we backed it up nightly on tapes. No one does that now.

though the age of tape backup is, I believe, coming to a close.

The federal government is, by and large, a very backward institution.

Things that government pioneers in one place (like NASA, where I worked), are often adopted by industry soon after invention, and only years and years later, filter back to the rest of government (like, say, the IRS, which is a good thing :) ).

The Whitehouse is an even stranger creature... it is not a federal agency and, while it does have it's own budget, it does not have infrastructure staff... so it calls on all of the other agencies to provide expertise for that function. My branch inside NASA (being the pioneers of the internet) was the office that actually ran the Whitehouse internet connection, their initial web server and email service (back when email and then web was a new thing). So it could take a long time for current industry standard practice to make it's way to someplace like the Whitehouse... or the VA, or some of the "non glamorous" agencies.

So I wouldn't be surprised at all to see a tape backup facility, complete with a tape robot, somewhere in the bowels of the Whitehouse... or even down the street at some other federal agency.

There is no excuse for this illegal action and they are misrepresenting the system.

The PROFS system used by the prior administration would have had to have been replaced and the law requiring the preservation of records ignored when implementing a replacement. That should lead to serious jailtime for a number of people.

-Hoot

really has nothing to do with data archival.

What I mean is that if someone "emptied the trash" (not only deleted email, but also hunted in the email servers for duplicates or "trashcan" type services and deleted that as well), the DR system would ensure that the updated databases (with the deletions) are duplicated across all DR member systems. This is the behavior you want from a DR system.

What the courts or congress are looking for is archived data, that is, data that once resided on the active system but was removed by someone. For that you need to troll through the system backups.

That the Whitehouse might still use a robotic tape system or even a manual tape system for creating archival data sets would not surprise me in the least.

That such a backup system would use tape cycles or tape rings for selecting the next tape to use for backup would be the old (only 10 year old) industry standard. This ensures that tapes are reused and, simultaneously, the number of rewrite cycles on each tape is kept to a minimum (by using a round robin selection).

But what WOULD surprise me is that they would not have kept a once a month or once every 4 month full backup tape as a permanent archive of the data that was present at the time the tape was created. That, too, would have been an industry standard practice. Otherwise, when somebody noticed that they "fumble fingered" and deleted an important file say 2 months ago, the backup system would NOT have it (as all tapes would be overwritten). As a person who was responsible for about 3 PB of government data it was stressed upon me that, above all else, losing someone's data was unacceptable, even if the user went to some trouble to delete the data that they now want to access.

Tapes don't cost so much that they could not have afforded the drain of 12 tapes a year.

Of course, whenever in the text I have used the term "tape", what I mean is a tape set, i.e. enough physical media to contain the entire contents of one complete copy of the file system or database, for relatively small email servers, covering the email of something like 300 or 400 or even 1000 users (I have no idea how many people use the "whitehouse.gov" email domain name now) a complete backup is likely to be less than a Terabyte in size, therefore a tape set is 10 to 20 tapes or so, using 10 year old tape recording technology. A Terabyte would give 1000 email users 1 GB of "current" undeleted email, including attachments.

Also, in this day and age, it would surprise me that the National Archives, interested in all things Presidential, would not have requested a complete copy of all emails to/from senior whitehouse officials and aides, to form a "witness to history" record. Not to mention the President himself, for his eventual Presidential library. Unless the people involved knew that they were doing or writing about crimes and wanted this never to be found.

Archival is required by law (Presidential Records Act?). W replaced a system that satisfied the law with one that doesn't.

Yes a DR site is worthless unless it's a mirror or can be restored to a reasonable timeframe. The point of my mentioning it is that they would have had off-site backups because of the DR activities. It's the obvious place to hook the archive tap.

-Hoot

If you had an offsite DR system, it's the obvious place to take a snapshot and then perform the backup from the snapshot.

I have no idea about the law (probably worth looking up).

My whole point was that

a) yeah, the backup system could have employed tape and reused tapes.

b) Still, standard practice would have been to keep a monthly full backup set offline.

c) While they may claim that the email was "lost" due to tape overwrites, and tape overwrites did happen, someone still had to go to some effort to make sure the data was destroyed.

d) Probably someone broke the law in erasing the old tapes or simply not employing a system that would ensure that every email written or received was not archived.

e) Breaking the law in the coverup is likely harder to prove than the breaking of the law contained within those emails, not to mention that there is some low level flunky who will be set up to take the fall for losing it.

:sarcasm:

I hope they all die from whatever scares them the most.