Welcome to the QA Tech-Tips blog!

Some see things as they are, and ask "Why?"   I dream things that never were, and ask "Why Not".  
Robert F. Kennedy

“Impossible” is only found in the dictionary of a fool.  
Old Chinese Proverb

Monday, November 17, 2014

 "Acceptable" Defects 
The Fast Lane to Disaster

Many years ago, I was invited to an interview in Boston by a company who wanted to bring on additional QA staff.  After the usual back-and-forth banter of the interview, the interviewer paused and asked me this question:

What is an acceptable defect?

I immediately replied  "There's no such thing.  It's an oxymoron, something that is inherently self-contradictory."

The interviewer persisted, wanting to know what an "acceptable defect" was.  I replied that there is no such thing as an "acceptable defect" as once the defect becomes "acceptable", it ceases to be a defect.  Or, as one of my favorite snarky QA quotes says:  "It's not a bug, it's a feature!"

The interviewer continued to press the point; and yes, I understood that what he was asking was not about defects being "acceptable", but rather he wanted to know at what point does a QA effort say it's time to stop?  When have you reached the level of "diminishing returns" and decide to release anyway, even though there may still be some bugs left to be resolved?

I mentioned this understanding to the interviewer, said that this was an entirely different question; whereupon I proceeded to answer it.  However I persisted in my original position that there is not, and there should never be, such a thing as an "acceptable" defect.

Needless to say, I didn't get the job.  And quite frankly, I wasn't too upset about it either.  As far as I was, (and still am), concerned, the idea of an "acceptable" defect presents a warped and possibly disastrous mind-set on the part of any QA team.



Why is this a problem?

Now I can hear everyone saying that I'm being too picky and pedantic; that I'm splitting hairs.  And maybe that's true, but I don't think so.

And why not?

It has always been my opinion, and my position within the greater QA community - not just "software" QA, but any kind of QA effort - that "defects" must never become "acceptable".  Because once a defect is categorized as "acceptable", it ceases to be an annoyance or an irritation in the back of our minds.  That nagging irritation goes away and we don't give it a second thought.

I have seen this time and time again, in hardware QA, software QA, process or manufacturing QA, or any other QA efforts I have been involved in.  And it has always, invariably, been the fast lane to disaster.

I go into the consequences of this kind of complacent, devil-may-care attitude in another article I wrote called The Cost of Complacency.  Go read it.



Again and again I return to the idea of defects never becoming "acceptable".  It is the responsibility of the QA community to ensure that defects, even seemingly small defects, never get "lost in the sauce", so to speak.

Even if we need to push on toward an ultimate release date, these defects should always "get in our craw", or be that annoying little pebble in our shoe.  They should always remind us that they are there, and we should continue be on the lookout for ways to mitigate them now, or if that's not possible, ways to avoid these issues in the future.

It's only by letting the seemingly "little things" get to us that we remain vigilant.  It is only by this continuing attitude of eternal vigilance that we in the QA community earn the respect of those around us.

By doing this, by always letting the "little things" bother us, we ensure that the people we work with, and those that depend on us, never sink into that abyss of complacency that has been the graveyard of those who have not heeded this call.

What say ye?

Jim (JR)

Wednesday, April 23, 2014

We Interrupt This Blog
For An Important Announcement!

Well, hello again!

If you have been following this blog, you know that I have some pretty big plans for this blog's future!

Part of those plans have been for me to build a WordPress "sandbox" server here in The Dungeon, try to migrate my blog to it, and try to do it in some semblance of order.  The idea being that if I'm going to make a complete balls-up of everything, (and I will, trust me!), I'd rather do it down in my basement where I can control the damage, instead of on a live production blog.

Of course, these things are much easier to say than do, and I am sure that the Wright Brothers had the same sense of angst:  "Hey!  If it's so easy for birds to fly, it should be a real snap for us too, right?"

Nope.

It's taken days, and days, and days, and days, and. . . .  Oh futz!  It didn't crash, again?!!

Needless to say, it's been a real learning experience, in more ways than one.



One of the resources I have been using is WordPress for Dummies (3rd edition) by Lisa Sabin-Wilson, and if you're even thinking about messing with WordPress, you can do far worse than to read her book.

One of the things I learned about was something called "comment spam".

Comment spam?!  Of all the. . . . ! ! !

And yes, it had me scratching my head too.

Here's the deal:
We all know about e-mail spam.  That endless stream of scams, frauds, and phishing attacks; not to mention the never-ending advertisements for "Penile Enhancement", Viagra, Cialis, and God Only Knows what else.

This is the "plain 'ole every-day" mail-type spam that we all know and love.  It's designed to get us to do something that will compromise us, so that we give away important personal information about ourselves, send money to someone via Western Union or MoneyGram, or open up our systems to being attacked, root-kitted, and perhaps used as a zombie to hack the Pentagon.

Comment spam is a horse of an entirely different hue.  And it's even nastier.

Comment spam is a way of getting MY BLOG to help the baddies clobber someone else.  The way they do this is by hiding vicious hyperlinks inside innocuous messages, masked in such a way that you don't really know what you're clicking on until after you've done the dirty deed.

An even more vicious form of comment spam is a comment that has, (what appears to be), a perfectly innocent hyperlink that leads you to a seemingly innocent site that will re-direct you to somewhere evil.

Translation:
You like my blog.
You read my postings.
You read the comments.
You click on what appears to be an innocent hyperlink in a comment.
YOU get hammered, and MY BLOG set you up for the kill.

The real epiphany was when I did a trial-import of all this blog's content and comments into my sandbox server.  Unlike Blogger, WordPress lets me see, (and do), anything I want with a message or its comments.  Nothing is hidden.  Nothing gets squirreled away.

Within WordPress, I could actually see the vicious bulldogs hiding behind the pretty flowers within some of the comments left on my site.  And it was scary!  Needless to say I was PISSED.



As a result I have had to do two things that I really did not want to have to do:
  1. I no longer allow "anonymous" comments on either of my blogs, since virtually all the nasties were hidden in anonymous comments.
  2. All comments, without exception, are now moderated.
    As unpleasant a task as it may be, I have no choice but to require every stinkin' comment to be quarantined until personally vetted by me.  And because I can't create a "white-list" of trusted posters, it's an all-or-nothing deal.
Additionally, I have to be really strict about embedded hyperlinks in comments.

Translation:
If the comment has an embedded hyperlink, it get's trashed.  Period.

Why?
Because I just spent the last half-day-or-so going through every single message on this blog, comparing every single comment here with the nasties I found when looking at the comments on WordPress there, and removing, one-at-an-effing-time, any suspicious comments that I found; sanitizing the comment stream so that no one who comes to my blog gets hammered.

And why is that necessary?
Because Blogger won't let me edit comments or view hidden content.  And because I can't view hidden content, I can't see what might be hiding behind an embedded hyperlink without clicking on it, and I won't risk what loyalty my blog may have for the sake of a few hyperlinks.

One potential exception is the plain-text in-line hyperlink that is visible to everyone.  I might be convinced to allow in-line hyperlinks that are plain-text, after I have personally tested them and verified that they are not harmful - if they are germane to the topic of the post.



It's a sad day when I have to spend more time babysitting the few baddies out there, then I spend actually creating useful content.  However, I'd rather do that than see those of you who read my blog get hammered because I'm asleep at the wheel.

What say ye?

Jim (J.R.)

Friday, April 11, 2014

Refilling Ink-Jet Cartridges
Is It Really Worth It?

So. . . .  You're thinking about refilling your ink-jet cartridges?

In concept, the idea has merit.  Since you often have a perfectly good cartridge, all it's missing is the ink that used to be in it.  And it makes sense to replace the ink instead of the cartridge.  Or, as one refilling site put it, you wouldn't replace your car just because you ran out of gas.  Right?

One of the points that the cartridge refill sites love to make is that most, (if not all!), ink-jet manufacturers sell cartridges that are woefully short of ink.  To illustrate this point the folks at Printer Filling Station showed the result of a cartridge tear-down.  What these folks did was to take an OEM cartridge that was brand new, right out of the box in its sealed foil packaging, and open it up.  What they discovered was that it was actually less than half-full.

Photos courtesy of  http://www.printerfillingstation.com

And that's exactly where the printer manufacturers want it, because when you think about it, the printer manufacturers aren't selling the printer, they're selling the supplies because that's where the huge profits are.  And it's a virtual lead-pipe-cinch that if you have two printers manufactured by the same company, purchased at the same time in the same store. they will not use the same cartridge, unless they're the same model, or you're extremely lucky.  So, you need to stockpile multiple cartridge types if you have multiple printers.

Another kicker is that ink-jet cartridges have a very limited shelf-life, (like six months or so), even if the sealed package is not opened.  If you are smart enough to have spares on hand, and if you bought them more than six months ago, there's a good chance they won't work, or won't work properly.  And that's that.

That being said, this leaves you with the following choices:
  • Punt and Go Retro:
    Pen-and-ink, as well as pencil and paper still work, and it's a lot less expensive than any of the other choices.  Not to mention that typewriters can be had for a song at garage sales and flea-markets.

  • Abandon ink-jet technology:
    Laser printers are often more cost-effective on a per-page basis, and the supplies don't usually have the shelf-life problems of ink-jet cartridges.  Unfortunately a good laser printer is not cheap, and a cheap laser printer is often not very good.

  • Cave in and take your lumps:
    You buy the "Genuine" manufacturers cartridges, (that might be significantly under-filled), at hugely inflated prices.

  • Aftermarket Supplies:
    You can buy a "refurbished" cartridge for about half the price of buying one brand-new.  This is an attractive option for those who don't want to mess with refilling cartridges themselves, but don't like the idea of being taken to the cleaners by the ink-jet manufacturers.

  • Refill the cartridge yourself:
    You can refill the cartridges on your own and save huge amounts of money.

The last bullet-point brings us to the topic of this post:  Is it really worth it to do manual refills yourself?



Like everything else in life, there are two sides to this question, and depending on what your priorities are, refilling cartridges yourself may, or may not, be your cup of tea.  So, let's take a look at the pro's and con's of refilling.

The Good:
  • Cost:
    Refilling a cartridge can be significantly less expensive than buying a new one.
    • The folks at Printer Filling Station, sell a refill kit with a full set of inks, (three colors and black), for my HP OfficeJet 6500, right at $40 if you use the pigmented black ink, (which, IMHO, is the best choice), or $35 if you get the "regular" black ink.  Having bought the kit, you can get four, five, or maybe even six refills depending on the cartridge, and the size of the ink bottles you bought.
    • Here's a comparison of "Club" prices for "OEM" 920XL ink for my HP6500 printer:
      • Sam's Club:
        920XL Black two-pack: $61.98
        920XL Color three-pack, (one each of C, Y and M) $51.48
        Total:  $113.46
        ($82.47 based on 1/2 the double cartridge price for the black ink)
        See Search Results Here
      • BJ's:
        920XL Black two-pack: $57.99
        920XL Color three-pack, (one each of C, Y and M) $42.99
        Total:  $100.98
        ($71.99 based on 1/2 the double cartridge price for the black ink)
        See Search Results Here
      • Costco:
        920XL Black two-pack: $61.89
        920XL Color three-pack, (one each of C, Y and M) $41.89
        Total:  $103.87
        ($72.84 based on 1/2 the double cartridge price for the black ink)
        See Search Results Here

    • A comparison of non-club retailers for the same supplies:
      • Wall-Mart:
        920XL Black single cartridge pack: $34.48
        920XL Color cartridge single-packs, one each of Cyan, Magenta, and Yellow at $15.98 each: $47.94
        Total:  $82.42
        See Search Results Here
      • Best Buy:
        920XL Black single cartridge pack: $34.99
        920XL Color cartridge single-packs, one each of Cyan, Magenta, and Yellow at $15.99 each:  $47.97
        Total:  $82.95
        See Search Results Here
      • Staples:
        (They also have a combo pack, one 920XL black, and three standard 920 cartridges for the three colors for $58.89)
        920XL Black single cartridge pack: $34.99
        920XL Color cartridge single-packs, one each of Cyan, Magenta, and Yellow at $15.99 each:  $47.97
        Total:  $82.96
        See Search Results Here

      • Newegg, Tiger Direct, and OfficeMax had identical pricing to Staples when I checked.

      • Micro Center did not stock the "genuine" HP 920xl inks when I checked them.  They did sell refilled cartridges for about half everyone else's OEM cartridge price.

      • Some suppliers also sell aftermarket cartridges, but I did not include that information here since I wanted to show the pricing for OEM cartridges.

    • Good sites also give you everything you need to refill your cartridges, as well as four bottles of ink - the three colored inks and a bottle of black.  They also give you the support you need to get you "over the hump" of refilling.

  • Ecological:
    I like the idea of replacing the ink, not the cartridge, since the cartridges themselves can often be re-used over and over and over again.  By simply replacing the ink, you keep a whole lot of cartridges out of the landfill.

The Bad:
  • It's time consuming.  Refilling a cartridge is not something you do in five minutes, or even fifteen minutes.  Especially if you're not experienced or, like me, you don't have the world's steadiest hands.

  • It's messy.  Some refill sites will tell you how clean, and easy, refilling is.  Don't you believe it!  When I refill cartridges, I have a big, thick towel to put on the desk, I wear a smock to protect my clothing, and I wear the crappiest, grungiest shirt and pair of pants I own.  And I don't wear good shoes.

  • It requires a bit of skill and patience.  Like I said before, this isn't a quick job.  If you want just "plug-and-play" you should go with refilled cartridges, or brand new.

  • Your printer might not want to use refilled cartridges.  Most printer manufacturers do NOT want to give up their ink-sale profits, so they program the printer's firmware to be really pissy and anal about cartridges and their replacements.
    • It is not uncommon for your printer to receive an "update" and afterward you discover that the refilled cartridges you have been using for years, stop working.
    • If the printer is totally anal about this, (there are some Epson and Canon models that do this), you can purchase a small electronic tool that resets the cartridges chip to read "full" again.
    • Other cartridges cannot be reset by reasonable means, but you can buy replacement "chips" that you use to replace the existing chip, allowing you to use the cartridge again.

The Ugly:
  • Printer manufacturers do not want you to refill your cartridges!  Why?  Consumables are a HUGE source of profits for the printer manufacturers.[1]
    • John Shane, a director at InfoTrends/CAP Ventures and an industry expert on the ink and toner market, had this to say about it on news.cnet.com
      The estimated retail value for cartridges used in HP inkjet machines in the United States in 2004 was about $6.3 billion, according to Shane. That's just more than half the $12 billion Shane estimates as the amount for all cartridges for all machines used for desktops last year.
    • This same blog posting has a number of user comments, most of which go something like this:
      I am convinced that HP is deliberately doing something to prevent the use of replacement cartridges. My HP Photosmart C4680 rejects replacement cartridges, even ones that go under their specs. I don't know if they're somehow installing something on my machine (especially during any updates) or software installations.
Assuming you have the patience to deal with all of this, then it is possible that refilling cartridges is right for you.



So, you want to take a try at refilling?  What's next?

Again, just like everything else, there are things you have to know before you just dive in.
  • Get GOOD supplies:
    There are a lot of refill sites, and there are a lot of refill kits available in stores, and most of them are pure junk.  The inks are total crap, the tools stink, and it's just a bad deal all around.
    • The solution is to do your homework, and find a refill site that sells good supplies, and gives you the information you need to make an informed choice.  One of the best I've seen, (and the one I use), is Printer Filling Station.  They're located within the US, (Georgia to be exact), so you don't have to wait for your stuff to come by Hong Kong Post.
    • These guys have the best ink I've seen - and I've seen a bunch.  (And I have the totally ruined shirts and pants to prove it!)
    • Their support is nothing less than amazing.  I had some troubles with some HP 920XL cartridges and Gordon, the "haid-man-boss" down there, was with me the whole way.  Making suggestions, shipping out replacement components; it was like I was dealing with a member of the family, not a merchant in a state hundreds of miles away.

  •  Make sure you have a clear and uncluttered work area that can get messy.
    • Despite your best efforts, you will have to deal with some ink going where it does not belong.
      • As a side note, Windex is an excellent solvent/cleaner for ink that goes awry.
    • Wear old clothes, or a smock/apron of some kind and - if you can deal with them - latex/nitrile gloves.
    • Keep rags or paper towels around for general cleanup.

  • It is extremely important that you avoid cross-contamination of colors at all costs, since contaminated colors, even slightly contaminated colors, are essentially useless.
    • This is usually accomplished by the use of separate, clearly labeled syringes and needles for each color as well as black, and making sure that the syringes are thoroughly cleaned after use.
    • It is important to make sure that you do not press the syringe plunger all the way down into the barrel of the syringe after cleaning.  Since the plunger will stick, pulling it back about a half inch or so while it is still slightly damp gives you room to push back in the next time you want to use it, freeing the plunger up.  If it is pressed all the way in, it is often impossible to remove the plunger without the rubber tip coming completely off.

  • If you are going to be refilling your own cartridges - or buying "reconditioned" cartridges - it is absolutely essential that you do not allow any firmware or software update downloads to your printer.  Despite what the manufacturers say, these updates often have little to do with the "performance" or "quality" of your printer.  Rather they are periodic updates to the firmware so that it will recognize, and reject, refilled cartridges.[1]  HP's own web site says it succinctly:
    The HP Cartridge Authentication feature for HP printers helps protect you ["You"?  They mean their 6.3 billion dollar annual profits!!] by checking and verifying the authenticity of each cartridge installed in your printer. HP Cartridge Authentication comes standard with all HP inkjet printers and All-in-Ones.
  • It is also important to carefully read, and follow, the instructions provided by whomever supplies the refilling supplies.
Done carefully, with a certain amount of forethought, ink cartridge refilling can be a very economical way to stretch your printing supplies dollar.

What say ye?

Jim (J.R.)

[1]  There are a number of web-sites out there that discuss the practice of printer manufacturers deliberately "killing" replacement cartridges to force you to buy only the higher priced OEM consumables.

Rumors Circulate That Lexmark Firmware Update Locks Out Non-OEM Supplies

http://inkandtonerexperts.com/support/troubleshooting.php where they say:
Turn off auto updates - If you have your printer set up to do automatic updates from the printer manufacturer, you need to turn that function "off" or to "manual" instead of automatic. The reason this is important is because some manufacturers have been known to use this software upgrade as a tool to identify refilled cartridges in your printer and "kill" the cartridge(s).
HP says I can't use refilled ink

HP cracks down on cartridge refill industry

Firmware Updates Prevent You from Using Refilled Cartridges

Tuesday, April 1, 2014

QA Tech-Tips
Where Do We Go From Here?

Well Hello Again, Everybody!

This blog has come a long, long way since I initially started it here at Blogger.com back in December of 2009,  The 28th of December, 2009, if my dates are correct.  And even further back if you consider the original e-mail based version of QA Tech-Tips that was circulating well before that time.

Since then this blog has begun to attract attention, though the folks at Google have nothing to fear from me.  At least not yet.

QA Tech-Tips is even beginning to get quoted as a source.  Though I am tickled pink when Microsoft TechNet quotes my blog, (or links to one of my articles!); what really makes me happy is when I hear from someone in the QA or technical community telling me that they referenced one of my articles in a bug report!  Booyah!!

While this is gratifying, it is also tremendously humbling to know that readers actually listen, and are interested in what I have to say.  As Voltaire once said, "With Great Power Comes Great Responsibility".

Though there may have been changes, (not the least of which is the improvement in my own writing skills!), I have tried to keep true to my original idea for the QA Tech-Tips blog; that is, present useful information about computers and software, as well as the Wild and Wooly things that can happen when you toss a computer and some code into the mix.  Not only present it, but present it in a way that is pertinent and useful down where "The Rubber Meets The Road", instead of some strange hair-ball theories masquerading as facts.



All that being said; both I, and this blog, have begun to outgrow Blogger.

First, there is the BIG issue of editorial control; especially over posted comments since Blogger won't let me edit a comment.

I am often torn between the Devil of throwing out a perfectly good comment because of something I consider inappropriate, or the Deep Blue Sea of allowing inappropriate content in blog replies.  I would greatly prefer to allow replies to stand, (though I might want to snip out the occasional self-serving hyperlink advertising something), instead of having to chuck the whole thing because I cannot edit out a #&%$#*&@!! or two.

Second, is my own desire to grow and improve the QA Tech-Tips experience.

There are things I would like to be able to do that Blogger does not support.  One of the most important improvements that I would like to make is the ability to break up longer posts into multiple pages.  Unfortunately, Blogger does not support multi-page articles in the way that other blogging environments do.

Not to mention a whole host of things that, though not fatal, are intensely annoying.  For example, I have to manually edit the article's underlying HTML code to force an underline on each and every hyperlink.  Doh!

I could go on and on with the things I'd like to see QA Tech-Tips do, that just can't be done here.



Now I want to make perfectly clear that I am eternally indebted to the folks here at Blogger.com, (and Google their parent), for allowing me to have a wonderful - and free! - blog here for years and years.  Were it not for them, it is very likely that the QA Tech-Tips blog would not be here at all today, let alone where it is after all this time.

Likewise, I have absolutely no hesitation whatsoever recommending them to anyone who might want to "get their feet wet" in the Blogosphere without dropping a lot of coin, (or time for that matter), into getting a blog up and running.  They implement a lot of nice features as pre-built widgets - things like user registration, RSS feeds, or Captcha checking - for everyone to use without the pain and grief of having to code it all.

They also take care of the noisy and messy "System Administration" kind of things - like security-hardening my blog - so that I can spend time on my blog's content instead of laying awake nights worrying about getting hacked.



Nonetheless, I am beginning to think that it's high-time to loose the training-wheels and move QA Tech-Tips to the next level.

And to do that I have begun thinking about things like getting my own web host, deciding on what blogging software / platform I should use, along with all the hairy "SysAdmin" kind of things that go Bump In The Night when you strike out on your own.

'Guess I'm gonna be eatin' some Prime Filet of Fingernail while I figure this all out!

What say ye?

Jim (J.R.)



Monday, February 3, 2014

OOPS! - When disaster strikes
Recovering Lost Files (Part 3 of a series)

This article discusses a particularly painful issue:  You've deleted something you really didn't want to, and you need to recover it, come hell or high-water.  So what's a poor fool to do?

Hopefully, if you paid any attention at all to the second article of this series, you'll have a good backup of what you wanted.  If that's the case, you're golden.  All you need to do is go to your backup, grab the missing file(s), and you're back in business - no muss, no fuss, as the laundry detergent commercials used to say.

If the missing file is something you retrieved or downloaded from somewhere - perhaps from the Internet - you're still golden as you can simply go back and grab it again.  Again the famous jingle of those old laundry commercials wafts its golden strains through the air.  (And perhaps you will save that file off-line somewhere in case the site goes down or they loose the file.)

However, if you find yourself in the incredibly unlucky position of having just deleted an absolutely irreplaceable file, have no backups, and "failure is not an option". . . .

Though you may feel thoroughly screwed to the wall, it is possible that all is not lost.



The descriptions and comments in this article are aimed primarily at the Windows user, because files lost in a Windows-type filesystem are fairly easy to retrieve.  If you're not a Windows user, read on anyway.  Though it may be much more difficult for you to recover a lost file, the hints and tips in this article apply to you as much as they do anyone else.



There are a number of file recovery methods.  Here's a list starting from the easiest to the most difficult:

The Recycle Bin
A feature that has become popular in most every modern operating system is the "Recycle Bin", "Trash", or whatever your particular flavor of operating system calls it.  This is a special, hidden, area within the filesystem where deleted files are stored, just in case you decide you want them back.  So long as you didn't do a "permanent delete" or a delete that "bypasses the Recycle Bin", you can restore the lost file by simply going to the Recycle Bin / Trash, and restoring the file.

Auto-Save files
Assuming that you are working within some kind of formal environment - a word processor, a spreadsheet, a programming development environment, a database, etc. - these programs can, and often do, create timed automatic backups while you work.

The advantage of this feature is that the program itself periodically takes a snapshot of what you are doing, and saves it in a special file, located somewhere on your hard drive.

In other cases, some programs keep a "one revision back" backup.  When you finally save your work, the original file is renamed with a ".BAK" extension and the modified file is written out as a new file with the original file name.

Automatic file backup software
Many backup utilities, Acronis True Image among them, have a feature where earlier versions of files can be automatically preserved.  In the case of Acronis, you can allow the software to maintain a special hidden partition at the end of the drive where various important things - like file versions - can be stored away.

Shadow Copies
Windows Vista, Windows 7, (and maybe Windows 8?), have a feature called "Shadow Copies" where the operating system itself maintains a series of older versions of certain files.  Mac users will recognize the same idea in their "Time Machine" feature.

It is important to note that the "shadow copy" feature of Windows is usually NOT enabled by default, and depending on your system's configuration, you may have to modify your computer's partition structure to enable it.  (Ouch!)

Scattered "temp" files
Occasionally programs like Word, or Word Perfect, don't clean up behind themselves very well and, (sometimes), they leave scattered "temporary" snapshot files laying around; usually beginning with a tilde. (~)  It's a bit dicey, but if you're lucky you can recover enough of a file to recreate what was lost.

Undelete Utilities
If all else fails, there are special software utilities that can scan your drive for deleted files and attempt to recover them.



Undeleting a file

All "undelete" utilities depend on one simple fact:  Files that have been really and truly deleted are - in fact - not really and truly deleted.  The directory entry for that file is marked as "deleted" and all the logical blocks allocated to the file are reset to the "available for use" status in the file system's volume bitmap.  However, the actual data that has been written to the disk is NOT deleted - a fact that has lead to many a criminal's downfall.  Not to mention things like divorces, getting fired, or even identity theft.

So, what REALLY happens when Windows deletes a file?
When Windows deletes a file, what happens is something like this:
  • First, Windows finds all the pieces of the file.  That includes the file's directory entry, along with each and every part of the file itself.
  • Then Windows places a special "mark" on the directory entry that says this file has been deleted.  Because of this, the "deleted" file disappears from the folder it was in.
  • Finally, Windows goes to the system's volume bitmap and - for each and every part of the file - turns the corresponding bits OFF, indicating that these parts are "empty", and are available to be re-used.
It's just like an apartment when someone moves out.  The apartment itself doesn't disappear, it's just that the "landlord", (Windows), puts a big "FOR RENT" sign on the apartment's door.

If you can get back to the file quickly enough - before someone else moves in! - you can take down the big "FOR RENT" sign, set all the volume bitmap bits for that file back ON again, and VoilĂ !  Your file is back.

The problem is this:  Windows is constantly doing things with the hard drive; moving things around, updating important information, figuring out what goes where, and deciding where it wants to put the next piece of whatever it's doing.  Because of this, "empty apartments" don't stay empty for long, and some of the individual locations and pieces that belonged to your file can get recycled and re-used by other files.  If that happens before you get back to the "apartment" you just left, you're out of luck.  It's already taken and it just stinks being you.

If that's true, how do undelete utilities work?
Undelete utilities can use different methods to do their job, and one possible way is like this:
  • The undelete utility scans the disk until it finds a directory entry that has been marked as "deleted".
  • If the directory entry itself isn't corrupted, it attempts to traverse the entire file, end-to-end, using the file's metadata within the directory entry, as well as the "next" and "previous" links in each of the file's individual parts, to verify the integrity of the prospective file.
    • If the program can traverse the entire file end-to-end, and the resulting file data matches all the file metadata in the deleted directory entry, (size, etc.), then classify this file as "excellent".  At this point you have a pretty good chance of getting the entire file back.
    • If the program can apparently traverse the entire file end-to-end and it appears to be complete, but there are discrepancies between the resulting file and the file's metadata, then classify the file as "good".  At this point you have a pretty good chance of getting, (at least), part of the file back.  Maybe it's OK, maybe not, but it's at least an even-money bet.
    • If the program can only traverse part of the file because the file appears to have been overwritten at some point, (i.e. segment "x" points to segment "y", but segment "y" does not point back to "x"), then classify the file as "poor".  At this point you know for certain that the file has been damaged, but depending on what kind of file it is, and how much of the file the undelete utility can find, it might still be usable - at least in part.
    • If the program cannot traverse any portion of the file because the very first segment of the file has been overwritten, then classify the file as "unrecoverable".  At this point you can pretty much throw in the towel unless you want to use the more advanced, (and chancy), features of the program.  Your ability to recover even part of the file is now in the hands of God, and the cleverness of the utility's programmer.
  • Then, go to the next directory entry marked as "deleted" and repeat all of the above; continuing until all "deleted" directory entries have been examined and classified.
  • Once everything it can find has been found and classified, you get the opportunity to select what you want to try and recover.

Even if the file is classified as "excellent", there is still a very real possibility that the file will be borked when you try to recover it.  Why?  Because Windows is still working with the disk even while YOU are trying to undelete something.  Not only is this possible, it's not all that uncommon either.  Like I said before, "undeleting" a file is a dicey proposition at best.

But what if all that effort doesn't work, and my file is still missing?
Many undelete utilities also include a special "advanced", "deep", "full", or "exhaustive" scan mode for finding files that the easier methods miss.  This is based, (at least in part), on the fact that files of a particular type have distinctive characteristics unique to that type of file.  For example, JPEG picture files begin and end with sequences of bytes that are specific and unique to JPEGs.

The advanced scan mode, (using the JPEG file type we mentioned above as an example), scans the entire drive, one tiny piece at a time, looking for the distinctive characteristics of a JPEG file.  If it finds the specific characteristic that indicates what might be the beginning of a JPEG file, it tries to follow the trail all the way to the end of that file.  Likewise, if it finds the other unique characteristic first, (indicating the possible end of a JPEG file), it tries to follow that trail back to its beginning.  And it's very likely to be a false scent.  If it is, you start all over again, continuing your search, one tiny piece at a time.  Again, and again, and again.

When you're doing this kind of microscopically detailed search, you've entered the wild, woolly, and wonderful world of "forensic analysis", and it is not unusual for some random sector to contain what appears to be the "magic bytes" for a particular file type.  Or, maybe you have found a JPEG file, but it's one that hasn't been deleted!  Because of all the random factors that can be encountered, searches of this kind can be very time consuming.  As in days, (or even longer!), depending on how much experience you have doing this kind of stuff, and the size of the hard drive you're searching.

If we assume that you're lucky enough to find what you believe are all the pieces of a particular file, the undelete utility assumes that you've found a complete file of that type and classifies the file as potentially recoverable.

If the file is not complete, but at least some of it is present, it classifies the found file as either "good" or "poor" depending on what, and how much, of the file has been found.

And, in some cases, the file is just gone; never to be seen again.  And that's that.

Once all that is done, if you decide to recover the file(s) that were found, the utility recovers, (or creates if necessary), the file's directory entry and marks all the file's pieces as "occupied" in the system's volume bitmap.



How to improve your chances for recovering a file or files

If you've gotten this far, and are seriously contemplating going the "undelete" path, there are things you can do that will dramatically improve your chances of getting your data back.
  • Preparation:
    Prior to needing them, get one or more undelete utilities that do not require installation, (this is important!), and save them somewhere you can get your greasy mitts on them in a hurry. (A thumb-drive, an external hard drive, a CD or whatever are all good choices.)
    • If you have the skill, time, and want to really cover your butt, you might want to consider creating a Bart PE rescue disk, (or something similar), with the undelete software included.  If you don't have the requisite skill, you might want to beg a more technical friend or colleague, pretty, pretty, pretty-please, to make one for you.  It's a really great idea.
  • Have more than one "gun" and plenty of ammo:
    Different undelete programs work in different ways, so just because program "A" doesn't find your file, does not mean that program "B" can't.
    • When you collect your undelete utility(s), get more than one.  Professional forensic analysts often use many different programs to examine a disk to increase their chances of getting what they want.
    • Consider investing in a combination of freeware, shareware, and possibly even payware programs.
    • If you're smart/lucky enough to have something like a Bart PE disk, put all the utilities you've collected on it so that they are all available when you need them.
  • Protection: 
    If you accidentally delete something important, it is absolutely crucial that the disk be taken off-line quickly!  You'll want to do this as rapidly as possible to prevent the sectors used by your deleted file from being overwritten.
    • If possible, shut-down the system, (if it's the system drive), or un-mount it quickly, (if it's a external or data-drive), to reduce the possibility of it being corrupted.
  • Investigation: 
    If at all possible
    , you'll want to take the drive and mount it externally on another system, or load an external operating system. (like a Bart PE disk for Windows.)
    • Mount the drive read only if possible to prevent data corruption.
    • Use the external system or CD's software, (if you prepared a Bart PE disk), to search for the files.
    • If you are using an advanced recovery method, and you know what kind of file you're looking for, (a JPEG for example), have the recovery utility search for only that type of file, (assuming that this type of file is defined in the utility's dictionary).  This will speed up the search process considerably, and will result in a much better chance of getting complete files.
  • Execution: 
    If you find your missing files, try to restore them to another disk drive, (like a thumb-drive), if your undelete utility supports that option.
  • Prayer: 
    I'm not including this just to be funny or sarcastic.  As you can see, undeleting something is a very iffy proposition at it's best, and a successful undelete depends just as much on luck as it does on skill.
  • Appreciation:
    Everyone likes appreciation, and the poor sod of a software writer likes it even more.  Especially since the only communication he's likely to get from his users are complaints!
    • In all seriousness, if the utility is shareware and it just saved your butt bigtime, dig out the Benjamins and buy the license.  It's the least you can do.
    • If it's freeware - and there's a "donate" option, cough up the dough.  At the very least buy him a six-pack of his favorite brew.  Or, absent that feature, send him an e-mail telling him how much you appreciate his skill and thoughtfulness.



Linux/Mac Users:

If you're dealing with a Linux/Unix/Mac type operating system, there's a really good possibility that you may just be plain-ole' hozed if you don't have something like "Time Machine" turned on.

Unix type systems have unique kinds of file systems, file deletion methods, and aggressive i-node recovery paradigms that they often use. (i-nodes contain the file descriptors, and all the pointers and metadata for a particular file or files.)  Because of their design, it may be almost completely impossible to recover files deleted on a 'nix system without resorting to extremely advanced techniques that are, (usually), beyond the ability and experience of even the most advanced system administrators.



Conclusion:

With any luck I have, (hopefully), scared you s**tless about the prospect of loosing important files or data.  And if I have done so, please, please, pretty-please consider using backups as a recovery strategy instead of "a wing and a prayer".  OK?

What say ye?

Jim (JR)