QA Tech-Tips

Tuesday, April 1, 2014

QA Tech-Tips
Where Do We Go From Here?

Well Hello Again, Everybody!

This blog has come a long, long way since I initially started it here at Blogger.com back in December of 2009, The 28th of December, 2009, if my dates are correct. And even further back if you consider the original e-mail based version of QA Tech-Tips that was circulating well before that time.

Since then this blog has begun to attract attention, though the folks at Google have nothing to fear from me. At least not yet.

QA Tech-Tips is even beginning to get quoted as a source. Though I am tickled pink when Microsoft TechNet quotes my blog, (or links to one of my articles!); what really makes me happy is when I hear from someone in the QA or technical community telling me that they referenced one of my articles in a bug report! Booyah!!

While this is gratifying, it is also tremendously humbling to know that readers actually listen, and are interested in what I have to say. As Voltaire once said, "With Great Power Comes Great Responsibility".

Though there may have been changes, (not the least of which is the improvement in my own writing skills!), I have tried to keep true to my original idea for the QA Tech-Tips blog; that is, present useful information about computers and software, as well as the Wild and Wooly things that can happen when you toss a computer and some code into the mix. Not only present it, but present it in a way that is pertinent and useful down where "The Rubber Meets The Road", instead of some strange hair-ball theories masquerading as facts.

All that being said; both I, and this blog, have begun to outgrow Blogger.

First, there is the BIG issue of editorial control; especially over posted comments since Blogger won't let me edit a comment.

I am often torn between the Devil of throwing out a perfectly good comment because of something I consider inappropriate, or the Deep Blue Sea of allowing inappropriate content in blog replies. I would greatly prefer to allow replies to stand, (though I might want to snip out the occasional self-serving hyperlink advertising something), instead of having to chuck the whole thing because I cannot edit out a #&%$#*&@!! or two.

Second, is my own desire to grow and improve the QA Tech-Tips experience.

There are things I would like to be able to do that Blogger does not support. One of the most important improvements that I would like to make is the ability to break up longer posts into multiple pages. Unfortunately, Blogger does not support multi-page articles in the way that other blogging environments do.

Not to mention a whole host of things that, though not fatal, are intensely annoying. For example, I have to manually edit the article's underlying HTML code to force an underline on each and every hyperlink. Doh!

I could go on and on with the things I'd like to see QA Tech-Tips do, that just can't be done here.

Now I want to make perfectly clear that I am eternally indebted to the folks here at Blogger.com, (and Google their parent), for allowing me to have a wonderful - and free! - blog here for years and years. Were it not for them, it is very likely that the QA Tech-Tips blog would not be here at all today, let alone where it is after all this time.

Likewise, I have absolutely no hesitation whatsoever recommending them to anyone who might want to "get their feet wet" in the Blogosphere without dropping a lot of coin, (or time for that matter), into getting a blog up and running. They implement a lot of nice features as pre-built widgets - things like user registration, RSS feeds, or Captcha checking - for everyone to use without the pain and grief of having to code it all.

They also take care of the noisy and messy "System Administration" kind of things - like security-hardening my blog - so that I can spend time on my blog's content instead of laying awake nights worrying about getting hacked.

Nonetheless, I am beginning to think that it's high-time to loose the training-wheels and move QA Tech-Tips to the next level.

And to do that I have begun thinking about things like getting my own web host, deciding on what blogging software / platform I should use, along with all the hairy "SysAdmin" kind of things that go Bump In The Night when you strike out on your own.

'Guess I'm gonna be eatin' some Prime Filet of Fingernail while I figure this all out!

What say ye?

Jim (J.R.)

Monday, February 3, 2014

OOPS! - When disaster strikes
Recovering Lost Files (Part 3 of a series)

This article discusses a particularly painful issue: You've deleted something you really didn't want to, and you need to recover it, come hell or high-water. So what's a poor fool to do?

Hopefully, if you paid any attention at all to the second article of this series, you'll have a good backup of what you wanted. If that's the case, you're golden. All you need to do is go to your backup, grab the missing file(s), and you're back in business - no muss, no fuss, as the laundry detergent commercials used to say.

If the missing file is something you retrieved or downloaded from somewhere - perhaps from the Internet - you're still golden as you can simply go back and grab it again. Again the famous jingle of those old laundry commercials wafts its golden strains through the air. (And perhaps you will save that file off-line somewhere in case the site goes down or they loose the file.)

However, if you find yourself in the incredibly unlucky position of having just deleted an absolutely irreplaceable file, have no backups, and "failure is not an option". . . .

Though you may feel thoroughly screwed to the wall, it is possible that all is not lost.

The descriptions and comments in this article are aimed primarily at the Windows user, because files lost in a Windows-type filesystem are fairly easy to retrieve. If you're not a Windows user, read on anyway. Though it may be much more difficult for you to recover a lost file, the hints and tips in this article apply to you as much as they do anyone else.

There are a number of file recovery methods. Here's a list starting from the easiest to the most difficult:

The Recycle Bin
A feature that has become popular in most every modern operating system is the "Recycle Bin", "Trash", or whatever your particular flavor of operating system calls it. This is a special, hidden, area within the filesystem where deleted files are stored, just in case you decide you want them back. So long as you didn't do a "permanent delete" or a delete that "bypasses the Recycle Bin", you can restore the lost file by simply going to the Recycle Bin / Trash, and restoring the file.

Auto-Save files
Assuming that you are working within some kind of formal environment - a word processor, a spreadsheet, a programming development environment, a database, etc. - these programs can, and often do, create timed automatic backups while you work.

The advantage of this feature is that the program itself periodically takes a snapshot of what you are doing, and saves it in a special file, located somewhere on your hard drive.

In other cases, some programs keep a "one revision back" backup. When you finally save your work, the original file is renamed with a ".BAK" extension and the modified file is written out as a new file with the original file name.

Automatic file backup software
Many backup utilities, Acronis True Image among them, have a feature where earlier versions of files can be automatically preserved. In the case of Acronis, you can allow the software to maintain a special hidden partition at the end of the drive where various important things - like file versions - can be stored away.

Shadow Copies
Windows Vista, Windows 7, (and maybe Windows 8?), have a feature called "Shadow Copies" where the operating system itself maintains a series of older versions of certain files. Mac users will recognize the same idea in their "Time Machine" feature.

It is important to note that the "shadow copy" feature of Windows is usually NOT enabled by default, and depending on your system's configuration, you may have to modify your computer's partition structure to enable it. (Ouch!)

Scattered "temp" files
Occasionally programs like Word, or Word Perfect, don't clean up behind themselves very well and, (sometimes), they leave scattered "temporary" snapshot files laying around; usually beginning with a tilde. (~) It's a bit dicey, but if you're lucky you can recover enough of a file to recreate what was lost.

Undelete Utilities
If all else fails, there are special software utilities that can scan your drive for deleted files and attempt to recover them.

Undeleting a file

All "undelete" utilities depend on one simple fact: Files that have been really and truly deleted are - in fact - not really and truly deleted. The directory entry for that file is marked as "deleted" and all the logical blocks allocated to the file are reset to the "available for use" status in the file system's volume bitmap. However, the actual data that has been written to the disk is NOT deleted - a fact that has lead to many a criminal's downfall. Not to mention things like divorces, getting fired, or even identity theft.

So, what REALLY happens when Windows deletes a file?
When Windows deletes a file, what happens is something like this:

First, Windows finds all the pieces of the file. That includes the file's directory entry, along with each and every part of the file itself.
Then Windows places a special "mark" on the directory entry that says this file has been deleted. Because of this, the "deleted" file disappears from the folder it was in.
Finally, Windows goes to the system's volume bitmap and - for each and every part of the file - turns the corresponding bits OFF, indicating that these parts are "empty", and are available to be re-used.

It's just like an apartment when someone moves out. The apartment itself doesn't disappear, it's just that the "landlord", (Windows), puts a big "FOR RENT" sign on the apartment's door.

If you can get back to the file quickly enough - before someone else moves in! - you can take down the big "FOR RENT" sign, set all the volume bitmap bits for that file back ON again, and Voilà! Your file is back.

The problem is this: Windows is constantly doing things with the hard drive; moving things around, updating important information, figuring out what goes where, and deciding where it wants to put the next piece of whatever it's doing. Because of this, "empty apartments" don't stay empty for long, and some of the individual locations and pieces that belonged to your file can get recycled and re-used by other files. If that happens before you get back to the "apartment" you just left, you're out of luck. It's already taken and it just stinks being you.

If that's true, how do undelete utilities work?
Undelete utilities can use different methods to do their job, and one possible way is like this:

The undelete utility scans the disk until it finds a directory entry that has been marked as "deleted".
If the directory entry itself isn't corrupted, it attempts to traverse the entire file, end-to-end, using the file's metadata within the directory entry, as well as the "next" and "previous" links in each of the file's individual parts, to verify the integrity of the prospective file.

If the program can traverse the entire file end-to-end, and the resulting file data matches all the file metadata in the deleted directory entry, (size, etc.), then classify this file as "excellent". At this point you have a pretty good chance of getting the entire file back.
If the program can apparently traverse the entire file end-to-end and it appears to be complete, but there are discrepancies between the resulting file and the file's metadata, then classify the file as "good". At this point you have a pretty good chance of getting, (at least), part of the file back. Maybe it's OK, maybe not, but it's at least an even-money bet.
If the program can only traverse part of the file because the file appears to have been overwritten at some point, (i.e. segment "x" points to segment "y", but segment "y" does not point back to "x"), then classify the file as "poor". At this point you know for certain that the file has been damaged, but depending on what kind of file it is, and how much of the file the undelete utility can find, it might still be usable - at least in part.
If the program cannot traverse any portion of the file because the very first segment of the file has been overwritten, then classify the file as "unrecoverable". At this point you can pretty much throw in the towel unless you want to use the more advanced, (and chancy), features of the program. Your ability to recover even part of the file is now in the hands of God, and the cleverness of the utility's programmer.

Then, go to the next directory entry marked as "deleted" and repeat all of the above; continuing until all "deleted" directory entries have been examined and classified.
Once everything it can find has been found and classified, you get the opportunity to select what you want to try and recover.

Even if the file is classified as "excellent", there is still a very real possibility that the file will be borked when you try to recover it. Why? Because Windows is still working with the disk even while YOU are trying to undelete something. Not only is this possible, it's not all that uncommon either. Like I said before, "undeleting" a file is a dicey proposition at best.

But what if all that effort doesn't work, and my file is still missing?
Many undelete utilities also include a special "advanced", "deep", "full", or "exhaustive" scan mode for finding files that the easier methods miss. This is based, (at least in part), on the fact that files of a particular type have distinctive characteristics unique to that type of file. For example, JPEG picture files begin and end with sequences of bytes that are specific and unique to JPEGs.

The advanced scan mode, (using the JPEG file type we mentioned above as an example), scans the entire drive, one tiny piece at a time, looking for the distinctive characteristics of a JPEG file. If it finds the specific characteristic that indicates what might be the beginning of a JPEG file, it tries to follow the trail all the way to the end of that file. Likewise, if it finds the other unique characteristic first, (indicating the possible end of a JPEG file), it tries to follow that trail back to its beginning. And it's very likely to be a false scent. If it is, you start all over again, continuing your search, one tiny piece at a time. Again, and again, and again.

When you're doing this kind of microscopically detailed search, you've entered the wild, woolly, and wonderful world of "forensic analysis", and it is not unusual for some random sector to contain what appears to be the "magic bytes" for a particular file type. Or, maybe you have found a JPEG file, but it's one that hasn't been deleted! Because of all the random factors that can be encountered, searches of this kind can be very time consuming. As in days, (or even longer!), depending on how much experience you have doing this kind of stuff, and the size of the hard drive you're searching.

If we assume that you're lucky enough to find what you believe are all the pieces of a particular file, the undelete utility assumes that you've found a complete file of that type and classifies the file as potentially recoverable.

If the file is not complete, but at least some of it is present, it classifies the found file as either "good" or "poor" depending on what, and how much, of the file has been found.

And, in some cases, the file is just gone; never to be seen again. And that's that.

Once all that is done, if you decide to recover the file(s) that were found, the utility recovers, (or creates if necessary), the file's directory entry and marks all the file's pieces as "occupied" in the system's volume bitmap.

How to improve your chances for recovering a file or files

If you've gotten this far, and are seriously contemplating going the "undelete" path, there are things you can do that will dramatically improve your chances of getting your data back.

Preparation:
Prior to needing them, get one or more undelete utilities that do not require installation, (this is important!), and save them somewhere you can get your greasy mitts on them in a hurry. (A thumb-drive, an external hard drive, a CD or whatever are all good choices.)

If you have the skill, time, and want to really cover your butt, you might want to consider creating a Bart PE rescue disk, (or something similar), with the undelete software included. If you don't have the requisite skill, you might want to beg a more technical friend or colleague, pretty, pretty, pretty-please, to make one for you. It's a really great idea.

Have more than one "gun" and plenty of ammo:
Different undelete programs work in different ways, so just because program "A" doesn't find your file, does not mean that program "B" can't.

When you collect your undelete utility(s), get more than one. Professional forensic analysts often use many different programs to examine a disk to increase their chances of getting what they want.
Consider investing in a combination of freeware, shareware, and possibly even payware programs.
If you're smart/lucky enough to have something like a Bart PE disk, put all the utilities you've collected on it so that they are all available when you need them.

Protection:
If you accidentally delete something important, it is absolutely crucial that the disk be taken off-line quickly! You'll want to do this as rapidly as possible to prevent the sectors used by your deleted file from being overwritten.

If possible, shut-down the system, (if it's the system drive), or un-mount it quickly, (if it's a external or data-drive), to reduce the possibility of it being corrupted.

Investigation:
If at all possible, you'll want to take the drive and mount it externally on another system, or load an external operating system. (like a Bart PE disk for Windows.)

Mount the drive read only if possible to prevent data corruption.
Use the external system or CD's software, (if you prepared a Bart PE disk), to search for the files.
If you are using an advanced recovery method, and you know what kind of file you're looking for, (a JPEG for example), have the recovery utility search for only that type of file, (assuming that this type of file is defined in the utility's dictionary). This will speed up the search process considerably, and will result in a much better chance of getting complete files.

Execution:
If you find your missing files, try to restore them to another disk drive, (like a thumb-drive), if your undelete utility supports that option.
Prayer:
I'm not including this just to be funny or sarcastic. As you can see, undeleting something is a very iffy proposition at it's best, and a successful undelete depends just as much on luck as it does on skill.
Appreciation:
Everyone likes appreciation, and the poor sod of a software writer likes it even more. Especially since the only communication he's likely to get from his users are complaints!

In all seriousness, if the utility is shareware and it just saved your butt bigtime, dig out the Benjamins and buy the license. It's the least you can do.
If it's freeware - and there's a "donate" option, cough up the dough. At the very least buy him a six-pack of his favorite brew. Or, absent that feature, send him an e-mail telling him how much you appreciate his skill and thoughtfulness.

Linux/Mac Users:

If you're dealing with a Linux/Unix/Mac type operating system, there's a really good possibility that you may just be plain-ole' hozed if you don't have something like "Time Machine" turned on.

Unix type systems have unique kinds of file systems, file deletion methods, and aggressive i-node recovery paradigms that they often use. (i-nodes contain the file descriptors, and all the pointers and metadata for a particular file or files.) Because of their design, it may be almost completely impossible to recover files deleted on a 'nix system without resorting to extremely advanced techniques that are, (usually), beyond the ability and experience of even the most advanced system administrators.

Conclusion:

With any luck I have, (hopefully), scared you s**tless about the prospect of loosing important files or data. And if I have done so, please, please, pretty-please consider using backups as a recovery strategy instead of "a wing and a prayer". OK?

What say ye?

Jim (JR)

Monday, October 7, 2013

Hot Smokin' Weapon! Award for October 2013
The Raspberry Pi

Yep. It's time for yet another Hot Smokin' Weapon award - and this month's winner is the Raspberry Foundation's pocket-sized computer, the Raspberry Pi.

Recently I was able to get my hot little hands on a Raspberry Pi board. Built and designed in the land of British pubs ~~and lousy beer~~, they're available here in the U.S. for about $35 each. For another five or ten dollars, you can get it complete with an SD card pre-programmed with a boot loader and five or six Linux distributions to play with.

Here's what a Raspberry Pi, Model B, looks like in real-life. Notice its size relative to the full-size SD card next to it. The Raspberry Pi people refer to it as a "credit card" sized computer. The red board on the right is the PI's GPIO breakout board that you can use to control external circuitry.

The layout illustration was taken from the Raspberry Pi web-site located at http://www.raspberrypi.org/faqs

And here's the layout, (minus the two flex connectors). The difference between the Model A, and B, is that the A version does not have the LAN port, has only half the onboard memory, and has only one USB slot instead of the two on the model B. Since, IMHO, the extra USB port and the LAN capabilities are important to me, I purchased the model B, and I am basing my review on that particular version.

This is the Ethernet / USB end of the board. You can see the full-sized HDMI connector on the left just above the Ethernet jack, (with the brown tape on it), as well as the yellow RCA composite video connector and a blue "headphone" jack for analog audio if you are not using the HDMI connector. Or, if you have connected it like me using a HDMI - DVI adapter plug, you need the aux audio connector for audio.

Here's a view from the other end. In this view you can see the two flex-cable connectors, (marked with red arrows), and the GPIO header on the left, marked with the green arrow. The bottom flex connector is for use with the Pi's camera module and the flex connector on the upper right is rumored to be for an LCD display module. The GPIO connector attaches to the GPIO breakout board using a standard 26 conductor ribbon cable and connectors.

The big square chip just above the lower red arrow is the brains of the beast. It is a Broadcom "System on a chip" that contains everything on the board, except the connectors, voltage regulator, decoupling capacitors, isolation resistors, and the status LEDs. The rest is in the chip - the ARM processor, the GPU, the Ethernet and USB controllers, and God only knows what else.

Here's a full-size SD card, the Raspberry Pi, and the GPIO breakout board shown next to a 12" ruler to give you a sense of scale.

This picture has the Raspberry Pi next to a plastic case you can buy for it. It has cutouts for all the connectors, the GPIO breakout board's ribbon cable, and a slot to put a SD card, (face down) at the back. Since, (obviously!), the Pi is a live circuit board, putting it in some kind of a case is a smart move to prevent accidental short-circuits from ruining it.

Now, let's insert a pre-loaded SD card, and fire the beastie up!

Note that the Raspberry Pi has no onboard BIOS of any kind, so there must be a SD card, with software loaded, before the thing will do anything at all. Especially since the SD card is the system's "hard drive" so to speak.

You can load software on the Pi in one of several ways:

If you are a total masochist, (or need special capabilities not available any other way), you can dig up the specs, an ARM assembler/compiler, and write "bare metal" code for the thing. There's a whole forum at the Raspberry Pi site devoted to just that kind of insanity. Since I've done that kind of thing before, (and no, I've never claimed to be sane), it sounds rather interesting. I'm planning to go wander around there, just as soon as I work up the nerve and get a couple of minutes to spare.

You can spool a byte-for-byte image of various distro's directly onto the SD card. Note that these images are bit-images, just like an ISO file for a CD, so they have to be written as a bit-image using special raw byte transfer software like "dd" in Linux. There are several software image writers for Windows, and you can use them as well.

Even though spooling a bit-image onto your SD card isn't really that difficult, it is a bit tricky, so the developers of the Raspberry Pi created a special boot-loader called NOOBS. (New Out Of Box Software) And yes, the pun is deliberate, since it is primarily intended for those of us who are not Certified Raspberry Engineers; to help us get up, running and productive with a minimum of grief.

Here's an image of the initial screen you see when NOOBS version 1.3.0 boots for the first time.

The first thing you notice is the list of distributions you can load, ranging from the very capable Raspbian re-spin of Debian Linux, all the way to the more esoteric RISC OS, while not for beginners, it's a full-blown re-spin of IBM's RISC operating system. It's funky, but fast. Unlike a Linux distribution, this thing boots all the way to it's GUI in less time than it takes your hand to move away from the power connector you just plugged in. Like I said, it's a bit wonky if you're not familiar with RISC systems, but its scary fast!

If you're looking carefully, you will notice a number of selection icons across the top of the dialog window. And yes, Virginia, that round thing that looks like a globe is the icon that launches the loaders built-in web browser.

Yep, you heard me right, the OS loader selection screen has a built-in web browser that can be used to surf the web in case you need any supporting information to get you started. And to make things better, the browser's home page is set to link directly to the Raspberry Pi help forum. And it's not just a crufty text-based browser either. It's the Real Deal. So far, the only things it doesn't do is allow you to download something, (though I might be wrong here), and play Flash video. Oh, in case you wondered, it does fully support PDF files, and they can be viewed directly within the browser itself.

In case you really need to get down-and-dirty, ALT-F2 brings you to a text based login screen - the user is "root" and the password is "raspberry" - where you can do pretty much everything you could do in a "normal" text based startup - or a terminal window - in Linux.

Here's a closer look at the distro selection screen showing the various distributions you can install. Oh, and by the way, the check-boxes are there for a reason. A new feature in NOOBS 1.3.0 is the ability to install more than one distribution at a time - limited only by the space on your SD card. And just to make sure you don't goof up, the area down at the bottom, labeled "Disk space" shows you both how much space you have to play with, and the total space needed for the selections you've made. If you select more than can fit - the total space number becomes a bold-faced red to warn you. I don't know if it prevents you from trying - I did not try it - but it is a thoughtful touch.

If you install Raspian, you see this configuration screen at first-boot where you can set a whole bunch of useful and important parameters - like your localization settings since everything defaults to settings appropriate for Great Britain. Unless you live there, you will have to set the system locale to one appropriate for you, your time zone, and your region's particular keyboard-map. Note that the locale settings also set the default language for the system, as run. And yes, they have support for just about every weird character set and language out there. I didn't try Arabic or Hindi, but they do a darn good job with Russian.

Once you're done, the system reboots and - if you have selected the option to boot directly to the GUI - you are presented with the Raspian desktop.

Of course, the background image is the Raspberry Pi logo, in living color, larger than life. Once you get tired of looking at that - which doesn't take long - you can substitute any image your little heart desires.

The desktop manager is LXDE, one of the "lightweight" desktop managers. Though it might be "lightweight" desktop in size, it's a fooler since it's most definitely not lightweight in capabilities. If you can do it in Gnome or KDE, chances are you can do it here too. Though you won't see all the "eye candy" that's in the larger desktops, (Face it, do you REALLY need bouncing desktop icons that spin and change colors?), it does a very competent job of doing what it does, being a desktop that lets you get work done.

Since the original design philosophy of the Pi was to use it as a teaching tool, it comes pre-loaded with a couple of programming environments: Scratch for the rank beginner who has never even thought of programming in his life, as well as Python, the "official" programming language of the Raspberry Pi in Raspian.

In case you're curious, Scratch is a very elementary programming language - not unlike the "turtle graphic" languages that were in vogue during the '90's, and it works a lot like a Lego building set. Each kind of statement, conditional, expression, or whatever has a particular shape. Syntax is taken care of by how the shapes fit. If they fit, it will work.

Another part of the language is a large catalog of "sprites" - graphical images that can be placed in the primary display area known as the "stage". Depending on the steps you put together, you can make the sprite do any number of things on the stage. You can put more than one sprite on the stage, change their position or direction of travel, and detect if they touch each other, or touch one of the stage's side walls.

I played with it for about 30 minutes or so, and I was able to get the main sprite - a cat - to walk around the stage, changing color as he walked.

Oh! I almost forgot to mention; what happens if you get heartily tired of the distro(s) you originally picked and want to do something different?

Every time the system boots, you get about five seconds of this - a way to abort the boot and return to the original NOOBS loader screen.

The good news: If you've gotten so tired of Raspian that you are about to scream, you can swap it out for any of the other distributions.

The bad news: Making new distribution selection(s) on the NOOBS loader screen is a destructive process. Any settings, files, and God Knows What Else you may have created are lost and gone forever, oh my darling Clementine. That is, unless you had the foresight to plug in a thumb-drive and copy your settings and files off the system before selecting a new distribution set.

The bottom line:

Though originally conceived as a teaching tool, the Raspberry Pi is a competent system in it's own right.

You can use it as a self-contained media center connected to your huge screen HDMI TV / Display - a job that it totally KICKS ASS at, making my larger computers with wicked nasty video cards look like pikers by comparison.

You can use it as a miniature, (in size only, not in capabilities), RISC "mainframe" system with multiple users logged in at the same time.

With a suitable distro installed - like Raspian - you can use it as a full-fledged workstation. So much so that there are a multitude of forum threads devoted to people who have decided to toss their big, noisy, power-hungry beasts and just use the Raspberry as their main computer system. And if you're like me, and won't miss the dancing bears, roving eyeballs, and bouncing icons, you will feel right at home sitting in front of your cigarette-case sized computer, kicking ass and taking names, right up there with the Big Dogs.

Don't take my word for it. Go out and get one - they're dirt-cheap by comparison - and give the beast a spin. Just be careful. It might be love at first sight!

Read all about it at http://www.raspberrypi.org

What say ye?

Jim (JR)

Friday, September 6, 2013

Migrating Outlook's User Settings
How to move your settings from one computer to another

In a previous article, I wrote about how to migrate Outlook's mail files from Windows XP to Windows 7, and it has been one of the most popular articles I've written based on web traffic. The most common question that this article spawned was, (something like), "OK, Einstein, I've moved all my files; now how do I go about moving my settings as well?!"

This is a perfectly legitimate question since, if you've had Outlook set up for longer than fifteen minutes, it's a lead-pipe cinch that you've developed a rather complicated set of settings, rules, accounts, and such-like over the years. And one of the biggest hurdles to migrating your e-mail from one system to another is not so much migrating the e-mail, (that is, your mail files and such), but recreating all the customizations, tweaks, changes, and such that make your installation of Outlook uniquely yours. Not to mention having to remember all the settings, passwords, security configurations, (etc. . .), for each and every e-mail account that you've associated with Outlook.

This article seeks to address that issue by showing you how to move not only your mail files, but all your customizations, rules, and settings as well.

Unfortunately, this article is Looooong. . . .

It's not that difficult to do, and it really doesn't take as long as you might think from reading this article. It's just not something that can be told in three or four steps.

Be patient, read through it two or three times, grit your teeth, and push onward!

Caveats:

This article assumes that you are using Outlook for local e-mail storage. (i.e. your mail files are stored locally on your system, you connect to your ISP's mail server using POP-SMTP, (or maybe even IMAP, though I have not tried it), and not as part of an Exchange Server system on a domain, etc. If your mail is stored on an external/corporate mail system this may not work. You can try it, and if it does work, please let us know!
My system is a Windows 7 - 64 bit system; though this should work on the 32 bit version of Windows 7, as well as both 32 and 64 bit versions of Vista. If anyone tries this on Windows 8 - 32 or 64 bit, please post a comment and let me know if it works there too.
I have not tried this migrating between 32 and 64 bit versions of Windows - in either direction. Because certain important metadata information about your e-mail setup is located in the Registry, and because I do not know if there are any differences between them on 64 or 32 bit systems, Your Mileage will most certainly Vary.
My system is a Win-7 / Vista generation system. Since I do not currently have a working installation of Outlook on anything older than that - like XP - I cannot say if this will work or not. However, knowing how Microsoft does things, and how their Registry is organized, I wold not be surprised if this works there too. Again, if you try this on XP, please post and let me know if it works there or not.

Also note that the locations of the various mail-files and such are different when migrating from XP to Vista/Win7/Win8. Pleas see my previous article for information on where the files are located on XP, and how to migrate these files from XP to something later. You should also be aware that these differences in file locations will, very likely, affect the Registry settings for Outlook too. Beware.

My version of Outlook is the Outlook 2010 version, however this should work on versions from 2007 onward. It might work with Outlook 2003, and I have not tested it with Outlook 2012, though it would not surprise me if it works there too.
My version of Outlook is the 32 bit version of Outlook 2010, since at the time I installed it Microsoft was recommending everyone install the 32 bit version unless you had a very specific need for the 64 bit variety of Outlook.
I am also assuming that your mail files are located in the standard locations where Outlook normally installs them. If you've moved your mail-files around, you'll have to substitute your unique paths and places when I tell you to either copy or paste the various folders.
You will be "mucking around" with your Registry, and if you're not careful, you can mangle a system in short-order. However, most of the things here are relatively benign, and if you follow the steps carefully, you should have no problems. Of course, Your Mileage May Vary, and if you bork your system. . . . Well you should have made backups first!

Here's how to do it.

Setup, prior to migration between the two systems:

First things first: Perform a full system backup, preferably a "bare metal" backup if at all possible, before doing anything else. These steps are fairly easy and the majority of Windows users should have no problems. However, good 'ole Mr. Murphy likes nothing better than to catch someone - no matter how experienced - with his pants down. Mistakes happen, so don't forget the backup!
Make sure that the account username is identical on both the original system and the target system. Likewise, make sure that both of the accounts are elevated to "Administrator" access. If you want to down-grade the account(s) to "Standard User" access, that can be done once everything is proven and working.

You can move between systems where the username is not the same, but you will loose a lot of the metadata and rule relationships simply because the paths to the various files are different.

In one case, I tried moving from a system where the username was "Jim" to one where the username was "Jim Harris", and because the user path isn't the same, things get interesting. In my case I ended up having to re-sync a number of my rules because the target folder of the rule changed due to the path change. Likewise, although all the account information came over, the passwords for the accounts were lost and I needed to re-enter them again.

Prepare the target system by installing the same version of Outlook as is on the original system before you migrate. Microsoft likes to make, (ahem!), "little tweaks here and there" between the differing versions of Outlook. If you don't believe me, go look at another article I've written on just that very subject. Verify that Outlook on the target system is fully installed and ready to have e-mail accounts set up. Don't set the accounts up just yet, just make sure that Outlook is at the point where accounts can be set up.
Make sure Outlook is not running on either of the two systems - both source and target.. In fact, while you're doing this, you'll probably want to either unplug the network cable, or turn Wireless networking off, (or both.), on both systems. The reason for this is that:

If the first system downloads e-mails while you're migrating to the second system, your two systems are now "out of sync", and you'll have to repeat much of this to re-sync them.
If Outlook is running on either system, you may not be able to copy, or instal, the necessary files and settings. And even if you do, you can't guarantee that they'll work right.

With networking / Wireless turned off on both systems, do what I call a "Frosty Cold Reboot". That is, do a full shutdown to the power-off state on both systems, count to five slowly, and then restart both systems, making sure that networking is still disabled and Outlook is not running.

Most of the things you'll be doing require full administrative rights on the system. If the account you're migrating from/to is just a "standard" account, you should, (temporarily), elevate it to an "Administrator" account while the migration is in progress.

Saving your mail-files:

This is where you'll backup all your important e-mail files.

Prepare a thumb-drive, (USB stick, flash drive, or whatever you want to call it), that is big enough to hold all your mail-files with about a gig or two left over.
Place the thumb-drive in your original system, and create three folders:

In the root of the thumb-drive, create a folder called "Outlook"
Inside the Outlook folder you just created, create two new folders called "Local" and "Roaming".

Make sure that Outlook is not running, (or shut it down if it is), then make the following file copies between your original system and the thumb-drive:
Go to the "C:\Users\[your user name]\AppData\Local\Microsoft" folder.

If you cannot see the hidden "AppData" folder from within your home directory, you'll have to open the "folder options" dialog and - under "Hidden Files and Folders", select the radio-button next to "Show hidden files, folders, and drives".
Likewise - from inside the same dialog - you'll have to un-check "Hide protected operating system files (Recommended)" Windows will complain bitterly about this, but go ahead and do it anyway. (You may want to go and set these two settings back to the way they were when you're finished migrating.)

Within the Microsoft folder, pick up and copy the entire "Outlook" folder to the \Outlook\Local folder on your thumb drive. Since this contains your mail files, as well as certain special indexing and metadata files, this might take a while. Note that the Outlook directory on my own system weighs in at just under two gigs. It's been around a while, but I've also done the periodic garbage collection - throwing out ancient and useless e-mails - as well as compacting folders to keep the amount of cruft and junk to a minimum.
Next, go to your "C:\Users\[your user name]\AppData\Roaming\Microsoft" folder and copy the following folders to the \Outlook\Roaming folder on your thumb-drive:

The "Outlook" folder
Both the "Proof" and "UProof" folders.
Both the "Signatures" and "Stationary" folders.

Saving your e-mail settings:

This is where you backup all the "hidden" settings and passwords from the Registry.

First: Open an elevated (Administrator), command prompt. This is important since Windows versions starting with Vista onward, "virtualize" important parts of the system. You need to be working with the real Registry, not some "virtual" copy of it.
If you've never done this before, here's how:

Click on the "Start" button and go to "All Programs" and then "Accessories".
Within the "Accessories" section, you'll see a black box icon named "Command Prompt" about three or four icons down from the top.
Right Click on the "Command Prompt" icon, and select "Run as Administrator" from the pop-up sub-menu that appears.
Click on "Yes" if a UAC prompt asks you for permission.
You should now see a "DOS" type command prompt window that has as it's title "Administrator: Command Prompt". This is very important! Verify that you have the correct command prompt window open before you proceed!

Within the elevated command prompt window, type "regedit". Once you do this, you'll see the standard Windows Registry Editor window appear.

Navigate to the following location within the registry:
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles.
(Whew! What a mouthful that was!!) Let's try that again, easier this time.

HKEY_CURRENT_USER
Software
Microsoft
Windows NT
CurrentVersion
Windows Messaging Subsystem
Profiles

When you expand the "Profiles" key, (by clicking on the small triangle), one of the sub-keys you should see is one named "Outlook". (On my system, "Outlook" is the only sub-key there. Your system may have others too.) This is the Registry key you're interested in, and this is the Registry key where all the magic is located. In the next few steps, you're going to make a copy of all the data in this key and save it on your thumb-drive so that you can move it to your new system.
Click on the "Outlook" sub-key so that it is highlighted, right click on it to expand the pop-up sub-menu, and then select "Export"
When you do that, you'll see a dialog box titled "Export Registry File"

Select the \Outlook folder on your thumb-drive as the location for the exported file.
Give the export file a useful name. Something like "Outlook Settings Export 06-21-2013.reg" is a good choice.
Make sure the "Save as type" is set to "Registration Files (*.reg)" - which is the default setting.
Make sure that the "Export range" is set to "Selected branch".
Click on "Save", wait a nanosecond or so, and the magic is done! You now have a perfectly preserved copy of all those mysterious settings in Outlook.

Danger, Will Robinson! Danger! Danger!

All of the previous steps assume that you have not opened Outlook since you started this process. Opening Outlook, (especially if you've received any e-mails between the time you copied the mail-files from ...\AppData\Local and now), will write various changes into many files on the system - especially within the "Outlook" sub-key in the Registry.

This actually surprised me as I had previously "assumed" ( ! ! ! ) that the Registry settings only contained account, setup, password, (and possibly), path information about your e-mail. When I re-exported my Outlook registry key today, (just to make sure I was using the correct sequence of steps in this article), I tried comparing it to another Outlook registry key export I did a week ago, "just for grins and giggles" - and damn if they weren't different!

Admittedly, the differences were (usually) relatively small in nature with only a few bytes different here and there; but any differences make me nervous. Especially since I don't know what those "few bytes here and there" might represent. And, AFAIK, the only differences between then and now is that I have received a fair amount of e-mail since I did the original Registry settings export.

If you have opened Outlook, it is very likely that your mail-files, metadata, and Registry have become unsynchronized. If you try to restore the files you copied earlier, along with the Registry settings that you've just now exported, the results may be, (ahem!), "interesting". To avoid this, you'll have to make sure Outlook is not running, then go back to the beginning and start re-copying all the files and settings all over again. Bummer. . . .

The last step in exporting everything from your "old" version of Outlook is to export all your rules.

I know that I have been screaming up-and-down about not opening Outlook, but this time you gotta do it, since (again, AFAIK), the only way to access these rules is via Outlook itself.

Now hold on! Before you get all eager and go starting up Outlook, you still have to make sure your e-mail sessions don't get outta whack. To do that, you need to make sure that Outlook can't get to the Internet, and the best way to do that is to disconnect from any and all network sources; either hard-wired Ethernet, or via WiFi.

Once you're absolutely and totally disconnected from ANY network source whatsoever, you can go ahead, start up Outlook, and export all your finely crafted rules.

Here's how to do it:

Start Outlook and go to your usual mail-view page where you read and send e-mails.
Once you've gotten your e-mail open, up on the "ribbon" near the right side, is an icon that looks like a folder, labeled "Rules".
Click on the little triangle under it to expose the drop-down menu, and select "Manage Rules & Alerts".
You should now see the "Rules & Alerts" dialog with the "E-mail Rules" tab selected, showing (the top of) all the rules you've defined.
On the top right-hand side, select "Options", it will open another dialog titled "Options", and within the top half of the dialog you'll see two buttons labeled "Export Rules" and "Import Rules". Obviously, what you want to select is "Export Rules".
When you click on the "Export Rules" button, you get the usual Windows "Save" dialog which defaults to the "C:\Users\[your user name]\AppData\Local\Microsoft\Outlook" folder. You don't want to save it there, you want to place it on your thumb-drive with everything else, so go find the thumb-drive on the left hand, ("places"), pane, and select the Outlook folder you created there earlier.
Give the exported rule file, (*.rwz), a useful name like "08_20_2013_rules_export.rwz", and save it to the thumb-drive.
Click "OK" on each of the dialog boxes to get back to Outlook's e-mail page, and then close Outlook.

At this point, you can do a complete shut-down on your system and remove the thumb-drive.

Now that you have all the data and metadata from the old system saved all snug and warm on your trusty thumb drive, you can set the old system aside, and bring the new system to center-stage.

At this point, you're going to move all of the carefully preserved data from the old Outlook installation into the new Outlook installation; making it, (hopefully!), an exact clone of the original Outlook install.

How to do it:

Setup prior to restoring the Outlook data:

First, make sure the new system has no active network connections whatsoever. Unplug the network cable and/or turn off wireless networking on the new system. This will prevent Outlook from trying to do things to you behind your back.
Make sure that the user on the new machine has the exact same name as the user on the old machine.
Launch Outlook, and verify that there are no accounts set up, but that it is ready to create new accounts. Then close Outlook.

Restore the saved Outlook mail-files and metadata:

Attach the thumb-drive (where you saved the data) to the new system, and navigate to the Outlook folder you created earlier.
Navigate to the \Outlook\Local folder on the thumb drive, and copy the "Outlook" folder you saved there to the "C:\Users\[your user name]\AppData\Local\Microsoft" folder on the new machine.

Windows may complain that there is already a folder with that name, and it may also complain that there are files there with the same names.
Allow Windows to "merge" the folders, and if there are any complaints about files being the same name, select the top option - copy the new file and replace the old one with it - since you want to totally erase whatever is there and replace it with what was brought over from the old system.

Next, navigate to the \Outlook\Roaming folder on the thumb-drive, and copy everything there into the "C:\Users\[your user name]\AppData\Roaming\Microsoft" folder on the new system. This should include:

The "Outlook" folder
The "Proof" and UProof" folders
And the "Signatures" and "Stationary" folders
If Windows complains about files or folders being the same, let Windows copy and/or merge as you did before.

On the thumb-drive, move up to the "Outlook" folder and find the saved registry file, (it will have an icon that looks like a Rubik's Cube with pieces missing), and double-click it.

You will (probably) get a UAC asking you for permission to update the system. Answer "Yes".
You will then see a "Registry Editor" dialog warning you that adding or modifying values in the Registry can do serious damage to your system. It will also ask you if you trust the source of the file you created just a few minutes ago, (of course you do!), so go ahead and let it install.

Sanity Check:

Before you do anything else, you should stop and make sure that things are going OK so far. To accomplish this, launch Outlook. What you should see when Outlook finishes launching itself, is that Outlook on the new machine looks exactly like the way Outlook looked on the old machine.

All your mail folders should be there and they should contain exactly what was in them before you began the migration.
The e-mail messages that were on the old system should be on the new system.
"Unread" messages on the old system should exist, and be "Unread" on the new one.
Likewise any garbage that was left in the "Junk" and "Deleted Items" folders on the old system should still be there on the new one.

In other words, what you see now, on the new system, should be exactly the same as what you saw on the old system before you shut Outlook down for the last time.

If they don't - STOP! - go back and verify that you've done things properly.

Verify that you copied the folders from the thumb-drive into the correct folders on the new machine.

Did you place the "Outlook" folder that was inside the \Outlook\Local\ folder on the thumb-drive inside the ...\AppData\Local\Microsoft\ folder on the new machine?
Did you place the five folders that you copied to the \Outlook\Roaming\ folder on the thumb-drive inside the ...\AppData\Roaming\Microsoft\ folder on the new machine?

Were you able to successfully import the Registry file you saved before?
Did you restore your Outlook settings to the exact same username on both systems?
Have you tried a system reboot?

It is important that you go back and resolve any issues now, before you continue.

Now that you're golden so far, let's do the last few tweaks to get Outlook running on the new machine.

Restore the saved rule file:

With Outlook still running and open to your e-mail, go back to the "Rules" icon on the ribbon.
Click on the small triangle under the "Rules" icon and select "Manage Rules & Alerts", "Options", and then "Import Rules".
Find the exported rule file, (*.rwz), that you saved in the Outlook folder on your thumb drive, select it and click "Open"
If all goes well, nothing will appear to happen when you import the rules file. Select "OK" on all the dialogs to bring yourself back to Outlook's e-mail page. You can verify that all is well by re-opening the "Manage Rules & Alerts" dialog to verify that all your rules were successfully imported.

Verify and Test the account settings:
Before you just "let Outlook go", you need to verify that the accounts, passwords, (etc.), are set up correctly.
Likewise, you should make a small settings change that will allow you to go back to your old system if something goes wrong.

With Outlook still running, and open to your e-mail, select the "File" tab on the far left side of the ribbon.
Within the "File" tab, select "Account Settings". This brings up a, (in my opinion), totally stupid and useless button that is - in essence - identical to the one you just selected. Go ahead and select that one too. (What where they thinking?)
You should, (finally!), be at the "Account Settings" dialog. Verify that each account that existed on your original machine is here on the new one.

If you do NOT see all your accounts, STOP! Close Outlook and repeat the steps for restoring the Outlook registry file.
If that does not fix the problem, go to your thumb-drive and open the Registry file with Notepad. You should see a lot of strange looking stuff that looks something like this:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook]

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook\05035e0babc255def9547eba6a273a93d]
"001f3006"=hex:4d,00,ff,00,ff,00,69,00,6c,00,65,00,21,00,41,00,64,00,64,00,72,\
00,65,00,73,00,73,00,20,00,42,00,6f,00,6f,00,6b,00,22,00,50,00,72,00,6f,00,\
76,00,69,00,64,00,65,00,72,00,90,00
"001f300a"=hex:6f,00,dd,00,77,00,88,00,70,00,2b,00,64,00,61,00,6c,00,00,00 (etc. . .)

and a lot more just like it. The file should be fairly big, mine is just a tad over 350k in size; but then again I have two different e-mail accounts, and some weird security settings for parts of them too. Depending on how many e-mail accounts you have, and their settings, your file may be a bit larger, or smaller, than mine. However it shouldn't be tiny, and it shouldn't be filled with unprintable garbage that looks like cartoon curse-words.
If you don't see text like I have shown above, or if it starts with something other than

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles]
or
[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook]

you should close Outlook, shut down the new machine, go back to the old machine, and re-do the steps for exporting Outlook's Registry profile.

Highlight your account, (or each account in turn if you have more than one), and then click on the "Change. . . ." item just above the account list.
Look at the first settings page, and verify that it looks like it should. Then clear the "Test account settings by clicking the 'Next' button" check-box. (You do this so Outlook doesn't try something stupid when you move from screen to screen. Ask me how I know. Go ahead, I dare you!)
Click on the "More Settings" button and click on each tab in turn, from "General" to "Advanced", and verify that the information on each tab is correct.
Once you get to the "Advanced" tab, look about 2/3 of the way down and select the "Leave Messages on Server" check-box. This is the important "small settings change" I mentioned before. By setting this, Outlook won't delete messages from your mail server when it reads them. This is important because if things don't go right, you can always go back to your old machine and all your messages will be there. (However, any replies you may have sent while on the new machine, won't be.)
Repeat these steps for each additional account you have, verify that all the information is correct, and make sure to select "Leave Messages on Server" for each account in turn.

Now, let's fire that puppy up and see if it really works!

At this point you've checked just about everything that can be checked without "going live" with Outlook.

Shut down Outlook.
Re-attach the Ethernet cable, or turn Wireless Networking back on, and wait for the connection to happen.
Verify a good network connection by launching your web browser and verify that you can go to various web pages.
Restart Outlook.
When Outlook restarts, (God willin' an' the Creek Don't Rise), you should receive any outstanding e-mails that have been queued on your various mail servers.

If you get errors, resolve them the way you normally would, though I'd be really surprised if you get anything significant at this point. Maybe a rule error or two, but nothing serious.

You should go to each of your accounts, in turn, and send a test message back to that account, along with one to every other account you have.

You should receive all the test messages from each and every account.
If you don't, go visit that account via it's web portal, (if it has one), and use normal troubleshooting techniques to discover what's happening. In my case, my Yahoo! account periodically goes brain-dead for no apparent reason when I try to download e-mail from it. No real reason that I can see, it just doesn't like POP mail sometimes.

If you can, call a friend or two and ask them to send you a test-e-mail. Or, be bold and send them a test e-mail asking them to reply!

By now, you should know if your e-mail migration has succeeded or not, and if you've gotten this far, it's a pretty fair bet that you're doing just swimmingly. Give it a few days, maybe a week or so, and if all goes well you can go back to each account's "Advanced" settings tab and un-check the "Leave Messages on Server" check-box to clear the cruft off of your mail server(s).

Drop me a line and let me know how things went!

Jim (JR)