Welcome to the QA Tech-Tips blog!

Some see things as they are, and ask "Why?"   I dream things that never were, and ask "Why Not".  
Robert F. Kennedy

“Impossible” is only found in the dictionary of a fool.  
Old Chinese Proverb

Thursday, April 1, 2010

Showdown at Tombstone
How to Resurrect An Active Directory

In a somewhat weird coincidence, it is interesting that in the week before Easter I should be writing about how to bring something back to life. . . .

Active Directory has a feature common to all complex databases that can be replicated from machine to machine - it implements a feature called "Tombstones"

Tombstones are a way of marking data that has been removed as "deleted" - without actually removing the data - so that the request to delete can spread to all other computers sharing this information.  Then - after a period of time long enough for all computers to have heard about it, the tombstoned data is actually and physically deleted.

This is needed because it is very difficult for databases to replicate the absence of data - to make replication more efficient, the only thing that gets replicated is the actual data itself.

In Windows*, the default tombstone lifetime is two months - 60 days - which under normal conditions is plenty long.  What this means is that if data has been unused for 60 days, it's automatically tombstoned, (marked for deletion), and after another 60 days, it's actually removed.  Note that when data is tombstoned, it's physically removed to a special place in the Active Directory - and for all intents and purposes, it's gone forever.  It's possible to "reanimate" a tombstoned object, but most of the characteristics of the object were stripped away when it was tombstoned, making "reanimation" a dicey proposition at best.

However - QA test environments are often used in ways that are much different from "normal conditions".  My friend Andy, for example, has a special test domain using two computers as domain controllers - one parent and one child - with other computers connected to them.  And it is not unusual for him to set this system up, perform a series of tests, and then need to tear that system down and restore the pristine network to do other testing on.  And there are some very nifty tools that allow him to do this in a very painless manner.

Unfortunately - some of these tests can take several months to complete.  Or he might get distracted by his boss to work on something More Important for a while.  The result is that the next time he "restores" the two computers in the domain from his most recent pristine backup, it's been longer than 60 days.  This results in the entire Active Directory tombstoning itself - in essence comitting ritual suicide - while he watches his network automagically reduce itself to a quivering lump of rubble.  And once that happens, there is little else to do but manually re-create everything from scratch - again! - to restore the pristine non-tombstoned status to the Active Directory.

There is a solution!

There is a special parameter within the Active Directory itself that tells Active Directory how long to wait before clobbering things, called the "Tombstone Lifetime", and it can be set to values that are reasonable for your situation.

Here's an example showing the problem and how to work around it:

I recently needed to restore a server - and my most recent backup was six months old.  If I just restore it and fire the computer up, it will destroy itself by tombstoning.  I can reset the computer's clock, but as soon as it gets on the network and gets the correct time - I'm dead again.

What you will need:
1.  A server running active directory.
2.  The "resource kit" or "support tools" distributed with your version of the server.  Don't use earlier or later tools, they won't work properly.

What I did:
1.  I set the computer's clock to a date six months ago that was just a bit later than the date the backup was taken.
2.  I restored the backup image.
3.  I removed the computer from the network (by unplugging it)
4.  I restarted the computer.

Once the computer came up, I logged in and waited for it to settle down.

5.  In the support tools for this server there is a program called ADSI EDIT - that can be used to make changes to the Active Directory in a manner similar to what REGEDIT does to the Registry.

You invoke it by either typing in "adsiedit" from a command line, or by finding the adsiedit executable and double-clicking on it.

6.  Once it opens, you have to navigate to the correct object:
  (a)  Configuration Container ---->
  (b)  CN=Configuration,DC=[your domain name],DC=com  (or net or whatever) ------->
  (c)  CN=Services  ------>
  (d)  CN=WindowsNT

7.  Once you get to the WindowsNT container, you will notice on the right an object called "Directory Service"  Right click on that object, and select "Properties"

8.  Inside the property sheet for that object is a drop-down list of properties. (a LONG list of properties. . .)  Select the property called "tombstonelifetime" and note what it is set to.  By default, on my machine, it was blank.  Change it to some value that is reasonable for you - I selected 360 - to represent 360 days, and then press "Set".

9.  Click OK a few times to get all the way back out, and you're set!

10.  You can now re-boot the box, catch it at the BIOS settings, reset the system clock to the correct date and time, and continue with your life unaffected by the tombstone lifetime setting.

What say ye?

Jim

* UPDATE (5/12/10)
This is actually true for the versions of Active Directory prior to Server 2003 - SP1. (See the TechNet article here.)  Windows Server 2003 SP-1 and later versions of Windows Server have Tombstone Lifetimes extended to 180 days by default.  This came about because Microsoft's many customers discovered that systems being built in one centeralized location, then shipped to distant branch offices - or built-to-spec abroad then shipped elsewhere, would end up Tombstoning themselves while in transit.

3 comments:

  1. Another case of MS making some weak decisions.
    Default should be a bit longer than 60 days..

    ReplyDelete
  2. Pat,

    You can't fault Microsoft for not having a crystal ball. And sometimes, what seems reasonable at one point in time, turns out to be less reasonable as times change. At least they published a process and provided a tool to fix it when necessary.

    And yes, the default tombstone lifetime on later versions of Windows Server can go up to 180 days. ( :-D )

    ReplyDelete
  3. Sometimes even if you format the computer some errors don't disappear some people said that is because of the hardware damage but is not, they are just errors that could not be deleted.

    ReplyDelete

Thanks for sharing your thoughts here at the QA Tech-Tips Blog!

Note:
This blog will not, repeat NOT, publish comments that contain ANY KIND OF HYPERLINK.

If your comment contains ANY KIND OF HYPERLINK it WILL BE DELETED.

Please read the article at How To Get Comments Approved On This Blog for additional information regarding this rule.

Thank you for understanding.

Jim (JR)