It's one of the strangest things that can happen in Active Directory (AD): You delete an object, such as a user, group, or organizational unit (OU), then a few minutes—or even days— later the object mysteriously reappears. Generally, deleting the object a second time makes it "stick," but reappearing objects—especially users and groups—can have serious security implications. After all, if a user account reappears, it can be used to access domain resources. So where do deleted objects come back from, and how can you prevent it from happening in the future?
The key to this puzzle lies within AD replication. Remember that every time an object is changed, its Update Sequence Number (USN) is incremented. When the object is replicated, it comes from the domain controller that has the highest USN for the object, and the object overwrites any older copies on other domain controllers.
When you delete an object from AD, it doesn't go away immediately. Instead, the object is tombstoned, meaning AD places a special marker on it indicating that the object is no longer active. The tombstone itself has a USN and is replicated to all domain controllers in the domain. 60 days after the tombstone, all domain controllers independently remove the object from their copy of the AD database.
There are a few situations in which the tombstone can be removed, and even situations in which a tombstoned and deleted object can mysteriously reappear. You'll need to review these possible situations to determine which occurred in your environment, and take steps to prevent it from happening again.
Some companies decide to keep a spare domain controller off the network as a last-resort backup, in case all of their active domain controllers somehow become corrupted or fail in some fashion. Some companies also ship new domain controllers to new branch offices, and unforeseen circumstances keep the domain controller offline for longer than intended.
Unfortunately, keeping an offline domain controller can cause deleted objects to return from the dead. Here's what happens:
The same scenario can occur if you restore an AD backup that's older than 60 days. For that reason, Windows' tools will try to prevent you from restoring a backup that is more than 60 days old. There are documented workarounds that can enable you to bypass this protective feature, but you shouldn't try.
How can you prevent this from happening? Take some basic precautions. First, make a backup of AD every single day. It only takes a few minutes and ensures that you have a recent backup to work from at all times. Second, never keep domain controllers offline for more than a few days. If you have to have a domain controller offline for a couple of weeks, ship a more recent AD backup that can be used to update the domain controller's database.
In Windows Server 2003, you can write the AD database backup to removable media, such as a CDROM. This media can be used to initialize a new domain controller's AD database. It's an ideal way to deploy domain controllers to remote offices: Ship a non-domain controller Windows Server 2003 computer to the office, use Dcpromo to install AD, and initialize AD from removable media. Doing so will prevent a heavy WAN load for initial replication while ensuring a recent copy of AD is placed onto the new domain controller.
Here's another scenario: Imagine you've got an OU named Sales, which contains a user named Maria and a user named Pete. You make a backup of the AD database on Monday morning. On Tuesday morning, Pete leaves the company, so you delete his AD account. Tuesday afternoon, someone accidentally deletes Maria's user account. To correct the problem, you decide to perform an authoritative restore of the Sales OU. Maria's account comes back, but so does Pete's.
This mistake seems like a fairly obvious gaffe: You could have performed an authoritative restore of just Maria's user account. However, it's a common situation. When you perform an authoritative restore, AD increments the USNs of all restored objects by several thousand, making them the highest USNs in the domain. This forces AD to accept the restored objects— which in this case have no tombstone—as authoritative, and they are replicated throughout the domain.
Authoritative restores from backups older than 7 days can cause trust and computer account problems. The passwords for trusts and computer accounts are automatically changed about every 7 days.
Had you performed a non-authoritative restore, neither object would have reappeared. That's because the USN on the backup tape was older than the current USN held by the domain's domain controllers. When you restored the objects, they would briefly reappear on the domain controller that you restored them to, but that domain controller would quickly re-replicate the higher-USN version of the objects from other domain controllers, and the objects would again be tombstoned.
The moral is to restore as few objects as necessary when performing an authoritative restore. When restoring an entire OU, you might not realize that other objects contained within the OU will be restored, so you'll also need to perform a follow-up inspection after your restore to see what's back. For example, suppose someone had deleted the Sales OU on Tuesday afternoon, rather than deleting Maria's user account. Obviously, you'll need to perform an authoritative restore of the entire OU at this point, and Pete's account is going to come back whether you like it or not. A quick follow-up inspection of the OU after the restore will let you re-delete Pete's account and correct the problem.
Windows Server 2003 provides a means for restoring accidentally deleted objects, effectively removing their tombstone. At least one domain controller in the domain must be running Windows Server 2003 to enable this functionality. A new user right, Reanimate Tombstone, provides the necessary permissions to the Domain Administrators group by default. Unfortunately, there's currently no user interface for using this feature; you'll need a custom utility to take advantage of it for now.
Windows 2000 (Win2K) Service Pack 3 (and some post-SP2 hotfixes) introduced a new feature known as Strict Replication Consistency. This feature can cause additional problems for out-of-date domain controllers, causing them to log event ID 1084 in the System event log.
The Microsoft article "Lingering Objects Prevent Active Directory Replication from Occurring" provides more information about this problem and describes how Strict Replication Consistency works.
You can also modify the tombstone lifetime in AD. The default 60-day value is designed to accommodate domain controllers that might be offline for an extended period of time and to ensure that backups are useful for a good, long while. However, you can reduce the value to as little as 2 days or even increase it. To do so, use the ADSI Edit snap-in, which Figure 14.1 shows, to edit the tombstoneLifetime attribute.
There's no default console that contains the ADSI Edit snap-in; you'll need to install the support tools from the Windows CD-ROM and run Adsiedit.msc.
Figure 14.1: Using the ADSI Edit console to modify the tombstoneLifetime attribute.
Few administrators prefer to shorten the tombstone lifetime; many, however, lengthen it to accommodate longer domain controller absences or to extend the useful life of AD backups. In most environments, 60 days is more than sufficient. 30 days is a reasonable minimum if you specifically want deleted objects removed more quickly and aren't concerned about backup lifetime. Anything longer than 90 days is probably excessive, and will keep deleted objects on your domain controllers a lot longer than they need to be.