Missing Files When Using DFS Replication

Sigurdur HaraldssonSenior System Administrator
Files go missing when using DFS (Distributed File System) Replication and how to recover them and fix it.

I recently had a case where a customer complained about missing data. We were using DFS Replication to replicate the data between three servers, two on-premise and one off-premise (which we used to backup the data with a backup software). We're using Shadow Copy as well on the fileservers.

So we were fairly confident that the missing files were not entirely lost. DFS Replication uses a folder, ConflictandDeleted, in which it stores all data that it cannot replicate or older versions of files that conflict with a newer one. So I went there to look for the missing files.

So I went to the DfsrPrivate folder, where ConflictAndDeleted resides. And I got an Access Denied! 

I went and double checked and yes, I'm logged on with Domain Admin permissions. I logged out and on again, just to make sure but alas, I got the same error again. After a short while searching the Internet, I found out that this is not uncommon. 

One solution is to use the UNC access path from another computer. 

Another solution is to run CMD or File Explorer in Admin mode if you're logged on locally on the server. (Note that in order to run File Explorer in elevated Admin mode, you have to run Task Manager as Admin, kill the Explorer.exe process and then click on File and select Run New Task.) 

So, after I got into DfsrPrivate, I was able to look at ConflictAndDeleted. There was about 3GB of files there, many since the day before and with a quick look, I found at least one missing file. But I wasn't done yet.

Restoring from ConflictAndDeleted is not straightforward. You cannot simply copy and paste from it. When there's a conflict, DFS puts the file into the folder with a GUID, so you can't see the whole name. It looks something like 01500BA3-{BF3C5C5A-F40E-4737-93C2-D08909129047}-v5071499.xxx (where xxx is the file extension). There's a file, ConflictAndDeletedManifest.xml, that keeps track of the GUID and the corresponding name, where you can see the name. But for all practical purposes, it's useless to restore more than a few files manually.

So in order to restore the files, I used the script from https://blogs.technet.microsoft.com/askds/2010/10/15/restoredfsr-vbs-version-3-now-available/. I copied the contents of ConflictAndDeleted into a new folder as well as the ConflictAndDeletedManifest.xml file.

I then edited the script to point to the right folders and the Manifest file and ran it. Unfortunately, the script requires you to confirm each file but it's easily "solved" by holding down ENTER while it is running. I restored to a specific folder and checked the restored files after completion. It restores the files and the folder structure so all there was left to do was to copy the folder to the live folder and skip the files that already existed.

On closer inspection, I found out that a lot of files had been moved to the ConflictAndDeleted folder. It's easy to see by filtering the DFS Replication log (under Applications and Services Logs) by event id 4412; 

"the Staging folder was too small". The guideline is that you take the 32 largest files in your replication and add their size together. That should be the size of the Staging folder.  You can run the following Powershell commands to calculate the size of the largest 32 files

$big32 = Get-ChildItem c:\temp -recurse | Sort-Object length -descending | select-object -first 32 | measure-object -property length –sum

$big32.sum /1gb

And what caused all this? We had changed permissions on the replicated folder on one server which caused massive replication and the Staging size was way under the recommended size. I was using 3GB where I should have been using 21GB. As the DFS service deletes the oldest files in Staging and it didn't manage to replicate fast enough, we "lost" a few hundred files. 

Sigurdur HaraldssonSenior System Administrator

Comments (0)

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.