Need help finding out what happened to 1.5TB of space on my harddrive

I have a 3TB drive as a separate data drive on Windows 8.1  This drive contains an old installation of windows 7.  The drive size reported in disk management is as expected 2794.39GB.    the Properties window shows that 2.5TB has been used and 231 GB is free for a total capacity of 2.72GB.    The problem is that I cannot figure out why it is showing 2.5TB used.    Unfortunately I did not pay attention to the growth rate and do not know when this occurred.  It may have been when i was still running that drive as my primary OS drive with Windows 7, or it may have occurred since adding it as a secondary drive to my Windows 8.1 system.

I've granted myself access to the System Volume Information folder and recycle bin so those folders are accounted for.
I've used WinDirStat and TreeSize so see where the space is being used up both report a total of 1.2TB in use.  So i should have roughtly another 1.5TB of free space left that is unaccounted for.
Used-Space.png
TreeSize.png
WinDirStat.png
cmbaker82Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CompProbSolvCommented:
Did you run TreeSize as an administrator?  It should have given you a prompt about that if not.
cmbaker82Author Commented:
Yes tree size was ran as administrator
pjamCommented:
How big is you WinSXS folder under Windows?
Have you installed the optional update kb2852386 which allows you to manage hidden SP an patch backup files no longer needed.
https://support.microsoft.com/en-us/kb/2852386
 64-bit is here:
https://www.microsoft.com/en-us/download/details.aspx?id=40445
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

cmbaker82Author Commented:
The windows installation on that drive is no longer in use, so I had already deleted the WinSXS folder on that drive.    The windows 7 installation is no longer in use, just some files remaining.    The current OS is Windows 8.1 installed on the C drive.  This is the D drive we are working with.
Lee W, MVPTechnology and Business Process AdvisorCommented:
Please see my article on getting an accurate reading using WinDirStat/Tree size:
https://www.experts-exchange.com/articles/28565/Using-Tools-To-Find-What's-Using-Your-Disk-Space.html
cmbaker82Author Commented:
As I already mentioned I did grant myself access to the system volume information folder.
Lee W, MVPTechnology and Business Process AdvisorCommented:
And you think that's the only place that my article can apply?  You realize you SHOULDN'T have done that in the first place.
cmbaker82Author Commented:
Unfortunately the article is reserved for premium members which I am not.  And why do you think I should not have done that when I was trying to determine how much space was being used?
CompProbSolvCommented:
@cmbaker:

I just read Lee's article and it appears to be excellent advice.  It shouldn't take too much effort to give the program a shot and see if it exposes the "hidden" used space on your drive.  I'll have to add it to my bag of "tricks".

@Lee:

Thank you for an excellent article.
Lee W, MVPTechnology and Business Process AdvisorCommented:
That's unacceptable to me.  I write these and place them on EE believing articles are publicly accessible.  If that's changed, I'll have to post them elsewhere as well.  Sorry, there will be a delay before that happens but it WILL happen.
cmbaker82Author Commented:
I will be more than glad to read it when it is accessible.    What program was suggested in the article?
Lee W, MVPTechnology and Business Process AdvisorCommented:
Summary - use PSEXEC to get a command prompt as the system account.  Then run Windirstat as System.  Administrator is not good enough.
cmbaker82Author Commented:
I will do this in the morning. Thanks.
noxchoGlobal Support CoordinatorCommented:
Can you post also a screenshot of Windows Disk Management? Looks like you have here a PartMgr problem where the used size reported by Windows Explorer and Windows Disk Management differs.
cmbaker82Author Commented:
Here is the disk management screenshot.
disk-management.PNG
cmbaker82Author Commented:
And here is the results of windirstat run as SYSTEM using psexec
WinDirStat-System.png
Lee W, MVPTechnology and Business Process AdvisorCommented:
Wow... ok.... have you run chkdsk on the drive?  Scanned with antivirus?
cmbaker82Author Commented:
Yeah I've checked for virus' and ran chkdsk.  I just ran chkdsk again just in case.  No errors found.  And it shows the free and used space same as explorer.
chkdsk.txt
noxchoGlobal Support CoordinatorCommented:
Can you brake the mirror and then convert the drive to basic mbr type and see if it has the same problem? Sure you need first to backup it.
There is something definitely wrong. The file system is smaller than partition size. This could be due to size of cluster.
cmbaker82Author Commented:
Yeah I need to wait on a new hard drive to come in for my backups before I want to risk messing with the raid even though it should be safe enough to do.
noxchoGlobal Support CoordinatorCommented:
Ok.
cmbaker82Author Commented:
Got the new drive in today, waiting on backup to complete
cmbaker82Author Commented:
So after several days of copying the data it appears there is an issue with the file structure.  I used robocopy to copy the data.    Reminder I have 1.2TB of data on a 3TB drive that only shows 200gb of free space.   So far robocopy has filled 2.5TB of data on the new drive and still going
cmbaker82Author Commented:
Now i'm going to delete (on the backup copy) each directory one at a time to see if i can narrow down which is using more space then it should
cmbaker82Author Commented:
So while deleting one of the directories on the backup drive (which is now having the same issue) it shows that there are 0 items remaining and 0 bytes and yet it is still deleting stuff. and there is still 100gb  in the directory it is deleting.
weird-deletion-items-remaining.png
noxchoGlobal Support CoordinatorCommented:
WOW! What are you programming with python? The item which is deleting - test_minidom.py is actually a test opensource software code.
cmbaker82Author Commented:
I'm a .NET programmer, but a while ago I had taken a few classes on python programming, and still have some class files on the drive.
noxchoGlobal Support CoordinatorCommented:
Well as you see on your screenshot it hangs on that file. Is there any active database generating script you've forgotten?
cmbaker82Author Commented:
it didn't hang on that file it just happened to be there when I didn't the screenshot.
noxchoGlobal Support CoordinatorCommented:
Ah, then sorry. So the data you have transferred with robocopy was more than 1.5TB, right?
cmbaker82Author Commented:
yeah it was 1.2tb of data and filled 2.5tb on the disk. I've since deleted as much as I could from the backup disk to where there is less than 100gb used and it still shows the drive having over 1TB used.
noxchoGlobal Support CoordinatorCommented:
The new drive, right? And what says TreeSize to this?
cmbaker82Author Commented:
Yes on the new drive.  So on the new drive (Drive E) I copied everything from the original drive (D) to a subfolder called "DataDriveCopy8-29-16"    The new drive is an 8TB drive and originally showed a capacity of 7.27TB with all of it free.  After copying the Data from the D drive to it, it showed about 2.5TB used.  I then deleted all the data on the E drive, except for a few files that are giving me some permission issues, and now it shows 6.04TB free for a total used of 1.23TB.  I've attached a screeenshot of treesize ran as system showing the E Drive
Treesize-after-deleting-most-everyth.png
noxchoGlobal Support CoordinatorCommented:
Habe you emptied the Recycle Bin?
cmbaker82Author Commented:
Yes
noxchoGlobal Support CoordinatorCommented:
But Treesize shows some data in Recycle Bin. Half in Windows folder and half in Recycle Bin.
CompProbSolvCommented:
Can you set TreeSize to display both the Size and the Space Allocated?  You want to make sure that there isn't something odd going on there.
cmbaker82Author Commented:
Currently treesize shows 55,802.7MB Allocated, and 55,704.8 as the size
cmbaker82Author Commented:
This looks like it may be an issue with the Application Data folders in programdata and all users.  Normally these have junctions pointing back to themselves but i think something may have gone weird
cmbaker82Author Commented:
The application data folders I mentioned are form an old windows 7 install, not the current installation
noxchoGlobal Support CoordinatorCommented:
Very weird issue. I would think about file system issues but you've used robocopy and file system should have not been transferred to this drive as well then.
Do you use any vms? Or are the snapshots of the system enabled?
CompProbSolvCommented:
It seems that you were on the right track with deleting folders and seeing when the missing space comes back.  Can you finish that process?
cmbaker82Author Commented:
working on it.  Having trouble removing the "All Users" folder.  After getting rid of program data it cleared up about 500GB.    Currently using robocopy to copy from an empty directory to the all users folder with the purge option to delete the stuff in it.
treesize-with-only-All-Users-left.PNG
CompProbSolvCommented:
In the earlier picture: https://filedb.experts-exchange.com/incoming/2016/09_w37/1115489/Treesize-after-deleting-most-everyth.png "Users" was empty.

Since then: https://filedb.experts-exchange.com/incoming/2016/09_w37/1115655/treesize-with-only-All-Users-left.PNG it has 52G of data.  

I'm missing how data got added to the Users folder between those two posts.

What does chkdsk report on the 8T drive?  This all looks as if there are folders on the drive that TreeSize isn't seeing.  Are you running TreeSize in admin mode or (probably better) through PSExec?
cmbaker82Author Commented:
I am running it as system using psexec. the earlier picture may have been before resetting permissions on the folder or something like that. no new data has been added. check disk reported no errors on either drive
CompProbSolvCommented:
What did chkdsk show as the total capacity and the available space?  Did the bytes in use seem correct or is the large amount "missing" as with TreeSize?
cmbaker82Author Commented:
Chkdsk reported the same used space approximately as the properties window for the disk did.  After deleting the "programdata\application data" and all of the "Users\All Users\Application data" folders the disk space used again matched the size reported by treesize on the external drive.   For some reason these two folders from an old windows installation were using up 1.5TB of data
cmbaker82Author Commented:
So now that I have cleaned up the external drive and figured out what caused the issue its time to make a new backup copy of the data so I can get back to fixing the issue on the original drive.    I believe the issue on the external drive was actually related to robocopy messing up the junctions in the folders I mentioned.  On the source drive I deleted the application data folders the same as I did on the external drive but here it only freed up about 500GB of data.   Before I do anything else on the source drive I am making a new backup copy to the external drive excluding the old windows installation files so if we break anything while working on the original drive nothing will be lost.  
Currently the original drive has about 1.2TB of files on it that treesize can see (ran as system or as administrator); however chkdsk and explorer are reporting 2TB used.    So about 800GB of missing space on a drive with a formatted capacity of 2.72TB
noxchoGlobal Support CoordinatorCommented:
Can you open the Windows installation folders and check one by one the directory properties by right click on each - properties? Maybe you can narrow the problem to its bottleneck by finding which folder exactly swallowed the space.
cmbaker82Author Commented:
When I do that, the total space used matches that shown by tree size.  The only place it shows the extra used space is on the drive properties
noxchoGlobal Support CoordinatorCommented:
This is definitely mysterious. Does it behave this way even if you copy only one folder?
cmbaker82Author Commented:
After deleting the the all of the old application data folders in the programdata and users\<username>\ folders the space is more correct.  I've also done some other cleanup so the latest results are:  
Disk Properties reports: 1,173,473MB used.  Treesize reports 1,161,156MB used.  This is a difference of only 12GB which is within a reasonable range for what might be stored in the system volume information folder.  

This is the solution (ONLY TO BE PERFORMED ON OLD WINDOWS INSTALLATIONS, NOT CURRENT OS DISK) ALSO BE SURE YOU HAVE SAVED ANYTHING YOU NEED FROM THE OLDER USERS DIRECTORIES (desktop, documents, videos, music, etc.)
1. create empty directory (d:\test)
2. Open elevated command prompt
3 robocopy /e /purge d:\test d:\users
4 robocopy /e /purge d:\test "d:\programdata\application data"
5. Fix permissions as needed
6. Repeat steps 3-5 until the folders are empty.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Gerald ConnollyCommented:
The size differences could just be down to rounding differences, or maybe the old 1000x1000 v 1024x1024 conundrum
cmbaker82Author Commented:
I was able to figure this out on my own
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows OS

From novice to tech pro — start learning today.