Solved

Need help finding out what happened to 1.5TB of space on my harddrive

Posted on 2016-08-24
54
69 Views
Last Modified: 2016-09-24
I have a 3TB drive as a separate data drive on Windows 8.1  This drive contains an old installation of windows 7.  The drive size reported in disk management is as expected 2794.39GB.    the Properties window shows that 2.5TB has been used and 231 GB is free for a total capacity of 2.72GB.    The problem is that I cannot figure out why it is showing 2.5TB used.    Unfortunately I did not pay attention to the growth rate and do not know when this occurred.  It may have been when i was still running that drive as my primary OS drive with Windows 7, or it may have occurred since adding it as a secondary drive to my Windows 8.1 system.

I've granted myself access to the System Volume Information folder and recycle bin so those folders are accounted for.
I've used WinDirStat and TreeSize so see where the space is being used up both report a total of 1.2TB in use.  So i should have roughtly another 1.5TB of free space left that is unaccounted for.
Used-Space.png
TreeSize.png
WinDirStat.png
0
Comment
Question by:cmbaker82
  • 29
  • 12
  • 6
  • +3
54 Comments
 
LVL 20

Expert Comment

by:CompProbSolv
ID: 41769153
Did you run TreeSize as an administrator?  It should have given you a prompt about that if not.
0
 

Author Comment

by:cmbaker82
ID: 41769159
Yes tree size was ran as administrator
0
 
LVL 17

Expert Comment

by:pjam
ID: 41769176
How big is you WinSXS folder under Windows?
Have you installed the optional update kb2852386 which allows you to manage hidden SP an patch backup files no longer needed.
https://support.microsoft.com/en-us/kb/2852386
 64-bit is here:
https://www.microsoft.com/en-us/download/details.aspx?id=40445
0
 

Author Comment

by:cmbaker82
ID: 41769656
The windows installation on that drive is no longer in use, so I had already deleted the WinSXS folder on that drive.    The windows 7 installation is no longer in use, just some files remaining.    The current OS is Windows 8.1 installed on the C drive.  This is the D drive we are working with.
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 41769744
Please see my article on getting an accurate reading using WinDirStat/Tree size:
https://www.experts-exchange.com/articles/28565/Using-Tools-To-Find-What's-Using-Your-Disk-Space.html
0
 

Author Comment

by:cmbaker82
ID: 41769760
As I already mentioned I did grant myself access to the system volume information folder.
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 41769762
And you think that's the only place that my article can apply?  You realize you SHOULDN'T have done that in the first place.
0
 

Author Comment

by:cmbaker82
ID: 41769764
Unfortunately the article is reserved for premium members which I am not.  And why do you think I should not have done that when I was trying to determine how much space was being used?
0
 
LVL 20

Expert Comment

by:CompProbSolv
ID: 41769766
@cmbaker:

I just read Lee's article and it appears to be excellent advice.  It shouldn't take too much effort to give the program a shot and see if it exposes the "hidden" used space on your drive.  I'll have to add it to my bag of "tricks".

@Lee:

Thank you for an excellent article.
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 41769768
That's unacceptable to me.  I write these and place them on EE believing articles are publicly accessible.  If that's changed, I'll have to post them elsewhere as well.  Sorry, there will be a delay before that happens but it WILL happen.
0
 

Author Comment

by:cmbaker82
ID: 41769771
I will be more than glad to read it when it is accessible.    What program was suggested in the article?
0
 
LVL 95

Assisted Solution

by:Lee W, MVP
Lee W, MVP earned 500 total points
ID: 41769778
Summary - use PSEXEC to get a command prompt as the system account.  Then run Windirstat as System.  Administrator is not good enough.
0
 

Author Comment

by:cmbaker82
ID: 41769798
I will do this in the morning. Thanks.
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41769849
Can you post also a screenshot of Windows Disk Management? Looks like you have here a PartMgr problem where the used size reported by Windows Explorer and Windows Disk Management differs.
0
 

Author Comment

by:cmbaker82
ID: 41770485
Here is the disk management screenshot.
disk-management.PNG
0
 

Author Comment

by:cmbaker82
ID: 41770490
And here is the results of windirstat run as SYSTEM using psexec
WinDirStat-System.png
0
 
LVL 95

Expert Comment

by:Lee W, MVP
ID: 41770503
Wow... ok.... have you run chkdsk on the drive?  Scanned with antivirus?
0
 

Author Comment

by:cmbaker82
ID: 41770646
Yeah I've checked for virus' and ran chkdsk.  I just ran chkdsk again just in case.  No errors found.  And it shows the free and used space same as explorer.
chkdsk.txt
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41770965
Can you brake the mirror and then convert the drive to basic mbr type and see if it has the same problem? Sure you need first to backup it.
There is something definitely wrong. The file system is smaller than partition size. This could be due to size of cluster.
0
 

Author Comment

by:cmbaker82
ID: 41771168
Yeah I need to wait on a new hard drive to come in for my backups before I want to risk messing with the raid even though it should be safe enough to do.
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41771334
Ok.
0
 

Author Comment

by:cmbaker82
ID: 41775150
Got the new drive in today, waiting on backup to complete
0
 

Author Comment

by:cmbaker82
ID: 41786659
So after several days of copying the data it appears there is an issue with the file structure.  I used robocopy to copy the data.    Reminder I have 1.2TB of data on a 3TB drive that only shows 200gb of free space.   So far robocopy has filled 2.5TB of data on the new drive and still going
0
 

Author Comment

by:cmbaker82
ID: 41786668
Now i'm going to delete (on the backup copy) each directory one at a time to see if i can narrow down which is using more space then it should
0
 

Author Comment

by:cmbaker82
ID: 41786686
So while deleting one of the directories on the backup drive (which is now having the same issue) it shows that there are 0 items remaining and 0 bytes and yet it is still deleting stuff. and there is still 100gb  in the directory it is deleting.
weird-deletion-items-remaining.png
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41786710
WOW! What are you programming with python? The item which is deleting - test_minidom.py is actually a test opensource software code.
0
 

Author Comment

by:cmbaker82
ID: 41786743
I'm a .NET programmer, but a while ago I had taken a few classes on python programming, and still have some class files on the drive.
0
Do email signature updates give you a headache?

Do you feel like all of your time is spent managing email signatures? Too busy to visit every user’s desk to make updates? Want high-quality HTML signatures on all devices, including on mobiles and Macs? Then, let Exclaimer solve all your email signature problems today!

 
LVL 46

Expert Comment

by:noxcho
ID: 41786764
Well as you see on your screenshot it hangs on that file. Is there any active database generating script you've forgotten?
0
 

Author Comment

by:cmbaker82
ID: 41786807
it didn't hang on that file it just happened to be there when I didn't the screenshot.
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41786810
Ah, then sorry. So the data you have transferred with robocopy was more than 1.5TB, right?
0
 

Author Comment

by:cmbaker82
ID: 41786811
yeah it was 1.2tb of data and filled 2.5tb on the disk. I've since deleted as much as I could from the backup disk to where there is less than 100gb used and it still shows the drive having over 1TB used.
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41786833
The new drive, right? And what says TreeSize to this?
0
 

Author Comment

by:cmbaker82
ID: 41786841
Yes on the new drive.  So on the new drive (Drive E) I copied everything from the original drive (D) to a subfolder called "DataDriveCopy8-29-16"    The new drive is an 8TB drive and originally showed a capacity of 7.27TB with all of it free.  After copying the Data from the D drive to it, it showed about 2.5TB used.  I then deleted all the data on the E drive, except for a few files that are giving me some permission issues, and now it shows 6.04TB free for a total used of 1.23TB.  I've attached a screeenshot of treesize ran as system showing the E Drive
Treesize-after-deleting-most-everyth.png
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41786852
Habe you emptied the Recycle Bin?
0
 

Author Comment

by:cmbaker82
ID: 41786947
Yes
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41787033
But Treesize shows some data in Recycle Bin. Half in Windows folder and half in Recycle Bin.
0
 
LVL 20

Expert Comment

by:CompProbSolv
ID: 41787041
Can you set TreeSize to display both the Size and the Space Allocated?  You want to make sure that there isn't something odd going on there.
0
 

Author Comment

by:cmbaker82
ID: 41787066
Currently treesize shows 55,802.7MB Allocated, and 55,704.8 as the size
0
 

Author Comment

by:cmbaker82
ID: 41787068
This looks like it may be an issue with the Application Data folders in programdata and all users.  Normally these have junctions pointing back to themselves but i think something may have gone weird
0
 

Author Comment

by:cmbaker82
ID: 41787069
The application data folders I mentioned are form an old windows 7 install, not the current installation
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41787347
Very weird issue. I would think about file system issues but you've used robocopy and file system should have not been transferred to this drive as well then.
Do you use any vms? Or are the snapshots of the system enabled?
0
 
LVL 20

Expert Comment

by:CompProbSolv
ID: 41787369
It seems that you were on the right track with deleting folders and seeing when the missing space comes back.  Can you finish that process?
0
 

Author Comment

by:cmbaker82
ID: 41788398
working on it.  Having trouble removing the "All Users" folder.  After getting rid of program data it cleared up about 500GB.    Currently using robocopy to copy from an empty directory to the all users folder with the purge option to delete the stuff in it.
treesize-with-only-All-Users-left.PNG
0
 
LVL 20

Expert Comment

by:CompProbSolv
ID: 41789859
In the earlier picture: https://filedb.experts-exchange.com/incoming/2016/09_w37/1115489/Treesize-after-deleting-most-everyth.png "Users" was empty.

Since then: https://filedb.experts-exchange.com/incoming/2016/09_w37/1115655/treesize-with-only-All-Users-left.PNG it has 52G of data.  

I'm missing how data got added to the Users folder between those two posts.

What does chkdsk report on the 8T drive?  This all looks as if there are folders on the drive that TreeSize isn't seeing.  Are you running TreeSize in admin mode or (probably better) through PSExec?
0
 

Author Comment

by:cmbaker82
ID: 41789868
I am running it as system using psexec. the earlier picture may have been before resetting permissions on the folder or something like that. no new data has been added. check disk reported no errors on either drive
0
 
LVL 20

Expert Comment

by:CompProbSolv
ID: 41790060
What did chkdsk show as the total capacity and the available space?  Did the bytes in use seem correct or is the large amount "missing" as with TreeSize?
0
 

Author Comment

by:cmbaker82
ID: 41794858
Chkdsk reported the same used space approximately as the properties window for the disk did.  After deleting the "programdata\application data" and all of the "Users\All Users\Application data" folders the disk space used again matched the size reported by treesize on the external drive.   For some reason these two folders from an old windows installation were using up 1.5TB of data
0
 

Author Comment

by:cmbaker82
ID: 41794947
So now that I have cleaned up the external drive and figured out what caused the issue its time to make a new backup copy of the data so I can get back to fixing the issue on the original drive.    I believe the issue on the external drive was actually related to robocopy messing up the junctions in the folders I mentioned.  On the source drive I deleted the application data folders the same as I did on the external drive but here it only freed up about 500GB of data.   Before I do anything else on the source drive I am making a new backup copy to the external drive excluding the old windows installation files so if we break anything while working on the original drive nothing will be lost.  
Currently the original drive has about 1.2TB of files on it that treesize can see (ran as system or as administrator); however chkdsk and explorer are reporting 2TB used.    So about 800GB of missing space on a drive with a formatted capacity of 2.72TB
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41794973
Can you open the Windows installation folders and check one by one the directory properties by right click on each - properties? Maybe you can narrow the problem to its bottleneck by finding which folder exactly swallowed the space.
0
 

Author Comment

by:cmbaker82
ID: 41794979
When I do that, the total space used matches that shown by tree size.  The only place it shows the extra used space is on the drive properties
0
 
LVL 46

Expert Comment

by:noxcho
ID: 41795002
This is definitely mysterious. Does it behave this way even if you copy only one folder?
0
 

Accepted Solution

by:
cmbaker82 earned 0 total points
ID: 41803369
After deleting the the all of the old application data folders in the programdata and users\<username>\ folders the space is more correct.  I've also done some other cleanup so the latest results are:  
Disk Properties reports: 1,173,473MB used.  Treesize reports 1,161,156MB used.  This is a difference of only 12GB which is within a reasonable range for what might be stored in the system volume information folder.  

This is the solution (ONLY TO BE PERFORMED ON OLD WINDOWS INSTALLATIONS, NOT CURRENT OS DISK) ALSO BE SURE YOU HAVE SAVED ANYTHING YOU NEED FROM THE OLDER USERS DIRECTORIES (desktop, documents, videos, music, etc.)
1. create empty directory (d:\test)
2. Open elevated command prompt
3 robocopy /e /purge d:\test d:\users
4 robocopy /e /purge d:\test "d:\programdata\application data"
5. Fix permissions as needed
6. Repeat steps 3-5 until the folders are empty.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 41803460
The size differences could just be down to rounding differences, or maybe the old 1000x1000 v 1024x1024 conundrum
0
 

Author Closing Comment

by:cmbaker82
ID: 41813565
I was able to figure this out on my own
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
If you get continual lockouts after changing your Active Directory password, there are several possible reasons.  Two of the most common are using other devices to access your email and stored passwords in the credential manager of windows.
The Task Scheduler is a powerful tool that is built into Windows. It allows you to schedule tasks (actions) on a recurring basis, such as hourly, daily, weekly, monthly, at log on, at startup, on idle, etc. This video Micro Tutorial is a brief intro…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now