Create a list of disk usage by directory

LrdKanien
LrdKanien used Ask the Experts™
on
I have a backup server and I want to create a list with data values for the disk space usage.  For example, the hierarchy is (drive letter:\sharename\servername\virtual\username) and I want to look at the user name directories for size and list them.

The backup server is Windows 2003.  The volumes are E, F, G, and H.  They are all 2 TB arrays.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Top Expert 2009

Commented:

#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use File::Find;
 
my %Users;
find(\&found, "$_:/sharename/servername/virtual") foreach (qw(E F G H));
 
foreach (keys %Users) {
	print "$_: $Users{$_}\n";
}
 
sub found {
	return unless $File::Find::name =~ m|^\w:/sharename/servername/virtual/(.*?)/|;
	$Users{$1} += -s $File::Find::name;
}

Open in new window

Author

Commented:
Adam thanks for the lightening fast reply...two issues.

1.  The Sharename and Servername in the path is variable.  I don't mind having to hardcode them, otherwise you have to grab a list of the directories inside the sub directory and use them as variables from an array.
2.  Where does it put the output?

I'll attach an example of it hardcoded.
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use File::Find;
 
my %Users;
find(\&found, "$_:/WINHOST-AR1/Maxima1/virtual") foreach (qw(E F G H));
 
foreach (keys %Users) {
	print "$_: $Users{$_}\n";
}
 
sub found {
	return unless $File::Find::name =~ m|^\w:/WINHOST-AR1/Maxima1/virtual/(.*?)/|;
	$Users{$1} += -s $File::Find::name;
}

Open in new window

Top Expert 2009

Commented:
If you don't want to hardcode, you could find everything on the drive:
    find(\&found, "$_:/") foreach (qw(E F G H));
    ...
    return unless $File::Find::name =~ m|^\w:/.*?/.*?/virtual/(.*?)/|;

If there are other directories on the drive, this would search them also, and the 4th sub-directory will be assumed to be a username.


CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

Author

Commented:
This is 10TB of data.  We are talking 15,000 user directories.  Where does it output? I need it to output to some kind of log file.

Author

Commented:
This is how I have the code now.
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use File::Find;
 
my %Users;
find(\&found, "$_:/") foreach (qw(E F G H));
 
foreach (keys %Users) {
	print "$_: $Users{$_}\n";
}
 
sub found {
	return unless $File::Find::name =~ m|^\w:/.*?/.*?/virtual/(.*?)/|;
	$Users{$1} += -s $File::Find::name;
}

Open in new window

Author

Commented:
Use of uninitialized value in addition (+) at C:\Documents and Settings\davidf\M
y Documents\datausage.pl line 16.
ozo
Most Valuable Expert 2014
Top Expert 2015

Commented:
could be a symbolic link that does not exist

Author

Commented:
this was the end result.  Can perl not grab the directory listing and pull the size without finding everything in the directories?  I put this question in perl, vbscript, and powershell forums, maybe perl is the wrong language to try this in.
Use of uninitialized value in addition (+) at C:\Documents and Settings\davidf\M
y Documents\datausage.pl line 16.
Can't opendir(E:/WINHOST-AR1/Maxima1/virtual/mwestray/site/Website Stuff/Website
 Stuff/59 Web Flash Templates Share By Lidapanther/Website Stuff/59 Web Flash Te
mplates Share By Lidapanther/59_Web Flash Templates_SHARE BY_LiDaPaNtHeR/Website
 Stuff/AQUA/Website Stuff/MUSIC1): No such file or directory
 at C:\Documents and Settings\davidf\My Documents\datausage.pl line 8
Can't cd to E:/WINHOST-AR1/Maxima1/virtual/mwestray/site/Website Stuff/Website S
tuff/59 Web Flash Templates Share By Lidapanther/Website Stuff/59 Web Flash Temp
lates Share By Lidapanther/59_Web Flash Templates_SHARE BY_LiDaPaNtHeR/Website S
tuff/AQUA/Website Stuff/MUSIC1../../../../../../../../../../.. at C:/Perl/lib/Fi
le/Find.pm line 940.

Open in new window

Suhas .Senior QA Manager

Commented:

use Filesys::DfPortable;
 
my $ref = dfportable("D:\\");
die "Could not get disk info:\n  $!\n  $@\n" unless $ref;
 
printf "Used : %12s\n", $ref->{bused};
printf "Total: %12s\n", $ref->{blocks};

Open in new window

Top Expert 2009

Commented:
The Filesys::DfPortable module will tell you info about the disk, not about usage in individual directories - so I don't think this is what you want.

The script I gave will generate the output on the screen.  This could be redirected to a file like so:
    script.pl > output.txt
Or, the script could be modified to output directly to a file, if you'd prefer.

>>Can perl not grab the directory listing and pull the size without finding everything in the directories?
No, but neither could any of the other languages.  The total size of all files in a directory tree is not stored anywhere, so the only way for any language is to search through all the subdirectories, and add it up.

The "Use of uninitialized..." error is because the program can not get the size of a file.  Not sure the cause of this.
The "Can't opendir..." and "Can't cd..." I believe are errors due to windows not supporting path names longer than 260 characters, which needs to include a terminating NULL(both of these paths are 259 characters excluding the NULL).  
    http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx
Is there another path to access these directories, such as through a different mapped drive?
If not, you might be able to do this:
  1)  Use the "subst" to create a new drive letter that refers to the drive:\sharename\servername\virtual\username
  2) Process that drive to get the usage for that user
  3) Use "subst /D" to delete new drive letter
  4) Repeat A-C for each user
Let me know if you want to give this a try.

Was there any other output?  Usually, if the File::Find has an error, it'll display the error (as you've seen), but then continue.
Top Expert 2009

Commented:
To deal with the long path names, you might be able to use the extended-length path syntax (described lower on the page I linked to above).  To try this, change line 8 to this:
find(\&found, "\\\\?\\$_:/") foreach (qw(E F G H));

Open in new window

Author

Commented:
Attached is the error generated from the code change suggested above.

Let's try something else..  Lets just attempt to do it for 1 path.  E:\WINHOST-AR1\Maxima1\Virtual\  I would like that path to then list the disk space usage of each directory in that path.


Can't stat \\?\E:/: No such file or directory
 at C:\Documents and Settings\davidf\My Documents\datausage.pl line 8
Can't stat \\?\F:/: No such file or directory
 at C:\Documents and Settings\davidf\My Documents\datausage.pl line 8
Can't stat \\?\G:/: No such file or directory
 at C:\Documents and Settings\davidf\My Documents\datausage.pl line 8
Can't stat \\?\H:/: No such file or directory
 at C:\Documents and Settings\davidf\My Documents\datausage.pl line 8

Open in new window

Top Expert 2009

Commented:
I don't think the problem is caused by the amount of paths being searched - it's by an individual path being to long (to long by windows requirements - not perl).

Anyways, to process just the 1 requested path, change line 8 to this:
find(\&found, 'E:/WINHOST-AR1/Maxima1/Virtual');

Open in new window

Author

Commented:
Can't opendir(E:\WINHOST-AR1\Maxima1\Virtual/mwestray/site/Website Stuff/Website
 Stuff/59 Web Flash Templates Share By Lidapanther/Website Stuff/59 Web Flash Te
mplates Share By Lidapanther/59_Web Flash Templates_SHARE BY_LiDaPaNtHeR/Website
 Stuff/AQUA/Website Stuff/MUSIC1): No such file or directory
 at C:\Documents and Settings\davidf\My Documents\datausage.pl line 8
Can't cd to E:\WINHOST-AR1\Maxima1\Virtual/mwestray/site/Website Stuff/Website S
tuff/59 Web Flash Templates Share By Lidapanther/Website Stuff/59 Web Flash Temp
lates Share By Lidapanther/59_Web Flash Templates_SHARE BY_LiDaPaNtHeR/Website S
tuff/AQUA/Website Stuff/MUSIC1../../../../../../../../../../.. at C:/Perl/lib/Fi
le/Find.pm line 940.
Top Expert 2009
Commented:
This directory is also 260 characters long - so I think that is the problem.  To confirm, try this:

Run the subst command to create an alias to the directory.  From a command prompt, type (use a diff drive letter if needed):
    subst Q: E:\WINHOST-AR1\Maxima1\Virtual
Edit perl code on line 8 like so (use same drive letter as above)
    find(\&found, 'Q:/');
Edit perl code on line 15 like so:
    return unless $File::Find::name =~ m|^\w:/(.*?)/|;
Run perl script

Author

Commented:
If you are saying the directory structure is limited to a set amount of characters, then this is never going to work.


Use of uninitialized value in addition (+) at C:\Documents and Settings\davidf\M
y Documents\datausage.pl line 16.
Can't cd to Q:/mwestray/site/Website Stuff/Website Stuff/59 Web Flash Templates
Share By Lidapanther/Website Stuff/59 Web Flash Templates Share By Lidapanther/5
9_Web Flash Templates_SHARE BY_LiDaPaNtHeR/Website Stuff/AQUA/Website Stuff/MUSI
C1../../../../../../../../../../.. at C:/Perl/lib/File/Find.pm line 940.

Open in new window

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial