Solved

Disk space confusion

Posted on 2006-07-24
18
541 Views
Last Modified: 2008-02-01
Hi, on my Netware 5 server, my volume "ONE" is reporting 94% full on 11GB . Yet through windows explorer if I right-click on some of the folders in that volume, some of them alone are more than 11GB. We recently archived about 4GB to CD and the volume is still reporting 93-94% full. Can anyone explain this difference in the disk space usage?

Thanks
Tim
0
Comment
Question by:timothyking
  • 8
  • 5
  • 4
  • +1
18 Comments
 
LVL 34

Expert Comment

by:PsiCop
Comment Utility
Well, you haven't said what interface is reporting the 94% full. MONITOR at the NetWare console? ConsoleOne? NWADMIN? What specifically is telling you that?

I *suspect* that the diffference can be accounted for if you have not purged the Volume. When you delete files in the default NetWare environment, the file is not actually deleted. Instead, it's marked as deleted, but the space it occupies is still allocated to it.

After a certain amount of time has elapsed (the amount of time is controlled by server- and fileystem-specific settings), then the space is made available for re-use, but still allocated to the file until it is actually overwritten. This allows files to be SALVAGED directly from the server without having to go to backup.

Purging the filesystem makes the space occupied by deleted files, that have met the time requirement, immediately available. The files would not be recoverable with a SALVAGE command.
0
 

Author Comment

by:timothyking
Comment Utility
I viewed this from MONITOR at the console. I did do a purge (from windows explorer) and it completed, and it still reports 94% full.
0
 
LVL 34

Expert Comment

by:PsiCop
Comment Utility
Under what user ID were you logged into eDirectory as when you performed the PURGE?

Did you perform the Purge at the root of the Volume, or in a subdirectory?

Did you purge the current directory only, or did you purge the current directory and all subdirectories?

When you archived the data recently, did you delete the directory structures that housed the data?
0
 

Author Comment

by:timothyking
Comment Utility
Under what user ID were you logged into eDirectory as when you performed the PURGE?

** I was logged in with administrator rights

Did you perform the Purge at the root of the Volume, or in a subdirectory?

** Both

Did you purge the current directory only, or did you purge the current directory and all subdirectories?

**Current and all subs

When you archived the data recently, did you delete the directory structures that housed the data?

** I deleted the top most directories and all below
0
 
LVL 35

Expert Comment

by:ShineOn
Comment Utility
How about compression?  More often than not, NW5x has compression turned on on the volumes.  If you remove files, it allows files to uncompress and remain uncompressed longer than if there's not enough room (in which case the files will remain compressed, only uncompressing to cache.)

If you're running the Novell client, there are Novell-specific context menus in Explorer that, when you right-click on a drive mapped to the root of a volume you can see a pie chart showing used space, free space, compressed space and purgeable space.  Take a look at that to see how much is compressed.
0
 

Author Comment

by:timothyking
Comment Utility
It looks like that might be it. Here is the breakdown:

Total space: 11,483,136KB
Free Space: 900,544KB
Compressed Space: 4,020,915KB
Purgeable Space: 139,520KB
Used Space: 6,422,157KB

Now, how do I reclaim the compressed space?

Thanks
0
 
LVL 35

Expert Comment

by:ShineOn
Comment Utility
You don't "reclaim" it - it's taking up less space than it would uncompressed.  The only thing you could reclaim would be the purgeable space, as PsiCop discussed already.  

The "used space" and "compressed space" get added together to make "unavailable" space, so to speak.  The purgeable and free added together make "available" space.  The "compressed" will convert to "used" based on your compression rules, but nothing will remain uncompressed while the free space is below a certain percentage of total, usually 10% but that's a threshold you can set.  It's not recommended to let your free space get below 10%.  First to go will be salvage, then things will be forced to stay compressed, and so on, and you'll start getting "volume running out of room" messages broadcast through the LAN...

You could, if you had to, change your compression criteria and "reclaim" some of the "used space" by having more files eligible for compression.

0
 

Author Comment

by:timothyking
Comment Utility
That's enough to make my head spin! I just don't get that I delete 4GB of files and my free space stays the same!
 I've deleted large chunks of data in the past and the % full would immediatley drop accordingly.

0
 
LVL 1

Expert Comment

by:pspencer53
Comment Utility
I don't think the concept of "reclaiming" compressed space is appropriate.   Basically, file compression saves space on a disk.  If you have compressed files on a NetWare server, you essentially can store more data than the file system can normally store.   If you want to free up more disk space, you would need to compress more files.

There are a couple fo NetWare concepts that explain the difference between "free disk space", "size of all the files", and "Disk space used".

First of all, NetWare 5.1 defaults to traditional volumes.  These volumes are created by default with compression enabled.  Compression occurs each night between 12:00 and 6:00am and is triggered by a file being inactive (or not having been touched) for a period of time, default I believe is 14 days.   Files are also automatically uncompressed (by default) on the second instance of it beign touched within the time period.  All of htese setting can be accessed in Monitor -> Server Parameters -> File System.  

Both file compression and also file uncompression take a lot of server resources.   For the last several years, I have always turned compression off.  I have run into too many nasty issues with compression.   Example: if you want to manually copy all the files to another server (say in a migration), you can do it once without triggering the uncompress function.  The second time you copy it, it will fill up your volume to 90% as it uncompresses all the files.

Second, the NetWare traditional file system uses 64K blocks and block sub-allocation.  What this means is the minimum disk space used by a very small file is 64K.   If a file is larger than 64K, it can fill the gaps left by the mostly unused space in these small block.        If you have a lot of small filles, then you have a lot of wasted space.  There is no way that I know of to know how much of this space has been allocated.

In summary then, if you look at the size of all the files by using DIR /S from the root of the volume, you will see the sum of all of the file sizes.   If you use windows explorer to list the properties of all of the files and folders on the volume,you will see 2 numbers.  One is file sizes and the other is "Size on Disk".  The irony is that this is a Microsoft calculation as if it were a Microsoft server, and is is not accurate.

If you use NDIR to look at the volume, you will see a confusing list of numbers, as follows:
Total volume space:                           6,179,968  100.00%
Space used by 34,499 entries:                   819,456   13.26%
Deleted space not yet purgeable:                     64    0.00%
                                          ------------- --------
Space remaining on volume:                    5,360,512   86.74%
Space available to ADMIN:                     5,360,512   86.74%

Maximum directory entries:                       82,432
Available directory entries:                     47,933   58.15%

Space used if files were not compressed:         84,992
Space used by compressed files:                  27,136
                                          -------------
Space saved by compressing files:                57,856   68.07%

Uncompressed space used:                        792,384

In this example from my lab server, I have a 6GB volume reporting 14% full in monitor.
It is saying that the size on disk is 820MB and the size of all of the files (if uncompressed) is 885MB.  

The thing of it is that Windows Explorer reports as follows:
Windwos Explorer:

      15,820 Files, 1,412 Folders

Size          828 MB (836,132,562 bytes)
Size on Disk 1.63 GB (1,758,6715,322 bytes)

and DOS /S report:
     Total Files Listed:
           15815 File(s)    860,385,972 bytes
            4230 Dir(s)   5,489,033,216 bytes free

The reason for the descrepencies is not exactly clear, but I believe it is in what each considers a file and how it understands name space etc.

In any case, the Novell NDIR /VOL command is the one that is giving you NetWare information.   The example above shows the following:
I have 820MB of disk space used, but the files that fill that space say that they are 885MB (820 + 65) .  If you were to copy these files to another volume, they would initially take up at least this much space.    If you copy them to a Microsoft share, they may take as much as 1.8GB, as reported by Windows Explorer (size on disk).

The best and most thorough way to purge all files on volume is to drop to a dos box and tehn at the root of the volume type PURGE /ALL.  You will see every file name go by as it is deleted.  If you only see directory names then there were no files to be purged.

The "Space not yet purgable" is files that have been deleted very recently.  NetWare holds these files for between 1 minutes and 5 minutes in case you want to undelete them.  Then they become purgable.

Hope this helps a little.


 
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 35

Expert Comment

by:ShineOn
Comment Utility
I disagree with your statement about 64K blocks and block suballocation.  I always did my traditional volumes with 4k blocks, and the block suballocation lets you suballocate a block to multiple files, all the way down to 512 bytes per suballocation unit.  It may default to 64K blocks - I don't recall - but you always had the option to choose a different block size.  64K blocks may help for speed in some instances but really adds overhead with the suballocation process, and if you don't suballocate is very wasteful of disk real-estate,, but if you're more concerned about space utilization than about performance, 4K blocks are good, and block suballocation, as I said, took the suballocation unit to 512 bytes.  In fact, for WinXP on a moderate-size disk (say 40GB) if you set the NTFS blocksize to 4K it actually improves performance as well as utilization, but not many know that can be done, or how to do it... ;)

Now, on the other hand, Windoze tends to be set with the blocksize way too high,  and doesn't suballocate, causing a small file to take up a whole block, which is why "file size" and "used on disk" is sometimes radically different as you explained.  Windoze Exploder doesn't "dig" block suballocation, so it often misreports "size on disk" because it tries to fit it into the Windoze wasteful methodology.
0
 

Author Comment

by:timothyking
Comment Utility
Is there any way I can decrease the "Compressed" space on this volume?
0
 
LVL 34

Expert Comment

by:PsiCop
Comment Utility
Volumes are either Compression-enabled or they're not.

If the Volume is Compression-enabled, then files on it will be Compressed, in accordance with the Compression scheduling settings on the server. Until a file meets the requirements, it will not be Compressed. Exceptions exist for Directories or Files that are flagged Immediate Compression or No Compression, but these flags must be explicitly set.

There is no "Compressed space" on a Volume. There is the space being occupied by Compressed files. If accessed/altered, these files will be unCompressed and stored in their unCompressed state until they again meet the criteria (or an exception applies). When that happens, the amount of space occupied by Compressed files will decrease (and the amount of space occupied by unCompressed files will *increase* by the same or greater amount).
0
 
LVL 34

Expert Comment

by:PsiCop
Comment Utility
Note that pspencer's comments on how long deleted files are held before becoming purgable varies based on settings made on the server hosting the Volume in which the files reside. These settings can also vary based on filesystem type (e.g. NSS or Traditional/FAT), and version of NetWare. Finally, the migration of deleted files from deleted to purgeable is affected by flags on individual Directories or files (e.g. Purge Immediate).

The point being is there is nothing carved in stone saying that the hold time is 1 minute or 5 minutes or whatever. It might very well be that in the Asker's environment, but we have no way of knowing that unless we know the various settings and flags that exist in that particular environment. We can only talk about general defaults.
0
 

Author Comment

by:timothyking
Comment Utility
Is there any way to force these files to become purgable?


Thanks
0
 
LVL 35

Expert Comment

by:ShineOn
Comment Utility
Which files?  The deleted files?  The compressed files?

For traditional/FAT, check your "monitor.nlm" server parameters, traditional file system screen to see what the two settings - "minimum file delete wait time" and "file delete wait time" to see what your delay before a deleted file becomes purgeable is set to.

You can, if you so desire, set them both to zero, but that will make a NetWare filesystem behave like Windoze - what's deleted is gone, too bad, so sad.  If you have it set fairly short, you should be able to wait that long before doing a "purge /all" yet still be able to say "oops, I didn't mean to delete that" and have the file salvageable for a couple of minutes, even if you're in that threshold window where deleted files get purged automatically because you're so low on free space.
0
 

Author Comment

by:timothyking
Comment Utility
Here is more info on the volume:

minimum file delete wait time: 1 min 5.1 sec
file delete wait time: 5 min 25.9 sec


J:\>ndir /vol

Statistics for fixed volume JONES_NOVELL5/ONE:
Space statistics are in KB (1024 bytes).

Total volume space:                          11,483,136  100.00%
Space used by 128,567 entries:               10,499,712   91.44%
Deleted space not yet purgeable:                     64    0.00%
                                          ------------- --------
Space remaining on volume:                      983,424    8.56%
Space available to KING:                        983,424    8.56%

Maximum directory entries:                      393,216
Available directory entries:                    264,649   67.30%

Space used if files were not compressed:     17,481,216
Space used by compressed files:               3,912,192
                                          -------------
Space saved by compressing files:            13,569,024   77.62%

Uncompressed space used:                      7,098,304

Name spaces loaded: OS/2

J:\>





0
 
LVL 35

Accepted Solution

by:
ShineOn earned 50 total points
Comment Utility
Well, as you can see, to hold all of the data in uncompressed form you'd have to add another 13 gigs of space to your volume.  You're getting great value in space savings from compression - you've got 17 pounds of sh ... er ... stuff in a 4 pound bag, so to speak ;)

If you're concerned about the performance impact of having that much compressed, then you should see about getting more disk space.  Add a volume or something.

Since you only show 64K of deleted space not yet purgeable, I wouldn't worry about the purgeable delay settings.
0
 

Author Comment

by:timothyking
Comment Utility
I think I understand now. Looks like I have to archive more data or add more disk space.

Thanks
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Marketing can be an uncomfortable undertaking, especially if your material is technology based. Luckily, we’ve compiled some simple and (relatively) painless tips to put an end to your trepidation and start your path to success.
HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now