Disk space confusion

Well, you haven't said what interface is reporting the 94% full. MONITOR at the NetWare console? ConsoleOne? NWADMIN? What specifically is telling you that?

I *suspect* that the diffference can be accounted for if you have not purged the Volume. When you delete files in the default NetWare environment, the file is not actually deleted. Instead, it's marked as deleted, but the space it occupies is still allocated to it.

After a certain amount of time has elapsed (the amount of time is controlled by server- and fileystem-specific settings), then the space is made available for re-use, but still allocated to the file until it is actually overwritten. This allows files to be SALVAGED directly from the server without having to go to backup.

Purging the filesystem makes the space occupied by deleted files, that have met the time requirement, immediately available. The files would not be recoverable with a SALVAGE command.

I viewed this from MONITOR at the console. I did do a purge (from windows explorer) and it completed, and it still reports 94% full.

Under what user ID were you logged into eDirectory as when you performed the PURGE?

Did you perform the Purge at the root of the Volume, or in a subdirectory?

Did you purge the current directory only, or did you purge the current directory and all subdirectories?

When you archived the data recently, did you delete the directory structures that housed the data?

Under what user ID were you logged into eDirectory as when you performed the PURGE?

** I was logged in with administrator rights

Did you perform the Purge at the root of the Volume, or in a subdirectory?

** Both

Did you purge the current directory only, or did you purge the current directory and all subdirectories?

**Current and all subs

When you archived the data recently, did you delete the directory structures that housed the data?

** I deleted the top most directories and all below

How about compression?  More often than not, NW5x has compression turned on on the volumes.  If you remove files, it allows files to uncompress and remain uncompressed longer than if there's not enough room (in which case the files will remain compressed, only uncompressing to cache.)

If you're running the Novell client, there are Novell-specific context menus in Explorer that, when you right-click on a drive mapped to the root of a volume you can see a pie chart showing used space, free space, compressed space and purgeable space.  Take a look at that to see how much is compressed.

It looks like that might be it. Here is the breakdown:

Total space: 11,483,136KB
Free Space: 900,544KB
Compressed Space: 4,020,915KB
Purgeable Space: 139,520KB
Used Space: 6,422,157KB

Now, how do I reclaim the compressed space?

Thanks

You don't "reclaim" it - it's taking up less space than it would uncompressed.  The only thing you could reclaim would be the purgeable space, as PsiCop discussed already.  

The "used space" and "compressed space" get added together to make "unavailable" space, so to speak.  The purgeable and free added together make "available" space.  The "compressed" will convert to "used" based on your compression rules, but nothing will remain uncompressed while the free space is below a certain percentage of total, usually 10% but that's a threshold you can set.  It's not recommended to let your free space get below 10%.  First to go will be salvage, then things will be forced to stay compressed, and so on, and you'll start getting "volume running out of room" messages broadcast through the LAN...

You could, if you had to, change your compression criteria and "reclaim" some of the "used space" by having more files eligible for compression.

That's enough to make my head spin! I just don't get that I delete 4GB of files and my free space stays the same!
 I've deleted large chunks of data in the past and the % full would immediatley drop accordingly.

I don't think the concept of "reclaiming" compressed space is appropriate.   Basically, file compression saves space on a disk.  If you have compressed files on a NetWare server, you essentially can store more data than the file system can normally store.   If you want to free up more disk space, you would need to compress more files.

There are a couple fo NetWare concepts that explain the difference between "free disk space", "size of all the files", and "Disk space used".

First of all, NetWare 5.1 defaults to traditional volumes.  These volumes are created by default with compression enabled.  Compression occurs each night between 12:00 and 6:00am and is triggered by a file being inactive (or not having been touched) for a period of time, default I believe is 14 days.   Files are also automatically uncompressed (by default) on the second instance of it beign touched within the time period.  All of htese setting can be accessed in Monitor -> Server Parameters -> File System.  

Both file compression and also file uncompression take a lot of server resources.   For the last several years, I have always turned compression off.  I have run into too many nasty issues with compression.   Example: if you want to manually copy all the files to another server (say in a migration), you can do it once without triggering the uncompress function.  The second time you copy it, it will fill up your volume to 90% as it uncompresses all the files.

Second, the NetWare traditional file system uses 64K blocks and block sub-allocation.  What this means is the minimum disk space used by a very small file is 64K.   If a file is larger than 64K, it can fill the gaps left by the mostly unused space in these small block.        If you have a lot of small filles, then you have a lot of wasted space.  There is no way that I know of to know how much of this space has been allocated.

In summary then, if you look at the size of all the files by using DIR /S from the root of the volume, you will see the sum of all of the file sizes.   If you use windows explorer to list the properties of all of the files and folders on the volume,you will see 2 numbers.  One is file sizes and the other is "Size on Disk".  The irony is that this is a Microsoft calculation as if it were a Microsoft server, and is is not accurate.

If you use NDIR to look at the volume, you will see a confusing list of numbers, as follows:
Total volume space:                           6,179,968  100.00%
Space used by 34,499 entries:                   819,456   13.26%
Deleted space not yet purgeable:                     64    0.00%
                                          ------------- --------
Space remaining on volume:                    5,360,512   86.74%
Space available to ADMIN:                     5,360,512   86.74%

Maximum directory entries:                       82,432
Available directory entries:                     47,933   58.15%

Space used if files were not compressed:         84,992
Space used by compressed files:                  27,136
                                          -------------
Space saved by compressing files:                57,856   68.07%

Uncompressed space used:                        792,384

In this example from my lab server, I have a 6GB volume reporting 14% full in monitor.
It is saying that the size on disk is 820MB and the size of all of the files (if uncompressed) is 885MB.  

The thing of it is that Windows Explorer reports as follows:
Windwos Explorer:

      15,820 Files, 1,412 Folders

Size          828 MB (836,132,562 bytes)
Size on Disk 1.63 GB (1,758,6715,322 bytes)

and DOS /S report:
     Total Files Listed:
           15815 File(s)    860,385,972 bytes
            4230 Dir(s)   5,489,033,216 bytes free

The reason for the descrepencies is not exactly clear, but I believe it is in what each considers a file and how it understands name space etc.

In any case, the Novell NDIR /VOL command is the one that is giving you NetWare information.   The example above shows the following:
I have 820MB of disk space used, but the files that fill that space say that they are 885MB (820 + 65) .  If you were to copy these files to another volume, they would initially take up at least this much space.    If you copy them to a Microsoft share, they may take as much as 1.8GB, as reported by Windows Explorer (size on disk).

The best and most thorough way to purge all files on volume is to drop to a dos box and tehn at the root of the volume type PURGE /ALL.  You will see every file name go by as it is deleted.  If you only see directory names then there were no files to be purged.

The "Space not yet purgable" is files that have been deleted very recently.  NetWare holds these files for between 1 minutes and 5 minutes in case you want to undelete them.  Then they become purgable.

Hope this helps a little.


 

I disagree with your statement about 64K blocks and block suballocation.  I always did my traditional volumes with 4k blocks, and the block suballocation lets you suballocate a block to multiple files, all the way down to 512 bytes per suballocation unit.  It may default to 64K blocks - I don't recall - but you always had the option to choose a different block size.  64K blocks may help for speed in some instances but really adds overhead with the suballocation process, and if you don't suballocate is very wasteful of disk real-estate,, but if you're more concerned about space utilization than about performance, 4K blocks are good, and block suballocation, as I said, took the suballocation unit to 512 bytes.  In fact, for WinXP on a moderate-size disk (say 40GB) if you set the NTFS blocksize to 4K it actually improves performance as well as utilization, but not many know that can be done, or how to do it... ;)

Now, on the other hand, Windoze tends to be set with the blocksize way too high,  and doesn't suballocate, causing a small file to take up a whole block, which is why "file size" and "used on disk" is sometimes radically different as you explained.  Windoze Exploder doesn't "dig" block suballocation, so it often misreports "size on disk" because it tries to fit it into the Windoze wasteful methodology.

Is there any way I can decrease the "Compressed" space on this volume?

Volumes are either Compression-enabled or they're not.

If the Volume is Compression-enabled, then files on it will be Compressed, in accordance with the Compression scheduling settings on the server. Until a file meets the requirements, it will not be Compressed. Exceptions exist for Directories or Files that are flagged Immediate Compression or No Compression, but these flags must be explicitly set.

There is no "Compressed space" on a Volume. There is the space being occupied by Compressed files. If accessed/altered, these files will be unCompressed and stored in their unCompressed state until they again meet the criteria (or an exception applies). When that happens, the amount of space occupied by Compressed files will decrease (and the amount of space occupied by unCompressed files will *increase* by the same or greater amount).

Note that pspencer's comments on how long deleted files are held before becoming purgable varies based on settings made on the server hosting the Volume in which the files reside. These settings can also vary based on filesystem type (e.g. NSS or Traditional/FAT), and version of NetWare. Finally, the migration of deleted files from deleted to purgeable is affected by flags on individual Directories or files (e.g. Purge Immediate).

The point being is there is nothing carved in stone saying that the hold time is 1 minute or 5 minutes or whatever. It might very well be that in the Asker's environment, but we have no way of knowing that unless we know the various settings and flags that exist in that particular environment. We can only talk about general defaults.

Is there any way to force these files to become purgable?


Thanks

Which files?  The deleted files?  The compressed files?

For traditional/FAT, check your "monitor.nlm" server parameters, traditional file system screen to see what the two settings - "minimum file delete wait time" and "file delete wait time" to see what your delay before a deleted file becomes purgeable is set to.

You can, if you so desire, set them both to zero, but that will make a NetWare filesystem behave like Windoze - what's deleted is gone, too bad, so sad.  If you have it set fairly short, you should be able to wait that long before doing a "purge /all" yet still be able to say "oops, I didn't mean to delete that" and have the file salvageable for a couple of minutes, even if you're in that threshold window where deleted files get purged automatically because you're so low on free space.

Here is more info on the volume:

minimum file delete wait time: 1 min 5.1 sec
file delete wait time: 5 min 25.9 sec


J:\>ndir /vol

Statistics for fixed volume JONES_NOVELL5/ONE:
Space statistics are in KB (1024 bytes).

Total volume space:                          11,483,136  100.00%
Space used by 128,567 entries:               10,499,712   91.44%
Deleted space not yet purgeable:                     64    0.00%
                                          ------------- --------
Space remaining on volume:                      983,424    8.56%
Space available to KING:                        983,424    8.56%

Maximum directory entries:                      393,216
Available directory entries:                    264,649   67.30%

Space used if files were not compressed:     17,481,216
Space used by compressed files:               3,912,192
                                          -------------
Space saved by compressing files:            13,569,024   77.62%

Uncompressed space used:                      7,098,304

Name spaces loaded: OS/2

J:\>

I think I understand now. Looks like I have to archive more data or add more disk space.

Thanks