What cluster size to choose for best NTFS performance with large files

Yep, you're using the right approach for high performance on large files (for large files, the cluster size should be as large as possible) Just a fyi compression doesn't work very well anymore when you use large cluster files on NTFS so don't use the NTFS file compression. chicagoan explained the rest about slack etc.

Converting cluster sizes

Scot’s Newsletter
January 31, 2002 - Vol. 2, Issue No. 20

By Scot Finnie

Converting cluster sizes. SFNL reader Rob Wilen sent me a simple but very good question. He's got PartitionMagic 7, Windows XP, and an NTFS partition that he notices has 512-byte clusters. So can PartitionMagic convert the cluster size to 4K for him? The answer is no. PartitionMagic can convert cluster sizes of FAT or FAT32 partitions, but it's not currently able to convert NTFS clusters, according to PowerQuest's Travis Eggett. Eggett says that most NTFS conversions result in 512-byte cluster sizes, but some result in 4K cluster sizes. In Eggett's experience, the performance differences between the two are negligible for desktop PC use.
I also checked this point with V-Com's Jeff Hyman. In addition to making the venerable System Commander multiboot utility, V-Com makes Partition Commander, a product that's been moving up in the polls (figuratively speaking). Jeff confirms that Partition Commander can't covert NTFS cluster sizes in it current version either.
But a reader named Tom Snyder located a program that does do the trick. It's Paragon Software's Paragon Partition Manager. According to Tom, he used the $40 program to convert his cluster size from 512-bytes to 4K and he's noticed a marked performance improvement. I haven't tried this product, but the marketing materials do specifically list NTFS cluster-size conversion and Windows XP support. It is also able to convert from FAT/FAT32 to NTFS and back. I can't vouch for the product's reliability the way I can with PartitionMagic. But I've sent an email to the company requesting a copy of the software. If I get that, I will try it and write about it in a future issue. If you try Paragon Partition Manager, please let me know about your experiences. I'm particularly interested in cluster-size conversion, but any thoughts you have would be interesting. Please send me info about what you learn.
NTFS performance. Microsoft's David Golds sent word that the NTFS article in the last issue was accurate except on one point. I mentioned that on larger (30GB or more) volume sizes, NTFS can cause an erosion of performance. Golds writes: "4K clusters are actually close to optimal for performance for most workloads. It's just that 512-byte clusters are a bit small. So NTFS retaining the 4K cluster size actually positions us really well for performance and for storage efficiency."
I think David put his finger on the issue. Many of the people who are noticing performance issues have 512-byte cluster sizes because they converted during or after an upgrade installation of Windows XP. It's probably much fairer to say that performance degradation occurs with 512-byte clusters.


Scot's Newsletter - 2-14-2002  
Date: 2/14/2002 8:13:32 PM Pacific Standard Time
From: sfnl@scotfinnie.com

HOW TO MAKE NTFS GO FASTER
-------------------------------------------------------------------

In the last two issues of this newsletter I've run one of my
inadvertent series -- this one on Windows XP's NTFS file system. I
had no idea I was about to touch off an avalanche of questions and
reveal a large problem for Windows XP users. Or even better, that I
was about to publish an NTFS performance solution.

I explained about cluster sizes in NTFS in the first in the series:

http://www.scotfinnie.com/newsletter/19.htm#filesys


In short, Windows XP's NTFS file system is even more storage
efficient than Win98's FAT32. And it can be at least as fast as
FAT32. It's also more reliable. But there is one very big problem.
When you install Windows XP as an upgrade of a previous version of
Windows running FAT16 or FAT32 and convert to NTFS as part of setup
or after the fact, in most cases you end up with tiny 512-byte
cluster sizes. This occurs because of the way the data is aligned on
the disk and the NTFS conversion process as carried out by
Microsoft's Convert utility. PowerQuest's PartitionMagic 7.0 uses
the Microsoft utility, so it has the same issues.

The surprising truth is that some new PCs also arrive with 512-byte
cluster sizes. So if you've got a new Windows XP box that runs slow,
you should definitely check out what I'm about to explain.

In the last issue, I alluded to a possible solution:

http://www.scotfinnie.com/newsletter/20.htm#filesys


A little-known program called Paragon Partition Manager, created by
a group of Russian programmers working for Paragon Software, has in
its latest version, 5.0, added the ability to dynamically adjust
cluster sizes.

Paragon Partition Manager 5.0:
http://www.partition-manager.com/n_pm_main.htm

Paragon Partition Manager 5.0 Features Details:
http://www.partition-manager.com/n_pm_requir.htm

Paragon Partition Manager isn't generally marketed in the U.S.,
although you can purchase it on the Internet, where it sells for
about $40:

Where To Buy Paragon Partition Manager:
http://www.partition-manager.com/n_pm_buy.htm

None of the popular disk utilities marketed in the U.S. is capable
of pulling off this feat yet. But Partition Manager does it, and
does it well.

To prove the point, I acquired a copy of Paragon Partition Manager
5.0 from the company and configured a test system. I had an existing
drive containing a clean Windows 98 Second Edition installation on
my trusty Compaq Armada 700 (the best Compaq product I've ever
worked with) notebook PC. I ran a standard Windows XP upgrade
installation, which took a while, but completed just fine. I also
converted to NTFS. When all the files were copied and the changes
made, it was immediately apparent to me that my performance eroded
markedly. It took Windows forever to load, and disk-intensive tasks
ran like molasses in January. In fact, I was surprised by how slow
the machine became. I had been led to believe that 512-byte clusters
slowed the machine down incrementally, but the reality was much
worse.

Next I used Windows' Disk Defragmenter to check the cluster size on
my hard disk. To do that, you open Disk Defragmenter from Start > 
All Programs > Accessories > System Tools > Disk Defragmenter.
Right-click the appropriate drive and choose Analyze. When the
analysis is complete, click the View Report button. There you see a
line that reads Cluster Size = XX KB. In my case, it showed 512-
bytes, the smallest, slowest cluster size NTFS allows. The optimum
size is 4K clusters.

I was ready to try Paragon Partition Manager. I'll tell you upfront
that there are two problems with this product. The first is the user
interface, which needs help. But it's usable. The second is that
before you make the cluster size change, block out several hours of
time for your PC. Overnight might be a good idea. You may save
yourself some time by running a defrag before you run the cluster-
size conversion, but you'll find that Disk Defragmenter also runs
very slowly when your cluster sizes are 512-bytes.

The time factor thing is variable. While it took me four hours to
convert the cluster size on a 12GB notebook drive, it took one SFNL
reader only a few minutes and another one over six hours. I asked
the Paragon people about that and they wouldn't commit to even a
range of time you can expect this process to take. Reading between
the lines, this large difference from PC to PC in the time it takes
to run the conversion is normal.

The user interface issue comes into play when you do the cluster
conversion because nothing says "Convert to 4K Cluster Size." But I
can save you that pain. Once you have Paragon Partition Manager
running, select the NTFS drive whose clusters you want to convert.
From the program menu, choose Partition > Modify > Change cluster
size. Dial the "Sectors/Cluster" spinner up to the number 8. Press
OK. (If you select 4 in this scroll box, you'll get 2K clusters --
not the desired outcome.) The conversion process requires that
Windows XP reboot.

Maybe you'll be lucky and have the fast-track conversion. If not, I
can promise you this, it'll be worth the wait. As soon as the
conversion completed for me my performance was back to FAT32 levels.
All that was left to do was run Disk Defrag again, both to check the
cluster size and also to defrag the disk. Do both things.

A couple final notes on NTFS this week. First, I've received a ton
of email about NTFS that I haven't had time to get to. Many offer
interesting info or questions deserving response. I will continue to
cover NTFS in future issues.

The other point is that Microsoft is continuing to investigate
issues people have had with slow NTFS performance on new Windows XP
PCs. The company is working with some of SFNL's readers on that
point. I hope to get some sort of report back from Microsoft -- and
if so, I'll publish it in a future issue. My take though? The steps
in this issue will probably fix your problem, assuming you're
willing to shell out for the Paragon product.


[SFNL] Scot's Newsletter -- 6/24/2002  
Date: 6/24/2002 9:24:47 PM Pacific Daylight Time
From: scot@scotsnewsletter.com

Fixing Sluggish XP Performance --
QUESTION: I am running XP Professional on a Pentium 4 Dell with
Preinstalled XP Pro. My applications include Office XP, IE 6, and MSN
Explorer. My hard drive's file system is NTFS. Performance is sluggish.
What services should I turn off to make the machine more responsive? --
Peter Luchansky

ANSWER: Because you have NTFS, I would first suspect that your
performance issues are related to small NTFS cluster sizes on your hard
drive. When a hard drive that was previously formatted with DOS tools
(Fdisk and Format, creating FAT16 or FAT32 partitions) is converted to
NTFS, there is frequently a problem that causes a tiny 512-byte cluster
size under NTFS. That's the smallest cluster size NTFS allows. I won't
go into a long explanation, but it's also the slowest cluster size. The
solution is to convert the cluster sizes to at least 4K. Many, many
people have reported that doing so solved their performance problems
completely. Microsoft is aware of the problem, but really hasn't done
much about it. It's also not warning people not to convert to NTFS on
Windows 9x upgrades, which routinely result in 512-byte cluster sizes.
This isn't just a Windows XP problem, either. Windows 2000 users face
the same issue.

It's also not just a problem for people who've upgraded to Windows XP
and then converted to NTFS either. Even some new PCs -- including name
brand models (especially in the early going right after Windows XP
shipped) -- had this problem. The DOS-based software tools some PC
makers use to image new hard drives are the culprit. If your new PC was
purchased recently from a local system integrator, for example, you
could have this problem. Most major PC makers have resolved this
problem (on PCs sold in 2002), though.

So far I have been unable to come up with a reliable solution that
doesn't require the use of a third-party disk-partitioning utility. So
the one I recommend is Paragon Partition Manager. This product was
developed by Russian programmers for the European market. You can buy
it on the Internet for about $40. What makes it different from most
other disk-partitioning utilities is that it can dynamically convert
cluster sizes under NTFS.

The following article in Scot's Newsletter describes where to get
Paragon and tells you how to use it to solve the 512-byte problem:

http://www.scotsnewsletter.com/21.htm#filesys

Before you rush off and buy Paragon, though, use the instructions on
this page to find out whether you have the 512-byte cluster size:

http://www.scotsnewsletter.com/20.htm#filesys


If your cluster size isn't less than 4K, then this isn't your problem.

If you already own PowerQuest's PartitionManager, you should read these
instructions.

http://www.scotsnewsletter.com/23.htm#filesys
 

While they don't work for everyone, they could save you the cost of the
Paragon product.

Finally, there are other disk partitioning utilities -- some of them
available for free -- that may also do the job. I have not tested those
products. But I wanted to let you know they existed if you want to do
your own research. I have personally solved the cluster-size problem
with both the Paragon and PowerQuest products. --S.F.

Refer following data
 
  Criteria      NTFS5      NTFS      FAT32      FAT16
Operting System      Windows 2000Windows XP      Windows NTWindows 2000Windows XP      Windows 98Windows MEWindows 2000Windows XP      DOSAll versions of Microsoft Windows
 
Limitations
Max Volume Size      2TB      2TB      2TB      2GB
Max Files on Volume      Nearly Unlimited      Nearly Unlimited      Nearly Unlimited      ~65000
Max File Size      Limit Only by Volume Size      Limit Only by Volume Size      4GB      2GB
Max Clusters Number      Nearly Unlimited      Nearly Unlimited      268435456      65535
Max File Name Length      Up to 255      Up to 255      Up to 255      Standard - 8.3Extended - up to 255
 
File System Features
Unicode File Names      Unicode Character Set      Unicode Character Set      System Character Set      System Character Set
System Records Mirror      MFT Mirror File      MFT Mirror File      Second Copy of  FAT      Second Copy of  FAT
Boot Sector Location      First and Last Sectors      First and Last Sectors      First Sector      First Sector
File Attributes      Standard and Custom      Standard and Custom      Standard Set      Standard Set
Alternate Streams      Yes      Yes      No      No
Compression      Yes      Yes      No      No
Encryption      Yes      No      No      No
Object Permissions      Yes      Yes      No      No
Disk Quotas      Yes      No      No      No
Sparse Files      Yes      No      No      No
Reparse Points      Yes      No      No      No
Volume Mount Points      Yes      No      No      No
 
Overall Performance
Built-In Security      Yes      Yes      No      No
Recoverability      Yes      Yes      No      No
Performance      Low on small volumesHigh on Large      Low on small volumes High on Large      High on small volumesLow on large      Highest on small volumesLow on large
Disk Space Economy      Max      Max      Average      Minimal on large volumes
Fault Tolerance      Max      Max      Minimal      Average
                             
                             

 
   
  Gam
 
 
 
   

gayatauk,

Next time provide a source to your information, I found it here:
http://www.ntfs.com/ntfs_vs_fat.htm

You might want to take a look at the guidelines about this:
http:help.jsp#hi125

>>We also interpret this clause to mean the theft of intellectual property by copying it to a thread and pasting it without acknowledgment of its original source. Plagiarism is not only actionable; it shows dishonesty, a distict lack of professionalism, and no class.<<

LucF

I have read few pages about cluster sizes (for example http://www.pcguide.com/ref/hdd/perf/perf/ext/fileCluster-c.html)

It seems that the only advantage of 2k cluster size (generally smaller size) is more effective use of disk space.

But in these days, price per GB is so low that if you compare gained performance and lost disk space, you really cannot say 2k cluster is better than 4k.

Sorry this comment had to be elsewhere...

Wondering what your file system's cluster size is? I haven't been able to find that information using the tools Microsoft provided (but if you know a way, please send it on and I'll print it next time).

Go to cmd prompt and run chkdsk (read-only if you want), wait until it's done and take the number of bytes in each allocation unit and divide by 1,024.