[Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Number of files affecting disk speed / system performance

Posted on 2009-12-21
5
Medium Priority
?
332 Views
Last Modified: 2012-05-08
Basicly the problem is this:
I have massive multi terabyte disks and I stored VMs, BIG backup files and replicated backup files. The problem is these disks get slow, REALLY slow after about 6 months.

Ive narrowed it down to any partition where I use the replicated backup files gets crazy slow. These replications are a 3rd party program that copies data to any destination I like using a single instance store. This can create a massive amount of data objects for a file system (I think?) and I think these massive amounts of objects are what causes the file system response to slow.

Here is an example:
1.      I replicate backup a directory containing 1,000 files.
2.      The destination creates 1,000 files the same is the original.
3.      The next day 100 files change, nothing added or removed (keeping it simple) the replication program creates a new directory on the destination NTFS file system and uses a Single Instance Store to create another 1,000 files, of the 1,000 files, 100 are new because they have changed.
4.      This continues about 14 times as the system creates: daily, weekly, monthly, quarterly and yearly backups. So the number of files I could browse would be 14 x 1,000 = 14,000 browseable files for 1,000 backed up files.

Everything I read says that the number of files on a partition doesnt matter. Wikipedia says the NTFS limit is 4 Billion files.

Since I use these backup jobs to backup multiple servers to a single destination the number of files can be astronomical.

The results seem undeniable, the file systems become turtle slow.
0
Comment
Question by:byteharmony
  • 2
  • 2
5 Comments
 
LVL 93

Assisted Solution

by:nobus
nobus earned 400 total points
ID: 26103215
it is the same if you put more and more files into 1 directory.
i had applications creating each time a txt file smaller than 1 kb
once you get over the 10.000 the file, you notice that problem, since each file is listed in the file structure, it must be read, in order to be accessible.
there are only a couple of solutions imo :
-1 keep the number of files as low as you can (eg delete older backups)
-2 use a faster disk i/o system.. to speed it up

0
 
LVL 69

Accepted Solution

by:
Callandor earned 1600 total points
ID: 26106867
Read item 7 about fragmentation of the MFT and how it can be avoided: http://windowsdevcenter.com/pub/a/windows/2005/02/08/NTFS_Hacks.html
0
 

Author Comment

by:byteharmony
ID: 26112752
Wow, you guys are great!

I want to leave this open for a little while longer to see if we get any other information. I will certainly test the Callander as the purpose here is to create the large number of files.

I also have a friend who has worked at MS for years and he is going to check if he can make windows faster for this need as well.

Thanks,
BK
0
 

Author Closing Comment

by:byteharmony
ID: 31668737
Thank you!
0
 
LVL 93

Expert Comment

by:nobus
ID: 26322300
you're welcome !
0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article shows how to use a free utility called 'Parkdale' to easily test the performance and benchmark any Hard Drive(s) installed in your computer. We also look at RAM Disks and their speed comparisons.
New style of hardware planning for Microsoft Exchange server.
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
Suggested Courses

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question