Solved

Delete all files in huge directory which are older than 6 months

Posted on 2006-10-26
9
1,040 Views
Last Modified: 2008-01-09
Hi there,

We have a database at my work which outputs around 10k-20k log files a day and it has been running around the clock for the best part of a year. Excellent stuff! A year of no downtime and almost constant load is punishing enough for any machine.

The downside of this is that there are something in the order of 4m log files now stored in a directory on that computer. Each file only contains a timestamp and a SQL statement, so the're mere bytes in size. Maybe 1kb at the most.

Now, I'm in the process of writing an app which removes any file that is older than six months in that directory.

I wrote something fairly elegant which gets a fileinfo array of the directory using getfiles(), and this obviously results in a massive array of information for 4m files when I only need their fullname and the creationtime. Because the database will be running constantly on a 1gb machine (and the db takes up about 600mb of mem), it's not practical to create such a large array, as the memory will run out.

So... Is it possible to do this in small measures so that the memory usage would never be over, say 100mb?

Here is my current code which, when run on a test directory, took about 30 minutes to process 130,000 files, using about 45MB of RAM, and using almost all the CPU for explorer.exe for 20 minutes. Doing a calculation to scale that to 4m made me a little concered that it wasn't efficient enough:

DirectoryInfo dir = new DirectoryInfo("C:\\log\\test");
Cursor.Current = Cursors.WaitCursor;
FileInfo[] arrFiles = dir.GetFiles("*.txt");

//dateTimePicker1.Value is a DateTime value which is given by the user from the datetimepicker control (it's set to 'DateTime.Now.AddMonths(-6)').
if (arrFiles[i].CreationTime < dateTimePicker1.Value)
  File.Delete(file);
0
Comment
Question by:DanBAtkinson
9 Comments
 
LVL 6

Accepted Solution

by:
nbkbar7 earned 500 total points
ID: 17810786

how about this...

foreach (string file in Directory.GetFiles(@"c:\log\test") {
    if (File.GetCreationTime(file) < dateTimePicker1.Value) {
        File.Delete(file);
    }
}

no need to allocate for the array, uses the list directly
That should reduce memory issue
not sure how fast it'll be...

I like to know the results...


Dave
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17810875
Came across an small app written by a bloke named Carl Daniel a while ago. His pattern doesn't require the use of a large amount of memory, like returning a string[] does. I'm not sure if it completely suits your needs, but have a go:

File System Enumerator using lazy matching
http://www.codeproject.com/cs/files/FileSystemEnumerator.asp
0
 

Author Comment

by:DanBAtkinson
ID: 17810933
Thanks guys.

I'm going to try out Dave's solution and see what happens. I'll report back with some stats.

Cheers.
0
Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

 
LVL 15

Expert Comment

by:SRigney
ID: 17811196
Here's a VBScript solution that uses WMI.  It's not c#, but sometimes the solution is to expand your knowledge and try something that you didn't think of before.

http://www.microsoft.com/technet/scriptcenter/resources/qanda/nov04/hey1104.mspx
0
 

Author Comment

by:DanBAtkinson
ID: 17811227
Ok... The results are in and Dave's was most efficient. Cycling through the array was the stinker. It just took ages to go through it. Removing the array was a better way of doing it!

Cheers,

Dan
0
 

Author Comment

by:DanBAtkinson
ID: 17811245
Um... Ok. I accepted the wrong comment here.

Damnit.
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17811275
Place a comment to the Community Support team:
http://www.experts-exchange.com/Community_Support/
0
 

Author Comment

by:DanBAtkinson
ID: 17811357
Aye. Cheers.

Thanks again folks.
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In order to hide the "ugly" records selectors (triangles) in the rowheaders, here are some suggestions. Microsoft doesn't have a direct method/property to do it. You can only hide the rowheader column. First solution, the easy way The first sol…
This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…
This video shows how to use Hyena, from SystemTools Software, to bulk import 100 user accounts from an external text file. View in 1080p for best video quality.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question