Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Delete all files in huge directory which are older than 6 months

Posted on 2006-10-26
9
Medium Priority
?
1,072 Views
Last Modified: 2008-01-09
Hi there,

We have a database at my work which outputs around 10k-20k log files a day and it has been running around the clock for the best part of a year. Excellent stuff! A year of no downtime and almost constant load is punishing enough for any machine.

The downside of this is that there are something in the order of 4m log files now stored in a directory on that computer. Each file only contains a timestamp and a SQL statement, so the're mere bytes in size. Maybe 1kb at the most.

Now, I'm in the process of writing an app which removes any file that is older than six months in that directory.

I wrote something fairly elegant which gets a fileinfo array of the directory using getfiles(), and this obviously results in a massive array of information for 4m files when I only need their fullname and the creationtime. Because the database will be running constantly on a 1gb machine (and the db takes up about 600mb of mem), it's not practical to create such a large array, as the memory will run out.

So... Is it possible to do this in small measures so that the memory usage would never be over, say 100mb?

Here is my current code which, when run on a test directory, took about 30 minutes to process 130,000 files, using about 45MB of RAM, and using almost all the CPU for explorer.exe for 20 minutes. Doing a calculation to scale that to 4m made me a little concered that it wasn't efficient enough:

DirectoryInfo dir = new DirectoryInfo("C:\\log\\test");
Cursor.Current = Cursors.WaitCursor;
FileInfo[] arrFiles = dir.GetFiles("*.txt");

//dateTimePicker1.Value is a DateTime value which is given by the user from the datetimepicker control (it's set to 'DateTime.Now.AddMonths(-6)').
if (arrFiles[i].CreationTime < dateTimePicker1.Value)
  File.Delete(file);
0
Comment
Question by:DanBAtkinson
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
9 Comments
 
LVL 6

Accepted Solution

by:
nbkbar7 earned 2000 total points
ID: 17810786

how about this...

foreach (string file in Directory.GetFiles(@"c:\log\test") {
    if (File.GetCreationTime(file) < dateTimePicker1.Value) {
        File.Delete(file);
    }
}

no need to allocate for the array, uses the list directly
That should reduce memory issue
not sure how fast it'll be...

I like to know the results...


Dave
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17810875
Came across an small app written by a bloke named Carl Daniel a while ago. His pattern doesn't require the use of a large amount of memory, like returning a string[] does. I'm not sure if it completely suits your needs, but have a go:

File System Enumerator using lazy matching
http://www.codeproject.com/cs/files/FileSystemEnumerator.asp
0
 

Author Comment

by:DanBAtkinson
ID: 17810933
Thanks guys.

I'm going to try out Dave's solution and see what happens. I'll report back with some stats.

Cheers.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 15

Expert Comment

by:SRigney
ID: 17811196
Here's a VBScript solution that uses WMI.  It's not c#, but sometimes the solution is to expand your knowledge and try something that you didn't think of before.

http://www.microsoft.com/technet/scriptcenter/resources/qanda/nov04/hey1104.mspx
0
 

Author Comment

by:DanBAtkinson
ID: 17811227
Ok... The results are in and Dave's was most efficient. Cycling through the array was the stinker. It just took ages to go through it. Removing the array was a better way of doing it!

Cheers,

Dan
0
 

Author Comment

by:DanBAtkinson
ID: 17811245
Um... Ok. I accepted the wrong comment here.

Damnit.
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17811275
Place a comment to the Community Support team:
http://www.experts-exchange.com/Community_Support/
0
 

Author Comment

by:DanBAtkinson
ID: 17811357
Aye. Cheers.

Thanks again folks.
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
Introduction Hi all and welcome to my first article on Experts Exchange. A while ago, someone asked me if i could do some tutorials on object oriented programming. I decided to do them on C#. Now you may ask me, why's that? Well, one of the re…
Have you created a query with information for a calendar? ... and then, abra-cadabra, the calendar is done?! I am going to show you how to make that happen. Visualize your data!  ... really see it To use the code to create a calendar from a q…
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …

610 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question