Solved

Delete all files in huge directory which are older than 6 months

Posted on 2006-10-26
9
1,032 Views
Last Modified: 2008-01-09
Hi there,

We have a database at my work which outputs around 10k-20k log files a day and it has been running around the clock for the best part of a year. Excellent stuff! A year of no downtime and almost constant load is punishing enough for any machine.

The downside of this is that there are something in the order of 4m log files now stored in a directory on that computer. Each file only contains a timestamp and a SQL statement, so the're mere bytes in size. Maybe 1kb at the most.

Now, I'm in the process of writing an app which removes any file that is older than six months in that directory.

I wrote something fairly elegant which gets a fileinfo array of the directory using getfiles(), and this obviously results in a massive array of information for 4m files when I only need their fullname and the creationtime. Because the database will be running constantly on a 1gb machine (and the db takes up about 600mb of mem), it's not practical to create such a large array, as the memory will run out.

So... Is it possible to do this in small measures so that the memory usage would never be over, say 100mb?

Here is my current code which, when run on a test directory, took about 30 minutes to process 130,000 files, using about 45MB of RAM, and using almost all the CPU for explorer.exe for 20 minutes. Doing a calculation to scale that to 4m made me a little concered that it wasn't efficient enough:

DirectoryInfo dir = new DirectoryInfo("C:\\log\\test");
Cursor.Current = Cursors.WaitCursor;
FileInfo[] arrFiles = dir.GetFiles("*.txt");

//dateTimePicker1.Value is a DateTime value which is given by the user from the datetimepicker control (it's set to 'DateTime.Now.AddMonths(-6)').
if (arrFiles[i].CreationTime < dateTimePicker1.Value)
  File.Delete(file);
0
Comment
Question by:DanBAtkinson
9 Comments
 
LVL 6

Accepted Solution

by:
nbkbar7 earned 500 total points
ID: 17810786

how about this...

foreach (string file in Directory.GetFiles(@"c:\log\test") {
    if (File.GetCreationTime(file) < dateTimePicker1.Value) {
        File.Delete(file);
    }
}

no need to allocate for the array, uses the list directly
That should reduce memory issue
not sure how fast it'll be...

I like to know the results...


Dave
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17810875
Came across an small app written by a bloke named Carl Daniel a while ago. His pattern doesn't require the use of a large amount of memory, like returning a string[] does. I'm not sure if it completely suits your needs, but have a go:

File System Enumerator using lazy matching
http://www.codeproject.com/cs/files/FileSystemEnumerator.asp
0
 

Author Comment

by:DanBAtkinson
ID: 17810933
Thanks guys.

I'm going to try out Dave's solution and see what happens. I'll report back with some stats.

Cheers.
0
 
LVL 15

Expert Comment

by:SRigney
ID: 17811196
Here's a VBScript solution that uses WMI.  It's not c#, but sometimes the solution is to expand your knowledge and try something that you didn't think of before.

http://www.microsoft.com/technet/scriptcenter/resources/qanda/nov04/hey1104.mspx
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:DanBAtkinson
ID: 17811227
Ok... The results are in and Dave's was most efficient. Cycling through the array was the stinker. It just took ages to go through it. Removing the array was a better way of doing it!

Cheers,

Dan
0
 

Author Comment

by:DanBAtkinson
ID: 17811245
Um... Ok. I accepted the wrong comment here.

Damnit.
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17811275
Place a comment to the Community Support team:
http://www.experts-exchange.com/Community_Support/
0
 

Author Comment

by:DanBAtkinson
ID: 17811357
Aye. Cheers.

Thanks again folks.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Article by: Ivo
C# And Nullable Types Since 2.0 C# has Nullable(T) Generic Structure. The idea behind is to allow value type objects to have null values just like reference types have. This concerns scenarios where not all data sources have values (like a databa…
Introduction This article series is supposed to shed some light on the use of IDisposable and objects that inherit from it. In essence, a more apt title for this article would be: using (IDisposable) {}. I’m just not sure how many people would ge…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, just open a new email message. In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now