Solved

Delete all files in huge directory which are older than 6 months

Posted on 2006-10-26
9
1,024 Views
Last Modified: 2008-01-09
Hi there,

We have a database at my work which outputs around 10k-20k log files a day and it has been running around the clock for the best part of a year. Excellent stuff! A year of no downtime and almost constant load is punishing enough for any machine.

The downside of this is that there are something in the order of 4m log files now stored in a directory on that computer. Each file only contains a timestamp and a SQL statement, so the're mere bytes in size. Maybe 1kb at the most.

Now, I'm in the process of writing an app which removes any file that is older than six months in that directory.

I wrote something fairly elegant which gets a fileinfo array of the directory using getfiles(), and this obviously results in a massive array of information for 4m files when I only need their fullname and the creationtime. Because the database will be running constantly on a 1gb machine (and the db takes up about 600mb of mem), it's not practical to create such a large array, as the memory will run out.

So... Is it possible to do this in small measures so that the memory usage would never be over, say 100mb?

Here is my current code which, when run on a test directory, took about 30 minutes to process 130,000 files, using about 45MB of RAM, and using almost all the CPU for explorer.exe for 20 minutes. Doing a calculation to scale that to 4m made me a little concered that it wasn't efficient enough:

DirectoryInfo dir = new DirectoryInfo("C:\\log\\test");
Cursor.Current = Cursors.WaitCursor;
FileInfo[] arrFiles = dir.GetFiles("*.txt");

//dateTimePicker1.Value is a DateTime value which is given by the user from the datetimepicker control (it's set to 'DateTime.Now.AddMonths(-6)').
if (arrFiles[i].CreationTime < dateTimePicker1.Value)
  File.Delete(file);
0
Comment
Question by:DanBAtkinson
9 Comments
 
LVL 6

Accepted Solution

by:
nbkbar7 earned 500 total points
ID: 17810786

how about this...

foreach (string file in Directory.GetFiles(@"c:\log\test") {
    if (File.GetCreationTime(file) < dateTimePicker1.Value) {
        File.Delete(file);
    }
}

no need to allocate for the array, uses the list directly
That should reduce memory issue
not sure how fast it'll be...

I like to know the results...


Dave
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17810875
Came across an small app written by a bloke named Carl Daniel a while ago. His pattern doesn't require the use of a large amount of memory, like returning a string[] does. I'm not sure if it completely suits your needs, but have a go:

File System Enumerator using lazy matching
http://www.codeproject.com/cs/files/FileSystemEnumerator.asp
0
 

Author Comment

by:DanBAtkinson
ID: 17810933
Thanks guys.

I'm going to try out Dave's solution and see what happens. I'll report back with some stats.

Cheers.
0
 
LVL 15

Expert Comment

by:SRigney
ID: 17811196
Here's a VBScript solution that uses WMI.  It's not c#, but sometimes the solution is to expand your knowledge and try something that you didn't think of before.

http://www.microsoft.com/technet/scriptcenter/resources/qanda/nov04/hey1104.mspx
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 

Author Comment

by:DanBAtkinson
ID: 17811227
Ok... The results are in and Dave's was most efficient. Cycling through the array was the stinker. It just took ages to go through it. Removing the array was a better way of doing it!

Cheers,

Dan
0
 

Author Comment

by:DanBAtkinson
ID: 17811245
Um... Ok. I accepted the wrong comment here.

Damnit.
0
 
LVL 8

Expert Comment

by:redpipe
ID: 17811275
Place a comment to the Community Support team:
http://www.experts-exchange.com/Community_Support/
0
 

Author Comment

by:DanBAtkinson
ID: 17811357
Aye. Cheers.

Thanks again folks.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Windows phone Development 11 29
Expando 4 35
String manipulation 15 49
Why don't I see this table in EDMX file? 2 20
Article by: Najam
Having new technologies does not mean they will completely replace old components.  Recently I had to create WCF that will be called by VB6 component.  Here I will describe what steps one should follow while doing so, please feel free to post any qu…
Introduction Although it is an old technology, serial ports are still being used by many hardware manufacturers. If you develop applications in C#, Microsoft .NET framework has SerialPort class to communicate with the serial ports.  I needed to…
It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

25 Experts available now in Live!

Get 1:1 Help Now