• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1078
  • Last Modified:

Delete all files in huge directory which are older than 6 months

Hi there,

We have a database at my work which outputs around 10k-20k log files a day and it has been running around the clock for the best part of a year. Excellent stuff! A year of no downtime and almost constant load is punishing enough for any machine.

The downside of this is that there are something in the order of 4m log files now stored in a directory on that computer. Each file only contains a timestamp and a SQL statement, so the're mere bytes in size. Maybe 1kb at the most.

Now, I'm in the process of writing an app which removes any file that is older than six months in that directory.

I wrote something fairly elegant which gets a fileinfo array of the directory using getfiles(), and this obviously results in a massive array of information for 4m files when I only need their fullname and the creationtime. Because the database will be running constantly on a 1gb machine (and the db takes up about 600mb of mem), it's not practical to create such a large array, as the memory will run out.

So... Is it possible to do this in small measures so that the memory usage would never be over, say 100mb?

Here is my current code which, when run on a test directory, took about 30 minutes to process 130,000 files, using about 45MB of RAM, and using almost all the CPU for explorer.exe for 20 minutes. Doing a calculation to scale that to 4m made me a little concered that it wasn't efficient enough:

DirectoryInfo dir = new DirectoryInfo("C:\\log\\test");
Cursor.Current = Cursors.WaitCursor;
FileInfo[] arrFiles = dir.GetFiles("*.txt");

//dateTimePicker1.Value is a DateTime value which is given by the user from the datetimepicker control (it's set to 'DateTime.Now.AddMonths(-6)').
if (arrFiles[i].CreationTime < dateTimePicker1.Value)
  File.Delete(file);
0
DanBAtkinson
Asked:
DanBAtkinson
1 Solution
 
nbkbar7Commented:

how about this...

foreach (string file in Directory.GetFiles(@"c:\log\test") {
    if (File.GetCreationTime(file) < dateTimePicker1.Value) {
        File.Delete(file);
    }
}

no need to allocate for the array, uses the list directly
That should reduce memory issue
not sure how fast it'll be...

I like to know the results...


Dave
0
 
redpipeCommented:
Came across an small app written by a bloke named Carl Daniel a while ago. His pattern doesn't require the use of a large amount of memory, like returning a string[] does. I'm not sure if it completely suits your needs, but have a go:

File System Enumerator using lazy matching
http://www.codeproject.com/cs/files/FileSystemEnumerator.asp
0
 
DanBAtkinsonAuthor Commented:
Thanks guys.

I'm going to try out Dave's solution and see what happens. I'll report back with some stats.

Cheers.
0
Cloud Class® Course: Microsoft Office 2010

This course will introduce you to the interfaces and features of Microsoft Office 2010 Word, Excel, PowerPoint, Outlook, and Access. You will learn about the features that are shared between all products in the Office suite, as well as the new features that are product specific.

 
SRigneyCommented:
Here's a VBScript solution that uses WMI.  It's not c#, but sometimes the solution is to expand your knowledge and try something that you didn't think of before.

http://www.microsoft.com/technet/scriptcenter/resources/qanda/nov04/hey1104.mspx
0
 
DanBAtkinsonAuthor Commented:
Ok... The results are in and Dave's was most efficient. Cycling through the array was the stinker. It just took ages to go through it. Removing the array was a better way of doing it!

Cheers,

Dan
0
 
DanBAtkinsonAuthor Commented:
Um... Ok. I accepted the wrong comment here.

Damnit.
0
 
redpipeCommented:
Place a comment to the Community Support team:
http://www.experts-exchange.com/Community_Support/
0
 
DanBAtkinsonAuthor Commented:
Aye. Cheers.

Thanks again folks.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now