troubleshooting Question

Copy many files without maxing out Kernel Paged Memory Pool?

Avatar of Intersection
Intersection asked on
.NET ProgrammingVisual Basic.NET
21 Comments2 Solutions1230 ViewsLast Modified:
Hello!

Question:
As in http://support.microsoft.com/kb/312362/ I am copying many files and running up my Paged Pool memory.
"there is a large probability that there are a very large number of files that are open on the server"
I start getting "Insufficient system resources exist to complete the requested service." error message, and then the computer crashes - sometimes destroying harddrives. :(
Even if I close the program before a crash, the Kernel Paged Memory values stays high.
(Note I monitor this in Task Manager > Performance > Kernel Memory > Paged value. And in VB.NET with System.Management.ManagementObjectSearcher("Select * From Win32_PerfRawData_PerfOS_Memory") and request "PoolPagedBytes".
)
If I start the program again without a reboot, I get the errors imediately. I guess because the Paged mem is still maxed out.

How in code can I close the "open files" that the memory system has open? That is, how can I reduce the level of the Paged Kernel Memory.
Is this possible?
Or, how can I copy many files without running up the Paged Kernel Mem in the first place?

Ive tried both of these calls to perform the copy  but the paged memory is still going up to 148916(KB).
System.IO.File.Copy(strFileSource, strFileDest, bOverwriteFile)
My.Computer.FileSystem.CopyFile(strFileSource, strFileDest, bOverwriteFile)

I have many files in some of the directories ie 19,400 - i dont know if this could have an impact at all.

Thank you for any guideance, advice, assistance!
-Topher


Background:
I need to copy many (100,000s to millions) of .jpg files from one drive to another. Off the shelf backup programs have failed (I assume because they try to create a massive list before starting), and other special requirements lead us to write custom software.

The software works as long as the number of files is not too great, but on operations lasting several hours sometimes the computer crashes - sometimes hosing the source and/or destination disk. Ive started debugging and have been getting "Insufficient system resources exist to complete the requested service." error message.

This seems to match up with http://support.microsoft.com/kb/312362/ or (http://support.microsoft.com/kb/304101)
This article recommends setting registry PoolUsageMaximum=60 and PagedPoolSize=-1

But I would like to fix the problem in code. (So that it is easy for customers to install - and im not modifying users computers.)
ASKER CERTIFIED SOLUTION
Intersection

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 2 Answers and 21 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 2 Answers and 21 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros