We are running Windows 2000 SP2 on several computers, and writing an application which seeks into and reads parts of several large files. File size totals about 60GB. A given "query" into these files typically reads a total of 500MB or so in large chunks.
Here's the problem: After performing 20 or so of these operations, the "Available Memory" in the system (as indicated by the task manager) drops by 500MB or so, and continues to drop steadily as more operations are performed. On a system with 512MB of memory, operations grind to a halt.
I expect file-caching to take up memory, but it is normally released when processes need memory for other purposes. However, this memory is not released until the process terminates (I haven't tested to see if it is released when the file is closed).
This is not a memory leak in the application! We know this because:
1) Changing the file-read mode from buffered to unbuffered (in the CreateFile call) makes the problem go away. But then we lose benefits of OS caching altogether.
2) The VM size reported for our process in the Task Manager hovers around 50MB consistently.
NT/2000 has a history of file-caching problems. But here's another wrinkle: this issue only seems to show up on dual-CPU machines and not single-CPU machines.
We have three systems like this exhibiting problems, which are very different. First is a Dell 650 workstation with a SCSI stripe-set. Second is a SuperMicro workstation with a 3Ware RAID controller. Third is an Intel Pro mobo system woth a Mylex SAN controller. Because of the independence of disk system, I conclude that this is not a disk-driver issue.
I'm looking for an OS patch or programming workaround to avoid the issue.