I need assistance identifying specifically what process/program is consuming resources until, ultimately, the multiple failures occur due to "insufficient resources".
I have generated tracking logs of memory usage using Performance Monitor and here is what I have learned so far. On boot the system is fine and paged kernel memory usage will run in the 60 MB range. During the evening the pooled paged memory usage will ramp up to about 120 MB and plateau there. Each evening in which I do a backup the memory usage ramps up to a new plateau. It never goes back down. Ultimately if I do not reboot the memory usage will hit the mid 300s and things go very bad quickly!
The ramping up appears to coincide with the period when the backup program is backing up local files on the server. It does not appear to occur when the backup program is backing up files from client servers over the network. Some testing I have done seems to indicate this problem is related to the antivirus program but this is not conclusive and I need to test further. The antivirus is supposedly configured to trust the backup program and I believe this is to prevent it from scanning all the files while they were being backed up. I will be contacting support representatives for the backup and antivirus programs but my gut experience tells me the better evidence I have in hand, the better chance I have of getting satisfaction from them. Due to my experience with these vendors I am not looking forward to either call.
I am going to continue monitoring and testing but am hoping that someone out there can point me to a tool or method that will identify what specifically has claimed all that memory and won't give it back.
I should mention that this is a Windows 2003 Server file and mail server running Exchange Server 2003, Retrospect Server backup software and McAfee Security as a Service antivirus software. The memory consumption does not begin when the Exchange store is backup up but rather when the files are being backed up.