Windows 2000 server - CPU 100% - SYSTEM Process

I have restarted the server once. However the CPU continued to be maxed out. I am going to remove the trend and restart the server tonight.

Use Sysinternals Process Explorer to hopefully see what exactly is causing your CPU to max out.

I've used sysinternals. No processes are displayed which are utilizing CPU except the SYSTEM processes. SYSTEM is at 100% and all other processes are reasonably low. Using Filemon I can see normal activity and nothing particularly unusual. Could this be a sign that the CPU is going bad? or could it possibly be one of the mirrored disks? I'm just at a loss for what to try next. I'm contemplating rebuilding the whole system but I'm hoping I can find an alternate solution.

Mirrored drive:  possible, but highly unlikely.  
CPU going bad:  again, possible, but also unlikely.
Most 100% CPU utlization problems are due to driver or DLL problems.  Possibly a recent update that didn't take well?  You might try uninstalling the most recent updates and/or hotfixes.

You could probably chase this problem until the bags under your eyes have luggage tags from every third world country you care to mention, but if it were me, I'd take the server down and try an OS repair using the installation CD and if that failed, I'd rebuild the server and restore the data from my backup source.

I'm considering an upgrade to Windows 2003 R2 sp2. Possibly this would take care of my issues?

BTW, we just had a Win2K workstation with a very similar problem and it turned out to be the motherboard.  Replaced the MOBO, using all other original hardware (CPU, memory, hard drives, etc) and the problem disappeared.   But, before you start swapping out hardware, I'd try he suggestions I mentioned in my prevous comment first.

RE the upgrade to Win2K3 R2:  maybe (assuming you don't have an underlying hardware issue), but you may also achieve the same resolution by reinstalling Server 2000.   Do a clean install in any case.

Well it's past time that we go to Win2003. A clean install I would lose all my current settings?

Yes, but if you've documented all your settings, you should be able to re-set them failry easily.  

Luckily this is a file server and not a domain controller, exchange or web server, otherwise it would be a lot more difficult and involved.  If you have space on another server, you might transfer the files temporarily and repoint your network shares to that to cover yourself while you build up the new file server under Win2K3.

All pertinent shares are on a san. However printers and a few apps (multifunction printing devices) reside on the server. I'm going to attempt the upgrade to win2003. However since there were really no updates around the time that this problem began I imagine that it could be hardware related. I have an extra server similar model type. What would happen if I simply pull the drives and swap them into the newer server. Swapping mobo's would probably be more convenient if possible.

Unless the two servers are exactly alike (with the exception that the NIC will have a different MAC address), moving the drives to the other server or swapping motherboards probably won't work because the hardware won't match your current HAL.  No harm in trying it anyway, though, I suppose.

I upgraded to 2003 last night. Seeing pretty much the same behavior. I'm beginning to think that hardware has got to be the culprit here.

Did you upgrade or do a clean install of 2003?

I'm going to proceed with a backup and restore on a different server tonight.  I did an upgrade as I simply did not have the time to worry about re-installing software, printers, and auditing settings. In addition I loaded the latest HP Proliant suport pack which updated all the pertinent drivers.

i'm not an expert in this area but why don't you get a spare hard drive and install a clean windows server in it? then you'll know if doing a clean install and resetting everything will be worth it or if the problem won't be solved that way.

also try to do that clean install with the least devices possible. if that's the cause, adding them one by one can give you a hint.

i don't know, it's just another method to approach the issue...

OK ... I have restored a backup of the c: drive and the system state to a new server. Minimal diffculties because it was almost identical hardware. I initially thought that this cleared up the issue but I am now seeing fluctuations in the CPU again. It's like cycles now. 40% - 60% - 99% - 60% - 40%. One difference in this server is it has dual CPU's. 100% on the old server should definitely not = 99% on this server. I am looking at a complete rebuild of the system drive on the server. My only concern is the numerous shares and NT permissions that are assigned to an attached SAN device. If I run a system state backup an do a clean install of windows then restore the system state backup that should restore the shares and permissions of my SAN attached drives while providing a clean install of the C: drive.  To be more thorough in my explaination I have:
C drive composed of mirrored internal hard drives. (system)
D drive composed of a LUN on my SAN for file storage.
F drive composed of a LUN on my SAN for file storage.

Previously I restored my old server's C drive and system state to a new server. Possibly carrying with it the source of my high CPU utilization. If I wipe the server and then do ONLY a system state restore would that restore my shares, printers, and NTFS permissions?

One of the most difficult things about this problem is that it only presents itself at about 9:00AM in the morning after my users are logged on. So I have to have the server fully operational before I know whetehr the problem is resolved or not.

If this only happens after your users are logged in, this would indicate a network & sharing issue.   If it were me, I'd look at all your shares and who's attached to them (and why).  I'm assuming you're running WINS and DNS on the server?   Do you use a custom HOSTS or LMHOST file on the workstations?  Static IP's (hopefully) or DHCP for the workstations?
What network antivirus are you running?   Are your workstations continually being scanned by the server?  Is your antivirus on the server scanning every shared file in use?
You mentioned that the problem only arose after you updated Trend Office to version 8.   If you disable Trend Office (temporarily), do you suddenly revert to normal CPU utilization levels?   If so, that is most likely your problem.

No I actually have uninstalled Trend completely thinking the same thing, but this did not resolve the issue. I upgraded trend from 7.3 and it inherited the same settings as previous. We were using the intelligent scanning option which scanned files on access I believe. But as I have completely uninstalled the product and the fluctuations were still present I don't see how that could be the cause. I've looked at shares under server management and it appears to be just normal use. We use DHCP here and the workstations all look at the DNS servers for name resolution. We don't scan the workstations from the server, but rather the client scans the workstation. The server side office scan product is on its own server. We have a about 500 users here and we do a manual scan of all the home folders at 3:00AM. I'm starting to wonder if it may be somethign that Trend is simply missing.  

DHCP DNS Exchange are all on seperate servers. None of which are having any performance issues.

OK ... after a week and a half of fighting this I came in this morning and did a capture af about 8 minutes on my file server using Ethereal. I immediately saw that SMB traffic was way high and it was being caused by client requests for file access on home folders. I noticed that the files that were all being requested from the root of the home drives and that the files were various dll's, exe's, cfg's. The files looked very familiar and I eventually recognized them as component files from TREND MICRO OFFICE SCAN. The clients were bombing my file server with requests for updates for TREND components. The fileserver is not the update server so I called TREND. They told me that this is a known issue - although I found it no where in there KB. The new version of TREND has a flaw that causes it to poll network drives for updates instead of the appropriate update source. The solution was going to Global Settings under Network computers and removing the check in the box next to "Notify clients that there pattern files are out of date after <?>  days." Immediatley after I removed the checkmark my CPU's dropped from 87%-99% range to a steady 20%.

Thanks for the help DrMicro47 and sticking with me throughout the course of the problem.