Intermittent slowdown of SBS 2003 server

Hello,

Dell Poweredge 2800
3GB ram
2- 3GHZ Xeon processors with hyperthreading on
8- 300GB SCSI HDD's in raid 5 configuration
SBS 2003 with Exchange
Trend C/S/M Security for SBS
Also used as a file server

I am having the following issue.

About once a week my server slows down to a crawl.  This can happen even when the server is not being heavily utilized. The problem is fixed by either rebooting the server or waiting about an hour or so and it clears up. Here's a couple of things I have found.

Performance monitor shows pages/sec and avg. disk queue length are constantly maxed on the graph.
The disk activity lights on the server flash a repeating sequence over and over. For example (disk 4, disks 1&2, disk 8, disks 6&7, disks 3&5)
From what I can see there are 2 processes that run when there is an issue and don't run when all is normal. dbserver.exe and dcstore32.exe
Application Event viewer shows 2 or more of the following Warnings every time.

Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      2003
Date:            11/30/2006
Time:            9:42:59 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The configuration information of the performance library "C:\WINDOWS\system32\aspperf.dll" for the "ASP" service does not match the trusted performance library information stored in the registry. The functions in this library will not be treated as trusted.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      1016
Date:            11/30/2006
Time:            9:43:26 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The data buffer created for the "EXOLEDB" service in the "C:\Program Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte boundary. This may cause problems for applications that are trying to read the performance data buffer. Contact the manufacturer of this library or service to have this problem corrected or to get a newer version of this library.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 7c e5 a5 01 04 15 00 00   |å¥.....


Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      1016
Date:            11/30/2006
Time:            9:43:39 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The data buffer created for the "MSExchangeIS" service in the "C:\Program Files\Exchsrvr\bin\mdbperf.dll" library is not aligned on an 8-byte boundary. This may cause problems for applications that are trying to read the performance data buffer. Contact the manufacturer of this library or service to have this problem corrected or to get a newer version of this library.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 98 54 a6 01 fc 2b 00 00   ?T¦.ü+..


Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      2003
Date:            11/30/2006
Time:            9:43:51 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The configuration information of the performance library "C:\WINDOWS\system32\inetsrv\w3ctrs.dll" for the "W3SVC" service does not match the trusted performance library information stored in the registry. The functions in this library will not be treated as trusted.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Hope this was enough information to get us started. Any help would be greatly appreciated. Thanks.

Steve

sirvodkaAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
myfootsmellsConnect With a Mentor Commented:
Hi Steve --

Try going to your Windows Task Manager and the Processes Tab.  Then click View > Select Columns.  You'll want to click Page Faults Delta.  Now sort from highest to lowest PFD.  Is there one that's constantly pegged very very high?  If so that could be a problem.

Let me know.

Michael
myfootsmells
0
 
sirvodkaAuthor Commented:
Michael,

Thanks for the response.

I will give your suggestion a try. Unfortunatly, this only happens about once a week but i'll keep you posted.

Thanks.

Steve
0
 
sirvodkaAuthor Commented:
I did download and install the Microsoft Process Monitor and ran a 3 minute capture while the server was slow. I don't know the program well enough to identify what was causing the problem. Is there any way I could attach the capture file to this thread to have someone who knows the program to take a look at it?

Steve
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

 
myfootsmellsCommented:
You can provide a link to the picture that might work.

Michael
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
Steve,

I think your point about dbserver.exe running is a good clue to the problem, and I'd suspect this is caused by Trend Micro.  Have you installed the latest patch?
http://www.trendmicro.com/download/product.asp?productid=39

dcstor32.exe is related to Dell Open Manage, and may not have much to do with this.

Also, I noticed on a previous question of yours you were asking about SPAM filtering... this also could be the issue if you didn't configure things right.  Are you using Trend Micro's SPAM filter?  Or something else?  

Which Service Packs are installed on your server?  Is SBS SP1?  Exchange SP2?

Jeff
TechSoEasy
0
 
sirvodkaAuthor Commented:
Jeff,

I am using Trend's Spam filter and it is set to high. All Trend patches are up to date.
SBS is SP1
c:\program files\exchsrvr\bin\store.exe properties show SP1. Thought I had installed SP2.

Michael,

Not sure if this helps but here are the process log summaries from Microsoft's Process Monitor.

During slowdown:

PID            PROCESS                            FILE TIME             FILE EVENTS
844      services.exe                0.0040571           135
7644      Explorer.EXE                0.0056661           103
856      lsass.exe                                2.7477895           246
792      winlogon.exe                0                           2
3328      tmlisten.exe                0.0005764           38
5856      csrss.exe                                0.0007693           42
2248      ofcservice.exe                0.0025818           73
7744      store.exe                                0.0097848           679
1060      svchost.exe                0.1297964           70
8924      Explorer.EXE                0.002815           133
4644      pccntmon.exe                0.0034013           140
6928      pccntmon.exe                0.0025744           147
2952      SMEX_Master.exe                0.0006868           23
1552      ntfrs.exe                                5.8486301           34
6584      qbupdate.exe                0.0000753           14
9644      qbupdate.exe                0.0000805           14
352      dcstor32.exe                0.0027159           24
6628      emsmta.exe                0.0001215           6
432      Dfssvc.exe                                0.0010537           68
8060      HPBPRO.EXE                0.0044965           147
752      csrss.exe                                0.0078863           430
5328      HPBPRO.EXE                0.0056982           147
8648      HPBPRO.EXE                0.0043856           147
4880      OfcPfwSvc.exe                0.0124209           2
9004      HPBPRO.EXE                0.004667           147
9756      HPBPRO.EXE                0.0055312           147
8836      taskmgr.exe                0.0000174           2
400      EUQMonitor.exe                0.0003045           12
7644      Explorer.EXE                0.9239298          3,985
2248      ofcservice.exe                0.0206055          724
844      services.exe                0.0048039          236
7744      store.exe                                0.6329706          1,296
3328      tmlisten.exe                0.0011666          73
6764      mmc.exe                                0.0002066          9
4644      pccntmon.exe                0.0072142          280
8924      Explorer.EXE                0.0059027          253
856      lsass.exe                                6.1551059          401
792      winlogon.exe                0.0005364          53
6928      pccntmon.exe                0.0049219          294
2952      SMEX_Master.exe                0.0161676          163
5068      wmiprvse.exe                0.000179          11
1060      svchost.exe                0.007359          114
6584      qbupdate.exe                0.0001511          28
1460      svchost.exe                0.584234          1,622
9644      qbupdate.exe                0.0001553          28
7664      wmiprvse.exe                0.1291076          45
6628      emsmta.exe                0.0002259          12
5856      csrss.exe                                0.0083839          422
9084      w3wp.exe                                0.7253762          8
2968      SMEX_SystemWatcher.exe   0.033961          1,051
6492      HPBPRO.EXE                0.0054421          147
1552      ntfrs.exe                                32.4889922         102
752      csrss.exe                                0.0131915          688
7180      HPBPRO.EXE                0.0058808          147
8524      HPBPRO.EXE                0.0062496          147
8836      taskmgr.exe                0.0000455          6
7944      HPBPRO.EXE                0.0048604          147
8632      HPBPRO.EXE                0.0047911          147
400      EUQMonitor.exe                0.000243          12
8296      mmc.exe                                4.5763386          578
688      inetinfo.exe                0.1532404          190
2996      omaws32.exe                0.014942          14
352      dcstor32.exe                0.0278154          248
432      Dfssvc.exe                                0.0009909          68
2696      DbServer.exe                0.59299          314
8576      HPBPRO.EXE                0.0050654          147
9580      HPBPRO.EXE                0.0052545          147
8700      HPBPRO.EXE                0.0045396          147
10164      cgiOnUpdate.exe                0.0120324          12

After Slowdown:

PID            PROCESS                            FILE TIME             FILE EVENTS
7644      Explorer.EXE                0.0005654            28
2248      ofcservice.exe                0.001208            29
844      services.exe                0.0018277            82
9644      qbupdate.exe                0.0000433            6
8924      Explorer.EXE                0.0011836            56
1060      svchost.exe                0.0016378            28
1460      svchost.exe                0.0006388            25
856      lsass.exe                                0.7358331            97
792      winlogon.exe                0.0006187            31
6764      mmc.exe                                0.0001695            9
4644      pccntmon.exe                0.0012119            60
6628      emsmta.exe                0.0000816            3
7744      store.exe                                0.0037359            255
6928      pccntmon.exe                0.0012654            63
3328      tmlisten.exe                0.0001628            15
7664      wmiprvse.exe                0.0008342            42
2952      SMEX_Master.exe                0.0034363            19
9112      HPBPRO.EXE                0.0043625            147
1552      ntfrs.exe                                0.181638            15
752      csrss.exe                                0.0032105            172
10092      HPBPRO.EXE                0.0051751            147
8836      taskmgr.exe                0.0000237            2
6584      qbupdate.exe                0.0000197            4
400      EUQMonitor.exe                0.0001829            12
0
 
sirvodkaAuthor Commented:
Sorry, the summaries didn't post too well.
0
 
myfootsmellsCommented:
No Steve that didn't help you need to do the following:

Try going to your Windows Task Manager and the Processes Tab.  Then click View > Select Columns.  You'll want to click Page Faults Delta.  Now sort from highest to lowest PFD.  Is there one that's constantly pegged very very high?  If so that could be a problem.

You need to check the Page Fault Delta.

Michael
0
 
sirvodkaAuthor Commented:
Michael,

The server is responding normally at this time. I assume looking at the Page Faults Delta right now will not help. Am I incorrect?

Steve
0
 
myfootsmellsCommented:
Yes that's correct.  You need to wait when it's slow.

Michael
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
Well, the obvious thing on that list to me was HPBPRO.EXE.  Which will bring up this forum thread when you search on it:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=370850&admit=-682735245+1165377794251+28353475

Looks like the solution is to not have the HP Software loaded on your server, which is always a good idea.  You never need to load printer software on an SBS because the printing isn't actually done from the SBS desktop, it's just acting as the print server.

Jeff
TechSoEasy

0
 
sirvodkaAuthor Commented:
Jeff,

I do not have any HP software loaded on the server. All printers were added using the Add Printer wizard.

Steve
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
HPBPRO.EXE is most definitely running on your server.

Jeff
TechSoEasy
0
 
sirvodkaAuthor Commented:
Jeff,

A search of my servers C: partition found the following 3 entries:

hpbpro.exe     C:\windows\system32\spool\drivers\w32x86\3
hpbpro.exe     C:\windows\system32\spool\drivers\w32x86\hewlet_packardhp_co723c
hpbpro.exe     C:\windows\system32\spool\drivers\w32x86\hewlet_packardhp_lab8ca_bprint

The extracted printer driver files for my Color LJ 3500 and LJ 9000 include hpbpro.exe

I am downloading the driver only from the HP website, not the printing system/toolbox, and installing through the add printer wizard and the .inf file.
If this procedure is causing hpbpro.exe to get installed I'm not sure how to prevent it.

One note. In reading the link you provided me the general problem that everyone seemed to have with hpbpro.exe was 100% CPU usage. When my server has it's slowdown there is not CPU percentage issue. It remains low. So I'm not sure if this will solve my issue but I will follow the suggestions in the link to prevent hpbpro.exe from running.

Ok. Two notes. Looking at the summary I provided the ntfrs.exe process seemed to take up the most file time during the issue. Not sure if that means anything or not.

Thanks

Steve
0
 
myfootsmellsCommented:
Still no page fault delta readings for us?
0
 
sirvodkaAuthor Commented:
Michael,

No. It's tough to catch the server when it slows down. I really don't know unless someone tells me it is slow or I notice it doing the funky light sequence. Unfortunatly, I'm out of town at our branch office. I am able to remote into the server though. I would like to setup a performance alert and log that will notify me when this is happening but due to lack of experience I'm not sure how to set that up, what counters to use, or what threshold settings to configure. Any idea's.

Thanks.

Steve
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
You don't have to "catch" it, just run Process Monitor and it'll log the activity.  http://www.microsoft.com/technet/sysinternals/Utilities/processmonitor.mspx

Jeff
TechSoEasy
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
You could also use AdventNet.com's ManageEngine which is free for SBS (up to 20 devices):
http://manageengine.adventnet.com/products/opmanager/performance-monitoring.html

Jeff
TechSoEasy
0
 
sirvodkaAuthor Commented:
Jeff,

The log summaries that I posted are from Process Monitor. Am I just not looking at the correct data or is it not configured to capture the correct data?

Steve
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
The Process Monitor I linked above is Sysinternal's which logs and holds historical data.  Why don't you take a look at that.

Jeff
TechSoEasy
0
 
sirvodkaAuthor Commented:
I'll take a look at it. Thanks.

Steve
0
 
sirvodkaAuthor Commented:
Jeff,

The link you provided is the Process Monitor I used to collect the data I posted.

Steve
0
 
Jeffrey Kane - TechSoEasyConnect With a Mentor Principal ConsultantCommented:
I'm do apologize... too many close terms, and too quick to respond on my part.  I had meant to suggest that you use Process Explorer instead... which, if you "choose columns", you can add information about all sorts of memory usage.  

http://www.microsoft.com/technet/sysinternals/Utilities/ProcessExplorer.mspx

Rather than running either of these for three minutes, you need to just let them run for 24-hours or more to get some valid data.

Jeff
TechSoEasy
0
 
sirvodkaAuthor Commented:
Jeff,

A couple of questions on Monitor Explorer. Besides PF Delta what memory columns do you suggest I use?

I have it running and displaying data right now but i don't see where it is "recording" anywhere. If I use file/save is seems to save a "snapshot" of what is happening at that exact time. How do I configure it to save historical data if possible? The help feature isn't much "help".

Thanks.

Steve
0
 
sirvodkaAuthor Commented:
Still haven't been able to catch the server when it slows down. I will be back in the main office Tuesday and will keep you posted.

Steve
0
 
sirvodkaAuthor Commented:
Well, the issue seems to have cleared up on it's own. I have not had any issues for a month. Wierd. Thanks for your efforts.

Steve
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.