Solved

Intermittent slowdown of SBS 2003 server

Posted on 2006-11-30
26
4,990 Views
Last Modified: 2011-10-03
Hello,

Dell Poweredge 2800
3GB ram
2- 3GHZ Xeon processors with hyperthreading on
8- 300GB SCSI HDD's in raid 5 configuration
SBS 2003 with Exchange
Trend C/S/M Security for SBS
Also used as a file server

I am having the following issue.

About once a week my server slows down to a crawl.  This can happen even when the server is not being heavily utilized. The problem is fixed by either rebooting the server or waiting about an hour or so and it clears up. Here's a couple of things I have found.

Performance monitor shows pages/sec and avg. disk queue length are constantly maxed on the graph.
The disk activity lights on the server flash a repeating sequence over and over. For example (disk 4, disks 1&2, disk 8, disks 6&7, disks 3&5)
From what I can see there are 2 processes that run when there is an issue and don't run when all is normal. dbserver.exe and dcstore32.exe
Application Event viewer shows 2 or more of the following Warnings every time.

Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      2003
Date:            11/30/2006
Time:            9:42:59 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The configuration information of the performance library "C:\WINDOWS\system32\aspperf.dll" for the "ASP" service does not match the trusted performance library information stored in the registry. The functions in this library will not be treated as trusted.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      1016
Date:            11/30/2006
Time:            9:43:26 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The data buffer created for the "EXOLEDB" service in the "C:\Program Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte boundary. This may cause problems for applications that are trying to read the performance data buffer. Contact the manufacturer of this library or service to have this problem corrected or to get a newer version of this library.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 7c e5 a5 01 04 15 00 00   |å¥.....


Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      1016
Date:            11/30/2006
Time:            9:43:39 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The data buffer created for the "MSExchangeIS" service in the "C:\Program Files\Exchsrvr\bin\mdbperf.dll" library is not aligned on an 8-byte boundary. This may cause problems for applications that are trying to read the performance data buffer. Contact the manufacturer of this library or service to have this problem corrected or to get a newer version of this library.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 98 54 a6 01 fc 2b 00 00   ?T¦.ü+..


Event Type:      Warning
Event Source:      Perflib
Event Category:      None
Event ID:      2003
Date:            11/30/2006
Time:            9:43:51 AM
User:            N/A
Computer:      SHEPSBS2
Description:
The configuration information of the performance library "C:\WINDOWS\system32\inetsrv\w3ctrs.dll" for the "W3SVC" service does not match the trusted performance library information stored in the registry. The functions in this library will not be treated as trusted.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Hope this was enough information to get us started. Any help would be greatly appreciated. Thanks.

Steve

0
Comment
Question by:sirvodka
  • 14
  • 7
  • 5
26 Comments
 
LVL 5

Accepted Solution

by:
myfootsmells earned 250 total points
ID: 18057385
Hi Steve --

Try going to your Windows Task Manager and the Processes Tab.  Then click View > Select Columns.  You'll want to click Page Faults Delta.  Now sort from highest to lowest PFD.  Is there one that's constantly pegged very very high?  If so that could be a problem.

Let me know.

Michael
myfootsmells
0
 

Author Comment

by:sirvodka
ID: 18060749
Michael,

Thanks for the response.

I will give your suggestion a try. Unfortunatly, this only happens about once a week but i'll keep you posted.

Thanks.

Steve
0
 

Author Comment

by:sirvodka
ID: 18060876
I did download and install the Microsoft Process Monitor and ran a 3 minute capture while the server was slow. I don't know the program well enough to identify what was causing the problem. Is there any way I could attach the capture file to this thread to have someone who knows the program to take a look at it?

Steve
0
 
LVL 5

Expert Comment

by:myfootsmells
ID: 18060881
You can provide a link to the picture that might work.

Michael
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 18061431
Steve,

I think your point about dbserver.exe running is a good clue to the problem, and I'd suspect this is caused by Trend Micro.  Have you installed the latest patch?
http://www.trendmicro.com/download/product.asp?productid=39

dcstor32.exe is related to Dell Open Manage, and may not have much to do with this.

Also, I noticed on a previous question of yours you were asking about SPAM filtering... this also could be the issue if you didn't configure things right.  Are you using Trend Micro's SPAM filter?  Or something else?  

Which Service Packs are installed on your server?  Is SBS SP1?  Exchange SP2?

Jeff
TechSoEasy
0
 

Author Comment

by:sirvodka
ID: 18079556
Jeff,

I am using Trend's Spam filter and it is set to high. All Trend patches are up to date.
SBS is SP1
c:\program files\exchsrvr\bin\store.exe properties show SP1. Thought I had installed SP2.

Michael,

Not sure if this helps but here are the process log summaries from Microsoft's Process Monitor.

During slowdown:

PID            PROCESS                            FILE TIME             FILE EVENTS
844      services.exe                0.0040571           135
7644      Explorer.EXE                0.0056661           103
856      lsass.exe                                2.7477895           246
792      winlogon.exe                0                           2
3328      tmlisten.exe                0.0005764           38
5856      csrss.exe                                0.0007693           42
2248      ofcservice.exe                0.0025818           73
7744      store.exe                                0.0097848           679
1060      svchost.exe                0.1297964           70
8924      Explorer.EXE                0.002815           133
4644      pccntmon.exe                0.0034013           140
6928      pccntmon.exe                0.0025744           147
2952      SMEX_Master.exe                0.0006868           23
1552      ntfrs.exe                                5.8486301           34
6584      qbupdate.exe                0.0000753           14
9644      qbupdate.exe                0.0000805           14
352      dcstor32.exe                0.0027159           24
6628      emsmta.exe                0.0001215           6
432      Dfssvc.exe                                0.0010537           68
8060      HPBPRO.EXE                0.0044965           147
752      csrss.exe                                0.0078863           430
5328      HPBPRO.EXE                0.0056982           147
8648      HPBPRO.EXE                0.0043856           147
4880      OfcPfwSvc.exe                0.0124209           2
9004      HPBPRO.EXE                0.004667           147
9756      HPBPRO.EXE                0.0055312           147
8836      taskmgr.exe                0.0000174           2
400      EUQMonitor.exe                0.0003045           12
7644      Explorer.EXE                0.9239298          3,985
2248      ofcservice.exe                0.0206055          724
844      services.exe                0.0048039          236
7744      store.exe                                0.6329706          1,296
3328      tmlisten.exe                0.0011666          73
6764      mmc.exe                                0.0002066          9
4644      pccntmon.exe                0.0072142          280
8924      Explorer.EXE                0.0059027          253
856      lsass.exe                                6.1551059          401
792      winlogon.exe                0.0005364          53
6928      pccntmon.exe                0.0049219          294
2952      SMEX_Master.exe                0.0161676          163
5068      wmiprvse.exe                0.000179          11
1060      svchost.exe                0.007359          114
6584      qbupdate.exe                0.0001511          28
1460      svchost.exe                0.584234          1,622
9644      qbupdate.exe                0.0001553          28
7664      wmiprvse.exe                0.1291076          45
6628      emsmta.exe                0.0002259          12
5856      csrss.exe                                0.0083839          422
9084      w3wp.exe                                0.7253762          8
2968      SMEX_SystemWatcher.exe   0.033961          1,051
6492      HPBPRO.EXE                0.0054421          147
1552      ntfrs.exe                                32.4889922         102
752      csrss.exe                                0.0131915          688
7180      HPBPRO.EXE                0.0058808          147
8524      HPBPRO.EXE                0.0062496          147
8836      taskmgr.exe                0.0000455          6
7944      HPBPRO.EXE                0.0048604          147
8632      HPBPRO.EXE                0.0047911          147
400      EUQMonitor.exe                0.000243          12
8296      mmc.exe                                4.5763386          578
688      inetinfo.exe                0.1532404          190
2996      omaws32.exe                0.014942          14
352      dcstor32.exe                0.0278154          248
432      Dfssvc.exe                                0.0009909          68
2696      DbServer.exe                0.59299          314
8576      HPBPRO.EXE                0.0050654          147
9580      HPBPRO.EXE                0.0052545          147
8700      HPBPRO.EXE                0.0045396          147
10164      cgiOnUpdate.exe                0.0120324          12

After Slowdown:

PID            PROCESS                            FILE TIME             FILE EVENTS
7644      Explorer.EXE                0.0005654            28
2248      ofcservice.exe                0.001208            29
844      services.exe                0.0018277            82
9644      qbupdate.exe                0.0000433            6
8924      Explorer.EXE                0.0011836            56
1060      svchost.exe                0.0016378            28
1460      svchost.exe                0.0006388            25
856      lsass.exe                                0.7358331            97
792      winlogon.exe                0.0006187            31
6764      mmc.exe                                0.0001695            9
4644      pccntmon.exe                0.0012119            60
6628      emsmta.exe                0.0000816            3
7744      store.exe                                0.0037359            255
6928      pccntmon.exe                0.0012654            63
3328      tmlisten.exe                0.0001628            15
7664      wmiprvse.exe                0.0008342            42
2952      SMEX_Master.exe                0.0034363            19
9112      HPBPRO.EXE                0.0043625            147
1552      ntfrs.exe                                0.181638            15
752      csrss.exe                                0.0032105            172
10092      HPBPRO.EXE                0.0051751            147
8836      taskmgr.exe                0.0000237            2
6584      qbupdate.exe                0.0000197            4
400      EUQMonitor.exe                0.0001829            12
0
 

Author Comment

by:sirvodka
ID: 18079566
Sorry, the summaries didn't post too well.
0
 
LVL 5

Expert Comment

by:myfootsmells
ID: 18079658
No Steve that didn't help you need to do the following:

Try going to your Windows Task Manager and the Processes Tab.  Then click View > Select Columns.  You'll want to click Page Faults Delta.  Now sort from highest to lowest PFD.  Is there one that's constantly pegged very very high?  If so that could be a problem.

You need to check the Page Fault Delta.

Michael
0
 

Author Comment

by:sirvodka
ID: 18079695
Michael,

The server is responding normally at this time. I assume looking at the Page Faults Delta right now will not help. Am I incorrect?

Steve
0
 
LVL 5

Expert Comment

by:myfootsmells
ID: 18080393
Yes that's correct.  You need to wait when it's slow.

Michael
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 18082450
Well, the obvious thing on that list to me was HPBPRO.EXE.  Which will bring up this forum thread when you search on it:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=370850&admit=-682735245+1165377794251+28353475

Looks like the solution is to not have the HP Software loaded on your server, which is always a good idea.  You never need to load printer software on an SBS because the printing isn't actually done from the SBS desktop, it's just acting as the print server.

Jeff
TechSoEasy

0
 

Author Comment

by:sirvodka
ID: 18086203
Jeff,

I do not have any HP software loaded on the server. All printers were added using the Add Printer wizard.

Steve
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 18089647
HPBPRO.EXE is most definitely running on your server.

Jeff
TechSoEasy
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:sirvodka
ID: 18090098
Jeff,

A search of my servers C: partition found the following 3 entries:

hpbpro.exe     C:\windows\system32\spool\drivers\w32x86\3
hpbpro.exe     C:\windows\system32\spool\drivers\w32x86\hewlet_packardhp_co723c
hpbpro.exe     C:\windows\system32\spool\drivers\w32x86\hewlet_packardhp_lab8ca_bprint

The extracted printer driver files for my Color LJ 3500 and LJ 9000 include hpbpro.exe

I am downloading the driver only from the HP website, not the printing system/toolbox, and installing through the add printer wizard and the .inf file.
If this procedure is causing hpbpro.exe to get installed I'm not sure how to prevent it.

One note. In reading the link you provided me the general problem that everyone seemed to have with hpbpro.exe was 100% CPU usage. When my server has it's slowdown there is not CPU percentage issue. It remains low. So I'm not sure if this will solve my issue but I will follow the suggestions in the link to prevent hpbpro.exe from running.

Ok. Two notes. Looking at the summary I provided the ntfrs.exe process seemed to take up the most file time during the issue. Not sure if that means anything or not.

Thanks

Steve
0
 
LVL 5

Expert Comment

by:myfootsmells
ID: 18090110
Still no page fault delta readings for us?
0
 

Author Comment

by:sirvodka
ID: 18090293
Michael,

No. It's tough to catch the server when it slows down. I really don't know unless someone tells me it is slow or I notice it doing the funky light sequence. Unfortunatly, I'm out of town at our branch office. I am able to remote into the server though. I would like to setup a performance alert and log that will notify me when this is happening but due to lack of experience I'm not sure how to set that up, what counters to use, or what threshold settings to configure. Any idea's.

Thanks.

Steve
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 18090500
You don't have to "catch" it, just run Process Monitor and it'll log the activity.  http://www.microsoft.com/technet/sysinternals/Utilities/processmonitor.mspx

Jeff
TechSoEasy
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 18090514
You could also use AdventNet.com's ManageEngine which is free for SBS (up to 20 devices):
http://manageengine.adventnet.com/products/opmanager/performance-monitoring.html

Jeff
TechSoEasy
0
 

Author Comment

by:sirvodka
ID: 18090519
Jeff,

The log summaries that I posted are from Process Monitor. Am I just not looking at the correct data or is it not configured to capture the correct data?

Steve
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 18090560
The Process Monitor I linked above is Sysinternal's which logs and holds historical data.  Why don't you take a look at that.

Jeff
TechSoEasy
0
 

Author Comment

by:sirvodka
ID: 18090576
I'll take a look at it. Thanks.

Steve
0
 

Author Comment

by:sirvodka
ID: 18091264
Jeff,

The link you provided is the Process Monitor I used to collect the data I posted.

Steve
0
 
LVL 74

Assisted Solution

by:Jeffrey Kane - TechSoEasy
Jeffrey Kane - TechSoEasy earned 250 total points
ID: 18091430
I'm do apologize... too many close terms, and too quick to respond on my part.  I had meant to suggest that you use Process Explorer instead... which, if you "choose columns", you can add information about all sorts of memory usage.  

http://www.microsoft.com/technet/sysinternals/Utilities/ProcessExplorer.mspx

Rather than running either of these for three minutes, you need to just let them run for 24-hours or more to get some valid data.

Jeff
TechSoEasy
0
 

Author Comment

by:sirvodka
ID: 18095836
Jeff,

A couple of questions on Monitor Explorer. Besides PF Delta what memory columns do you suggest I use?

I have it running and displaying data right now but i don't see where it is "recording" anywhere. If I use file/save is seems to save a "snapshot" of what is happening at that exact time. How do I configure it to save historical data if possible? The help feature isn't much "help".

Thanks.

Steve
0
 

Author Comment

by:sirvodka
ID: 18156776
Still haven't been able to catch the server when it slows down. I will be back in the main office Tuesday and will keep you posted.

Steve
0
 

Author Comment

by:sirvodka
ID: 18444421
Well, the issue seems to have cleared up on it's own. I have not had any issues for a month. Wierd. Thanks for your efforts.

Steve
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

In the event you manage a Small Business Server 2003, and you are audited for PCI compliance, there are several changes you must make in order to pass the audit. I can take no credit for discovering any of these fixes or workarounds, but there is no…
You may have discovered the 'Compatibility View Settings' workaround for making your SBS 2008 Remote Web Workplace 'connect to a computer' section stops 'working around' after a Windows 10 client upgrade.  That can be fixed so it 'works around' agai…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now