Solved

Network aplications slow on Microsoft Windows Network, XP, SBS 2003

Posted on 2008-06-12
44
907 Views
Last Modified: 2013-11-09
Hi,

   I have a problem that I can't fully identify, and for which I've spent a lot of time testing and debugging, and nothing helped.

   Computer network description:

   Freshly (6 months) installed computer network, freshly wired network, managed 3Com switch, new IBM server x3200 series, all clients have XP on them. All service packs and updates are installed, all network card drivers are latest versions.

   The problem is with two accounting applications that communicate with the server over shared folder mapped as a hard drive in My Computer. Both programs are flat file database based.

   The problem is when programs slow down. Something happens and both of them become unbelievably slow, and then clients start calling me.

   On all of the computers ESET NOD32 is installed, on the server NOD32 antivirus application options everything is disabled except file checking, and on the client computers options are left default.

   There is SBS Domain installed in the network, and all computers log in in that network. There is no folder redirection.

   Even Viewer:

Applications: No problems, except yesterday there were some problems with Exchange. I have installed some updates, and restarted and everything is OK now.

Directory service: All OK.

DNS Server: No problems.

File replication services: OK

Internet Explorer and Security: All OK.

System: All OK, except some printer problems with driver not installed on server. Mainly have connection with me connecting to terminal administration console.

netcps gives median transfer of about 32 megabytes per second on gigabit clients (All).

There is no firewall application installed, except the one built in SBS 2003.

Computer is defragmented, users normally use their files located on the share with no problem or speed slowdowns.

I have even gone so far to move all users from normal wired network to completely new CAT5 cables built in the electronic shop. Everything worked fine after I've installed that cables, and now people are again reporting slow performance.

The "SLOW" is defined by users click on some operation in in a program, and then everything halts for a period of time. That is extremely annoying and occurs on all computers in the network.

When I thought it is the Switch, I've replaced it with another one, and the problem persisted.

On the server there is Broadcom NetXtreme Gigabit Ethernet LAN card installed, newest IBM drivers, there is a screen shot attached of how the card is configured.

Other card is D-Link used for access to a ZyXel aDSL router that connects the LAN to the network. The Internet is over the SBS server.

The server is sufficiently powered, with two mirrored arrays, Dial Core Xeon, 2 GB of RAM.

Every computer and all network equipment is connected to an APC ups, server, switch and router to the Smart UPS that delivers pure sinewave and corrects the current.


I don't know what to add more, except - please help!

This problem is driving me nuts!


screen1.jpg
0
Comment
Question by:mrmut
  • 29
  • 14
44 Comments
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21767305
Two quick things I'd like to know first - Do ALL the clients start running slow at about the same time, and continue doing so until you take some sort of action (server reboot or whatever)? Or does it happen to single random clients whilst others are still running fine?

And secondly, when all is running incredibly slow, what is the status of the server performance wise? i.e. have you checked mem and cpu utilisation during these periods, to ensure there is plenty of memory and cpu resources available to the server?

I know it seems obvious, but I didn't notice it stated above, so thought I'd ask before doing anything else!

Thanks,

Pete
0
 

Author Comment

by:mrmut
ID: 21767472
1. Yes.

2. Practically ide. No problems whatsoever.


I know it sonds stupid, but I am completely short out of ideas.

Newest thing I am trying (starting today) is to get one client to work locally on the terminal to see will all work OK.


Thank you!
0
 

Author Comment

by:mrmut
ID: 21767664
NEW INFO!

The client logged into the server to his remote desktop. The started the application and everything worked great; snappy and fast, but at one moment clicking on one command caused the slowdown.

The slowdown manifested itself with rising System processes to 50% (one whole CPU) while the application functioned with normal load of about 0% to 4%.

Any ideas?

Thank you,

Borislav
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21767706
Did the system process drop back to normal on it's own or require intervention to bring it back down again?

And was this command anything particularly intensive, a large DB query etc?
0
 

Author Comment

by:mrmut
ID: 21767742
Yes, it did drop back to normal without any intervention. The CPU load rise is even hard to catch, as does pass by later on and everything continues to work to a second slowdown.

The command was certainly not intensive - the problem happens from time to time. The command was exiting the one tab of the program.
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21767855
You could try using a tool like Sysinternals Process Monitor (downloadable from the MS website) to get a better idea of what's happening with the processes when the problem occurs?

Not really sure where to go though, if something the application is doing is causing it to drain all CPU resources for 1 of the 2 cores, it would suggest some kind of problem with the app itself, but I'm afraid I don't have much experience with DB apps...

Might be an idea to try proc mon though, unless someone else comes along with a better idea?

Pete
0
 

Author Comment

by:mrmut
ID: 21767879
OK - new info;

   it seems that some system resource is periodically eating one CPU. Unfortunately, I can not say which one. - Is there any way to detect what is the problem?

   *Note that users are using the File Shares during my overviews. However, with normal fileshare use, with dedicated RAID and server class LAN card, no file transfer should take more than few percent, even with Antivirus running at 100% all the time.

   And as I've mentioned later - AV program is not using practically any CPU time, few percent at most.
0
 

Author Comment

by:mrmut
ID: 21767905
OK - there is even newer info;

   for some reason SQL server occasionally uses 50+ CPU. This incident is different from earlier one, but it does rise both System and SQL process.

   Screenshot attached.
0
 

Author Comment

by:mrmut
ID: 21767913
Sorry, here is the screenshot.
highCPU.jpg
0
 

Author Comment

by:mrmut
ID: 21768600
Here is what perfmon catched.

What the heck is happening? - It seems that some process, supposedly SQL or sth. is taking CPU time with high priority.

Any idea how can I find out what is happening?


Thanks.
problem.jpg
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21768674
Hi, is that the chart for just one of the cores? Or can I take it that the 50% jumps are basically 1 core at 100%?

I've seen on servers before, and was told at the time that it some apps do this periodically when carrying out periodical tasks such as indexing - An index running in a large DB will often cause CPU spikes like that, and if what I was told is true then it's nothing to worry about and perfectly normal etc...

Only periods I'm concerned about are the ones where it lasts a little longer, as the spikes I refer to above should be no more than a few seconds(ish). And if it's eating an entire core for several minutes, and that happens to be the only core than  SQL is using (again apologies but I don't know much about SQL so not sure if it happily multi-threads without any extra config etc) then it would obviously cause large detriment to performance for those few mins, then go back to normal...

Is any of what I've said making any sense?

Pete
0
 

Author Comment

by:mrmut
ID: 21768717
Well, yes - but I have concluded so even before.

The chart is for both cores, 50% is one entire core, and the pattern repeats itself, causing BIG problems.

Now - I really don't have a clue what is behind, as the server shouldn't be doing anything at all in the meantime.

Do you have any suggestion?


0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21769066
I'm afraid you'll probably need someone with more knowledge of SQL to figure out what it's doing... As I say I think it's normal for it to continuously spike (it may not happen on everyones SQL box but I don't think that it's anything to worry about) but the longer periods that last several minutes are where the problem is...

You probably need an SQL guru to make some suggestions, as I simply don't have enough experience with it...

Take a look at this too, another person with a similar problem - http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/Q_22868929.html
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21769096
Just as a warning, it's a LONG thread, and it's not answered yet, but it contains a lot of info on troubleshooting this sort of problem that I think may be useful to you!
0
 

Author Comment

by:mrmut
ID: 21769228
Thank you a lot!
0
 

Author Comment

by:mrmut
ID: 21773244
OK, I have some new info:

Ive captured one executable periodically consuming 100% of CPU: w3wp.exe.

It seems that some system service, sql and w3wp work in conjunction in a way, but I don't know what should I do now when I located the potential problem.

I would shut the bastards off this instance, but that probably wouldn't be such a smart thing to do.

Any hints?

0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21776663
Sorry to be brief, but take a look through these -

http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Microsoft_IIS/Q_23108115.html

http://forums.asp.net/p/917713/1113492.aspx

http://www.webhostingtalk.com/showthread.php?t=567795

That'll get you started, I'll keep searching for more relevant information, but unfortunately I now also have work to do! :)

Pete
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21776694
Unfortunately though, if you google "w3wp.exe consuming 100% cpu" there are MANY links... All the same sort of problems, in different environments etc... You may need to trawl through these links to find the one that's closest to your environment before you find anything you can use!

However that first link I posted (from the EE website) could be useful?? Depends how involved you are with SQL... :)

Pete
0
 

Author Comment

by:mrmut
ID: 21776798
Hi Pete, and thank you for your help! It is much appreciated, I suppose that you are also aware of the impossibility of explaining that some problems need time.

However, it seems that I have tracked down the bad processes!

I have used Process explorer and watched constantly over CPU utilization graph. Then I have suspended the most likely candidates - and VOILA!

CPU utilization is bas to normal, no periodic spikes.

The screenshot is included, suspended stuff are grayed out, namely:

1. svchost.exe with two w3wp.exe instances
2. sqlservr.exe (one of three)


Is there any way to learn what a service does?

Regarding all the stuff you gave links for - I've read them, but I didn't get it. I know that Google does return myriads of w3wp.exe results, but it would be much nicer if there are only a few, and one of them possible solution.

Thank you a lot!
proces-explorer.jpg
0
 

Author Comment

by:mrmut
ID: 21776888
OK - now I've resumed all the processes to see what will happen.

Still waiting for a burst, feeling like a fisherman.
0
 

Author Comment

by:mrmut
ID: 21776913
Got this from: http://groups.google.com/group/microsoft.public.windows.server.sbs/browse_thread/thread/968c88da75b4480d#

Here's how I would start with this:

- make sure that your AV exclusions are configured correctly for SQL and
Exchange (those w3wp's are IIS application pools, which is the reason I
mention Exchange).  I tend to blame AV for a lot of otherwise unexplained
problems, and sorry to say I'm often right.

- open Task Manager, go to the Processes tab, and add PID to the view if
necessary.  Get the PID for the SQL instance that's acting up.  Open a cmd
prompt and type "tasklist /svc."  Find the PID in question, and it'll tell
you which SQL app it is.  At least that will narrow down your
troubleshooting somewhat.

- get Process Explorer and run it on the server.  Look under System to see
specifically what is using the CPU cycles.  Again that'll help narrow things
down, and you might get some additional information about the SQL thing
there as well.

Process Explorer is free, and there's no installation - you can just run the
executable.  I run it directly from the USB key that contains all my server
tools.
http://technet.microsoft.com/en-us/sysinternals/cb56073f-62a3-4ed8-9d...
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21777064
Well I'm glad we're getting there! Process Monitor is a very useful tool, I've used it (and it's predecessors, reg mon and file mon) many many times! Always helps to be narrowing down where the problem is coming from...

With regards to "Is there any way to learn what a service does?" - Have you tried PROCESS MONITOR? I'm not sure if it's the same as Process Explorer, as it looks quite different... (another sysinternals tool).

This is a little more complicated to use I think, but tells you exactly what each process is doing and whether it succeeded or failed... See the screenshot for an example.

Bare in mind that this program is constantly capturing events (until you stop it) and within seconds of starting the program it has well over 30,000 events captured, so the filters are crucial. The best way to use it is run it during the problem just for a 5 seconds or something, then stop it capturing, and then identify how to use the filters to only see the things you want to see.

Give that a go and let me know... :)

Pete



But if you're asking whether you can find out what a process is actually doing
ProcMon.jpg
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21777071
Apologies, I got a little confused above in the first few comments etc... Process Explorer is what you've been using, Process Monitor is what I'm suggesting you try now... :) Ignore me babbling on about proc mon as if you'd been using it all along at the beginning of the post! :)
0
 

Author Comment

by:mrmut
ID: 21777228
Thank you,

   will go testing right now!

   Here is the link: http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx?PHPSESSID=d926


0
 

Author Comment

by:mrmut
ID: 21777418
Here is the screenshot.

I have tracked down the problem to the Scheduled Task: Update Services Synchronization task

That spike that can be seen is every five minutes caused by this task. I have disabled the task and it is peace now.

Will see what will happen now.
0
 

Author Comment

by:mrmut
ID: 21777421
Sry, screenshot.
PID.JPG
0
 

Author Comment

by:mrmut
ID: 21777428
What I've just found:

http://blogs.mcbsys.com/mark/post/Setting-Up-ESET-NOD32-Antivirus-30-Business-Edition-on-SBS-2003.aspx

Apparently, NOD32 can cause a lot of problems, so I have removed the listed equivalents of the files listed in NOD32 to see what will happen now.

The exclude list:

D:\Mail Server\Exchsrvr\MDBDATA\*.*
C:\Program Files\Exchsrvr\Mtadata\*.*
C:\Program Files\Exchsrvr\MCB03.log\*.*
C:\Program Files\Exchsrvr\Mailroot\*.*
C:\Program Files\Exchsrvr\Mdbdata\*.*
C:\Program Files\Exchsrvr\Conndata\*.*
C:\Program Files\Exchsrvr\srsdata\*.*
C:\WINNT\system32\inetsrv\*.*
C:\WINNT\IIS Temporary Compressed Files\*.*
C:\WINNT\NTDS\*.*
C:\WINNT\sysvol\*.*
C:\WINNT\ntfrs\*.*
C:\WINNT\security\edb*.log
C:\WINNT\security\tmp.edb
C:\WINNT\Security\Database\secedit.sdb
C:\WINNT\system32\CertLog\*.* - added 3/13/2008 (Certificate Authority files)
C:\WINNT\system32\dhcp\*.*
C:\WINNT\system32\wins\*.*
C:\Program Files\Microsoft SQL Server\MSSQL$BKUPEXEC\Data\*.*
C:\Program Files\Microsoft SQL Server\MSSQL$SBSMONITORING\Data\*.*
C:\Program Files\Microsoft SQL Server\MSSQL$SHAREPOINT\Data\*.*
F:\MSSQL2000\MSSQL\Data\*.*
C:\WINNT\System32\ntmsdata\*.*
C:\Program Files\Microsoft Windows Small Business Server\Networking\POP3\Failed Mail\*.*
C:\Program Files\Microsoft Windows Small Business Server\Networking\POP3\Incoming Mail\*.*
C:\WINNT\SoftwareDistribution\DataStore\*.*
C:\pagefile.sys
C:\WINNT\system32\licstr.cpa
C:\WINNT\system32\lls\*.* - corrected 3/10/2008
G:\*.*
H:\*.*
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21777472
Interesting... Let me know how it turns out, it certainly sounds like we're on the verge of having this sorted now anyways! :)

Pete
0
 

Author Comment

by:mrmut
ID: 21777567
OK - new news;

Now I still have periodic bursts of CPU activity, but they appear without order, and users can work in their program.

Also, - I don't get any w3wp.exes anymore, or at least I can't detect them to cause any inconvenience. I have noted that update services cumulatively create the most CPU activity, mainly in one of the SQL instances, but it seems that that doesn't create problems.

I use WSUS 3.0 SP1 and I have purged unneeded updates (it lasted all night tonight).

Today I have also restarted so we'll see what will happen now. I will leave it unattended for a day or two.

Will post any new finding.
0
 

Author Comment

by:mrmut
ID: 21807632
OK, a few days passed with new settings and it seems that set of actions I've made fixed the problem.

The CPU log is attached, server back to almost constant idling, with just an occasional system activities.


cpu-normalised.jpg
0
 

Author Comment

by:mrmut
ID: 21807636
Will wait for a few days more, and if all works, that's it. :)
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 21811454
Great news! But don't get too excited just yet... :) We all know what a let down it is when you really think you've fixed a very long winded and annoying problem, only to find that it rears it's ugly head again a week later!

Glad things SEEM to be working out now though!

Pete
0
 

Author Comment

by:mrmut
ID: 21811521
Thank you a lot Pete, although you haven't brought me a Holy Grail of Solution, I don't think I would do it without your help! :)

We'll see what will happen.

Will keep this post for a while with "PING!".
0
 

Author Comment

by:mrmut
ID: 21853808
PING!
0
 

Author Comment

by:mrmut
ID: 21874296
PING! - The problem is back.

The slowdown occurs here and then - not often as before, but it is here.

Just for thest, I have suspended all three sqlserv.exe instances and both w3wp.exe instances under syvhost.exe instance. Now it seems everything works, will try to debug further and will report what I've found.


suspended.jpg
0
 

Author Comment

by:mrmut
ID: 21961074
PING!
0
 

Author Comment

by:mrmut
ID: 22344270
Unfortunately, I must conclude that although overall helpful, I have not came to the solution and will have to reinstall server.
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 22345489
Apologies mrmut, but I'm utterly out of ideas anyway...

I hope the reinstall resolves the issue! :)

Pete
0
 

Author Comment

by:mrmut
ID: 22345667
Thanks Pete! :)
0
 

Accepted Solution

by:
mrmut earned 0 total points
ID: 22574274
Solution found!

I have uninstalled SharePoint services, and corresponding SharePoint MYSQL driver from Add/Remove programs, and that solved this utterly annoying problem.

I don't think that I will any time soon deploy any SBS machine.
0
 

Author Comment

by:mrmut
ID: 22574288
Solution found, no need for deletion of the question.
0
 
LVL 19

Expert Comment

by:PeteJThomas
ID: 22581939
Excellent news! This question should definitely be left here in case anyone else faces the same problem. It's the utterly annoying ones that you're dying to find already answered on these sites! :)

Well done!

Pete
0
 

Author Comment

by:mrmut
ID: 23990720
Solution has been found.

There probably are some other solutions to this problem, tho time would not allow to research all possibilities.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

this article is a guided solution for most of the common server issues in server hardware tasks we are facing in our routine job works. the topics in the following article covered are, 1) dell hardware raidlevel (Perc) 2) adding HDD 3) how t…
A quick step-by-step overview of installing and configuring Carbonite Server Backup.
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now