Solved

Slow launching application across network (and sometimes, unexplicably fast)

Posted on 2011-03-10
21
2,017 Views
Last Modified: 2012-05-11
I'm experiencing a strange problem opening a set of applications on a file server. In brief, opening the app takes a long time. But then, if I happen to be copying a file (a large one), and only while I'm copying the file from that same network server, the same application now opens in half the time. What gives?

The Environment:
- Windows 2008 Server Standard, Windows XP SP3 etc. etc. clients.
- Symantec Endpoint Security 11.0.6a (almost be the latest release).
- Hamachi VPN2 on the server and workstation(s).
- Gigabit network (seems to be operating at gigabit speed).
- The application in question is Dentrix (a dental practice management software). The executable is on the local C: drive, while the *flat* data files reside on a shared network location. No known access rights issues.
- It appears like once accessed, file access is "normal". Can't be 100% sure of this though because I have nothing to compare it to.
- Local access to the same Dentrix app on the server alone is fast (faster than the "fast" achieved while copying the large file).
- The difference in access speed is considerable: clearly 50% faster (5-6 seconds vs. 13 seconds). Local access on the server: 3 seconds.
- The server and workstation are fast (quad core), with plenty of processing power, including RAM.
- DNS is running on server, and forwards internet requests to 68.94.156.1 (AT&T). No obvious problems. Hosts file is also configured on workstations 192.168.169.3 = server.domain.local as well as LMHosts 192.168.169.3 - server
- Workstations have DNS set to just the internal DNS server (previously had that plus 68.94.156.1).
- All static IPs.
- Offline files enabled to one of the server shared folders, but not to the Dentrix Data folder (obviously accessed by several other workstations).

What I tried:
- Turned off (smc -stop) antivirus on both workstation and server.
- Turned off Hamachi VPN (on workstation only).
- Made changes to DNS (Hosts was not initially configured).

I ran into this discovery by accident, and I have always wondered why this application takes so long to open. What am I missing?????
0
Comment
Question by:btm02sf
  • 15
  • 5
21 Comments
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Have you mapped the remote files to a drive (net use G: \\server\share) or are accessing the file directly from the share name?

If you are using the share name directly, try mapping it to a drive or just doing a new use for the share:

    net use \\server\share

0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
Mapped as drive x:. I will play with the shares and will let you know how it goes. I may also create a new share and remap.
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
Sorry for the delay - haven't gotten to it yet, possibly today. Thank you for your suggestions so far!!!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
On the Workstation:
- I disabled SEP, then removed all other mapped drives. No change.
- Then, I changed the path from X:\Data to \\Server\Dentrix\Data (using UNC instead). No change in symptoms (slow then fast while I'm coping the large file).
- Connected to a brand new share straight to the subfolder, and mapped to it: Y:\ (instead of X:\Data). No change.

On the server:
- SEP not running.
- The new share I created has "everyone" full access privileges.

So, still no joy. Where can I look next? Thank you!!!!!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
Ok, now I'm completely puzzled. Here's more empirical evidence of .... what? So I was in the process of reconfiguring various network parameters. Added the WINS role to the server and configured Netbios over TCP/IP (default option), all to no avail.

However, at least at some point I decided to check whether the network card is in full or half duplex mode (gigabit). It was in full duplex mode, so I left it that way, but then I -casually- checked one of the configuration checkboxes. In this case it was "Wake on link". Boom! It was immediately after saving this configuration setting that file access started working super-fast. It also became immediately apparent that the application (Dentrix) became very, very responsive.

Unfortunately, that didn't hold - rebooting resulted in the original problem resurfacing. However, I'm able to replicate the fix every time I reboot, and it does not have to be that specific configuration checkbox, it's pretty much any I have randomly toggled!!! Usually the most innocuous and seemingly irrelevant will do the trick.

A couple of more notes: I'm using a DELL PowerConnect 2716 gigabit switch which I was ready to replace before I ran into this last discovery. The systems use Broadcom 82567LM-G Gigabit network adapters or very similar. Again, these are 1 year old Optiplexes (960 or 780).

And one more thing: actually rebooting the card (by disabling and re-enabling it) did not result in the temporary "fix".

Help, please!!!
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Is the NIC configured as fixed full duplex, or were you just checking the status and it was fixed?

If the NIC is configured for fixed full duplex the switch port should be configured fixed full duplex too.  

The NIC and the switch port should be configured the same.  If one is AUTO the other should be AUTO.  If one is fixed full, the other should be fixed full.

It could be a mismatch on duplex.  However, your symptoms are opposite of what normally happens when there is duplex mismatch.  Normally when there is a mismatch on duplex, the more traffic the slower it gets instead of the faster.
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
Yes, I actually had tried playing with that setting initially, and didn't notice anything different, but this time I did a bit more scientific testing and here's what I got:
- Factory default, AUTO negociation (which is what I had initially): well, this describes the symptoms I was reporting. Application opens in 3 seconds once I toggle "Wake up on link" (this is the "fix"). After rebooting, app opens - again - in about 13 seconds.
- 1Gbps Full duplex - same as for factory default (auto negociation).
- I don't have a 1Gbps Half duplex option.
- 100Mbps Full duplex - just slow (20 seconds or longer), no possible "fix" by toggling a setting.
- 100Mbps Half duplex - slow (20 secs to open app), then fixed, application opens in about 6 seconds after "fix".

So basically no definite conclusion, although it appears like the link negotiation just doesn't work well at 100Mbps full duplex. Any ideas?

I'll also try a couple of other different switches - although it's still mystifying as to what exactly happens when I make such a small change to the adapter settings? Note that changing the link speed will not fix anything, not to mention that the change only appears to take effect if I reboot.

Thanks!!!!
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
--> although it appears like the link negotiation just doesn't work well at 100Mbps full duplex

If you set the NIC to a specific speed link negotiation does not occur.


Are you changing the switch settings also?  You need to make sure both the NIC and the switch are set the same.
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
No, no changes on the switch - and frankly I didn't think I even had to do that. I'll check tomorrow and see if there's anything I can adjust, and try a different switch just for the sake of it. Thanks!
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
I guess I was not clear.  The NIC settings and the switch port setting MUST match.

If NIC is set to 100 Full, the switch port must be 100 Full.  If the NIC is set to Auto Speed/Auto Duplex, the switch port must be set to the same.

I would try setting the speed to the highest speed both devices support and full duplex.
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 1

Author Comment

by:btm02sf
Comment Utility
Ah, what I meant is that I never modified the port settings on this switch so I only assumed that they must be set to the defaults of auto negotiation for duplex and speed - but then again, the switch has been originally set up before my time. BTW - these settings should have matched the default settings for the network cards when I first noticed the problem.

I'll keep you posted - likely on Monday! Thanks so much for your help!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
Ok, so I did check, and before even I set the switch to managed mode the lights indicate full duplex (solid green on the right), and transmitting at 1000Mbps (right flashing green LED) or 100Mbps (a few computers do not have the gigabit cards.

So there's nothing obviously wrong with the switch. I will be replacing it temporarily with a gigabit Netgear switch (and something else) - and see what happens, and then I may play around with the managed mode (I'll turn it on).

I don't see how this is a factor given all of the sympthoms so far, but I'll also make sure that in at least a couple of instances I'll bypass the existing CAT5/6 cabling and connect directly to the computer(s).

I'll be able to work on this this coming Friday or Saturday.

Thanks so much for your assistance!!!!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
One more "development". I'm now noticing that in fact there's no need to make any changes to the Network card properties to get the response time of the network to "speed up". In fact it will speed up if I only go to the local area network properties then click on Configure next to the Intel 82567LM-3 network adapter, and then simply click Cancel (yes, not actually making any changes)!!!!!

Go figure!

I'm also going to try out a couple of other gigabit network adapters on the computers that still have the 100Mbps adapters. Thanks!
0
 
LVL 1

Expert Comment

by:itconsultant7717
Comment Utility
I feel for you, I installed a state of the art network for a dentist and when it came to equipment didn't skimp on anything. However, just like you when it came time to install and run Dentrix we encountered nothing but problems just like you are, and strange problems not seen before. We went through all the equipment out of desperation but the thing is I've designed and managed many networks and the difference with Dentrix is the sales reps and tech support changed their tune after the network was designed. One example is they said there has to be a WINs server and my first thought was NOTHING uses a WINs server anymore, I mean you can find devices that do but even if it's not configured they run well. Anyway, we installed WINs and it worked well for awhile until more problems. It got to a point where just about everything was replaced when I said enough is enough, I'm sorry but the answer can't continue to be "Dentrix can only work like this or with this". I turned the project over to someone else and they ended up removing the domain and going to 10 workstations in a work-group because Dentrix said their product doesn't work well in a domain environment. I'm sure this isn't helping you much and I'd love to know your final answer, just letting you know Dentrix was extremely difficult to work with. Good luck, I'm looking for my notes and the specific problems encountered and will pass anything along I can that may help.

Good luck.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Well, although it may not help you may want to do a packet capture.  I use Wireshark (http://www.wireshark.org).

I would install it on the workstation first and just try an use the application without doing anything "magic" that would cause the application to act normal.

I really can't tell you exactly what to look for  but a few things:

     DNS requests that never get a response
     WINS requests that never get a response
     NetBIOS name lookups that never get a response
     long delay between packets request or responses

I know it not much.  

I can't think of too much difference between a domain and a work-group environment except how user-id authentication works.  In a domain environment, everything should be done through the DC's.  In work-group environment it is done using the target computer's local security.  There are other difference, but that is really the biggie.
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
THANK YOU for all of the input - it's all very valuable and I don't feel like I'm completely in the dark.

- WINS was not installed on the server (2008 Standard), so I did install it. Then added WINS to the TCP/IP advanced configuration (default radio button then tried second radio - enable NETBIOS over TCP/IP). None of that had any effect.
- I then turned on a VMWare virtual machine that I use for testing (an XP workstation with the Dentrix client installed) and the results were the same (10 seconds +).
- I will try Wireshark and see if it points to anything at all that looks suspicious - although then what do I do with that information? So let's first see what I get, hopefully tomorrow).
- I did not mention that I've setup a very similar environment for the same business (different location) - just Windows 2003 Domain. Wins is running but I never paid attention to it and now I realize that it only shows the server itself as an active registration, nothing more. Then the virtual machine I just mentioned is working normally (fast) in this other environment (this VM has not been joined into any of the 2 domains). I can't think of anything that might be configured differently at this office.
- Finally, I'm still wondering if the Hamachi2 VPN client may not be causing any issues - although it wouldn't make sense why it doesn't in the 2003 environment. I turned it off at times, and the VM doesn't have it installed.

I don't know about Dentrix "domain" issues, and reconfiguring the environment as a workgroup is not really an option. Not to mention that there's always the chance that the simple fact of reinstalling Windows is what "fixed" the the scenario I was given (above).

So the plan for the next troubleshooting round is to run wireshark (not very hopeful), and then I may have another option since we have Acronis Backup & Recovery running at both locations which create virtual machines for both environment. I could then simply run that and reconfigure the VM server as much as it takes to get to a conclusion, and even run the server at the alternate location (if the symptoms carry over then I can pretty much rule out the wiring, workstations and their configuration and other related hardware - switches).

Thanks again!!!!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
And let's not forget the 2 "actions" that do provide a temporary fix that I've uncovered so far:

1. While copying a large file (for the duration of the copy process) - only on the Gigabit cards. The behavior is "normal" for the few computers with 100Mbps cards (Dentrix slows down when opening while copying the large file).
2. If I simply "configure" the network adapter, without saving anything - but only on the newer Dell systems that have the Intel 82567LM-3 Gigabit adapters.
0
 
LVL 1

Accepted Solution

by:
btm02sf earned 0 total points
Comment Utility
It turns out that I have found something tangible. Disabling opportunistic file locking on the Windows 2008 Server resulted in a significant boost in application response. In essence it all went down from 13 seconds to about 5 when opening any of the Dentrix modules. Here's an article on how to disable opportunistic file locking: http://support.microsoft.com/kb/296264

Disabling opportunistic file locking on the Windows side had absolutely no additional effect.

I'm coming a little short of being 100% happy about it because it appears that there are still some features within the application that run significantly slower than in environments running Windows 2003 server & XP. For example a specific patient search feature is quite slow, as well as opening a specific module.

Still, this is as close to a solution as I could get to on this exact topic, so I want to say thank you again to all that contributed in solving it!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
CORRECTION: I don't know what exactly I found... It turns out that while the obvious consequence of what I did was to get the apps to start faster, at least a couple of features (Complex Patient Search and Treatment Planner) slowed down to an unbearable level. Toggling the Opportunistic file locking demonstrated a clear effect on the application start time and these two features - however the infuriating part was that as stated the effect was opposite of each other.

After talking to Dentrix, they actually did have a batch file that in fact ENABLED opportunistic file locking by removing all of the associated registry entries (see Knowledge base article above) that I had so painstakingly tested and setup. So basically I had to choose the lesser of two evils and basically reverted to the original configuration!!!!! In fact now that seems like nothing compared to the complaints I was getting regarding the Treatment Planner.

I'm almost inclined to say that the solution to my problem is in fact to wipe everything out and start over using a downgrade option to Server 2003 (as itconsultant77.. implied above)...

I'll leave this open for a while in case there are any other ideas floating out there, but I'm frankly a little burned out on this.

Thanks!!!
0
 
LVL 1

Author Comment

by:btm02sf
Comment Utility
NOTE: removing the opportunistic file locking registry entries reverts everything to the MS defaults (OFL enabled).
0
 
LVL 1

Author Closing Comment

by:btm02sf
Comment Utility
All of the suggestions were legitimate, but ultimately I found a solution along a completely separate avenue.
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Don’t let your business fall victim to the coming apocalypse – use our Survival Guide for the Fax Apocalypse to identify the risks and signs of zombie fax activities at your business.
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now