Solved

Users at a remote location slow to download a web server's page

Posted on 2013-06-30
18
463 Views
Last Modified: 2014-02-25
We have a few IIS web servers (all in same subnet/VLAN)
 that are load-balanced using NLB.

Users at a remote office (of only 3 PCs) have been complaining of
frequent slowness in loading a web page: not even doing data query
but simply upon login to the main menu page (ie doesn't involve
DB servers yet).

These 3 users would access a url of the NLB & when this happens,
we tried accessing from the users' PCs different web servers'
url directly (to bypass the NLB) & could reproduce this problem:
2 of the web servers are slow to load the main page (25-39 secs)
while the rest of the web servers would load the same main
menu page within 1 sec.

I installed httpwatch basic edition on one of the user's PC to test out.

Attachment 1 is the screen showing web007's fast loading time.
Attachment 2 is the screen showing web010's slow loading time.

occasionally the web010 would load within 2 secs but at most
times, it would take 25-39 secs.

To isolate out the DNS, I've hardcoded web010 & web007's IP address on
that user PC's  c:\windows\system32\drivers\etc\hosts   but web010 still
takes as long to send back the mainmenu.aspx.

During this testing, I issued 'pathping -q 10  IP_addr_of the web servers'
& they all returned back with good timings of less than 3 secs for every
hop.

During these slow loading times, the CPU/RAM utilization of all the web
servers are less than 20%.  Also users from other remote locations
could load very fast the very same web010 page.  I also tried using the
user's PC to load a .js file (from web010) that's about 15 times larger
than  mainmenu.aspx & it loads in less than 1 sec.

Someone in EE suggested this could be an MTU issue but I've checked
the "show int xxx" outputs of the Cisco ports of the 3 PCs as well as
the Cisco ports that the web servers are all connected: the counter/
column for "Giants" is 0 (so I suppose this means no MTU issue at
both the servers & PCs ends).

When web010's page was consistently slow to load, if I entered web010's
IP addr in the IE browser to load the same url, it loads within 1 sec
consistently.  Strange that when I define/hardcode web010's IP in the PC's
hosts table (& I verified the hosts' table entry takes effect by intentionally
entering an unused IP & did a ping), it loads slow (25-39secs): so I really
can't conclude if this is a DNS issue.

I've seen one case when pinging web010 on this PC, it resolves to one
IP addr & the next minute, it resolves to a different IP addr.  However,
ipconfig/all on the PC showed the DNS used is still the same.

How else can I troubleshoot this further?

Would the Event Viewer logs in the DNS server help?  If so, I'll ask the
DNS admin to provide
web07.png
web10.png
0
Comment
Question by:sunhux
  • 13
  • 5
18 Comments
 
LVL 2

Assisted Solution

by:eexchangetech
eexchangetech earned 480 total points
Comment Utility
on the client install http://fiddler2.com/ , open the website.
This tool will list how time was consumed at various stages e.g. DNS querry, display page etc.

It's a fantastic tool for debugging such issues.
0
 

Author Comment

by:sunhux
Comment Utility
Actually my 2 development colleagues had previously installed this tool
on that PC but they got nothing out of it.   They've since deleted the
fiddler outputs.

 Is there anything specific that you would like to capture so that this time
round, the fiddler output will give more clues.
0
 

Author Comment

by:sunhux
Comment Utility
The attachments I uploaded are httpwatch, which I feel is more
user-friendly but if you prefer fiddler, I'll get it installed again but
do tell me what are the things you would like me to test out so as
to be able to get more fruitful diagnosis this round
0
 
LVL 2

Accepted Solution

by:
eexchangetech earned 480 total points
Comment Utility
If you install fiddler2, the out put will be some what like (tab statistics)
ACTUAL PERFORMANCE
--------------
ClientConnected:      03:57:09.811
ClientBeginRequest:      03:57:09.827
GotRequestHeaders:      03:57:09.827
ClientDoneRequest:      03:57:09.827
Determine Gateway:      0ms
DNS Lookup:             0ms
TCP/IP Connect:      258ms
HTTPS Handshake:      522ms
ServerConnected:      03:57:10.093
FiddlerBeginRequest:      03:57:10.093
ServerGotRequest:      03:57:10.093
ServerBeginResponse:      00:00:00.000
GotResponseHeaders:      00:00:00.000
ServerDoneResponse:      00:00:00.000
ClientBeginResponse:      03:57:10.093
ClientDoneResponse:      03:57:10.093

      Overall Elapsed:      00:00:00.2656250

In the attached screen show, it isn't clear at what primary stage the time is consumed..
as compared to above fiddler output.

even am not sure about the test, trying to give you a helping hand.

Btw, make sure you fix IP name resolution  and run these command on prompt from client pc
ipconfig /flushdns
nbtstat -R
arp -d *
0
 

Author Comment

by:sunhux
Comment Utility
Ok, finally managed to get permission for access & install
fiddler to the users' PCs.  Notice the 2 results from fiddler
below: 1st one took about 21 secs while the 2nd one, 1 sec:

When browsing/loading the webserver by using
its FQDN in IE, got the following:

ACTUAL PERFORMANCE
--------------
ClientConnected:      23:15:19.933
ClientBeginRequest:      23:15:19.933
GotRequestHeaders:      23:15:19.933
ClientDoneRequest:      23:15:19.933
Determine Gateway:      0ms
DNS Lookup:             0ms
TCP/IP Connect:      2ms
HTTPS Handshake:      0ms
ServerConnected:      23:15:19.933
FiddlerBeginRequest:      23:15:19.933
ServerGotRequest:      23:15:19.933
ServerBeginResponse:      23:15:41.554
GotResponseHeaders:      23:15:41.554
ServerDoneResponse:      23:15:41.569
ClientBeginResponse:      23:15:41.554
ClientDoneResponse:      23:15:41.569

      Overall Elapsed:      00:00:21.6364382

RESPONSE BYTES (by Content-Type)
--------------
text/html: 53,573
~headers~: 530


==================================

When browsing/loading the webserver by using
its IP address in IE, got the following:

Request Count:   1
Bytes Sent:      321            (headers:321; body:0)
Bytes Received:  710            (headers:627; body:83)

ACTUAL PERFORMANCE
--------------
ClientConnected:      23:20:42.224
ClientBeginRequest:      23:21:27.604
GotRequestHeaders:      23:21:27.604
ClientDoneRequest:      23:21:27.604
Determine Gateway:      0ms
DNS Lookup:             0ms
TCP/IP Connect:      0ms
HTTPS Handshake:      0ms
ServerConnected:      23:20:42.240
FiddlerBeginRequest:      23:21:27.604
ServerGotRequest:      23:21:27.604
ServerBeginResponse:      23:21:27.604
GotResponseHeaders:      23:21:27.604
ServerDoneResponse:      23:21:27.604
ClientBeginResponse:      23:21:27.604
ClientDoneResponse:      23:21:27.604


RESPONSE BYTES (by Content-Type)
--------------
~headers~: 627
text/html: 83
0
 

Author Comment

by:sunhux
Comment Utility
& the commands you suggested, a couple did
not work (as I probably don't have admin rights ):


C:\fiddler2>nbtstat -c

Local Area Connection:
Node IpAddress: [10.231.6.54] Scope Id: []

                  NetBIOS Remote Cache Name Table

        Name              Type       Host Address    Life [sec]
    ------------------------------------------------------------
    KAMKCCN12060199<20>  UNIQUE          10.231.6.55         507

C:\fiddler2>arp -a

Interface: 10.231.6.54 --- 0x12
  Internet Address      Physical Address      Type
  10.231.6.1            00-07-b4-00-05-02     dynamic
  10.231.6.55           ac-16-2d-10-c0-03     dynamic
  10.231.6.255          ff-ff-ff-ff-ff-ff     static
  224.0.0.22            01-00-5e-00-00-16     static
  224.0.0.252           01-00-5e-00-00-fc     static
  239.255.255.250       01-00-5e-7f-ff-fa     static
  255.255.255.255       ff-ff-ff-ff-ff-ff     static

C:\fiddler2>
C:\fiddler2>ipconfig /flushdns
The requested operation requires elevation.

C:\fiddler2>nbtstat -R
    Failed to Purge the NBT Remote Cache Table.

C:\fiddler2>arp -d *
The ARP entry deletion failed: The requested operation requires elevation.
0
 

Author Comment

by:sunhux
Comment Utility
Some observations that I've just noted:

For the slow web server, ping & nslookup gives
different IP addresses :

C:\Windows\System32\drivers\etc> ping web010
Pinging web010 [10.198.11.142] with 32 bytes of data:
Reply from 10.198.11.142: bytes=32 time=2ms TTL=120 <== this is HB IP of web010
Reply from 10.198.11.142: bytes=32 time=2ms TTL=120

For a web server that loads fast (ie <1 sec), both ping
& nslookup gives the same IP address:
C:\Windows\System32\drivers\etc> ping web007
Pinging web007.shsh.sss.com [10.198.11.87] with 32 bytes of data:
Reply from 10.198.11.87: bytes=32 time=2ms TTL=120
Reply from 10.198.11.87: bytes=32 time=2ms TTL=120


> web010
Server:  ad2.kkk.com
Address:  10.188.1.88

Non-authoritative answer:
Name:    oascghweb010.shes.shs.com.sg
Address:  10.198.10.101   <==direct login to web010 gives this IP; ping gave another IP

> oascghweb007
Server:  kkhdrad2.kkh.shs.com.sg
Address:  10.188.1.88

Non-authoritative answer:
Name:    oascghweb007.shes.shs.com.sg
Address:  10.198.11.87  <== this is the same IP as what ping gave
0
 

Author Comment

by:sunhux
Comment Utility
Provided all cache are cleared (for consistent test results),
I've also noted that if I browse web010 by the IP addr
10.198.10.101, it takes 21-35 secs to load but loading
using web010's the other IP, ie 10.198.11.142, it's always
loading in less than 1 sec
0
 
LVL 2

Assisted Solution

by:eexchangetech
eexchangetech earned 480 total points
Comment Utility
looks like you hit the nail. The time is cosumed at initial stage and as you've mentioned but different IP's being resolved, this could be the reason why much of the time is wasted.

Do this, on one of the client system
leave IP/mask/gateway as it is
Change the dns to use public dns e.g. 8.8.8.8 only
remove wins ip (if configured)

edit host file and map server's ip

What we are trying is isolating your local dns/wins because, these two are the only component that will perform name resolutions.
also verify by ping -a serverip and ping serverip
0
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 

Author Comment

by:sunhux
Comment Utility
> edit host file and map server's ip
Have tried the above a few times, did not help.
Suspect the return packets referred to DNSes & took a
long way for the return packets to come back: I noted
when the users' PC tried to load the webpage, within
1 sec, ' netstat -an | find "user_PC_IP" ' would show the
connection as established but it's the return packets
that took more than 21 secs (sometimes 30 over secs)
to get back to the user's PC/browser.

I checked the numerous DNSes & they define those web
servers that are slow to load to 10.198.10.x addresses
while those  web servers that are fast to load, they're
defined as 10.198.11.x addresses : one set of addresses
are the actual IP address while the other set is the
heartbeat IP addresses.

I don't have admin rights (only normal user rights that don't
allow me to change IP/DNS etc) to amend the configs on the
users' PCs.  Will attempt what you've suggested on Mon
when I get in touch with the PCs' admin.
0
 

Author Comment

by:sunhux
Comment Utility
Give me 1 more week: raising CR to update this suspected
incorrect entry, thereafter, will update in this thread again.
0
 

Author Comment

by:sunhux
Comment Utility
Now, even entering the IP address  of 10.231.11.x
which used to load fast has now become slow to load.
The users' frustration has escalated.

I've just done another fiddler which did not reveal much:
Request Count:   1
Bytes Sent:      2,869            (headers:2,869; body:0)
Bytes Received:  8,607            (headers:582; body:8,025)

ACTUAL PERFORMANCE
--------------
ClientConnected:      22:37:21.546
ClientBeginRequest:      22:38:30.905
GotRequestHeaders:      22:38:30.905
ClientDoneRequest:      22:38:30.905
Determine Gateway:      0ms
DNS Lookup:             1ms
TCP/IP Connect:      11ms
HTTPS Handshake:      0ms
ServerConnected:      22:38:30.918
FiddlerBeginRequest:      22:38:30.918
ServerGotRequest:      22:38:30.918
ServerBeginResponse:      22:38:30.961
GotResponseHeaders:      22:38:30.961
ServerDoneResponse:      22:38:49.446
ClientBeginResponse:      22:38:30.961
ClientDoneResponse:      22:38:49.446

      Overall Elapsed:      00:00:18.5410000 <== 18 secs

A good response is usually <1sec


I've got our DNS admin to deregister the heartbeat address
from DNS but no joy.



Also tried the suggestions below:
> Do this, on one of the client system
> leave IP/mask/gateway as it is
> Change the dns to use public dns e.g. 8.8.8.8 only => yep, done & tested it took effect
> remove wins ip (if configured) => under the NIC properties, WINS not configured
>                                          ==> but ipconfig/all still shows  WINS servers are there

> edit host file and map server's ip
Done the above for web007 & it loads very slow (of > 30 secs)
0
 

Author Comment

by:sunhux
Comment Utility
Ok, just found that to get rid of the 2 WINS, I'll just have to key
in a public DNS IP in its place & the 2 internal WIN servers will
be gone & replaced with the public DNS IP
0
 

Author Comment

by:sunhux
Comment Utility
There's 2 processes in the users' PCs:
BESClientUI.exe  &
BESPower.exe

What are these?  I've heard of Norton Antivirus
causing slowness by hundreds of times.
0
 
LVL 2

Assisted Solution

by:eexchangetech
eexchangetech earned 480 total points
Comment Utility
Software from IBM. IBM Tivoli Endpoint Manager (BigFix)
0
 

Author Comment

by:sunhux
Comment Utility
I've set the 2 BigFix processes to Lowest priority, no joy.

Is there anything else I can do to isolate further?
0
 
LVL 2

Assisted Solution

by:eexchangetech
eexchangetech earned 480 total points
Comment Utility
I would like to know if you systems are now set with correct IP settings, ping with IP , ping with name, ping -a etc are responding with correct IPs and Names?
0
 

Author Comment

by:sunhux
Comment Utility
By hard-coding into hosts file, the users PCs are now pinging
to the correct & consistent IP: but there are still some servers
out there that resolves to different IP address : would this matter?
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Join & Write a Comment

Don’t let your business fall victim to the coming apocalypse – use our Survival Guide for the Fax Apocalypse to identify the risks and signs of zombie fax activities at your business.
If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now