We help IT Professionals succeed at work.

Website request slow

274 Views
Last Modified: 2016-03-09
Hi Experts,

we are migrating to a new system with web servers and a website.
The old system runs under WIN2008 servers and IIS and on physical nodes.
The new system runs under WIN2012R2 and IIS and on VMs.

The website is start itself is ok but when I log in to my personal place to see my bookings or want to book something, this is nearly 10 times slower.

What can I check to speed up the content requests from my SQL to my website ?
Do you have any ideas or tools to check ?
Comment
Watch Question

Zac HarrisSystems Administrator

Commented:
Have you tuned your SQL database to restrict the amount of RAM it can use? SQL is a memory hog, it will eat RAM...not letting go of it until it has to. SQL loves to cache queries in true Microsoft fashion.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
What is the difference between the following server resources?  Physical and Virtual

1. CPU
1a. How many sockets (processors)
1b. How many cores per socket
2. RAM
2a.  How much in each server?
3. Network connection
3a.  how fast is the connection between the VMWare host and the database server?
3b.  how fast is the connection between the old physical server and the database server?

4. What features are installed in IIS on the new server?
5. What language is the app written in?  ASP.NET, PHP, etc.

What pages in the http logs have a large time-taken value?

Dan
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
If the SQL server is not being mirgated and the connection between the old server and website functions fine... but with the new VM there are delays in serving pages, then the issue is most likely not SQL or the SQL Engine configuration.

The SQL Server is not being migrated... right?

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Zac,

I have installed 128 GB RAM on each SQL Server.
The SQL Server act in one AG.
The Memory is restricted to 100 GB each.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Performance of Datastore?

CPU and Memory performance in vCenter ?

I'm not sure, you'll ever managed to see the same performance, as your physical server!
Zac HarrisSystems Administrator

Commented:
So are your SQL servers VMs or just the web servers. I wasn't really clear from your question. Just want to make sure I am getting the whole picture.

How much RAM have you allocated to the web servers? I built a virtual environment once and had plenty of physical RAM on my SQL servers but I forgot about the VMware host and was pegging the physical memory limits of it which slowed down my database calls.
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

the old system is all physical.
The machines have all 2 CPUs.
The old SQL servers are replaced by new physical server with 128 GB RAM each.
The new SQL servers have also 2 CPUs each.

The old Webserver is also physical and has 32Gb RAM
The new Webserver VM has now 16GB RAM and 2 CPUs but no IO yet. And I cannot see any memory peaks.
Eprs_AdminSystem Architect

Author

Commented:
3. Network connection
3a.  how fast is the connection between the VMWare host and the database server?
The speed is 1Gbit from the host uplink to the switch and to the SQL server.

3b.  how fast is the connection between the old physical server and the database server?
also 1Gbit to the switch and to the old SQL server.
Eprs_AdminSystem Architect

Author

Commented:
4. What features are installed in IIS on the new server?
Just IIS and Fileservices are installed
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

where can I check this last question from you ?
What pages in the http logs have a large time-taken value?
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
Ah, you didn't mention that the SQL Servers are also being migrated.

1. What does the network connectivity between your VM hosts and the physical SQL Servers look like?
2. Have you looked into the http logs to find what page(s) is taking a long time to deliver?
3. Have you profiled the new SQL Servers while you are trying to use the app on the new servers?

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Zac,
my new SQL servers are physical and each has 128GB RAM.
The WEBSERVER VM has now 16GB RAM.
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

Performance of Datastore?
I have checked this on my vcenter with selected webserver vm.
The latency has an avaerage of 0,5 milliseconds


CPU and Memory performance in vCenter ?
I cannot see no peak in memory and CPU on the webserver, check from the vcenter performance tab.
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

1. What does the network connectivity between your VM hosts and the physical SQL Servers look like?
On the hosts I have two networks, LAN and DMZ.
The webserver is in the DMZ and the other server in LAN.
Both networks are connected via uplinks to my cisco switch with trunks and port channels.
The cisco switch cpu is not on the limit, I have checked this.
The physiacl SQL servers are connected via NIC Team 2Gbit to the CISCO switch.


2. Have you looked into the http logs to find what page(s) is taking a long time to deliver?
Where do I find the logs ?

3. Have you profiled the new SQL Servers while you are trying to use the app on the new servers?
not yet profiled, but these are big machines
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
5. What language is the app written in?  ASP.NET, PHP, etc.
What is the typical extension for a web page?  ".aspx" ".php"

What pages in the http logs have a large time-taken value?
If you have logging enabled in the default location, your logs should be at:  c:\inetpub\logs\LogFiles'.  There you'll find 1 or more "W3SVC" numbered directories.  The http logs should be there.  The corresponding number for the W3SVC directory is the site ID displayed in IIS Manager.

So for the Default Web Site, the log directory is W3SVC1.

The field named time-taken is recorded in milliseconds.  It is usually the last column in the row.

Dan
Eprs_AdminSystem Architect

Author

Commented:
5. What language is the app written in?  ASP.NET, PHP, etc.

What is the typical extension for a web page?  ".aspx" ".php"

Yes it is .aspx !!
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,
I have found the log.
But I am not sure to find the exact time-taken stamp....these are long lines.
Eprs_AdminSystem Architect

Author

Commented:
Can it be, the logs are not written online to the file, right ?
It takes some time.
Eprs_AdminSystem Architect

Author

Commented:
What I have seen, the time in the log is not the same time on the web server.
But the webserver has correct time.
Why in the log is different ?
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
The logs are written to every 60 seconds or so.  Depends on the server activity.

The IIS writes UTC in the logs.

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

the SQL servers are configured like this.
I have  5 partitions.

C: OS  -> RAID1-> 10KSASDisks -> 2 disks
D: SQLDATA -> RAID10 -> 10KSASDisks -> 8 disks
E: TEMPDB -> RAID1-> SSDDisks -> 2 disks
F: BACKUP -> RAID1-> 10KSASDisks -> 2 disks
T: SQLLOGS -> RAID10-> 10KSASDisks -> 8 disks

All disks local in the SQL Server not on the storage.
Eprs_AdminSystem Architect

Author

Commented:
Here is the end of the log

.../Sites/User/Bookings.aspx 200 0 0 48771

Open in new window


When I change between addBooking and MyBookings, it takes ruthly 25 seconds.
This is really poor for the users.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
Ok, can you turn on all fields in the http logs?  All that says is tha the bookings page was successfully delivered.

Dan
Eprs_AdminSystem Architect

Author

Commented:
Ok, can you turn on all fields in the http logs?  All that says is tha the bookings page was successfully delivered.

How do I do this ?
Where to turn on all fields ?
I just have this text file.
Eprs_AdminSystem Architect

Author

Commented:
ok done
Eprs_AdminSystem Architect

Author

Commented:
all fields now on.
And I am looking for a IIS log analyser to understand the logs better.
Eprs_AdminSystem Architect

Author

Commented:
now all fileds are on

2016-02-17 16:54:18 W3SVC2 NTS-WEB01 172.16.1.73 GET /Sites/user/AddBooking.aspx - 443 11000634 10.5.1.72 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64;+Trident/7.0;+Touch;+rv:11.0)+like+Gecko ASP.NET_SessionId=lmy5uy5cqdtikptozeqx;+.ASPXAUTH=84032EA362E0AED6B4527B108BDABA2B816A3284A847CF662988D2F925EEFA90A1A69D08A211ACFE3DDFABE62B6A5C12383562F3763700D6BED71B1D41C64FF01B355ECF7FFC6702234BDBB652318586C11EACE389F4B2C0F7F1F1E3DD9BA704A6690FAD1571A907385D76C61D04C77E00EC35B3F7D9276C32AEBB0FD15129E1DFDEE953E55C0BBEAD1FF3E43D4 https://mysite.at/Sites/User/Bookings.aspx waschtage-test.hausbetreuung-wien.at 200 0 0 18026 764 48241

Open in new window


Where is the field TIME-TAKEN and how to reduce it ?
gr8gonzoConsultant
CERTIFIED EXPERT

Commented:
My own first suggestion would be to get a developer involved - whoever wrote the .aspx (ASP.NET) scripts, and have them place inline-logging with timestamps at major points throughout the script that is running slow, and also provide a write-up of the configuration (e.g. "script uses ADO.NET to connect to SQL server X at hostname x.y.z using these credentials: <user> <pass>") and the results of the logging.

The logging should narrow down exactly which areas of the script are taking a long time to execute. There's an assumption here that it's a problem with the database servers, and that's probably somewhat relevant, but it's still an assumption.

For example, let's say the logging reveals that each query takes 3-5 seconds to complete, and it runs 5 queries, which results in the slow results. To me, that says that it's either a problem with the queries themselves (which can be confirmed by running SQL profiler on the queries), or a problem with the transport level (the data packets traveling between the database server and the web server).

Alternatively, let's say the logging reveals that the script takes 20 seconds to connect to the database and < 1 second to run the database queries after the connection is established. This would indicate more of a problem with the connection technology itself. For example, the new server might have IPv6 turned on but not configured properly, and so the connection routine wastes time trying to look up the right hostname using IPv6, fails, and falls back to IPv4 and works. The fastest way to confirm if it's a IP or DNS issue is to have the script connect directly to the IPv4 address of the new server when making the database connection. If it eliminates the problem, then the next step is to have an ASP.NET script that does nothing but resolve the hostname of the new server and tell you how long it took to do that and what the results were. If it takes a long time, it's likely an issue between the internal DNS server and the new database server.

Again, a lot of helpful information will come from simply having the developer put in some temporary logging into the script that has the problem and then reproducing the problem.
Eprs_AdminSystem Architect

Author

Commented:
Hi Gonzo,

the developers also in the boat.
The same script is running on our old environment with physiacl servers and it is 10 times faster.

On our new environment just the SQL Servers are physical and the rest are VMs.

In this case I think, the webserver script is ok.
But why it takes so much longer ?

The webserver is in the DMZ and the SQL servers and application server not, they are in the LAN.
The PING from webserver to application server is < 1ms
The PING from application server to SQL server is also < 1ms

For me it is strange, because it brand new hardware from DELL. All R730 servers with 128GB RAM. Storage is a EQUALLOGIC PS4000 series with DUAL Controller and 10Gbit connection.

When I copy data between the VMs, the speed is about 300MB a second.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
Wow... 48 seconds sounds like an issue.

I would recommend 2 things.  ID what that page is doing in the code.  ID any db calls & have the dba run a profile on what is happening in the database during the page's operation.

Definitely bring in the developer.

What .NET Framework version was this written in?

How is the AppPool for the site setup?

Dan
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
On the SQL servers... The same OS as the old ones?  Same SQL server version?  Or have you upgraded the OS & SQL versions?

Dan
gr8gonzoConsultant
CERTIFIED EXPERT

Commented:
Hi Eprs,

It's good that the developer is a part of this, and many developers will just say "the code didn't change so it's not the code". However, just because the same code worked before with the physical servers doesn't mean the problem isn't with the code.

Don't get hung up on the hardware - your hardware is probably all fine. It sounds like you have plenty of RAM and bandwidth, which usually means it's just a configuration problem. The question is, "Which configuration is the problem?"

That's why I suggested that the developer add logging lines into the script - to help narrow down the possibilities and provide a list of how long each section of the code takes to run. Or you can always just use a tool like dotTrace to do an extensive profiling session:
https://www.jetbrains.com/profiler/

It will tell you exactly how long the script is taking with each line of code. This will help pinpoint where the configuration problem lies.

My hunch is that it's an issue with IPv6 configuration/binding and/or DNS. If you can, run a test on this by disabling IPv6 (both on the network adapter protocol and also in the SQL Server config section for protocols), then restart your network adapter. (Or again, you can just have the script connect to the new server's IPv4 address as a test).

I'd also recommend turning off your old physical servers if they're still connected to the network, and then tell your DNS and DHCP server to flush its DNS cache and renew the IP address leases. Also have your web server and SQL server flush their DNS caches.

I've seen some situations where you have two servers that are both competing for the same network resources (e.g. the same hostname), causing intermittent/random network behavior. You want to make sure that the web server can communicate with the database server without any problems.

Let us know when you have the results from all the various tests.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
What are the processors in the old physical server, and what is the CPU utilisation ?

How many sockets have you given to the new VM server?

What is the underlying datastore, and is there any read and write latencies on the datastore.
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

the old SQL servers are SQL2008 and WIN2008, not R2.

The new SQL servers are SQL2014 with availability Groups and WIN2012R2.
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

the old physical SQL server have 2 CPUs, Hardware: HP DL380 G5, RAM 20GB
The new physical SQL servers (NO VMs) have also 2 CPUs and 20 Cores, DELL R730 with 128 GB RAM each.

The CPU utilisation is always less than 40%.
This is the production system.

The datastore is a DELL EQUALLOGIC  PS4210X-21.6TB- Dual-24-2.5"
The latency is less than 1 ms for my VMs
Eprs_AdminSystem Architect

Author

Commented:
Hi Gonzo,

I have some test results for you regarding this :
My hunch is that it's an issue with IPv6 configuration/binding and/or DNS. If you can, run a test on this by disabling IPv6 (both on the network adapter protocol and also in the SQL Server config section for protocols), then restart your network adapter. (Or again, you can just have the script connect to the new server's IPv4 address as a test).

IPv6 is disabled on all network adapters and the NIC team on the SQL servers. IPv6 checkbox is not checked.

Before I wanted to start my tests today with your script, I found out the following.
I was logged on on several VMs and physical machines.
I was connected to the website and each request was very quick, less than a second. I was surprised.
Then from the webserver I just started a copy-job of 1 GB to my application server.
During this copy-job I tested again the website, and it was slow like before, each request takes again 25 seconds.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
How were the databases migrated?  Just a detach/attach process or was the database redone to use new SQL Server 2014 functionality?  Did you run the upgrade advisor prior to the upgrade?

Here is blog link about items to check during the upgrade process.

Link:  http://thomaslarock.com/2014/06/upgrading-to-sql-server-2014-a-dozen-things-to-check/

The MSDN article about changes in SQL Server 2014.

Link:  https://msdn.microsoft.com/en-US/library/ms143729%28v=sql.120%29.aspx

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Gonzo,

regarding your advice :
I'd also recommend turning off your old physical servers if they're still connected to the network, and then tell your DNS and DHCP server to flush its DNS cache and renew the IP address leases. Also have your web server and SQL server flush their DNS caches.

I cannot switch off the old system because it is in production.
The new system is a complete new setup and works in paralell mode.
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

the old DB was backed up and on the new SQL server it was restored.
Is this not ok ?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
The VM does have the VMXNET3 interface.

Also try increase the vCPU sockets to 4.
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

yes I always use VMXNET3.

Do you mean the sockets or core on the vm ?
Because now, we use 1 socket with 2 cores for each VM.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
My experience has shown that a simple backup/restore of a database from an exist version of SQL Server to a newer version is something that should be first done in a DEV environment in order to verify all functionality and to remediate depreciated functionality and known issues between versions.

I would always check for issues when moving/migrating existing databases to a newer version of SQL Server.  I recommend working with the DBAs/DBEs on investigating the functional differences between SQL Server 2008 & 2014.

I would also review the articles I posted as they explain the new functionality in SQL Server 2014.

On the IIS side, you could configure Failed Request tracking to watch for long executing pages.  When the booking page runs long, IIS will write a type of debug log about what is happening on that page.  This will help ID where you could invest time in the troubleshooting process.

Dan
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Well cores were implemented by VMware, to address licensing requirements, when you have a cores license for the VM.

It's best practice, to use Sockets x 1 Core. and the reason is because by default when you select sockets x 1 Core, This enables vNUMA to select and present the optimal configuration for vNUMA, under the physical topology.

If you select sockets x more than 1 core, optimally this is not best for performance.

You can read more here, in VMware's Blog

https://blogs.vmware.com/vsphere/2013/10/does-corespersocket-affect-performance.html

and even VMware Support Engineers are known to give the wrong information on this subject!
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

regarding your answer:
On the IIS side, you could configure Failed Request tracking to watch for long executing pages.  When the booking page runs long, IIS will write a type of debug log about what is happening on that page.  This will help ID where you could invest time in the troubleshooting process.

Where do I configure this to troubleshoot ?
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
Enabling Failed Request Tracing:

1. Open IIS Manager
2. Expand the IIS Server tree to show all objects
3. Under Sites, select the website you want to monitor.
4. In the center, open the feature "Failed Request Tracing Rules"
5. On the right-hand side panel, at the top, if an Alerts states the feature is not enabled for this site, following the instructions listed to enable it.  ** click on the message **
6. Click the Enable box, note the location of the logs, click OK
7. On the right-hand- side panel (the actions panel) click Add
8. Select ASP.NET to trace, NEXT
9. uncheck status codes and check Time take and enter a number in seconds. I'd start with 10, based on the http log entries.  NEXT
10. on the Select trace providers, select only ASPNET and WWW.  Leave logging in verbose mode.
11. FINISH

After it is setup, hit the booking page several times and let FRT collect info.

Then you can review the trace logs for more detailed information about what was happening.

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

point 4 is not on my screen.
4. In the center, open the feature "Failed Request Tracing Rules"

I just see REQUEST FILTERING.
But Failed Request Tracing Rules is not shown.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
The you need to add the feature to IIS.  You can Install it from the Server Manager GUI on Server 2012 R2.

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

ok thanks, data is coming in now into the FRT logs.
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

for the socket and cores just to understand.
I sit better to have

2 sockets and one core
or
1 sockets and two cores ?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
If you post a log from the FRT, I can take a look into it to see if anything jumps out as unusual...

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

today at 0830 the website was bloody quick.
I tested from several servers and from external, still it was quick.

Then I started a copy job from my web server to my application server.
Then the website was very slow again, each request takes now 30 seconds again :-(

This is really weired.
Still, when the copy job is finished, the website is slow.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
Can you post the FRT logs?  Also, can you post a shot of the Task Manager, Performance tab?  Also a shot of the Processes tab sorted by Working Set (Memory).

On the Processes tab, can you show the following columns:
- image Name
- PID
- User Name
- CPU
- CPU Time
- Working Set (Memory)
- Memory (Private Working Set)
- Handles
- Threads

It will help get a view into the server's state.

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,
first of all here is the FRT Log.
fr000027.xml
Eprs_AdminSystem Architect

Author

Commented:
Here is the rest. I hope this helps.
WEB01-Processes.JPG
WEB01-Perfromance.JPG
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
THE FRT log didn't hit the booking page.  I only see a request for the logout page.

Can you hit the booking page and see if FRT catches a log?

Dan
Eprs_AdminSystem Architect

Author

Commented:
Here is the new log....
fr000031.xml
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

thanks a lot for your help.
I will show this to the developers and come back to you.
:-)
Consultant
CERTIFIED EXPERT
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
Eprs_AdminSystem Architect

Author

Commented:
Hi Gonzo,

thanks for your post.
I have checked the sp_who but on the new SQL server is noone connected just me and some employees to test.
So there is no much IO on it.

When I start some REQUESTS to the website , I have checked the TCP Connections and the bandwith utilization.
Both was low  ,

network utilization was between 200-600 kbps
Network utilization always 0%
TCP connections about 26
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
so did you change to 4 vCPUs for testing purposes, and re-run your tests, and did it perform faster or slower ?
Eprs_AdminSystem Architect

Author

Commented:
@all

tomorrow we have a meeting with the developers of the application.
I will let you know the result of it.

If you have some special questions to ask, just let me know :-)
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Have they tested this application, on a VMware vSphere platform before, and if so, what was the specification of the Virtual Machine, and the underlying host, storage, and network infrastructure, and Do they have experience of running this application under a Hypervisor ? and if so which one ?

Please provide reference sites!
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

I have tested also with 4 vCPUs but the performance is the same.

The application never run before on a hypervisor, so this is new for the developers.

But first of all the yshould explain the FRT logs from the IIS, because here we have lost 22+5 seconds.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
So maybe this performance loss is to be expected under a hypervisor, and is the baseline, when the server is virtualised.

Is this affecting service ?
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

I think not, because in the morning the website can be very quick.
But for some reason, it is getting more worst and the system is not coming back to normal.
This is so strange.
Eprs_AdminSystem Architect

Author

Commented:
Hi Gonzo,

I have installed the WIRESHARK on the webserver.
I cannot see any strange data but I am also not very familiar with it.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
OK, not for nothing... but are you sure the new websites are properly configured to point to the new SQL Server databases?

Dan
Eprs_AdminSystem Architect

Author

Commented:
Hi Dan,

yes it is.
The old system is still running on old hardware.
The new system is running just on new hardware and configured for parallel usage.
Dan McFaddenTechnical Lead - Active Directory
CERTIFIED EXPERT

Commented:
So while actively monitoring the new SQL Servers, you log in on the new web servers and see no activity on the new sql servers?

If the new web servers are hitting the new SQL servers, then there should be activity in SQL.

Dan
gr8gonzoConsultant
CERTIFIED EXPERT

Commented:
For the wireshark data, basically you want to go to the View option, then to Time Display Format and you want to select "Seconds Since Previous Displayed Packet". This will set up the Time column to display the number of seconds between packets. Here's where you want to look for anything that took over 1 second to execute.

If you can take and attach screenshots of those results, that may help. Don't upload the raw capture data, since it will likely contain data that should not be shared publicly.
Eprs_AdminSystem Architect

Author

Commented:
Thanks a lot to all.
The developers are working on it.
The still think the vmware hardware is too slow but in my eyes it si bullshit.

I have learned alot from you and I will close this ticket now, thanks.
gr8gonzoConsultant
CERTIFIED EXPERT

Commented:
Everything you've said indicates that the hardware is more than sufficient for most situations. If it were truly a hardware issue, you wouldn't see the discrepancy of it working quickly at one time and then back to a whopping 20-some seconds the next time.

The only way it could be hardware in this situation is if you have a lot of other busy VM instances running on this same physical servers (you said that you had two new physical servers with 128 gb ram each, and implied they were dedicated to SQL, so I'm assuming that's not the case). Ultimately, VMs are borrowing physical resources from the host.

It still sounds a lot like a configuration issue to me. I hope the developers will be able to figure it out.
Eprs_AdminSystem Architect

Author

Commented:
Thanks Gonzo.

Yes on the performance tab, no hardware issues to see, no balooning no peaks.
I also hope :-)
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Is there a solution here ? or fix ?
Eprs_AdminSystem Architect

Author

Commented:
Hi Andrew,

not yet, the developer are working on it.

But there is one strange situation.
When you install wireshark on the APP01 and on the WEB01, the requests are bloody quick.
Do you know anything about changes on the TCP stack with an install of wireshark ?
gr8gonzoConsultant
CERTIFIED EXPERT

Commented:
Wireshark doesn't change the stack - it just monitors it. There's usually a driver involved (PCap), but that shouldn't fundamentally change anything about how the packets are originally handled. It might be a coincidence - you mentioned that it runs fast at other times, too.
Eprs_AdminSystem Architect

Author

Commented:
Hi Gonzo,

yes when you reboot the VM and you start wireshark, then the traffic and request are very fast.
Without wireshark not.
Strange or ?
Eprs_AdminSystem Architect

Author

Commented:
Hi all,

here is my follow up from this case.
As you know we had very bad website requests, sometimes a request took 30 seconds.
Then we had strange behaviors with wireshark installed and not.

Now we have fixed the problem.
The website requests and all other VMs now acting fast and constant.

The solution was, we have changed the vmware nic from VMXNET3 to E1000.