Our proprietary application we use (in house) is performing slow. HELP!!

This is our in house web app we use within our company to allow our employees to run background checks, as that is what we do. The server is a Win2k3 Standard OS and is a VM running on ESXi 5. As of late, our application has become very slow for our. I am the sysadmin for my company, and is currently working with the development team who built the app. From a server standpoint, everything seems fine. Development see's no issues with their codes. There has been no current deployment that would cause any issue. I've even tried v-motioning the VM to another host to no avail. We look at the storage layer, networking, codes, SQL DB to no avail. We cannot determine the root cause. It's been a week. It's driving us nuts! Being that this is an in-house app, I would think it will be difficult for anyone on Expert Exchange to comment or give advice on what steps to perform next. I don’t expect anyone to provide a fix for the app as it is our app. However, if there is any suggestion as to what to look for to determine root cause, that would be appreciative. Maybe some advice on the server, network, SQL, or ESX layer?

Here are steps we've performed thus far to troubleshoot:

 - J2EE server and performance was showing as normal.

- The server was then restarted

- The logs on the server were reviewed and a few errors were found that were forwarded to Development for further review. Per Development, the errors seem as usual behavior and were no indication of the slowness issue.

- Profiler was running the entire morning, from 8 am, without seeing any deadlocks. The SQL server logs were also clear of deadlock entries.
Who is Participating?
here are some questions for you.

1) Prior to the app slow has anything changed on the server: ie: Windows updates, java updates, AV installs, driver updates etc....

2) Does you server have ample free space?

3) What version of J2EE are you using?

4) How is memory usage on the server Vm and host?

5) When was the last time the server was rebooted?

6) What type of storage are you using?

7) What type of vNic is attached to the VM?

8) How many VM are connected to the same vSwitch or vDS and are the running without issue?

9) Is the VM running on a snapshot?
Dave BaldwinFixer of ProblemsCommented:
How are you determining that it is slower?  Anecdotal evidence is not good enough because people get used to an app and then 'think' it is going slower because they are impatient.  You really need to collect some numbers for many different aspects before you can effect a change.

Do you have any way to compare the current traffic level with the traffic from "when it was faster"?  Do you have more people using the app now?  Are there other processes using more time on that server?
5itfaceAuthor Commented:
It is a fact that the app is slower. We've tested it and with our users. It's intermittent most of the time, and sporadic. The server seems fine. Since I am not the network guy, what can i do to compare the traffic as you mentioned?
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Dave BaldwinFixer of ProblemsCommented:
It is not a useful fact until you have a measurement by the numbers.  The reason that it is not is because you can't fix a mere observation.  "app is slower" and "server seems fine" are not facts you can use to make a change.

If you have a network guy, you should involve him in the situation.  He definitely should be able to make useful measurements.  I have been fixing and repairing electronic equipment since 1967.  You can't fix something until you can make a measurement that tells you what is wrong.
5itfaceAuthor Commented:
I've already looked into this with our networking guy, and our network service provider (COX). Everything seems fine on the networking layer. We have a few VMs on the ESX host, and I would think that if it's related to network, other VMs on the same vswitch or host would be affected as well.
5itfaceAuthor Commented:
By the way, would applying Windows Update help?
Dave BaldwinFixer of ProblemsCommented:
Since you don't know what is causing the problems you see, there is no way to tell what will help.  While I normally install all the Windows updates, sometimes people find that their application was using a feature that was changed by an update.

Until you are able to make measurements when there is a problem and can identify what is wrong or different, you simply have nothing to go one.  If there is truly a problem but everything "looks fine" that means you haven't looked at it at the right wrong time.
5itfaceAuthor Commented:
It appears the problem has resolved after we've done a deployment and rebooted the machine. A deployment meaning some jboss stuff that the developers deploy. However, this deployment carries no fix, but just a regular maintenance deployment. We are left to now finding the root cause. We've spent a couple of days, but still cannot find a root cause.
5itfaceAuthor Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for 5itface's comment #a39938450

for the following reason:

no one helped.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.