Solved

Virtual Environment

Posted on 2012-12-21
12
433 Views
Last Modified: 2016-11-23
Hello,

Our company has been using an Intranet ASP.NET 1.1 application (in house) for the last 8 years. It queries a SQL Server 2000 installation with the actual DB being only 5 GB Max. The Intranet application will only work correctly with IE 6 and they have been running on old Dell Poweredge 700 series servers running 4GB RAM (32 bit) on Server 2003. These two servers have been our bread and butter and have made our company alot of money. This application and SQL server is called on by only 30 employees at any given time so the resource footprint has never been that big.

During the last year (before me and the other admin got here) our developer convinced our owners that going "virtual" will revolutionize our speed and availability. and this "virtual environment will make our CSR's more efficient.

So after many round tables with dell the company decided upon the consultation of our developer to purchase 3 - R710 Servers with 96GB of RAM running VSphere,  2- Force 10 Switches and an Equalogic SAN (4TB) all coming to a cost of $140,000 (yes that's right ONE HUNDRED FORTY THOUSAND DOLLARS)

Now the app RAN fine for 8 years with the the two $200 Poweredge 700's. so as soon as he deployed his .NET 1.1 application to the 64BIT Server 2008R2 "Virtual" Environment he begin to get 500 Errors.

The SQL Server Portion has been ok and the ASP.NET server is hosting the following:

IIS 7 With 4 Application Pools
Print Services for 40 Printers

This virtual server has been allocated 16 GB of RAM and he locked in the server with 60GB
of hard drive space out of 4TB of SAN space unused.

The 500 errors wont stop and when I look in event viewer I am seeing a consistent 5077 error about ASP.NET and WAS
"A worker process with process id of '%1' serving application pool ASP.NET1.1 has requested a recycle because it reached its virtual memory limit."
This warning is happening every minute and I'm thinking that maybe this is his "work around" to make the .net1.1 play nice with the 64 bit?

This DEV will not use newer .NET framework and refuses to upgrade to a newer webrowser. I think its not playing well with the X64 environment but thats just me.

Now at one time I noticed that he was using 54 GB of the 60 GB and when I brought in to his attention he didn't like that too much. He has since deleted some files because there's 16gb of free space now.

So basically now he's blaming myself and another administrator that our DC's are causing the problem but we didn't build the two DC's until 2 1/2 months AFTER his 500 errors began.

Me and the other Admin basically told them (the owners) yesterday that they spent about $130,000 too much and the whole virtual environment has to be re-engineered because we cannot change the size of the 60GB partition (we tried already) and his virtual environment is sharing all three physical servers so our production would be affected.

We told them that the whole virtual thing has been misrepresented

The developer even put "Now loading virtual environment" on the login page of the app. for a psychological suggestion to the owners.

When the app was first deployed the Owner told the Dev "I thought this was supposed to be faster?" and "They are getting errors"

Me and the other admin told them that this whole "VIRTUAL" word that has been thrown around as the savior was mis-sold because Virtual is great for companies who need many servers for less hardware such has hosting companies which require a smaller fiscal footprint when buying thousands of servers.

Now the latest thing this DEV is doing now is blaming our TeamViewer Hosts for slowing down our network. He's flatout lying and using the owners lack of knowledge to attack our technical credibility. We service 80 users and we install the lightweight teamviewer hosts on the machine for desktop support.

We've only been here for 6 months and he's been here for 8 years and is the "guru" so they believe him.

What do you guys suggest?
Thank you in advance.
RightFlank
0
Comment
Question by:Rightflank
  • 7
  • 2
  • 2
  • +1
12 Comments
 
LVL 95

Expert Comment

by:Lee W, MVP
Comment Utility
I disagree with you both.

In my opinion, Virtualization makes sense in almost all cases.  HOWEVER, the implementation recommended was FAR more then necessary.  Using an appropriate design I think it could have been done right and reliably for about $15,000-25,000.

I also think you need to get a new developer.  IE6 is a security risk now - it's not supported - not even by google.  If this is a public facing web site, you're insisting your clients put themselves at risk using an antiquated browser.  If I were a consultant of one of your clients, I'd be urging them to look elsewhere for a competitive product that runs on a modern web browsers, preferably manufacturer independent (meaning IE, Firefox, Chrome, Safari, Opera, etc).

You're MOST LIKELY right in that nothing you've done is causing the issue.  But while I see a lot of frustration, I don't know what exactly you're looking for?  A solution to the 500 errors?  Advice on dealing with the person?  Whether you should leave the position?
0
 
LVL 24

Expert Comment

by:lionelmm
Comment Utility
The best thing to do it to show the owner other examples of environments that are using virtualization successfully and at a much lower cost. The 140,000 is outrageous, no question about that. From what you've described with should of been able to be done with no more than 6 to 8, 000 in hardware and then techie time charges, well under 140K. There are tons of video on youtube that you could check out and put together a few for the owners to check out so they can better understand what virtualization is. The main thing is to get solid facts to the owners and show them you are interested in solving the problem and not trying to prove the developer was wrong; solving the problem will do that on it own merits. Good luck and stay focused on the solution and not the developer.
0
 
LVL 8

Expert Comment

by:gsmartin
Comment Utility
Let me start out by saying that I am an IT Manager with over 18 years of experience. I have a HP (C7000 w/ (10) BL490 G7), (3) DL380 G7/ (2) Compellent SANs (130TB RAW combined storage) /Brocade (4//8/16Gb FC/ BigIron RX-16 (w/10Gb Connectivity)/ VMware Enterprise virtualization (VM/HA/DRS/SRM) platform of 180 virtual machines on 10 physical ESX servers with 128GB Memory per server.
We are using over 200Gb of 10Gb network (LAG configuration) and 4/8Gb Fiber Channel connectivity (and soon 16Gb FC on new SSD SAN).  Offsite SAN replication, and more...  I can elaborate more, but i think this a good start.

So, I hate to disagree with everyone, but 140k for an Enterprise virtual environment is fairly reasonable.  (3) Dell Servers, (10) Dell Switches (using ISCSI), 4TB ISCSI SAN, plus VMware licensing.  You are using and paying for reputable Enterprise hardware where you can rely on the vendor for support.  FYI... Virtualization has many benefits and is also the way of the future for the majority of IT shops.  However, going virtual will not speed up performance!  Virtualization is inherently slower in a number of ways.  One of the primary objectives of virtualization is to make better use of server and storage resources.  A dedicated or stand-alone server tend to be greatly under utilized from a (memory and CPU) perspective, which helped pave the way for making virtualization possible.

Now, going from a dedicated hardware to a shared resource model such as with your SAN and server virtualization you will need to be aware of the saturation points for some of your resources (Memory, CPU, Disk I/O, Network).  For your environment with an ISCSI SAN, even running at 10Gb, your two biggest contention areas will be Disk IO and network latency.  The network latency will depend on your network architecture and weather you separated/isolated your ISCSI environment.  My preference is a dedicated Fiber Channel Fabric vs. ISCSI.  

Nevertheless, one of the main ways you can improve performance is a dedicated server and/or SSDs or NAND Flash drives (such as Fusion IO) for your primary database.  This would provide a significant performance increase and mask the inheritant performance trade offs of a virtualization.  Note virtualization environments also provide many inheretant benefits from a server/storage/backup administration and management perspective.
0
 
LVL 8

Expert Comment

by:gsmartin
Comment Utility
Now to address your primary issue.  I would have agree the Developer needs to go.  After 8 years of supporting a legacy .NET platform
he's impeding the progress of the business.  
He apparently either doesn't have the aptitude and/or hasn't been able to advance his skills to be able to support and implement newer web development platforms.  One problem that stsnds out, is if the local browser of the web server can't open and run the web application then you definitely have issues.  IE6 is out dated and should be changed to support a more current version of IE and perferrably more universal support accross other browsers.  

Sometime a business needs to "Change the Chef in the Kitchen" and this sounds like one of those times!  The question becomes 'if not now, at what point will the developer decide to develop and implement a more current development platform?'  There are many more savy developers on the market.  By the time he's ready the business will have himridge and lost alot of money.

However for now on the Infrastructure side, I would focus on the issue at hand that is more likely related to the Windows 2008 64-bit platform vs your Active Directory servers.  The first thing I would try would be to run a 32-bit Windows 2003 virtual web server to see if it resolves the error message(s).
0
 
LVL 8

Expert Comment

by:gsmartin
Comment Utility
For any database performance issues I would recommend SSDs or NAND Flash, but for 30 concurrent users I would expect that to be necessary.

Can't you add a new LUN to the web server to be used for the log files?  You want to keep your Web environment on it's own LUN (not shared on the C: drive) and separate from the log files.  This should help Logan's other files (such as temp) from directly affecting the web application.
0
 
LVL 8

Expert Comment

by:gsmartin
Comment Utility
How many virtual machines in the (3) server virtual environment?
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 8

Expert Comment

by:gsmartin
Comment Utility
As far as Teamviewer is concerned, I highly doubt you guys are running enough concurrent sessions to saturate your network.  Any network issues in your environment would be more likely the result of introducing ISCSI on to your Network.  Hopefully, you are not using VOIP as well?  If you also have network issues you should setup WireShark and monitor the traffic.  You will probably need to monitor uplink switch ports to see traffic on various VLANs.  You should also use NetFlow and/or SFlow monitoring and tools to give you better visibility of your network traffic and conversations.  ManageEngine.com has one called NetFlow Analyzer that's also available as a plug-in in there OpManager application that I use.
0
 
LVL 95

Assisted Solution

by:Lee W, MVP
Lee W, MVP earned 166 total points
Comment Utility
gsmartin,

could you elaborate on how a $140,000 SAN Virtualization solution makes the remotest amount of sense for a system that had been running on "Dell Poweredge 700 series servers running 4GB RAM (32 bit) on Server 2003. These two servers have been our bread and butter and have made our company alot of money. This application and SQL server is called on by only 30 employees at any given time so the resource footprint has never been that big."

It sounds like they RIDICULOUSLY oversized this.  I suppose if they moved it to DUAL Fiber Channel SANs the price could creep up NEAR there... but again, if they are using a 5 GB database with 30 users, how would dual iSCSI SANs that were a mirror and two host servers that need LITTLE in the way of resources (low disk, low RAM, low CPU given what they had been running) run the price up to $140,000?  The Servers should be $5-7K each and the iSCSI SANs should be $7-15K.  Ok, so maybe my initial estimate was a little off... but even with services to have the equipment installed, the total cost should have topped out at $55,000... and even a $25,000 solution would have been several orders of magnitude more reliable than what they had been using.

At the end of the day, if they can justify it because the system allows them to make them $1000 per minute in net profit, then 2.5 hours of down time saved and the system pays for itself... BUT, If they were doing well before, I would SERIOUSLY question the logic and motivations of anyone recommending a system for this costing $140,000.
0
 
LVL 8

Expert Comment

by:gsmartin
Comment Utility
Bottom line, as long as you don't have the proper tools to provide you visibility into your environment your going to continue to have the finger pointing.  He obviously doesn't have the answer and is going to continue to try to point the finger at everything other than what he recommended.
0
 
LVL 24

Accepted Solution

by:
lionelmm earned 167 total points
Comment Utility
I have to agree with leew and strongly disagree with gsmartin--why? As pointed out by leew the issue is not that the cost of a virtualization solution can never be $140,000 but that for the situation explained to us by Rightflank--based on what they had before and what they are doing to go from that to a $140,000 solution is a horrible waste capital resources, cash. If this is such a great solution for them then why is the MASSIVE INCREASE in computing and network power causing all these problems. A "simple" upgrade of the current servers is at minimal a better solution.
0
 
LVL 8

Assisted Solution

by:gsmartin
gsmartin earned 167 total points
Comment Utility
First, I am not saying at ALL that virtualization was the right solution to address the initial databse problem.  I was just making a point of the cost of that type of solution.  Sorry,  I know that I explained a lot of things and my real point may have gotten lost within the details.  

I think we all agree that virtualization was completely the wrong approach by the Developer who clearly dosn't understand the pros and cons of virtualization; or the how and when to use it.  Now he's probably trying to save his A**!  Virtualuzation has it's caveats so you need to have a good understanding before using or considering it.  

My point is don't use virtualization or shared SAN storage to address a database performance issue.  In my own environment with all the virtualization our primary databases are all on dedicated hardware with SSDs in a RAID 10 configuration, which we get very good performance.  However, if your database server is already virtualized then you can still leverage SSDs in a RAID 10 configuration with RDM LUNs mounted up to the database server VM for the database(s) and transaction logs to improve performance.  Will be as fast as dedicated hardware with SSDs.  Simple answer, No!

SSDs should have been added to the original server environment to improve performance at a significantly lower cost; recommending virtualization was completely way off base and cost the company a lot if money (well over a 130K+).  In most environments that is enough for a dismissal.
0
 

Author Comment

by:Rightflank
Comment Utility
Thank you all for the great answers. We are going to be getting a 2nd developer who has been told to give an honest assessment of the environment. We have shown the owners of what the application was running on,  and I was mistaken I told you it was a poweredge 700 series but it was better. It was a dell poweredge sc1425 valued at $175 dollars on ebay with 3 GB of RAM.  I was starting to question myself why this 140K solution was needed but the answers you guys gave brought me back to sanity : )
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

The password reset disk is often mentioned as the best solution to deal with the lost Windows password problem. In Windows 2008, 7, Vista and XP, a password reset disk can be easily created. But besides Windows 7/Vista/XP, Windows Server 2008 and ot…
Whether you've completed a degree in computer sciences or you're a self-taught programmer, writing your first lines of code in the real world is always a challenge. Here are some of the most common pitfalls for new programmers.
Windows 8 comes with a dramatically different user interface known as Metro. Notably missing from the new interface is a Start button and Start Menu. Many users do not like it, much preferring the interface of earlier versions — Windows 7, Windows X…
Windows 8 came with a dramatically different user interface known as Metro. Notably missing from that interface was a Start button and Start Menu. Microsoft responded to negative user feedback of the Metro interface, bringing back the Start button a…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now