Avatar of Rightflank
Rightflank
 asked on

Virtual Environment

Hello,

Our company has been using an Intranet ASP.NET 1.1 application (in house) for the last 8 years. It queries a SQL Server 2000 installation with the actual DB being only 5 GB Max. The Intranet application will only work correctly with IE 6 and they have been running on old Dell Poweredge 700 series servers running 4GB RAM (32 bit) on Server 2003. These two servers have been our bread and butter and have made our company alot of money. This application and SQL server is called on by only 30 employees at any given time so the resource footprint has never been that big.

During the last year (before me and the other admin got here) our developer convinced our owners that going "virtual" will revolutionize our speed and availability. and this "virtual environment will make our CSR's more efficient.

So after many round tables with dell the company decided upon the consultation of our developer to purchase 3 - R710 Servers with 96GB of RAM running VSphere,  2- Force 10 Switches and an Equalogic SAN (4TB) all coming to a cost of $140,000 (yes that's right ONE HUNDRED FORTY THOUSAND DOLLARS)

Now the app RAN fine for 8 years with the the two $200 Poweredge 700's. so as soon as he deployed his .NET 1.1 application to the 64BIT Server 2008R2 "Virtual" Environment he begin to get 500 Errors.

The SQL Server Portion has been ok and the ASP.NET server is hosting the following:

IIS 7 With 4 Application Pools
Print Services for 40 Printers

This virtual server has been allocated 16 GB of RAM and he locked in the server with 60GB
of hard drive space out of 4TB of SAN space unused.

The 500 errors wont stop and when I look in event viewer I am seeing a consistent 5077 error about ASP.NET and WAS
"A worker process with process id of '%1' serving application pool ASP.NET1.1 has requested a recycle because it reached its virtual memory limit."
This warning is happening every minute and I'm thinking that maybe this is his "work around" to make the .net1.1 play nice with the 64 bit?

This DEV will not use newer .NET framework and refuses to upgrade to a newer webrowser. I think its not playing well with the X64 environment but thats just me.

Now at one time I noticed that he was using 54 GB of the 60 GB and when I brought in to his attention he didn't like that too much. He has since deleted some files because there's 16gb of free space now.

So basically now he's blaming myself and another administrator that our DC's are causing the problem but we didn't build the two DC's until 2 1/2 months AFTER his 500 errors began.

Me and the other Admin basically told them (the owners) yesterday that they spent about $130,000 too much and the whole virtual environment has to be re-engineered because we cannot change the size of the 60GB partition (we tried already) and his virtual environment is sharing all three physical servers so our production would be affected.

We told them that the whole virtual thing has been misrepresented

The developer even put "Now loading virtual environment" on the login page of the app. for a psychological suggestion to the owners.

When the app was first deployed the Owner told the Dev "I thought this was supposed to be faster?" and "They are getting errors"

Me and the other admin told them that this whole "VIRTUAL" word that has been thrown around as the savior was mis-sold because Virtual is great for companies who need many servers for less hardware such has hosting companies which require a smaller fiscal footprint when buying thousands of servers.

Now the latest thing this DEV is doing now is blaming our TeamViewer Hosts for slowing down our network. He's flatout lying and using the owners lack of knowledge to attack our technical credibility. We service 80 users and we install the lightweight teamviewer hosts on the machine for desktop support.

We've only been here for 6 months and he's been here for 8 years and is the "guru" so they believe him.

What do you guys suggest?
Thank you in advance.
RightFlank
Microsoft Legacy OSMicrosoft ApplicationsMicrosoft DevelopmentDell

Avatar of undefined
Last Comment
Rightflank

8/22/2022 - Mon
Lee W, MVP

I disagree with you both.

In my opinion, Virtualization makes sense in almost all cases.  HOWEVER, the implementation recommended was FAR more then necessary.  Using an appropriate design I think it could have been done right and reliably for about $15,000-25,000.

I also think you need to get a new developer.  IE6 is a security risk now - it's not supported - not even by google.  If this is a public facing web site, you're insisting your clients put themselves at risk using an antiquated browser.  If I were a consultant of one of your clients, I'd be urging them to look elsewhere for a competitive product that runs on a modern web browsers, preferably manufacturer independent (meaning IE, Firefox, Chrome, Safari, Opera, etc).

You're MOST LIKELY right in that nothing you've done is causing the issue.  But while I see a lot of frustration, I don't know what exactly you're looking for?  A solution to the 500 errors?  Advice on dealing with the person?  Whether you should leave the position?
Lionel MM

The best thing to do it to show the owner other examples of environments that are using virtualization successfully and at a much lower cost. The 140,000 is outrageous, no question about that. From what you've described with should of been able to be done with no more than 6 to 8, 000 in hardware and then techie time charges, well under 140K. There are tons of video on youtube that you could check out and put together a few for the owners to check out so they can better understand what virtualization is. The main thing is to get solid facts to the owners and show them you are interested in solving the problem and not trying to prove the developer was wrong; solving the problem will do that on it own merits. Good luck and stay focused on the solution and not the developer.
gsmartin

Let me start out by saying that I am an IT Manager with over 18 years of experience. I have a HP (C7000 w/ (10) BL490 G7), (3) DL380 G7/ (2) Compellent SANs (130TB RAW combined storage) /Brocade (4//8/16Gb FC/ BigIron RX-16 (w/10Gb Connectivity)/ VMware Enterprise virtualization (VM/HA/DRS/SRM) platform of 180 virtual machines on 10 physical ESX servers with 128GB Memory per server.
We are using over 200Gb of 10Gb network (LAG configuration) and 4/8Gb Fiber Channel connectivity (and soon 16Gb FC on new SSD SAN).  Offsite SAN replication, and more...  I can elaborate more, but i think this a good start.

So, I hate to disagree with everyone, but 140k for an Enterprise virtual environment is fairly reasonable.  (3) Dell Servers, (10) Dell Switches (using ISCSI), 4TB ISCSI SAN, plus VMware licensing.  You are using and paying for reputable Enterprise hardware where you can rely on the vendor for support.  FYI... Virtualization has many benefits and is also the way of the future for the majority of IT shops.  However, going virtual will not speed up performance!  Virtualization is inherently slower in a number of ways.  One of the primary objectives of virtualization is to make better use of server and storage resources.  A dedicated or stand-alone server tend to be greatly under utilized from a (memory and CPU) perspective, which helped pave the way for making virtualization possible.

Now, going from a dedicated hardware to a shared resource model such as with your SAN and server virtualization you will need to be aware of the saturation points for some of your resources (Memory, CPU, Disk I/O, Network).  For your environment with an ISCSI SAN, even running at 10Gb, your two biggest contention areas will be Disk IO and network latency.  The network latency will depend on your network architecture and weather you separated/isolated your ISCSI environment.  My preference is a dedicated Fiber Channel Fabric vs. ISCSI.  

Nevertheless, one of the main ways you can improve performance is a dedicated server and/or SSDs or NAND Flash drives (such as Fusion IO) for your primary database.  This would provide a significant performance increase and mask the inheritant performance trade offs of a virtualization.  Note virtualization environments also provide many inheretant benefits from a server/storage/backup administration and management perspective.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
gsmartin

Now to address your primary issue.  I would have agree the Developer needs to go.  After 8 years of supporting a legacy .NET platform
he's impeding the progress of the business.  
He apparently either doesn't have the aptitude and/or hasn't been able to advance his skills to be able to support and implement newer web development platforms.  One problem that stsnds out, is if the local browser of the web server can't open and run the web application then you definitely have issues.  IE6 is out dated and should be changed to support a more current version of IE and perferrably more universal support accross other browsers.  

Sometime a business needs to "Change the Chef in the Kitchen" and this sounds like one of those times!  The question becomes 'if not now, at what point will the developer decide to develop and implement a more current development platform?'  There are many more savy developers on the market.  By the time he's ready the business will have himridge and lost alot of money.

However for now on the Infrastructure side, I would focus on the issue at hand that is more likely related to the Windows 2008 64-bit platform vs your Active Directory servers.  The first thing I would try would be to run a 32-bit Windows 2003 virtual web server to see if it resolves the error message(s).
gsmartin

For any database performance issues I would recommend SSDs or NAND Flash, but for 30 concurrent users I would expect that to be necessary.

Can't you add a new LUN to the web server to be used for the log files?  You want to keep your Web environment on it's own LUN (not shared on the C: drive) and separate from the log files.  This should help Logan's other files (such as temp) from directly affecting the web application.
gsmartin

How many virtual machines in the (3) server virtual environment?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
gsmartin

As far as Teamviewer is concerned, I highly doubt you guys are running enough concurrent sessions to saturate your network.  Any network issues in your environment would be more likely the result of introducing ISCSI on to your Network.  Hopefully, you are not using VOIP as well?  If you also have network issues you should setup WireShark and monitor the traffic.  You will probably need to monitor uplink switch ports to see traffic on various VLANs.  You should also use NetFlow and/or SFlow monitoring and tools to give you better visibility of your network traffic and conversations.  ManageEngine.com has one called NetFlow Analyzer that's also available as a plug-in in there OpManager application that I use.
SOLUTION
Lee W, MVP

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
gsmartin

Bottom line, as long as you don't have the proper tools to provide you visibility into your environment your going to continue to have the finger pointing.  He obviously doesn't have the answer and is going to continue to try to point the finger at everything other than what he recommended.
ASKER CERTIFIED SOLUTION
Lionel MM

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
SOLUTION
gsmartin

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Rightflank

ASKER
Thank you all for the great answers. We are going to be getting a 2nd developer who has been told to give an honest assessment of the environment. We have shown the owners of what the application was running on,  and I was mistaken I told you it was a poweredge 700 series but it was better. It was a dell poweredge sc1425 valued at $175 dollars on ebay with 3 GB of RAM.  I was starting to question myself why this 140K solution was needed but the answers you guys gave brought me back to sanity : )
Your help has saved me hundreds of hours of internet surfing.
fblack61