Does VMware slow down machines?

VMWare will slow things down a bit, but unless one of the VM is stressing the system (either CPU, RAM, or I/O) it should not be noticeable.

However, you do have 12 GB of RAM allocated on a physical server that only has 8GB.  You may want to check to see how much RAM each of the VM's are using.  If they are using the full amount of RAM allocated to them, then the host  system would be paging which would cause performance issues.

Qiltir is right, you have overallocated memory, and at least exchange will grab all memory it can get. By the way: There is no 585G2, I gues you have a G5?

yes, I would agree with that, but look at the attached file, there is almost no toll on the server
vmware.png

ollfried:  I think it is a G2.  look at the png file.

Strange. What storage do you use?

it has on board 8x146Gb disks, 15K RPM.

Do you have BBWC for you SmartArray? If not, you have no write-back cache, which is a serios performance impact.

I think it does, but this server was a stand alone exchange server last year, and it was lightning fast.  now add vmware and 2 more machines, and they are all (at least from the console aspect) slow.

I want to see some screenshots of your performance for CPU Storage and Memory.  Please post some.

This can be acquired from the Performance tab.

There is a DL585 G2:

     http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-3328412-241644-3328422-3219717.html

If Exchange is using 6GB of RAM (and ollfried is correct Exchnage will take everything you give it) and any of the others wake up you could start having problems.

Over allocating RAM is not that bad of a think, as long as you know the normal RAM usage pattern of the servers you are virtualizing.

Your snap shot showed only 7 GB of RAM in use by ESX, which is not really that bad.

How is the  networking setup?  When you say it is slow, can you define slow?  Are you RDP'ing over the LAN, or over a WAN?

Do you have vmware tools installed in the VMs?

Rdping over LAN, it's the feel you get when it's a slow machine, logging in is slow, opening up my computer takes 20 seconds etc, opening exchange managemt  console takes
a long time etc.  

This might have nothing to do with vmware. Does your name resolution work fine? Do you have errors in your event logs?

Yes Vmware tools are on all machines.  I have a similiar setup at a colo, on a HP ML350 G5, 24gb ram, and it also
like this as I know this does t happen with these machines as standalone.  

Other than RDP are there any other performance issues?

Does Exchange work fine?

Do the DC's seem to keep up with the workload?

On the screen shot there was 4 VM's. Is the 4th active?  If so, what does it do?

Do you have a ftp server on any of the VM's?  If so, can you ftp a few files to see if it could be network related?

No FTP server but I you want let me know what files and I can get them to you

No FTP server but I you want let me know what files and I can get them to you
as far as performance, there are not that many users and it seems to be fine, which is why I thought it may be a console thing not background services.

No special files, I wanted to see the network performance.  CIFS/Samba copies are slow, so seeing how long it takes to transfer a file using copy doesn't really do any good.

FTP can push the network and/or the disk system to their limit fairly easily in most cases.  The exception is if you gigabit network paths and are not using jumbo frames.

However, since you don't have ftp server, you may want to try coping a few large files, at least 100 MB, and see if they seem to take longer than you would expect.

Yes, I copied a quickbooks installation, about 600mb, from dc2 to dc1 and it took around 6-7 minutes.  Over the network it takes much less...  What does that mean?

yo sbsc,
how did u prepare the disks raid?
best practice is to move all VMs on dedicated disk array/LUN
even better if exchange DB on its on dedicated disk

Based on what you have stated so far DC2 and DC1 are sharing the same disk subsystem, so this means you were reading and writing to the same place and thus creating contention.

However, I will say copying 600 MB in 6-7 minutes between two VM's on the same host is real SLOOOOOOOOOW.  Unless my math is way off that is between 11-13 Mbits/second.

Are all of the VM's on the same IP subnet?

There is one RAiD 5 with a hotspare which is around 800Gb
on which all the VM's reside.

What about the esxi itself?

It's on the RAID 5 volume as well

Even with everything on the same RAID 5 drive, 11-13 Mbits/sec is still WAY slow.  

In many clients t i supported, the host is normally on its own raid 1 disk, VMs on raid 5 array, DB on raid 5(tier 1)
So the host is actually better to run on its own dedicated disk
If you overcommit ram, its normal to experience a bit of slowness as the host & vm guests need to do swapping to disk, since all in the same disk, they are fighting to use the same spindle to read/write swap files
If you dont want to reinstall your esxi, add more ram to your host and avoid over commit of ram

I have another machine on standby, is there a better way to move VM's from one host to another? I used the converter, and exported the VM's from the old machine to the new one, but I can now possibly
export them back, and rebuild this DL585.  Since I have 3 machines there (the 4th one is idle) is the best way to do it is using the VMconverter?  
 
I will then rebuild the machine to have the host esxi on 2 disks, and the VM's on a RAID5.  is  that a better solution than adding memory you think?

If you are fine with downtime, you can just download to backup all the VMs data via veeam fastscp
You dont have to convert them, once the host is ready, just upload them again and reregister each VM vmx file

ryder0707; Im not so familiar with VMware, what is veeam fastscp ? i know scp from unix, where can I get this tool?

Its a free tool to manage esx/esxi files, its similar to scp but a lot faster becoz it doesn't use encryption
Get it from http://www.veeam.com/vmware-esxi-fastscp.html

Create pools . see attachments

give at your dc's at least the minimum needed memmory (depending of your active directory) and 1 cpu .

every pool sould be expandable .that means , if the resource is on the edge automaticaly the pool will request resources from the his above pool.

As for your exchange i recomend to give him 4gb (depend of your company size) and put it on a low pool.
Your current configuration dosnt load balancing the resources on your ESX server and your ESX tries to give 24/7 full resources.

Regards
tmc
pools1.JPG

edit 2
pools2.JPG
pools3.JPG

There are no pools in ESX free edition. I am still thinking of the SmartArray and I guesss it has noch BBWC an though noch write back cache.
Regarding exchange: It's nonsense to create separate disks for db and logs, as they will reside on the same physical spindles.

One would have to look at several performance values of the host an VMs, e.g. cpu ready time. iometer might be a good idea, too.

hm. never played with free edition on prodactive enviroment , anyway ,


The DL585 G2 includes a Smart Array P400 Controller which supports SAS storage. Some configurations of the DL585 G2 include a battery-backed write cache (BBWC) on the P400 controller. For models that do not include the BBWC, it is available as an option.

Sbsc can u define that pls? can u give a screenshot from the ilo ?

@ollfried

"Regarding exchange: It's nonsense to create separate disks for db and logs, as they will reside on the same physical spindles"
It is widely practice in enterprise environment, that is why there is a concept called "storage tiering", you can google for whitepapers & best practice

enjoy this and if you think separation of db & log to its own disks is nonsense, tell that to vmware http://www.vmware.com/files/pdf/solutions/08Q4_VM_Exchange_Server_2007_VI3_WP.pdf

The science is simple, more dedicated spindles means more performance

Cheers!

@olfried ......
again

My dear friend . attached is a screenshot from my lab esxi4 ......

Guess what , FREE Version called esxi4 SUPPORTS POOLS. SO if u dont have a clue about virtualization please do not reply .
pools4.JPG

@sbsc,

I would suggest you monitor the performance of the VM's and ESX to see if there are problems that you may not be seeing right now.  I can't think of anything that would cause RDP to be slow and everything else perform fine.  The next time RDP appears is slow you may want to jump on ESX and see if something else is going on.  

Although having one RAID array for all of this is not the best idea, unless something is really beating the heck of the the I/O subsystem it should not impact RDP traffic, at least not just RDP traffic.  It should impact everything.

Do you have any performance monitoring tools for Windows that show history?  If not, you may want to look into getting some.  You could install the SNMP agent on the Windows VM's and get MRTG and set MRTG up to monitor the each individual VM for CPU, RAM, and I/O stats.  Then do the same for the ESX host.  That way you get a total view of what is going on.

@tmcgr,

I see you are fairly new here.  Please refrain from questing others experience and knowledge.

Currently there is only 1 spindle, so creating multiple disk partitions on the same spindle is of no real use, which is what I beleive ollfried was referring to.  Now, redoing the whole array to create multiple spindles could help improve performance, but if the only current issue is RDP performance, that seems a bit over kill.

IIRC the free version of ESX V3 did not include pools, but I will say that I have only played with ESX of any version just a little.  
Knowing virtualization and knowing all of the specific functions and features of specific virtualization products (and all of their versions and releases) are two different things.  

You have come into the middle of a post about performance issues using RDP to VM's and made some good suggestions about changes to the hardware and ESX setup to help with performance of the VM's.  

However, as of right now sbsc has stated there are no reported performance issues with the VM's themselves.  I do feel 6-7 minutes to copy 600 MB of files between two VM's on the same box may be a bit long. You also have to realize that since they were both on the same box that would be 1.2 GB of I/O (600 MB of read on one VM and 600 MB of write on the other).  So we are talking about 1.2 GB of data transfer in 6-7 minutes.  Also, we have no clue how many files there were in that 600 MB.  It takes a lot longer to copy six-hundred 1MB files than it does to copy one 600 MB file.  So if there were a lot of little files, then 6-7 minutes may be O.K.

Therefore with no other performance issue reported, wiping out a whole enviroment and rebuilding seem a bit extreme to me.  

Depending on the growth of the enviroment your suggestions could prevent future problems.

EE welcomes all to participate in helping out, but you may want to refrain from commenting on other's knowledge and experience until you have been around for awhile.

@tmcgr: Yawn.
Seems like I mixed up Resource Pools on unmanaged ESXi and DRS-enabled clusters, my apolohgies for that. Anyway, ESXi does not mean it is free, you can license ESXi like ESX standard in enterprise environments. And you will need to if you want to use things like HA und DRS. So please, don't attack me like that.
And, again, using different virtuak disks residing on the same logical volume is nonsense.

The poster said, opening Explorer is slow and things like that. This stills seems to be an effect of resource contention to me. Either disk, or memory. The question about BBWC is still open, memory is definetely overallocated. That's why I spoke about cpu ready time for the VMs.

tmcgr: can you send a screen shot of what your High Resource pool looks like?  I created the high resource pool, see picture, are those settings right?
vmh.png

I plan to grow this installation up to about 20-30 users, then Ill buy a new server, set it up the way its recommended, and either export the VM's either by converting them or SCP'ing them.  however to get
decent performance for now, I think setting up the resource pools is a good idea.  Ill see how that works.

The file I was copying was 137 Files, 19 Folders.  So it may have been taking longer.

@giltjr
Maybe i was a bit aggressive, sorry for that . bad day after 4 days off from work (monday always the worst )

@ofried
apologies, (look above)

@sbsc
Yes keep that confiiguration for the time ,
Also , a sily question  , Check if by any chance u have Hyper v on the Microsoft Server accidently(just a thought)

tmcgr: no, I never installed HyperV.  

ok :)

I must be doing something wrong with these resource pools, I have 3 resource pools, see pix, but except for the exchange server, the DC is now a crawl, took like 2 minutes to log into the console (RDP)
 
vmware1.png
vmware2.png

To be honest, there's nothing much you can improve since you clearly over commit ram, RP is just to allocate/limit physical resources to pool, since you dont actually have enough physical resources, both host & VMs will go to disk for swapping
I'm quite sure your problem is due to over committing of ram and excessive disk swapping, perhaps you should start monitoring these

You should really start monitoring your resources and find the cause of your performance problems. Resource Pools will not help you, if your VMs use more resources than you have physically. What could help is reserving memory and cpu for the VMs.

Over committing RAM is not necessarily bad.  Its only bad if the total amount of VM RAM being used is greater than physical RAM for a long period of time.

However, as I stated before and other have stated, you will need to monitor all resources (CPU, RAM, I/O) of both the ESX server and each of the VM's.

When it was talking so long to RDP to the VM in question did you look at CPU/RAM/IO the on the ESX server?

For some reason, Im seeing alot of swap file used by esxi, I dont know why that would be.  It seems to me that 8Gb for these 3 machines should suffice, or am I wrong?

THe DC1 has 1 CPU and 4Gb Ram
The DC2 has 1 CPU and 3Gb RAM
the exchange has 1 CPU and 4Gb RAM now.

The DC1 and Exchange work fine now, much better after this resource pool setting, but the DC2 is lethargic....
see pic
swap.png

Do your DCs really need that much RAM? They should also work with 1GB, give it a try. Better let windows swap a bit, than forcing ESX to swap.

ok, I will try it now, should I give all these machines 2 CPU's or leave it at 1?  

I have 8Gb physical memory on the server, so I can give them 2Gb a piece, and they should still be fine, as
they will be using exactly the physical memory, right?

The DC1 I have is also a fileserver, and a print server, as well as DHCP and DNS.  thats why I want to allocate more memory to it.

Giving a VM 2 CPUs will allow it to take twice as much physical CPU resources as a VM with 1 CPU. For a simple DC I would take just 1.
Regarding RAM: VMware produces some overhead, so you can not use all of your 8GB. But, with vmware tools you take benefit from transparent page sharing, so commiting 8GB to your VMs should be okay. Just monitor at least the swapping.

Also check, if the DCs really need that miuch memory. Even with DNS/DHCP/CIFS it should run with much less than 4GB - and you do not have that much RAM physically.

O.K, you have swapped some, up to 1.7GB worth.  If you are constantly swapping, this is BAD.

You have 4+3+4, which equals a total of 11 GB allocated.  If each server only uses 75% of their allocated memory that over 8GB of RAM needed.

This means, as other have stated, that you need to either decrease the RAM you have allocated to the VM's, or increase the amount of physical RAM.

If you lower the amount of RAM for the DCs (as ollfried and tmcgr have suggested) and performance improves, then my guess is that you have been having performance issues with your DC's and Exchange all along but you just have not noticed.

How is your ESX swap file allocated?

You may need to go back and take tmcgr and ryder0707 and redo your SAN setup to create better allocation so that even if you are doing lots of I/O you can spread it over a couple of spindles to improve performance.

well, i disabled the one VM that's not used, and put 3Gb on DC1 and 1Gb on DC2, and see attachd pic:
new-resource.png

Wow, what?  I asked if the 4 VM was active earlier and you said it was not.  How much RAM did it have allocated to it and since it was active, how do you know it was not used?

You do realize that even if you think that a VM is not being used as long as it is active it still takes up resources (RAM, CPU, I/O).

Based on the drop in swap file in use that VM probably had 2GB of RAM allocated to it.

actually it had 3Gb, but it was just sitting there.  it seems like a new problem arose, I reduced DC1 to 2Gb and it doesnt boot up, it goes in loops and keep rebooting, I now pushed it to 3Gb to see if it makes a difference.

Regarding IO: We still do not know if the SA hast BBWC. Anyway, it is quite obvious, that memory is the botteneck for now.

which is curious to me, 8Gb is quite a bit isnt it?!   if not, what amount would you put for this setup?

exchange server to support up to 100 users (in the next 3 months)

and regular DC services as described above.  I thought 8Gb is an overkill! :)

The DC is still not coming up, I gave it back its 4Gb, but it reboots after staying at "Applying computer settings"   any ideas?

8GB is not much these times. I would use at least 24GB, or 32. Memory is quite cheap.

Boot your DC in safe mode and check the event logs.

I changed a registry value, for RPC, that must have done it.  Im more of a Unix guy, Im not so much into the intricate world of windows.....

What is the best way to back these VM's up? to take snapshots and store them elsewhere? will a snapshot taken will be able to fully restore a server like a backup?  

What RPC registry value did you change and what problem did it resolve?

The amount of physical RAM will depend on how much RAM each VM uses.  I'm surprised that a DC would need 4GB of RAM to even boot.

As for backing up:

http://www.vmware.com/pdf/vsphere4/r40/vsp_vcb_15_u1_admin_guide.pdf

I hope I am not wrong again, but VCB requires a licensed ESXi. Please correct me if I am wrong. Backup is one of the things you can't do very good with the free edition - except if you treat a VM like a physical server.

What I did that seemed to hang the server is add this registry value:  

 a. Be sure that there is an REG_MULTI_SZ entry created named NSPI interface protocol sequences at HKLM\System\CurrentControlSet\Services\NTDS\Parameters\ and the value set to "ncacn_http:6004"

this is from http://messagexchange.blogspot.com/2008/12/outlook-anywhere-failing-rpc-end-points.html

as Outlook anywhere is not working.

i agree with giltjr..... 4gb mem for a dc? to much for almost 20 users .u actully need 1gb per 1000 accounts.
the is something wrong with the installation of the DC's .
Did u virtualized your current enviroment or its from the scratch ?
If so , is it possible to install a new DC from the scratch at ESX with 1gb for installtion and 512  with 1 cpu .after the installation .
currently im at home and im boring to log at ESX to sent u screen shots.
I will sent you tomorrow my exact config per machine , storage and esx configuration. at least to show u how mine works.

I beleive that VCB does require a license.  I did not read the doc I referenced in detail, but I though it covered various methods of backups.

If you can survive a outage, you can always shutdown the VM and copy the VM image file.

But also this copy has to be done manually using the datastore browser.

yes, after the registry fix, I put 2Gb on DC1, 1Gb on DC2 and there is 4Gb on the exchange, so now its performing good.   The point of VMware though is to share all these resources seamlessly I thought... so if you have 2 servers each with 2Gb of RAM, but they use their resources 50/50, I thought 2Gb would be enough.
Their claim is that exchange performance is much higher on vmware than on a dedicated box, but Im not sure if that includes other VM's on the same host....

The claim that exchange's performance is better in a VM than on a physical box is not correct, if  we talk about similar hardware.

Was outlook working?

If you are having issues and you think it is with the RCP ports you may want to read:

http://technet.microsoft.com/en-us/library/aa997836%28EXCHG.65%29.aspx

The purpose of VMWare is sort of resource sharing, but it really is targeted at CPU sharing.  To a point sharing I/O and RAM.

The main purpose is to consolidate servers that are not using the full resources of the physical servers.  

You had Exchange running on physical server with 8GB RAM and two dual-core CPU (basically a 4-way box).  It obviously was not stressing the system's CPU or RAM or you would have never considered using it as your VM server.

If you beleive all of the papers on virtualization 90% of all distributed servers avg. between 5 and 10% CPU utilization with some spikes into the 50%.  Well based on that you could put 5-10 servers on a single physical box and save money (software costs and environmental).  As long as not all of the server hit their peak CPU requirments at the same time, everything is fine and dandy.

One issue with virtualization in the distributed world is that most distributed platforms have serious I/O constants, so you can't do anything that needs a lot of I/O.  Over committing RAM can generate I/O load due to swapping.  This is why you have to be careful.

I originally did not think your over committing was cauing the problem because I though "there's no way a DC need 3 GB of RAM" so I assumed that your two DC's were only using around 1 GB each.  Even if Exchange was using all 6GB with the DC's using my assumed 1GB each that would put you at 8GB, even with ESX's overhead it should have been able to handle that.

Some platforms (like IBM's zSeries, a.k.a. mainframes) have much better I/O capabilities and their virtualization OS (zVM) has much better paging and can over commit "RAM" in larger increments.  I know of one site that for their production enviroment over commits 2:1 and for their test enviroment they are at 5:1.  We are talking 700+ virtual machines for product on two zSeries servers (350 per server).  In the three years of doing this they saved USD $15 million and prevented USD $100 million in data center upgrades.

Oh, while we are at it, do you have a 3rd DC.  If not, its not really a good idea to have both your DC's virtualized  on the same physical box.  If you loose that physical box, you loose both your DC's, not a good thing.  :)

Just to clarify, vcb itself doesn't require license as it is only part of any licensed vSphere edition
It is actually possible to backup esxi with vcb but it's illegal to do so according to vmware and vmware dont actually hardcode the esxi to prevent you from doing so, i've tested this in lab environment
So if you are using the free esxi, you cant use vcb to backup your VMs, you cant even download the framework directly but ofcoz there is a workaround but it can be too complicated for a starter

If you need a quick backup of all your VMs, stop all the VM and download all VMs data via the free tool i mentioned earlier
Its good to see that your performance has improved so as you can see, over commit ram has it limits, its not wrong to use it but very important to understand limit, and if you ask my opinion about DC with 2GB, i think it is way too much
I can you tell that none of pure DC(no other services) in my clients environment has a DC with more than 1GB, so you can actually reduce it to 1GB each to free up ram for future VMs you might want to create.

Let me know if other things related to esx is not clear to you, I'll try my best to assist you.

Cheers!

ESXi has no ServiceConsole, so there is no ssh access for using WinSCP or FastSCP, unless you use unsupported workarounds,

:)

I know what I'm talking about dude
http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-articles/general/how-to-access-the-vmware-esxi-hidden-console.html

Fyi, veeam doesn't require you to enable SSH

any other comment?

@ollfried

You can learn how to setup ssh for esxi at http://www.yellow-bricks.com/2008/08/10/howto-esxi-and-ssh/

Let me know if you need help or post q Q under vmware zone :)

I know one can enable SSH in ESXi, but it is unsupported and I prefer to use supported configurations. I see that I am not too familar with the free edition as I normaly use licensed ESX(i) in managed environments. I still had in mind, that ESXi free has read-only access for rcli, but I see this is not true any more. Anyway, I think it is not the best idea to show a beginner how to use unsupported or even illegal methods to use a free product. If he needs enterprise functions, he should pay for it. He can still move to e.g. XEN which provides more functions for free.

Well did i say to use unsupported? Show it to me then

As mentoned earlier, he doesn't need to login to the unsupported mode, veeam fastscp will work just fine without it
But its a good knowledge to know how/what you can do in unsupported mode

Any other comment?

"Anyway, I think it is not the best idea to show a beginner how to use unsupported or even illegal methods to use a free product."

Fyi, unsupported is not illegal, I've been working with vmware consultant we always use unsupported mode for in depth troubleshooting
Good example, see this question https://www.experts-exchange.com/questions/24991151/ESXi-PANIC-Failed-to-find-HD-boot-partition.html
In that Q, vmware engineer actually requested the user to go into unsupported mode to check partition table, there is no way you can check/fix partition table just using vSphere client

and fyi, VCB is actually NOT the best idea to show a beginner how to backup VM
VCB itself is huge topic, if you really understand how VCB works, it requires 10x times more work then just copying files using veeam fastscp

and if you are not aware, it is not illegal to use veeam, its good product from one of vmware premier partner

As i said, anything you are not sure about vmware, you can always post a Q in EE under vmware zone

we should not bug this Q, excuse us sbsc

Cheers mate!

Great shots tmcgr.

sbsc if you look at the first 3 pictures tmcgr posted you will see what I am talking about dealing with memory over commit.

His host box as 20GB of RAM and (if I am reading the pictures correctly) has about 18GB reserved which is less than his physical RAM, so he is not over committing.  However, if you look at the pictures all of the VM's combined are currently only using about 2GB of RAM combined.

So, since all of his VM's combined are only using about 2GB of RAM he has about 16GB unused.  He could either give some of the VM's more memory or add a few VM's without stressing the system at all.  This does assume that at any single point in time that the combined RAM requirements does not exceed the 20GB on the system for any long periods of time where response time is critical.