vmware san and fiber channel monitoring

rschwab
rschwab used Ask the Experts™
on
Hello all
My question is that i have a client running a foxpro database in dbf format.   They are acessing this data via network share.  It is virtual server utilizing vmware the server image and all the dat data resides on SAN, acessed via fiber channel.
the problem is the application is dragging and access time isn't what it should be.  My feelings are that a dbf style data base that needs a client on workstation to perform the processiing is taxing the SAN with just way to many read, right and data transfers to the client.  This coupled with their entire infrastructure servers (that are virtualized) could be causing some type of issue.
the performance monitors show no excessive utilization in processing, disk activity nic or memory.
I guess i'm wondering can the fiber channel bus  be saturated while the disks show low utilization in perfomance monitor?  And can this be viewed from a vmware management console
Or am i reaching hear?    I look forward to any and responses  
 Thank you all for your time and input
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Are you looking at local workstation performance monitor or at the server values?

I would guess the fiber channel is not the problem but some server diagnostic software will tell more.

Following points are focussed on the application itself but without its detail knowledge it will be difficult to help.

The application speed depends mainly on its architecture and the way of data access. As a rough estimation if your application is slow or not you can test the sequential data read/write speed by copying some file between SAN and your local workstation.

The FoxPro application:
1) Should be installed on local workstation
2) Place all temporary files on the local workstation
3) Should use indexes to optimize data access performance
4) Should not open/close shared files periodically
5) Should lock data records for the shortest possible period of time
6) Should not lock whole files in multiuser environment
etc.

How many concurent users are working with same data? You could also measure some multiuser data access (for different number of users) versus exclusive data access. (Exclusive access is much faster always but not 10 times - it could point to bad network/system settings.)
VFP being a file server database, will utilize the network more that a client server database. You will have to check the indexes and optimize the database.
Olaf DoschkeSoftware Developer
Commented:
As far as I know about SAN it's seperating the storage from the rest of the LAN and can make several raids or servers look as one towards clients, it's doing load balancing, mirroring etc. to speed up storage and data access.

There's one bad thing about this in regard to foxpro I think: Foxpro is doing some of it's essential things, eg locking table headers or records for saving data, via the file system. For example when locking a record (manually or automatic) VFP does a bit of trickery, instead of locking the bytes of the record to lock, it locks the byte with offset 2G minus rrecord number, even if the file is not that long. -because if VFP would lock the real bytes of the record it couldn't write to them. Other VFP clients trying to write to the record will also try to lock that byte at the end of the file. It might be, that this mechanism does not translate well to a SAN.

It's more of a "feeling" than knowledge, I don't know how file locks translate on a SAM. But whatever really is the matter, your experience shows the SAN does no good to the VFP clients and or database, so can you put the VFP databases out of the SAN into a separate server? It can be a virtual one, VFP has no problems running on VM as client or having data on a VM file server, at least that should not be the big problem. Of course it causes a llittel overhead and makes it slower than with a real server.

I second pcelba in all his recommendations, also try to measure the throughput of the network. I can't explain what you observer, that the fibre channel is saturated, while there is little processing or disc activity. Let me perhaps try to make a guess anyway:

You're having many single reads and locks and unlocks, DBF files might be cached in memory so the disc activity is low. Procesor load should be expected as low, because VFP is no real server, even stored procs code is loaded from DBC file to clients and excuted there, it's just you have a central file the code is stored in, so VFP is letter by letter implementing "stored" procedures. Don't expect a performace advantage from multiple processors on servers for a foxpro application, that's only good for doing several virtual VMs on one hardware server. SAN fibre channel and LAN performance obviously seems the bottleneck. Some of the receommended actions to do from pcelba will reduce the traffic, eg correct indexing. This might solve your problem, that this is the bottleneck.

Bye, Olaf.
Success in ‘20 With a Profitable Pricing Strategy

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Olaf, do you have some link or source of the "locking behind the end of file" concept? It sounds like a nonsense to me because if you lock something else than the physical record itself other (non-fox) applications could lock the same record physically and write to it...

I could imagine the lock of one byte with offset 2G minus record number as a quick test of record availability but after such test the physical record locking must follow (and only this physical lock allows the only one FoxPro instance to write data into the record).

I agree the locks after the EOF could cause some system slow downs but I have no data to confirm it.

Olaf DoschkeSoftware Developer

Commented:
This concept of VFP's locking is documented, for example here: http://fox.wikis.com/wc.dll?Wiki~RLOCK
The wiki also suggests to use FileMon to see what happens and explains why this is done this way, to still allow reading the locked record.

Bye, Olaf.

Author

Commented:
Guys thank you so much for your comments,   The fox pro app i'm refering to is an app that been sold to  around 200 customer it's been in service for about 8 years  each customer implement this app in their own environment.  on average each customer has anywhere from 20 to 80 users accessing it concurrently.   so the app is rather stable.   It just this environment that seem s to be dragging it down.

My thought is that the disk utilization within the vm environment is low but the fiber bus is what's saturated.
there is the possibility of network issues as well.  my other thought was to put the data on a physical box and move the server to the physical location of the bulk users.
Thanks Olaf, I have to confirm, you were right, VFP locks bytes from "2G down" only... And the lock means reserved access for one process. I supposed the file lock prevents write access on OS level. I don't know why I missed this info earlier.

Interesting behaviour is when you issue FLOCK():  VFP decided to lock 85 MB block somewhere around 2GB offset in my less than 1 KB dbf.

This simply means we can write routines which will ignore VFP file and record locking (maybe such routine already exists and its name is Clipper :-)

Still learning.
If SAN is the only environment component which causes slow downs then you have to find out appropriate settings how to avoid this behaviour.

I still think the fiber channel connection should be faster than anything else.

The vmware monitor must exist, experts are just too silent.

The test with another network disk is a good idea.

You may create a small test of FileLocking inside the file and behind EOF. You may use Win32API described e.g. here: http://www.news2news.com/vfp/?group=-1&function=230
depends on how many users are accessing that share, but it could be a matter of looking at how your servers are accessing those LUNs.  if your servers have two channel HBAs, it might be worth looking at the preferred paths for each, and trying to balance them out to spread the load.  what SAN do you have?  is there not diagnostic software for it?  when we had performance issues with a psql database we submitted the service log to IBM and they had some minor tweaks for us to try, although in the end it was a database problem, we were still able to fine tune our setup a bit with their assistance.

Author

Commented:
I guess i'll find out tomorrow,   There're placeing the data on a dell 2950 stand alone.  it's easy test so well see.  Stephen    it's a  Dell/EMC array CX3-20.  It's not my box but i'll get a better look at it tommorrow.  it wasn't clear if th were giving me performance info off the san or the virtual image
could always go the process of elimination route.  load esxi onto a box with local disk and import the vm from the SAN to server with local disk.  if problem persists, install OS from scratch on server with local disk and install foxpro database and see what happens.  sure it's not the best way, but you'll know exactly where your problem lies.

personally, i wouldn't run any database or high disk access application in a vm. i've had experience before with performance relating to sql and psql databases and the cause was vmware; the moment we switched them to standalone servers (still connecting to the SAN, not local disk) all the issues disappeared.

if you want failover and redundancy etc, you're better off with two stand alone servers running in a cluster when it comes to databases in my opinion.

Author

Commented:
Thank you guys the combination of all you knowledge help me to eliminate the vmware san issue at this point.  After getting both both termserver and production data off the the San the performance remained the same.   It seems a Dell 5424 power connect switch maybe the culprit.  Further actions are underway to segment the device from the switch.  as well as Dell providing software upgrades  "hhhhhhhhmmmmm wonder why that is "    Thanks again for all your help and confirming some of my actions    

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial