Issues with isolated network segment for SAN for VMware environment

We have created an separate network for our Eqaullogic SAN traffic to our VMware 4.0 environment.  The network consists of 2 cisco 2960G switches (SanSwitch1 & SanSwitch2) that have ports 20-24 configured as etherchannels between the two switches.  These switches are configured for multi-path I/O.   Each of the three host servers (Dell 2950) have a total of 4 NICS installed.  Nic1 & Nic2 are connected to SanSwitch1 and SanSwitch2, respectively.  The Equallogic SAN has active/standby controllers. Controller A has 3 ports port 1&2 connect to SanSwich1; port 3 connects to SanSwitch2;  The standby controller B also has a total of 3 ports.  Ports 1$2 are connected to SanSwitch2 and port 3 is connected to SanSwich1.  In VMware each of the hosts are configured with multiple vswitches.  Vswitch0 is for the production network, Vswitch1 for the SAN network, and Vswitch2 is set to connect to our secure network.  Vswitch1 consists of a Port group called Guest SAN Network and that port group has the two nics assigned.  Everything seemed to be working well.  Or so I thought.  I introduced a forth server to build a small test environment, loaded an eval version of vmware.  I setup a separate network on an independent switch to mimic our Production environment,  But the I created connections to the SAN based on the same settings as the production servers.  No connection to the secure network was neccessary.  Again, everything looked good up to the point I introduced a copy of our domain controller into the test environment and received a warning that the computer name existed on the network.  Now I am trying to figure out how production traffic is getting onto the SAN network.  The guests on the Vm ESXi servers all use the SAN to boot.  I can only guess that somehow the traffic is getting directed on one of the guest servers.
dalsymDirector of ITAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Paul SolovyovskySenior IT AdvisorCommented:
Sounds like your SAN network and your production may be routing on the physical switches.  Make sure you cannot ping the standalone vlan.

What I would recommend for the test vswitch is to create one on the test ESX host with no nics attached, just servers to test.  You can then clone your production VMs and do all the testing without impacting production network.  

See if it's routing and we'll go from there
dalsymDirector of ITAuthor Commented:
More information:  I cannot ping any address on the SAN from a Guest O/S in the production environment.  I also negelected to mention that we have two hyper-v servers in the production environment as well.  These servers also have connections to the SAN network.  I can ping an address on the SAN network from a guest O/S on the the Hyper-V server.  I have shut one of the servers down and can still ping.  I cannot shut the other one down just yet to test.

I did verify that the traffic is going over the SAN network by diabling the vswitch on the test box.  If disabled then no traffic was passed.  I will setup the vswitch with no nics.  This will not let me test connections with the SAN.
Paul SolovyovskySenior IT AdvisorCommented:
Check if you have a VM with nics on both subnets (SAN and Production Network) It may be bridging the two

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
dalsymDirector of ITAuthor Commented:
Well, this is embarrassing.  I have gone through each of the host ESXi servers and reviewed their nic setups,  reviewed each of the host machines running and also reviewed the Hyper-v servers and did not located any bridge (side note:  one bridge does exist on a Hyper-v server but it is bridging two nics that are both connected to the SAN Network on two different switches i.e., sanswitch1 and sanscitch2).  I made no changes to the envirnonment.  I started testing again on Friday and the issue has gone away.  What is truly worse is that because I don't know what the source of the connection into the production network I am very uneasy with my test environment.  I am going to keep looking though.
dalsymDirector of ITAuthor Commented:
We didn't solve the source but paul provided direction to review.  Points awarded for best and only attempt to try and solve issue
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Architecture

From novice to tech pro — start learning today.