Question

Network problems

Asked by: sminfo

I have a complex network configuration in my server farm. I have to Cisco Catalyst 6500, with two uplink in trunk (each uplink is LACP 802.3ad). These links are connected to Ethernet ports of each HP Virtual Connect of the c7000 enclosure. The link aggregation work fine and each uplink is active and in balancing load, the "Enable VLAN Tunneling" is enable because I need a trunk in the server NICs. Each server of the Enclosure receive in each NIC a trunk, of each HP Virtual Connect. The ESX 3.5 are installed in each server, the NICs form a teaming to load balance for the Port Group. Exist many Port Group to work with vlan. More or less this is the configuration and work fine, but I have two issues.
1.-Some days ago we most to do some change of links for migration in the server farm, and many of my ESX servers stop working. The Virtual Center lost the contact with the ESX host, and the Virtual Machine lost network connection too. I entered via iLo and the configuration seems to be ok, nothing change in the configuration at all. Curiously I can't make ping to any host in or out of the subnet, and in the physical switch does not appear the MAC address of any of this host or VM. I make a restart of network "service network restart" and "service mgmt-vmware restart" but nothing happened, but when I reboot the ESX server with no change in the network configuration the problems was solve. Everything returned to normal. Any idea about this problem???
2.-I am also having problems with VMotion now, the process of migration succeed but I lost the network connectivity of this VM with the external subnet, some time try to Vmotion this VM for another host and the problem continues, but when the machines back to the original host everything work fine again.
these are the problems that I would like to resolve, thanks in advance.
Best Regard.

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-02-05 at 09:51:34ID24116545
Tags

VMware ESX 3.5

,

HP Virtual Connect

,

Cisco Catalyst.

Topics

VMware

,

Networking Hardware

Participating Experts
5
Points
500
Comments
14

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. VMware ESX 3.0.1 No Ping
    I have a Dell PowerEdge 6850 that came preinstalled with Winddows Server 2003. I have removed Windows Server 2003 and installed VMWare ESX Server 3.0.1. The problem that I am having is that I am unable to ping the ESX Server, or on the ESX Server, I am unable to ping the ga...
  2. Vmware ESX 3.5 &  DHCP problem
    Hi, we currently running a ESX 3.5 Update 4 up-to-date with Windows Machine on it. The physical server is an HP DL380 with a broadcom NetXtreme card. Network driver card in windows machine is a Intel Pro 1000 MT as recommended by Vmware. We have a teaming of both network card...
  3. Network issue with Solaris 10 on VMware ESX
    I have installed Solaris 10 U6 on VMware VI3 ESX 3.5 U4. Both are fully patched and as up to date on drivers as possible. The server hardware is HP DL580 connected to the network with copper and 6 NICs or a HP Blade environment using the Flex 10 virtual connection. The cor...
  4. VMWare: Presenting more than one vLAN to VM Network
    Dear Experts We are experimenting with ESX 3.5 and have a dual nic host connected to a physical switch. At the moment they are connected to our server vLAN but i'd like to be able to present more than this vlan to the 2 nics. So in network adapters ive got vmnic0 - vswitch0...
  5. VMWare ESX IP Addresses
    I have a VMWare ESX server. The server has two nic, VMNIC0 and VMNIC1 The server VMNIC0 COS IP is 192.168.1.5 The server has 4 virtual machines running from 192.168.1.10-13 and all use VMNIC0 I would like to have a virtual machine use VMNIC1 and have an IP of 192.168.118.20 T...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: za_mkhPosted on 2009-02-05 at 10:09:16ID: 23562206

Well, the issue definitely has to be what you did to your cisco links for your migration. You will need to sort that out because if nothing else has changed on the ESX then the phyiscal network needs to be looked.

Since it is a bit difficult for me to understand your actual request, I would recommend you 'start' again with checking all your cisco switch configurations.

 

by: sminfoPosted on 2009-02-05 at 13:29:24ID: 23564519

I really don't think that the problems is on Cisco switches, because some ESX server work fine during the change, but other not. Even I have another c7000 Enclusure without virtualization (the configuration of the Cisco switches are the same in both case), and nothing happened after the change. I even came back to the previous network configuration but nothing work either for this server with problems. One of my question is that all the necessary for restart network service for ESX server are the commands "service network restart" and "service mgmt-vmware restart", I think not, because with the same configuration I reboot the server and everything work fine again . The other question is about the VMotion, why when I WMotion a VM the network stop working until come back to the original server.
Sorry for my poor English.
Best Regard

 

by: sminfoPosted on 2009-02-05 at 14:30:32ID: 23565057

I send a diagram to facilitate understanding of the network configuration.

 

by: cat6509Posted on 2009-02-05 at 15:42:29ID: 23565577

Nice Diagram!

a question about it though, I have the same enclosures with Pass-thru ethernet to a 6500, are you using the pass-thru ? If you are using the HP integrated switches, I belive they may connect on the backplane making a STP loop, which could give you similar symptoms to what you are describing

 

by: cat6509Posted on 2009-02-05 at 15:46:21ID: 23565597

sorry to add to my above:

do
sh spanning-tree detail

find your ports (lots of output sorry) and note the values for

   Number of transitions to forwarding state: X

Do you show a lot of transitions ? Are they incrementing? this can show a STP problem.
 

 

by: paulsolovPosted on 2009-02-05 at 19:04:46ID: 23566507

Check the simple things:

On ESX vswitches that are attached to your physical nic ports (which are aggregated via LACP on the switch side) make sure you're using IP HASH on the vswitches, otherwise this may cause funny things to happen

 

by: za_mkhPosted on 2009-02-06 at 01:05:13ID: 23567942

If I can ask a question on this too ... does STP not have to be disabled on the ports connecting to the ESX Servers? We have to disable STP for all our MLT links going to our ESX servers. We use Nortel Switches..

 

by: markzzPosted on 2009-02-06 at 06:57:22ID: 23570268

I believe PaulSolov has the correct answer for you..use IP HASH on the vswitches.
Another though.
Ensure your etherchanel groups are unique per physical switch and logical server.

 

 

by: markzzPosted on 2009-02-06 at 07:04:42ID: 23570350

VMotion wise, I'd suggest you consider separating your service console and VMotion activities from other Virtual Machine network activity.
By this I mean putting the Service Console and vMotion into their own dedicated interface and VLan which ideally would not be part of your trunk or etherchanel group..

 

by: sminfoPosted on 2009-02-06 at 08:50:35ID: 23571529

 
Thanks for answering.
Hi cat6509,
I do "sh spanning-tree detail" this is the output for the vlan of the Service Console:
Port 838 (Port-channel6) of VLAN0270 is forwarding
 Port path cost 3, Port priority 128, Port Identifier 128.838.
Designated root has priority 32768, address 0011.5df5.190e
Designated bridge has priority 32768, address 0011.5df5.190e
Designated port id is 128.838, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
The port is in the portfast mode by portfast trunk configuration
Link type is point-to-point by default
Bpdu guard is enabled
BPDU: sent 37903, received 0
This is the output of the vlan for the Service Console, in other vlan the "Number of transitions to forwarding state: 1" are the same. I'm pretty sure that the problem is not in Cisco. The configuration of the Port Channel is:
interface Port-channel6
 description "Encloser Konoha"(2/13-14)
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 spanning-tree portfast trunk
 spanning-tree bpduguard enable  
The HP Virtual Connect recognizes the Link Aggregation pretty fine (Active/Active), and is configured to work in pass-through with the Enable VLAN Tunneling enable.  
Hi za_mkh,  
As far as I know the HP Virtual Connect does not participate in Data Center STP, neither the ESX server.  
Hi paulsolov,  
Until now I always use the Load Balancing Routed based on source MAC address, I try with this now.  
 
Looking for an answer I see the following in each server with problems. One of the two has a correct range Observed IP ranges but the other card sometimes not see any range at all. Im trying to figure it out why.  
Thanks for everything; I hope that we keep talking.  

 

by: paulsolovPosted on 2009-02-06 at 09:13:25ID: 23571776

The route based on mac address will give you problems in a etherchannel/lacp configuration.  The IP hash should be used.

 

by: sminfoPosted on 2009-02-06 at 11:19:27ID: 23573144

Thanks paulsolov, I try your recommendation for now on.
Again about the "Observer IP range", something really estrange is happens. When I look for the range of certain NICs they are wrong. For example my IP class is B, and some VLAN has IP range in class C in my ESX hosts, this IP address does not exists in company. I have a doubt, I understand that the virtual switch know the MAC-address of all the VM connected to it. But how does the vswitch know the VLAN??? There is a command to erase this IP table??? Obviously the problem is that. When I try a VMotion to an ESX host with bad IP range in a NIC the VMotion is successful but I lost the network connection, so when I try with ESX server where the IP range in the Active NIC are right everything is ok.
To see the range what I talking about, follow this step:
1.      Click on host.
2.      Click on configuration, networking, Properties.
3.      Click on the tab Network Adapter.
4.      The last field Observed IP range.
Best Regard.

 

by: sminfoPosted on 2009-09-18 at 21:01:43ID: 31543339

I solved the problem

 

by: mikeleahyPosted on 2011-10-04 at 09:01:06ID: 36911052

how did you solve it?

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...