Link to home
Start Free TrialLog in
Avatar of compdigit44
compdigit44

asked on

Tracking Down IP in VMware

I have run into an issue in my environment where an IP was allocated to some device that was already assigned else where.

I knew there was problem when I tried to assign the correct IP to the rebuild VM and it would not connect to the network but change the IP to another one on the same vlans works. When I ping the correct IP it does respond and returns an address the is a VMware address 00:50:56xxxx. I worked with my network team and the helped me narrow this down to one blade. Here is the odd part I have gone through each VM and confirm the MAC assigned to the problem MAC is not listed. I even wrote a powercli script to check my who VMware environment and the MAC is not found yet you can ping the address. I cannot connect to the computer via UNC, Computer Management, SSH etc... but believe it is a windows server since  when I did a telnet to port 80 it is open and see it list Microsoft-HTTPAPI in the banner message.

I already try to run a port scan on the IP and it is useless and did not tell me much. Outside of doing a packet capture on the host any other ideas how I can find this problem server?
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

This is not assigned to a VMKernel Portgroup ?
Avatar of compdigit44
compdigit44

ASKER

Nope...
Thanks for your help as always Hancock. I thought about this as well by the host has over 25 Vms running on it and will take a while to do this one by one???
You should be able to create a VM, quick test VM, which has an OS which supports Wireshark.

Install Wireshark, connect the network interface to a vSwitch, using 4095 (ALL) VLAN Tag, that will sniff out all the traffic, connect to vSwitches, set to Promiscuous Mode, and let run....and log.

if you've done all the obvious, dump ALL MAC Addresses from VMX files, have Forged Transmits set, so there is no spoofing going on inside a VM? - checked this ?

then, dump and check your esx.conf, this is where ALL the MAC Addresses are held for HOST - you'll be surprised what you find in here!

Also if you want post the MAC Address here, and I'll delete it, as soon as I see it, and anon it!
I will try again what you suggest and post my results shorty...

Thanks Again..
One thing I noticed is the port group in question is part of a vDS which has been around for a while and the version upgraded a couple of times. Right now the port group is set to allow mac address changes and forged transmits..ugh

I did create a new port group with promiscuous mode enabled and seeing traffic from this IP but unable to see which VM it is really coming from ie: port
On another note when setting up the new port group it would not let me assign vlan 4095.. The highest it would allow me to set it was 4094
You should be able to assign the special VLAN ALL (4095)

User generated image
so if you have forged transmits allowed, an OS could override it's assigned MAC Address in the VMX file.

So you can detect traffic from this VM and MAC Address using wireshark ?
Always, thank you for your help. I am seeing traffic for the suspect IP/MAC but the VM name is thinks it coming from is not correct.

For example packet trace are showing IP 1.1.1.1 resolving to server when in fact serverA is really using another IP and MAC which I have confirmed in the guest OS since the server was rebuilt. Also we have checked DNS and wins and nothing is listed....

This is so frustrating..
I already tried to turn off the VM which Network Monitor resolved the IP to and it did not help... which proves that something else is spoofing the mac and iP
Right now I am logging into all VM that are use the vLan of the problem IP MAC and so far have found nothing..... ugh
Stupid question. when using promiscuous mode on a port group , the boundary is the vDS correct? It will not see traffic from another vDS correct?
Would doing a packet capture on the host help me to narrow down which VMware port the problem VM is using?
As I test I tried to disabled forged transmits then running the capture again and still setting traffic flowing from this IP and MAC.... :o(
Not really. Unless you start swapping things around, and turning on and off stuff.

You've identified this MAC Address is definitely coming from a VM ?

you did check your esx.conf for the mac address ?
I know for sure it is a VM since the MAC starts with 00:50:56...

I even disabled forged transmits and mac changes and still seeing traffic... I will check the esx.conf now
is this a MAC Address which is not observed other than in traffic, e.g. you can ping it , and the arp cache reveals this MAC Address ?

Well I could spoof a physical machine with the MAC Address 00:50:56

But you've seen this on your network, and tracked it down to using a vmnic on this blade ?

so it exists inside this blade ?
I checked the host in the problem cluster and did not see the MAC address in question listed. Also when I use powercli to search my VMware environment for this MAC is does not find anything... THis is just crazt
this is a puzzle, if it's a VM, which is spoofing this MAC Address ?

how is it getting an IP Address, Static or DHCP ?
The IP range affect do not use DHCP
I was going to suggest using http://www.xangati.com/

BUT, it does not look like, they automatically give you a FREE 14 day trial anymore!!!!

You could just use the software for 14 days..... now someone wants to call you!!!
yes.. so I do not know if someone misconfigured something or what
how about a quick WMI script to run through and inventory all the VMs ?

if the MAC Address is configured in the OS, it should appear to an Audit/Inventory....

BUT....this has me thinking, VMware Tools, should report ALL the IP Address VMware Tools find in the VM....

so you have the IP Address ? and MAC Address ?

So does it not display in VMware Tools e.g. inventory ?
Good thought about the script. VMtools is not show the IP in question
That is also odd, because VMware Tools, effectively just does a IPCONFIG /ALL and reports ALL IP Addresses, it's very dump, it's not doing anything like working out if they are in use.

So you will often get odd IP Addresses appearing in VMware Tools, which are not in use, e.g. internal, loopback, IPv6, Cluster Private addresses!

In fact we have a Datacentre, where we came across an IP Address we could not use, it conflicted with an "IP Address in use", but it had NO MAC Address in any ARP table we can find....

In our Inventory, against this IP Address which is black listed, we wrote "Dragon's be er!"
I am completed frustrated and stumped at this point
One thing I have noticed it this problem mac / IP is sending a lot of UDP traffic to 224.1.0.22 which I believe is a multicast address. But do not help me find out what is using this ip / mac
You've not got Network Load Balancing configured have you ?

This needs a custom MAC address, for Mutlicast Operations!

which should be reserved in DHCP, and DNS Entry for the IP Address.
NO Windows NLB setup in the cluster the VMs are in...
What I find troubling about this issue is the fact when I disabled allowing MAC Changes and Forged transmits on the port group using the vLAN in question it had not affect and could still see traffic flow from the IP..

This tells me with the MAC is not a VM or the VM is on another vDS that is using the same vLAN.
Yes, that's possible.

have you had a look at physical switches, and checked where the MAC Address appears , on which port ?
yes we have and the switch port our network team did narrow this down to a blade but we checked all VMs on that blade and found nothing. Also I exported a list of all VMs from each vDS using the vLan in question and still did not find the IP or MAC in question...

So just for my own knowledge..
1) With promiscuous mode the boundary is not the vDS any if more global if that correct? For example if we have multiple vDS that have similar port groups and vLAN the sniffer could see traffic from the other vDS's correct?
Well here is bazaar one for you. Our network team was certain the IP/MAC was coming for one specific blade so this evening we powered down all VMs on the host and could still ping the IP. It wasn't until I rebooted the blade did the pings stop for good.

How and why? I know ESXi runs in memory is it possible something screwing is happening here hardware wise?
ASKER CERTIFIED SOLUTION
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
wow!!!!!

Thanks again for all of your help as always
no problems, keep a check for if it comes back....