Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Dropping Aironet device tanks SQL comms

Posted on 2013-06-11
7
Medium Priority
?
329 Views
Last Modified: 2015-06-22
Have a very strange network issue that I have not been able to track down. I was tasked with replacing an older aironet device with a newer device. I have the new device up and running, however when I drop power to the old device, within seconds the company's main line of business application goes down with issues talking to the SQL back end. Power up the Aironet, problem disappears.

Topology:
2k8r2 server, dc, running hyperV, dhcp server, dns server  at .3 on the scope
2k8r2 VM, dc, running MS SQL 2008 at .4 on the scope
2k8r2 VM, member server, running an app not yet deployed to floor at .6 on scope
Aironet 1200 series at .242 on scope,
( I didn't design it, I just service it, so be kind on your criticism)

Captures on the server links showed only one comm to the Aironet Ethernet address, that being a single ping. There does seem to be an increased amount of Arp traffic on the servers when the device is dropped. Hooked up a workstation on the ip  to see if there was any traffic targeting that address. Only comms seen in the packet trace were DNS queries and some RDP traffic back to one of the servers. I need to repeat this experiment, as the system seemed to operate with the workstation holding the 242 address open, however there is a gray area from when I dropped the address and started seeing the failure again.

The devices talking via the wireless on the floor are milling machines, wnd there does not seem to be any push communications that would require a constant link. (in addition, all the mills appear to be routing through the new aironet, leaving only someone's droid on the old system)
0
Comment
Question by:box-bb-car
7 Comments
 
LVL 17

Accepted Solution

by:
TimotiSt earned 1000 total points
ID: 39240372
Maybe the device at .242 is doing ICMP redirect or proxy ARP for some weird traffic?
Search the registry of the SQL server for the IP address, maybe you get something useful?
Mirror the port on the switch, capture traffic with a linux+wireshark combo, to make sure you capture everything?

Interesting, anyway... :)

Tamas
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 39240470
Is the old one the default gateway for something?
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 39243916
As TimotiSt said, the Aironet AP is probably running Proxy-ARP.

For some strange reason, Aironet APs run Proxy-ARP by default in some firmware versions.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:box-bb-car
ID: 39244295
Never thought about the proxy arp. Would explain the increased around on the server. Am also looking at the virtual switching/networking as am seeing some event log entries with the mac mapping. Mirroring the ports not an option with what they have in place, will have to see if I can get a temp device in. Thanks all, will update when I get a chance to try these suggestions
0
 

Author Comment

by:box-bb-car
ID: 39337068
Arp may or may not have been the cause, probably will never know, as problem resolved after I flushed the arp caches on reboot and left the device off . Sill going to award points at least for the effort
0
 

Author Closing Comment

by:box-bb-car
ID: 39337076
As stated, cannot prove this was the answer, but am awarding points
0
 

Author Comment

by:box-bb-car
ID: 40844377
Update on this issue. Though the accepted solution did give brief relief, the issue raised it's ugly head again several months later. The client had since hired an in house IT person, however the issue was beyond his expertise and we were called in to assist. During the subsequent investigation (over a period of two days) we were able to further characterize this issue, and discovered a flood of discovery packets from a recently added MakerBot. It gave the appearance that we had a small workgroup switch had gone awry and was flooding the network. The in house IT swapped out the switch and removed the makerbot driver from all units to verify we did not see the flood again. Late that same day the issue occurred again. The in house IT had initiated a couple of coops tracing down and identifying cable runs, and during their plugging and unplugging, the net had gone down, and they discovered a previously unknown piece of equipment on the network. The device had been hidden underneath a wire bundle, and had been put into place by one of the machine vendors to translate RS232 coming from the machines to ethernet. In looking up the specs on the device, we discovered that of the two ports on the device, one was for uplink to the network, the other was for daisy chaining subsequent devices. The Vendor had plugged both into the local switch creating a broadcast loop. If ANY device on that switch restarted, the suspect device, essentially a hub, would broadcast, hear it's broadcast and repeat, in a loop flooding the network. It would create so much traffic that it would overwhelm the virtual switches in HyperV, shutting down access to all servers on the host. The Makerbot broadcasts, even though not attached to the same switch initiate the same affect through the device. We have since superseded the in house personnel and have a more controlled grasp on network operations.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article provides a convenient collection of links to Microsoft provided Security Patches for operating systems that have reached their End of Life support cycle. Included operating systems covered by this article are Windows XP,  Windows Server…
This program is used to assist in finding and resolving common problems with wireless connections.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…

971 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question