Major Wifi issues leaving me scratching my head

I have a campus of about 3 building. I have a total of 22 AP. A mixture of AP541N-A-K9 Dual Band Access Points and Enterasys RoamAbout AP1002 AP's. Almost dead even at 11 on each kind. I have been running 2.4 gtz wifi singnal for about 2 years now. I have a multi OS enviroment from Macs, PC's, iPhone's, iPads, and misc network devices. I run about 240+ workstations as well as a free wifi signal which gets about 600 - 800 connections each day. I run my DHCP lease time at 24 hours for this reason. Never had an major issues with this setup.

Here is where things get weird.

2 weeks ago Fri March 30th 2012 at around 5:00 pm my wifi goes down. I spent all night trying to get it back up and I couldn't figure it out. I went home to bed and was going to come in to work on it the next day. I get in and what do you know, the wifi is working fine. I check the logs and I can pull up is a ton of clients authenticating just fine to the wifi.

Yesterday BOOM!! the wifi goes down again. Same thing - so now I run some more "process of eliminations". Here is what I have discovered so far

Hardwired workstations not effected
Hardwired workstations connect and receive IP's just fine from DHCP server
No device can connect to 2.4 GHZ 802.11b/g/n
If I switch to 5 GHZ for Wifi, I can get supported devices to connect??????????????????????

If I run a rouge detection I get nothing on my network, but there is a Non Broadcasting SSID full bars that my AP's are picking up

Here is the entry from my AP picking it up (Full Bars)
00:18:e7:d5:18:5b	100	AP	(Non Broadcasting)	On	On	2.4	6	1		2393	Sun Jul 19 10:03:27 1970	1,2,5.5,11,6,9,12,18,24,36,48,54

Open in new window

I just dont get it, 2.4 nothing works. 5 it works. Am I have an interference issue? that would knock out 3 buildings within 3-4 acres? I need some ideas :)
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Kyle AbrahamsSenior .Net DeveloperCommented:
Was the 2nd event around 5:00 PM as well?

Can you confirm the outage is across the entire campus?  What happens if someone was connected previously to the outage?  Did they notice an interruption?
cbielichAuthor Commented:
I cant confirm the 2nd time. It could have been 5:00 PM the previous day but didnt realize it till I came in the next morning.

Yes I cant get any devices to connect campus wide to 2.4 in any building

Previous users do notice the service interuption
Kyle AbrahamsSenior .Net DeveloperCommented:
You said they were all APs . . . Can you provide a generic network map?

What do they all have in common?  Are they all on the same switch or anything?

I wonder if some signal is being sent out to cause the interuption.  

Interesting problem to say the least.
Defend Against the Q2 Top Security Threats

Were you aware that overall malware worldwide was down a surprising 42% from Q1'18? Every quarter, the WatchGuard Threat Lab releases an Internet Security Report that analyzes the top threat trends impacting companies worldwide. Learn more by viewing our on-demand webinar today!

Is it possible that the non-broadcasting AP is interfering with your AP broadcasts?

When you say non-broadcasting, I presume you mean that it's simply not broadcasting its network name - it's clearly broadcasting a signal and a very strong one at that, if full signal strength bars are showing.

Has anyone on the campus decided to do his own thing? Perhaps someone who has moaned in the past about the wireless, and decided to "fix" it himself, by getting his own supa-dupa access point and plugging it into a LAN socket somewhere? As you have a MAC address for the AP, you should be able to find out who makes it, which might make identifying it a bit easier, especially if it ain't one of yours...
cbielichAuthor Commented:
Yes the SSID is not showing. I did find the device, it was just this AP for some sound equipment. I unplugged it and still no luck. Even if it was that would it effect 3 buildings that are at least 400 meters apart from each other.

All my building are connected through fiber

Let me ask this:
Is there something that could cause a device not to connect to a  2.4 signal that can be disrupted from something on the LAN? For instance a broadcast storm (which I am not picking up any).

Are there any suggestions to software I can run on a mac that can scan signals for me so I can troubleshoot that. I used to use one and I cant remember for the life of me what it was called.
Kyle AbrahamsSenior .Net DeveloperCommented:
I was taking it from this approach (again only theory):

Something broadcast over LAN which tells APs to reboot 2.4ghz or drop connection or attempt update.  

Also the obvious question:  What was the last thing done to the network?  Was a new AP added?  Something configured?  New Firewall Rules?  ISP Change?  AntiVirus Rollout?

I don't know of anything that would interfere that far across the 2.4ghz band.  Is the campus a school or involved in any research?  

Wondering if someone can be "flipping a switch" and in advertently broadcasting something.
cbielichAuthor Commented:
Thinking as I type :)

Whats the weirdest is I have 2 different types of AP's. They don`t talk to each other yet both experience the same issue. I am convinced my AP's are not the problem.

Cant connect to 2.4 but can connect to 5, if one set of AP's were having the issue then maybe could be the AP's but since all of them are having the exact issue again I am ruling this out.

I think logically I can rule out the AP's as the problem. In doing a wifi scan with wifi scanner. I am not picking up any other strong signals in my area.

I think I am left with something in the LAN is causing my issues. Yet no computers on the LAN side are effected in any way that I can tell.

Just talking out loud :)
When you say the wireless goes down, does that mean no signal, or that clients can't connect?

Do the APs get their information from a central source?
cbielichAuthor Commented:
Signal is up and seeable, clients cant connet ONLY to 2.4. Not 5

Enterasys Roamabouts do not have a central

Ciscos cluster together so I can log into any interface and change a setting and it will push it out to all the others.
Kyle AbrahamsSenior .Net DeveloperCommented:
Are we talking two different subnets / vlans for the different IPs?

Just to confirm when it goes "down" you can still see the wireless network as an option, but upon trying to connect it won't.

I don't think it's a problem with the APs themselves but something triggering the APs to a certain behavior.
cbielichAuthor Commented:
Ok just now my co worker was able to get a PC to connect to the signal and get an IP. Did nothing special but it did connect. The signal was so weak (1 meg) that he cant even ping anything.
Kyle AbrahamsSenior .Net DeveloperCommented:
It's currently happening?  Are you experiencing any sort of brownouts or anything?

If you reboot one of the APs . . . does that fix the signal?
cbielichAuthor Commented:
No brownouts, the signal is to weak to scan with it and I have rebooted many times :) no fix
cbielichAuthor Commented:
Is there software that can scan frequincies?
Kyle AbrahamsSenior .Net DeveloperCommented:
Fred MarshallPrincipalCommented:
Get NetStumbler.

The type of thing you describe is most likely radio frequency interference (RFI) which can be generated by noisy devices or by radios on the same frequency.

You haven't mentioned which 2.4GHz bands you're using.
If many then the "noise" would be broadband.
If few, then have you tried other bands?
Why did you select the bands you did select?
Or, are these things on "auto" as far as band selection.  I've seen auto cause dropouts so I tend to not use it.  I suppose some day I may try again.  Usually I just pick a band that is empty or with the lowest signal levels.
NetStumbler will show you which bands have signals and will show you the signal to noise ratio for each particular "station" / SSID.

I don't think you can do without it here......

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
cbielichAuthor Commented:
So after a long hard 1.5 weeks of racking my brain trying to figure out what was happening to my wifi issues I finally found my problem and fixed my network. I want to share my story because it was nothing I had come across before in 12 years of Systems Administration.

Basic story was that all of a sudden wireless devices could not get connected to my 2.4 ghz frequency, if I switched to 5 ghz users connected fine. At the same time all hard wired users were fine as well. Of course thinking this was a frequency issue that was happening to all my devices across 3 buildings within a 6 acre area. In talking with some neighboring buildings they were not experiencing any disruption in their 2.4 ghz frequency so I was going nuts. I was trying to figure out why users could not connect to my 2.4 ghz, but could connect to my 5 ghz wifi.

Here is what happened:

I finally convinced my company to let me purchase a very good and pricey software Solar Winds Orian Network Monitor. I installed it and discovered all my SNMP devices on my network. From Switches, Printer, Server...everything. Once I had it running I realized there was a bandwidth utilization being broadcasted at about 15% across the board. I was able to identify the switch and port where this data was coming from. Once I found the outlet there was a Mac Mini connected in a soundbooth in which is used for media production. The Mac mini was running Quicktime Broadcaster which is used to stream live video. The media department was experimenting with streaming some services we provide live. Well the quicktime broadcaster was left running for this entire time. Someone forgot to shut it off. For what ever reason the broadcaster was using about 100 mb of bandwidth which it was shooting across the network. Once I shut it off BOOM! the wifi came back to life. Without doing to much research on QT Broadcaster here is my idea what was happening.

Because 2.4 ghz utilizes a max of 54 mb of bandwidth, with users on average connecting at 40 mbs this would cause users to have a hardtime connecting to the network because of the bandwidth being used exceeded what the frequency could handle. Since 5 ghz can connect at a higher rate those users were able to push through the bandwidth utilization and get a connection. Also since my wired computers connect at 1 gig there was no competition for them.

So there you have it, I hope this can help someone in the future :)
Fred MarshallPrincipalCommented:
Glad you found it.  Thanks for the points.

PRTG, the free version, would likely have done the same thing for you.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Wireless Networking

From novice to tech pro — start learning today.