Domain Computers Randomly Lose Internet Connection

I am running Windows SBS 2011 w/ Exchange on 1 server. This server provides DHCP for about 50 nodes. Random computers at random times will lose internet access (yellow explanation point in network icon). All intranet services will work, but user cannot access internet.
1. This happens randomly
2. After a while the computer will reconnect to internet
3. I think it is a DNS issue
4. Server DHPC does not show anything abnormal
5. User can ping internal IP's EXCEPT the gateway (gateway is a cloud firewall - which has been tested no issues)
6. Release and Renew IP does nothing to fix it
7. grabbing a new IP does not fix it

I need some help please.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

This sounds like a problem that I used to see on firewalls configured with a 10 user limit.  In order to enforce the 10 user limit, the vendor simply limited the number of ARP entries to 10.

This doesn't sound like a IP conflict to me, or else you'd be having trouble reaching things other than just the gateway.

Run arp -a on a machine experiencing the issue, and on a machine not experiencing the issue and see whether they both have the same entry for the gateway.  If you can check the arp table on the gateway at the same time, that would be ideal.
njpetrucelliAuthor Commented:
Hi asavener,

Thank you for the quick response.
I did as you asked and one node (experiencing the problem) did not list the gateway IP in the arp -a.

The gateway is a cloud firewall (managed by ISP) I have been through the run around with them, but they state traffic is going in and out properly. I would not expect the nodes (experiencing the problem) to ping or contact the gateway when this issue is happening.

i am really not thinking this is an issue with the router (gateway).

It is acting like the name is not resolving properly (DNS issue)
njpetrucelliAuthor Commented:

I found another post on a similar issue having to do with WiFi users.
I disconnected both my AP's and low and behold - the problem went away.

Perhaps an android or IPhone user is causing an IP or MAC duplication?

I need these AP working - how do I go about figuring out which phone is causing the issue?
The AP's do not hand out DHCP, just WiFi to users. both AP's have a static IP and are excluded from DHCP on the server. Could I have a rouge phone causing this issue?

Need More Insight Into What’s Killing Your Network

Flow data analysis from SolarWinds NetFlow Traffic Analyzer (NTA), along with Network Performance Monitor (NPM), can give you deeper visibility into your network’s traffic.

njpetrucelliAuthor Commented:
Never mind: It is still happening even with the AP's unplugged. Back to the drawing board.

It is really strange. One node I am working on had internet access without a problem, and randomly just stopped and threw an explanation point through the network icon.

DNS has both a forward and reverse A record for the IP. Weird
It's not a name resolution/DNS issue.  The client has no need to resolve the name of the gateway to an IP address.

Try adding a static ARP entry to an affected machine.  That MIGHT be a workaround for the problem.

I suppose it is possible that another machine is getting assigned the IP address of the gateway, but that would result in it having a different result in the ARP table.
njpetrucelliAuthor Commented:
Thank you for the suggestion, interestingly enough, I reset my switches this morning and everything cleared up.
I do not think I have a bad switch (all 3 are brand new) but I had them daisy chained - now each are plugged directly to the adtran, maybe this will help me localize the issue (if it is in fact a switch).

Could a bad switch cause random issues like this? Wouldn't you expect it to work or not work, not be random? (these are unmanaged)
It doesn't sound like a bad switch.  Typically what you would see is a bad port or bad bank of ports, where there's no traffic and/or no links.

Connecting the switches in the wrong way would typically result in a broadcast storm, and little or no traffic would make it through.

Really looks like an ARP resolution issue.
njpetrucelliAuthor Commented:
When I try the arp -s workaround, am I adding the ip and mac of the router (gateway)?

Is this a router issue, do you think?
Yes, add the IP and mac of the gateway.

My guess is that it's an issue with the router, although there could be something else causing the problem on your network.  

Are you familiar with how ARP works?  

If not, here's an overview:
njpetrucelliAuthor Commented:
I just got off the phone with ISP, they also think it is an ARP issue (kuddos asavener)

The adtran has a limit of 100 ARP MACs, there is a chance I am exceeding that limit. The adtran, according to the ISP, has a MAX limit of 100 and cannot be increased. However, I have a hard time believing that I am hitting that 100 max mark that often, if at all. I could see if the device held the addresses for extended periods of time, but the leases are 10 min.

Why do you think restarting the switches fixes the problem temporarily?
njpetrucelliAuthor Commented:
arp -a on down devices shows the correct IP and MAC for the gateway (I checked that last night)
Yeah, if the limit is on the adtran then the PCs could have a valid ARP entry and still not be able to communicate.

Your best bet is to just put a router between the adtran and the rest of your network.  Then the adtran only has one MAC address to keep track of.

Probably can be a very simple router.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
njpetrucelliAuthor Commented:
Thank you, this has resolved the issue!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.