Something is closing ports on our network.

I've set up 7 RFID readers on our network.  They are generating more traffic than normal as I am polling them one a second to grab any RFID tags they may have read.

All of a sudden, after 15 hours or so, all readers stopped working with a "socket" issue.

All readers have a static IP address and operate on the same subnet as the reset of the network.

I though that maybe our firewall detected this extra activity and, perhaps, shut down the ports.  I reached out to our firewall people and they said that this wouldn't be a firewall issue as this is all internal traffic.  

I changed the IP address of one of the readers and it started working again.  Changed it back and it quit.

The firewall people stated "This would be more so pointing to the switch than the firewall as all local broadcast traffic takes place on the switch and would't be touching the firewall. You may check in to broadcast storm or some kind of switch setting that would be dropping the traffic. You may even consider peeling this system out on to its own network / interface and decrease the size of the broadcast domain to help improve performance. Hope this helps steer you in the right direction."

The switch shows "Ingress Control Mode", under Storm Control, as Disabled.

Any thoughts here?
Sheldon LivingstonConsultantAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Nick UpsonPrincipal Operations EngineerCommented:
does the network have an intrusion detection system?
0
Sheldon LivingstonConsultantAuthor Commented:
No IDS but I did find a second switch... an HP 1820 that I, of course, don't have the correct password for, that could be the culprit.
0
Wayne88Commented:
If these are internal devices then yes it wouldn't have anything to do with the main firewall.  It would be internal.  Try turning off Windows firewall and any Antivirus or Endpoint Security Software for testing.  These can block workstation ports.  Do you used managed switches?  Can we get an idea of how big your network is and the layout?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Sheldon LivingstonConsultantAuthor Commented:
I believe this now to be an HP 1820 switch issue.  But alas... I don't have the password to it.  The switch doesn't have a "clear password" method.
0
nociSoftware EngineerCommented:
It should be able to be reset to completely empty..., then you loose the complete confiiguration incl. VLAN's...
The exact procedure depends on model, firmware etc. See here:
https://theitbros.com/reset-a-hp-procurve-17001800-switch/
0
Sheldon LivingstonConsultantAuthor Commented:
Yea... just not sure of the config...
0
nociSoftware EngineerCommented:
I have seen issues with HP Procurve switches loosing UDP traffic with non standard priorities. ( SIP traffic).with loop control enabled.
But i haven't seen them loose TCP traffic.

Can you make traffic traces (tcpdump/tshark/wireshark from a copy/mirror port?) that might show more then just trying to guess.
 reset, icmp ... etc. should be visible.  (Try to get as close as you can to your RFID reader (network topology). and log on your server.
If there are differences in traffic patterns it should show. (ie. a device in between interfering). If traffic patterns are equal  it is within the endpoints...) as seen from the point of view from you measure points.
0
Sheldon LivingstonConsultantAuthor Commented:
Turned out to be the switch.  I had to reset/reconfigure the switch and now all is good.
0
Wayne88Commented:
Glad you were able to solve it and thanks for getting back. Cheers!
0
nociSoftware EngineerCommented:
thanks for the update.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Networking

From novice to tech pro — start learning today.