Intel Switch "Out of Pools" error
Posted on 2006-05-01
We have some ancient Intel 510T switches that used to work great but are now having some problems. The whole switch will randomly started dropping packets, and will report "Receive Discards - Out of Pools" error on only one particular port at the same time (it is one of the busiest port because the load balancer is plugged in there). Looking at the switch's manual, it says these errors "shows there are no memory pools left because there are so many frames stored." And then the next line says: "Significance: The switch tries to cause collisions to increase the number of frames rejected; this gives the pools time to empty."
Since the timing of the packet loss corresponds to our load balancer log (about it getting disconnected from the firewall, likely the result of packet loss) as well as the switch increasing the count of "Out of Pools Receive Discards", I presume they are related... However, what does that error really means, what may have caused it, and how do I fix it? Does it means I am reaching the limit of this switch and I will have to upgrade? Would changing the switch mode from "cut-through" to "store and forward" help the issue? I didn't know if "out of pools" is a standard networking-speak that have common solutions out there... especially the manual's explaination sounded kind'a vague to me? Any advise will be greatly appreciated.
By the way that port in-question normally only puts out ~120 total packets per second, and when the error occurs the swich is usually not under load. We have load-tested it to about 1800-2200 packets per second doing several large downloads at the same time, so it would seems to me that the problem is not throughput related... I mean, 120 packets per second should be tofu for these switches, no?