e1000 tuning

Dooglave
Dooglave used Ask the Experts™
on
Anyone know where I can educate myself on how to calculate proper values for the three parameters below.

RxDescriptors = 4096 max (default 256)
TxDescriptors = 4096 max (default 256)
RxIntDelay

How do I take into account throughput, latency, packet mix, etc.. and calculate the best guess value to use.

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Duncan RoeSoftware Developer

Commented:
The best guess is to not specify any of them (so take defaults). This advice is also to be found in /usr/src/linux/Documentation/networking/e1000.txt (or wherever you have your source tree).
It's a matter of optimization, mainly CPU, vs network bandwidth.
A small interrupt delay should lead to more CPU and good network latency even in low usage situation.
A longuer interrupt delay optimize the CPU (less interrupts) and allows for a maximum network bandwidth but deliver longuer latency in low usage situation.

You can read the full INTEL description of the e1000
Basically, you can raise the descriptors at the small cost of more system memory but should let the RxIntDelay to its default 0 value if you don't want to risk loosing packets
Top Expert 2015

Commented:
Linux TCP stack gas transmit queue. tx buffers would back tx queue. rx buffers would queue a lot of packets for apps to use. (at least a half second of data with 4k buffers.) and you lose interactivity of VoIP etc.
I would suggest to keep options at their defaults unless solving an immediate problem.
Become a Microsoft Certified Solutions Expert

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

Top Expert 2014

Commented:
Are you having a problem?  Or just trying to prevent one?

Author

Commented:
Solving a problem. RX drops, little CPU utilization.
Another server, RX drops, high cpu utilization.

I'm going to read the doc from BigSchmuh now, it looks promising.

Author

Commented:
Ok I read through this http://download.intel.com/design/network/applnots/ap450.pdf  -Very interesting.

I see people setting the RX and TX descriptors, along with RXIntDelay at the same time they are setting InterruptThrottleRate.    Does this make sense?

It seems like it would make since given InterruptThrottleRate deals with time and quanity of packets. While the Descriptors deal with size in KB.

If the buffer isn't large enough to allow the adaptive mode to do it's job then it wouldn't work as expected. -My opinion.

and it doesn't seem that you could set the RX and TX descriptors too high if the adaptive mode were always triggering the interupt before you reached the max buffer value.

and it seems setting RXIntDelay wouldn't make sense at all if adaptive modes were in use.

What do you think?

Author

Commented:
Another twist, "interrupt coalescing"

clustermonkey thinks this is the same thing as InterruptThrottleRate, but I think he might be wrong.

http://tweaks.clustermonkey.net/index.php/Small_GigE_Switches

Author

Commented:
Na, I think "Interrupt Coalescing" and "Interrupt Moderation"  are the same thing.
Top Expert 2015
Commented:
What are you receiving when you get drops?

Maybe
sysctl net.core.somaxconn=32768
cures if your apps are slow reading from sockets.

Author

Commented:
I'm reading about the net.core.somaxconn but I think the Intel Interrup Moderation guide helped understand this quite a bit.  I was more looking for a mathematical formula for calculating the best values. I'm going to see if I can turn on the "Interrupt Coalescing" and "Interrupt Moderation" then I wouldn't have to calculate it.

I think I should open another question just to chat about other tunning ideas/experience like, sysctl net.core.somaxconn=32768, this parameter is new to me and I want to see more stuff like this.
Top Expert 2015

Commented:

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial