Solved

GRE/IPSEC performance issues

Posted on 2013-10-22
14
2,805 Views
Last Modified: 2013-10-31
I am troubleshooting a performance issue on a GRE over IPSEC tunnel between two sites.
After a lot of research, I ended up lowering the MTU size and configuring an mss value and this has reduced 99% of the fragmentation I was experiencing. The issue now is that the throughput is only a portion of the WAN link. Both sites are connected by a 100 Mbps metro e but when doing testing on it I can't seem to get any higher than 25-30 Mbps. I figured I would lose some due to the tunnel but not as much as I am.

See the configuration of the tunnel interface below:

Router 1
interface Tunnel0
 bandwidth 100000
 ip address X.X.X.X X.X.X.X
 ip mtu 1400
 ip tcp adjust-mss 1360
 delay 1
 keepalive 10 3
 cdp enable
 tunnel source GigabitEthernet0/0
 tunnel destination X.X.X.X X.X.X.X

and the physical interface:

interface GigabitEthernet0/0
 description BlahBlah
 ip address interface GigabitEthernet0/0
 ip address X.X.X.X X.X.X.X
 duplex auto
 speed auto
 crypto map CM

 
Router 2

interface Tunnel0
 bandwidth 100000
 ip address X.X.X.X X.X.X.X
 ip mtu 1400
 ip tcp adjust-mss 1360
 delay 2970
 keepalive 10 3
 tunnel source GigabitEthernet0/0
 tunnel destination X.X.X.X

Physical interface:

interface GigabitEthernet0/0
 ip address X.X.X.X X.X.X.X
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
 crypto map CM

Any ideas as to how to address this problem?

Thanks
0
Comment
Question by:troubleshooter141
  • 6
  • 4
  • 4
14 Comments
 
LVL 26

Expert Comment

by:Soulja
ID: 39594154
What hardware are you running this vpn on?
0
 
LVL 32

Assisted Solution

by:harbor235
harbor235 earned 250 total points
ID: 39594165
I would try re-adjusting upwards, you should be using the most efficient values possible.

IPSEC tunnel mode adds another IP HDR = 20bytes
GRE adds 4bytes

1500-24 = 1476
TCP HDR = 20 bytes

ip tcp adj-mss 1456

See if that improves things, if fragmentation starts up again adjust down by 10byte increments and retry.

harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
ID: 39594185
The hardware is Cisco 2951 on one side and Cisco 2921 on the other.
0
 
LVL 3

Author Comment

by:troubleshooter141
ID: 39594481
I keep seeing this in the logs:

Oct 22 19:52:44.250: %CERM-4-RX_BW_LIMIT: Maximum Rx Bandwidth limit of 85000 Kbps reached for Crypto functionality with securityk9 technology package license.

Could this be related? I have read that the SEC-K9 license limits encrypted throughput to less than or equal to 85-Mbps unidirectional traffic.
I just expected to see higher numbers than 30
0
 
LVL 26

Expert Comment

by:Soulja
ID: 39594486
Check out this doc. I believe the limiting factor is your hardware.

http://www.cisco.com/en/US/partner/prod/collateral/routers/ps380/performance_2013.pdf
0
 
LVL 3

Author Comment

by:troubleshooter141
ID: 39594500
I am not able to read the document. Apparently my account is not entitled to see that document.

Can you give a brief description?
Thanks
0
 
LVL 26

Accepted Solution

by:
Soulja earned 250 total points
ID: 39594587
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 
LVL 32

Expert Comment

by:harbor235
ID: 39594673
The SEC-K9 license does limit you to less than or equal to ~85Mbps, but you are not even close. HSEC-K9 license provides up to ~170Mbps. Did you adjust your settings as I posted earlier? Your devices should have no problems getting to those numbers.

What kind of traffic are you sending? is this test data or production data? I ask this because if you are testing with small packets your performance will vary greatly.

Please define how you are validating performance, I would also maximize IP mtu and tcp mss.

harbor235 ;}




harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
ID: 39594905
Harbor235

I appreciate the suggestions. To answer your question, no I haven't tried further adjusting the mtu and mss values as you suggested. I can't Do that until later tonight.

Also the test I am running is with iperf. See below:

C:\>iperf -c 10.16.16.18 -p 7575 -i 60 -d
------------------------------------------------------------
Server listening on TCP port 7575
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.16.16.18, TCP port 7575
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[176] local 10.16.22.56 port 34080 connected with 10.16.16.18 port 7575
[200] local 10.16.22.56 port 7575 connected with 10.16.16.18 port 4276
[ ID] Interval       Transfer     Bandwidth
[200]  0.0-10.0 sec  24.9 MBytes  20.9 Mbits/sec
[176]  0.0-10.0 sec  24.9 MBytes  20.8 Mbits/sec
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39594940
Try using a larger tcp window size,

iperf -c 10.16.16.18 -p 7575 -i 1 -w 64KB


harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
ID: 39595170
Definitely better results:

C:\>iperf -c 10.16.16.18 -p 7575 -i 1 -w 64KB
------------------------------------------------------------
Client connecting to 10.16.16.18, TCP port 7575
TCP window size: 64.0 KByte
------------------------------------------------------------
[164] local 10.16.22.56 port 36071 connected with 10.16.16.18 port 7575
[ ID] Interval       Transfer     Bandwidth
[164]  0.0- 1.0 sec  4.08 MBytes  34.2 Mbits/sec
[164]  1.0- 2.0 sec  2.75 MBytes  23.1 Mbits/sec
[164]  2.0- 3.0 sec  2.41 MBytes  20.3 Mbits/sec
[164]  3.0- 4.0 sec  2.75 MBytes  23.1 Mbits/sec
[164]  4.0- 5.0 sec  4.48 MBytes  37.6 Mbits/sec
[164]  5.0- 6.0 sec  6.47 MBytes  54.3 Mbits/sec
[164]  6.0- 7.0 sec  2.79 MBytes  23.4 Mbits/sec
[164]  7.0- 8.0 sec  6.87 MBytes  57.6 Mbits/sec
[164]  8.0- 9.0 sec  2.37 MBytes  19.9 Mbits/sec
[164]  9.0-10.0 sec  6.41 MBytes  53.8 Mbits/sec
[164]  0.0-10.0 sec  41.4 MBytes  34.7 Mbits/sec

What exactly does this mean? Since this is test data, we can manipulate the settings to try different vaules but what happens to real production data?

Thanks harbor235 you are being very helpful
0
 
LVL 26

Expert Comment

by:Soulja
ID: 39595207
Still looks like its hanging around that 50Mbps range that the document depicts for the 2921.
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39598264
And you still have not tweaked the IP mtu and the tcp mss, correct? you should see better numbers there as well.

its hard to determine if additional BW can be achieved without understanding the environment in more detail.

Good luck,

harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
ID: 39598449
I did tweak the mtu sizes last night but it still created issues. Basically I put everything back to defaults. I tested from my pc sending ping packets of different sizes with a DF bit set and then configured those values.
Fragmentation was still occirring so I lowered it progressivley until I reached 1420 mtu. I tested from a PC on that side of the tunnel and everything seemed ok but this morning I had to change things back to 1400 and 1360 as there were multiple users who had no internet access as a result of the changes

My understanding was that the normal mtu is 1500, GRE takes 24 (4 for GRE and 20 for IP) so that would be 1476 and then IPSEC would take 20 for IP and some additional for IPSEC depending on the encryption selected (I think I read some where that it is an average of 56 or so bytes but I could be wrong). The from my reading also determined that the mss value is approximatelly 40 bytes less than the MTU.
Long story short, it appears that the 1400 and 1360 values are working best.

I appreciate everyone's input and help. I am starting to belive that the throughput we are getting is as good as it is going to get based on the current settings and hardware limitations.

Someone on a Cisco forum suggested hardcoding the speed on the physical interface since it is setup to auto and is prbably negotiating at a higher rate than the 85Mbps IPSEC limitation of the Sec-K9 license so I may try that next.
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

If you use NetMotion Mobility on your PC and plan to upgrade to Windows 10, it may not work unless you take these steps.
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

896 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now