Solved

GRE/IPSEC performance issues

Posted on 2013-10-22
14
2,734 Views
Last Modified: 2013-10-31
I am troubleshooting a performance issue on a GRE over IPSEC tunnel between two sites.
After a lot of research, I ended up lowering the MTU size and configuring an mss value and this has reduced 99% of the fragmentation I was experiencing. The issue now is that the throughput is only a portion of the WAN link. Both sites are connected by a 100 Mbps metro e but when doing testing on it I can't seem to get any higher than 25-30 Mbps. I figured I would lose some due to the tunnel but not as much as I am.

See the configuration of the tunnel interface below:

Router 1
interface Tunnel0
 bandwidth 100000
 ip address X.X.X.X X.X.X.X
 ip mtu 1400
 ip tcp adjust-mss 1360
 delay 1
 keepalive 10 3
 cdp enable
 tunnel source GigabitEthernet0/0
 tunnel destination X.X.X.X X.X.X.X

and the physical interface:

interface GigabitEthernet0/0
 description BlahBlah
 ip address interface GigabitEthernet0/0
 ip address X.X.X.X X.X.X.X
 duplex auto
 speed auto
 crypto map CM

 
Router 2

interface Tunnel0
 bandwidth 100000
 ip address X.X.X.X X.X.X.X
 ip mtu 1400
 ip tcp adjust-mss 1360
 delay 2970
 keepalive 10 3
 tunnel source GigabitEthernet0/0
 tunnel destination X.X.X.X

Physical interface:

interface GigabitEthernet0/0
 ip address X.X.X.X X.X.X.X
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
 crypto map CM

Any ideas as to how to address this problem?

Thanks
0
Comment
Question by:troubleshooter141
  • 6
  • 4
  • 4
14 Comments
 
LVL 26

Expert Comment

by:Soulja
Comment Utility
What hardware are you running this vpn on?
0
 
LVL 32

Assisted Solution

by:harbor235
harbor235 earned 250 total points
Comment Utility
I would try re-adjusting upwards, you should be using the most efficient values possible.

IPSEC tunnel mode adds another IP HDR = 20bytes
GRE adds 4bytes

1500-24 = 1476
TCP HDR = 20 bytes

ip tcp adj-mss 1456

See if that improves things, if fragmentation starts up again adjust down by 10byte increments and retry.

harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
Comment Utility
The hardware is Cisco 2951 on one side and Cisco 2921 on the other.
0
 
LVL 3

Author Comment

by:troubleshooter141
Comment Utility
I keep seeing this in the logs:

Oct 22 19:52:44.250: %CERM-4-RX_BW_LIMIT: Maximum Rx Bandwidth limit of 85000 Kbps reached for Crypto functionality with securityk9 technology package license.

Could this be related? I have read that the SEC-K9 license limits encrypted throughput to less than or equal to 85-Mbps unidirectional traffic.
I just expected to see higher numbers than 30
0
 
LVL 26

Expert Comment

by:Soulja
Comment Utility
Check out this doc. I believe the limiting factor is your hardware.

http://www.cisco.com/en/US/partner/prod/collateral/routers/ps380/performance_2013.pdf
0
 
LVL 3

Author Comment

by:troubleshooter141
Comment Utility
I am not able to read the document. Apparently my account is not entitled to see that document.

Can you give a brief description?
Thanks
0
 
LVL 26

Accepted Solution

by:
Soulja earned 250 total points
Comment Utility
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 32

Expert Comment

by:harbor235
Comment Utility
The SEC-K9 license does limit you to less than or equal to ~85Mbps, but you are not even close. HSEC-K9 license provides up to ~170Mbps. Did you adjust your settings as I posted earlier? Your devices should have no problems getting to those numbers.

What kind of traffic are you sending? is this test data or production data? I ask this because if you are testing with small packets your performance will vary greatly.

Please define how you are validating performance, I would also maximize IP mtu and tcp mss.

harbor235 ;}




harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
Comment Utility
Harbor235

I appreciate the suggestions. To answer your question, no I haven't tried further adjusting the mtu and mss values as you suggested. I can't Do that until later tonight.

Also the test I am running is with iperf. See below:

C:\>iperf -c 10.16.16.18 -p 7575 -i 60 -d
------------------------------------------------------------
Server listening on TCP port 7575
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.16.16.18, TCP port 7575
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[176] local 10.16.22.56 port 34080 connected with 10.16.16.18 port 7575
[200] local 10.16.22.56 port 7575 connected with 10.16.16.18 port 4276
[ ID] Interval       Transfer     Bandwidth
[200]  0.0-10.0 sec  24.9 MBytes  20.9 Mbits/sec
[176]  0.0-10.0 sec  24.9 MBytes  20.8 Mbits/sec
0
 
LVL 32

Expert Comment

by:harbor235
Comment Utility
Try using a larger tcp window size,

iperf -c 10.16.16.18 -p 7575 -i 1 -w 64KB


harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
Comment Utility
Definitely better results:

C:\>iperf -c 10.16.16.18 -p 7575 -i 1 -w 64KB
------------------------------------------------------------
Client connecting to 10.16.16.18, TCP port 7575
TCP window size: 64.0 KByte
------------------------------------------------------------
[164] local 10.16.22.56 port 36071 connected with 10.16.16.18 port 7575
[ ID] Interval       Transfer     Bandwidth
[164]  0.0- 1.0 sec  4.08 MBytes  34.2 Mbits/sec
[164]  1.0- 2.0 sec  2.75 MBytes  23.1 Mbits/sec
[164]  2.0- 3.0 sec  2.41 MBytes  20.3 Mbits/sec
[164]  3.0- 4.0 sec  2.75 MBytes  23.1 Mbits/sec
[164]  4.0- 5.0 sec  4.48 MBytes  37.6 Mbits/sec
[164]  5.0- 6.0 sec  6.47 MBytes  54.3 Mbits/sec
[164]  6.0- 7.0 sec  2.79 MBytes  23.4 Mbits/sec
[164]  7.0- 8.0 sec  6.87 MBytes  57.6 Mbits/sec
[164]  8.0- 9.0 sec  2.37 MBytes  19.9 Mbits/sec
[164]  9.0-10.0 sec  6.41 MBytes  53.8 Mbits/sec
[164]  0.0-10.0 sec  41.4 MBytes  34.7 Mbits/sec

What exactly does this mean? Since this is test data, we can manipulate the settings to try different vaules but what happens to real production data?

Thanks harbor235 you are being very helpful
0
 
LVL 26

Expert Comment

by:Soulja
Comment Utility
Still looks like its hanging around that 50Mbps range that the document depicts for the 2921.
0
 
LVL 32

Expert Comment

by:harbor235
Comment Utility
And you still have not tweaked the IP mtu and the tcp mss, correct? you should see better numbers there as well.

its hard to determine if additional BW can be achieved without understanding the environment in more detail.

Good luck,

harbor235 ;}
0
 
LVL 3

Author Comment

by:troubleshooter141
Comment Utility
I did tweak the mtu sizes last night but it still created issues. Basically I put everything back to defaults. I tested from my pc sending ping packets of different sizes with a DF bit set and then configured those values.
Fragmentation was still occirring so I lowered it progressivley until I reached 1420 mtu. I tested from a PC on that side of the tunnel and everything seemed ok but this morning I had to change things back to 1400 and 1360 as there were multiple users who had no internet access as a result of the changes

My understanding was that the normal mtu is 1500, GRE takes 24 (4 for GRE and 20 for IP) so that would be 1476 and then IPSEC would take 20 for IP and some additional for IPSEC depending on the encryption selected (I think I read some where that it is an average of 56 or so bytes but I could be wrong). The from my reading also determined that the mss value is approximatelly 40 bytes less than the MTU.
Long story short, it appears that the 1400 and 1360 values are working best.

I appreciate everyone's input and help. I am starting to belive that the throughput we are getting is as good as it is going to get based on the current settings and hardware limitations.

Someone on a Cisco forum suggested hardcoding the speed on the physical interface since it is setup to auto and is prbably negotiating at a higher rate than the 85Mbps IPSEC limitation of the Sec-K9 license so I may try that next.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Let’s list some of the technologies that enable smooth teleworking. 
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now