MTU in the internal chain of routers starting with PPPoE

I understand that the MTU for PPPoE is usually 1492.
What I don't understand is how that affects everything inside my network.

I sorta reckon that if packets are limited to 1492 then a device with MTU set at 1500 should have no trouble.  But, I'm not sure.
I've seen some very poor performance when the numbers aren't right but I can't remember the details as this happened years ago.
So, it's time to get things straight as I'm dealing with a throughput problem right now.

I have a fiber link from an ISP that is connected with a router via PPPoE.  Let's say the MTU on that router is set to 1492.

1492 is the number I get using the typical ping test from a computer at the end of the chain.  I don't care much about my workstation computer because there are LOTS of computers that will possibly be affected.  I'm going to have to assume that their settings are OK for now.

Between the modem-like router with 1492, there is a firewall.
The firewall defaults to MTU=1500 but I can set it manually.
What should I do with the firewall re: MTU?
What should I do with the modem-like PPPoE router re: MTU?

I don't understand the interim device setting requirements.
Most of the stuff I find addresses single computer tweaking.
Is there a good, clear discussion of these things?
LVL 27
Fred MarshallPrincipalAsked:
Who is Participating?
TimotiStDatacenter TechnicianCommented:
@fmarshall is mostly right, a few details:
- All end-to-end communication needs to be done with the smallest MTU in the path (otherwise packets won't fit the pipe).
- Normally devices either do path-MTU-discovery (PMTU), which finds the smallest MTU. This uses ICMP, so if ICMP is firewalled, it might break.
- Routers along the way can send back ICMP "Too large, fragmentation needed" messages. Again, the firewalling issue.
- Routers might fragment IPv4 packets if they want, but they usually don't want, as it takes a lot of CPU and RAM. In IPv6, I think this is explicitly forbidden.
- Most UDP and ICMP traffic is small packets, so they are usually not really affected by a bit smaller than default MTU sizes.
- For TCP, you can tweak the max-mss-size during the initial SYN handshake as @fmarshall says. Cisco and Linux can do this if configured, as well as most cheap DSL routers automatically do this to make life smooth for the home user.
- For DSL, 1492 is usually okay, as PPPoE takes 8 bytes. I usually go with a more conservative 1442. It doesn't really affect the throughput, but is safe in case someone along the line has some weird/misconfigured system.
- As @thinkpads_user says, if you start doing weird stuff like GRE-over-IPSec-over-PPPoE, you need to count your bytes to figure out the exact MTU you should use.

My favorite explanation from the man pages of iptables:

       This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface's MTU minus 40 for  IPv4  or  60  for  IPv6,  respectively).   Of
       course, it can only be used in conjunction with -p tcp.  It is only valid in the mangle table.
       This  target  is used to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too Big" packets.  The symptoms of this problem are that everything works fine from your Linux
       firewall/router, but machines behind it can never exchange large packets:
        1) Web browsers connect, then hang with no data received.
        2) Small mail works fine, but large emails hang.
        3) ssh works fine, but scp hangs after initial handshaking.
       Workaround: activate this option and add a rule to your firewall configuration like:

               iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN
                           -j TCPMSS --clamp-mss-to-pmtu

JohnBusiness Consultant (Owner)Commented:
I am not a detailed expert on this issue, but for me it works like this:

1. Default MTU is almost always 1500. Without VPN, 1500 is good everywhere.

2. If you have VPN's set up, 1492 allows for the way VPN packet overhead works. Then all the routers in the VPN change would want the same MTU.

If you do not have VPN, I don't think a change to 1492 causes poor performance.

If you do have VPN tunnels set up, an MTU of 1500 can cause poor performance compared to an MTU of 1492.

A bit of experimenting is always required (at least for me).

... Thinkpads_User

If you send a packet thats too large for the link and its don't fragment bit isn't set in the IP header then the packets could get fragmented. If the don't fragment bit is set then sessions can be terminated.

On a Cisco router you can use the "ip tcp adjust-mss" interface command to adjust the TCP mss option in a TCP SYN segment so that the correct maximum segment size is used for your PPPoE interface. You have to specify what the mss should be taking into consideration the size of the headers.

The following link gives some information about mtu and mss sizes for a PPPoE link.
Simple Misconfiguration =Network Vulnerability

In this technical webinar, AlgoSec will present several examples of common misconfigurations; including a basic device change, business application connectivity changes, and data center migrations. Learn best practices to protect your business from attack.

Fred MarshallPrincipalAuthor Commented:
I'm not planning on sending any packets of any particular size unless I'm doing a ping test with packet size specified in order to detect the size causing fragmentation.

I just want to know how to set the interim router(s).

One guess is to set them all the same as in 1492.

Another guess is to set them at the edge at 1492 - I believe this is best or necessary? practice.  And then, set all those inside at 1500 - how can that hurt?  They don't initiate packets - or do they?
JohnBusiness Consultant (Owner)Commented:
I would set them all the same size. If no VPN and 1500 works, use it everywhere. That is, after all, the default. There is nothing wrong with it. Make sure your server is not sending Jumbo packets. Otherwise 1500 everywhere is good.

.... Thinkpads_User
Fred MarshallPrincipalAuthor Commented:
Well, there is definitely something wrong with 1500 with PPPoE.  I just wish I knew enough to be able to say what can happen and where and .......
I imagine the biggest problem is that a host could launch with 1500 only to hit the PPPoE interface and then what?
I have certainly see BIG differences in speed if the setting is wrong.  I'm just trying to undersatnd.  
Most commodity routers don't give you much other than MTU to play with.  And, this includes Cisco Small Business routers.

I don't get a feeling that there is a notion of a best practice here .. yet.

My ignorance is getting in the way .. thus the question.
Presumably if a packet is launched that will have no further headers applied then:
- if it meets the minimum MTU of the path then all is good.
- it it exceeds the minimum MTU of the path then something has to happen: fragmentation or dropping/error.

But how is one to know if a packet might not arrive at a "system" whose size is bigger than the system MTU?  And, conversely, how is one to know if a packet being launched is too big for the destination system?  Or, is PMTU prevalent?

Or maybe I've been led down a path because the Linksys/Cisco Small Bus. routers may have had a bad "Auto" MTU function.  I just changed one from Auto to 1492 with what appears to be noticeable results.  Having done that, I decided it was time to check them all.
JohnBusiness Consultant (Owner)Commented:
PPPoE is DSL and an MTU of 1492 usually works best. Normally I see this with VPN connections, but it may help you here as well.

(Added - Remember from above that it is probably best to have all the same MTU unless that causes problems)

.... Thinkpads_User
JohnBusiness Consultant (Owner)Commented:
@fmarshall - Thank you and I was pleased to help you with this.

.... Thinkpads_User
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.