I have created a supposed L2 bridge between two remote sites, yet I get truncated results on the client end. Using pfsense and OpenVPN.

Mike Smith
Mike Smith used Ask the Experts™
on
Every request on the server side I can watch in realtime get received incomplete on the client side... help!

00:21:36.185844 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->4520)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:36.656109 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->4551)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:36.656152 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->4551)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:37.878673 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->44ef)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:37.878976 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->44ef)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:40.869245 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->4481)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:40.869623 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->4481)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:43.784240 IP0 bad-hlen 4
00:21:43.785039 IP0 bad-hlen 4
00:21:43.786308 IP0 bad-hlen 4
00:21:43.786348 IP0 bad-hlen 4
00:21:48.739724 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->449a)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156
00:21:48.740448 IP15 truncated-ip - 65475 bytes missing! (tos 0xff,CE, ttl 86, id 65535, offset 640, flags [none], proto unknown (156), length 65535, options (EOL), bad cksum c0a1 (->449a)!)
    8.6.0.1 > 8.0.6.4: ip-proto-156

I tried:
setting mss at 1200 on all interfaces had no effect, same errors.

I always seem to get bytes missing on the remote side of the l2 bridge.

Pls help shed some light on this.
Captureovpn.PNG
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
View Solution Only
Scott SilvaNetwork Administrator

Commented:
That really sounds like a bad net card in one of the PFSense boxes, or you are using cheap host based cards like realtek...
Mike Smith

Author

Commented:
its not a hardware problem, these are the packets coming over the openvpn bridge that seem to have some data missing when they arrive... the interfaces work perfectly on both sides, its something in the middle.
Scott SilvaNetwork Administrator

Commented:
Do the systems you are using have adequate CPU power? The encryption /decryption is very processor intensive.... Also check memory... I am assuming you are using a PC at each end and not the hardware that PFSense sells...

I know it should work because I ran L2TP tunnels for years on commodity boxes....

Maybe try L2TP instead of open vpn? It is older and more stable...
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Find out now!
Scott SilvaNetwork Administrator

Commented:
From PFSense site:

Network Card Selection

Selection of network cards (NICs) is often the single most important performance factor in your setup. Inexpensive NICs can saturate your CPU with interrupt handling, causing missed packets and your CPU to be the bottleneck. A quality NIC can substantially increase system throughput. When using pfSense software to protect your wireless network or segment multiple LAN segments, throughput between interfaces becomes more important than throughput to the WAN interface(s).

NICs based on Intel chipsets tend to be the best performing and most reliable when used with pfSense software. We therefore strongly recommend purchasing Intel cards, or systems with built-in Intel NICs up to 1Gbps. Above 1Gbps, other factors, and other NIC vendors dominate performance.
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Bridge configuration?  These are real strange numbers, who produces these numbers tcpdump?
can it be that packets are encapsulated but that it is not noticed? ppp used to have issues like that.

or adapter is missing bytes from the serial stream and the resulting packet data is corrupted.
Mike Smith

Author

Commented:
I still haven't been able to figure this out... I'm using intel NICs on both sides. I prefer openvpn because then I can have two failover internet connections on the client side (which is required for this application).

here's the configs:

server
dev ovpns1
verb 4
dev-type tap
dev-node /dev/tap1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local <my external ip>
engine cryptodev
lport 1194
management /var/etc/openvpn/server1.sock unix
secret /var/etc/openvpn/server1.secret
comp-lzo no
mssfix 1366
~

client
dev ovpnc1
verb 4
dev-type tun
tun-ipv6
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local x.x.x.x <- my local ip
engine cryptodev
lport 0
management /var/etc/openvpn/client1.sock unix
remote x.x.x.x 1194
secret /var/etc/openvpn/client1.secret
comp-lzo no
mssfix 1366
resolv-retry infinite
Software Engineer
Distinguished Expert 2018
Commented:
I think the problem is one side has a tap and the other side has a tun interface.
tap = ethernet bridging.
tun = IP network routing.
Mike Smith

Author

Commented:
really? oh man! that's a basic error... let me check that.
Mike Smith

Author

Commented:
That helped a LOT! but now I can only see ARP traffic on the client end of the bridge, all the IP traffic seems to not be broadcasting across... I can put my test laptop on the same IP range as the LAN on the server end and ping only the LAN address...

config is like this:

server
em0 WAN (real IP)
em1 LAN internal IP 10.0.0.2
em2 BRIDGE (no ip, promisc)

client
em0 WAN (real IP)
em2 LAN 192.168.1.1
em3 BRIDGE (no ip, promisc)

I put a laptop on em3 on client end and setup 10.0.0.92/24 with GW of 10.0.0.1 (my internal NAT Firewall on the server side) I cannot ping 10.0.0.1 or anything else from client side but I can ping 10.0.0.2

very wierd.

NAT is disabled on both sides and routing shouldnt matter if the l2 bridge is working properly right?

I see TONS of IP traffic if I tcpdump em2 on server side but only the ARP traffic seems to be making it across to em3 on the client side.

Firewall rules are set to allow all from any ipv4 on all interfaces involved in the tunnel (for now during testing) Im missing something here, hopefully you can help.
Mike Smith

Author

Commented:
it's working, I can ping stuff on the other side and talk to it but I can't get out to the internet... must be the other firewall/router which runs pfsense as well which is stopping the traffic.
Mike Smith

Author

Commented:
I see tons of arp requests reaching the server ethernet port which is bridged but it doesn't seem to be getting any responses... any ideas on that?

    7.7.10.92.54569 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.54569 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.54569 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.54569 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
20:53:45.652316 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 7.7.10.40 (14:fe:b5:c7:af:b3) tell 7.7.10.92, length 46
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.63227 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.63227 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.63227 > 224.0.0.252.5355: UDP, length 22
    7.7.10.92.63227 > 224.0.0.252.5355: UDP, length 22
20:53:46.160670 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 7.7.10.1 tell 7.7.10.92, length 46
20:53:46.160868 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 7.7.10.1 tell 7.7.10.92, length 46
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
20:53:47.153162 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 7.7.10.1 tell 7.7.10.92, length 46
20:53:47.153365 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 7.7.10.1 tell 7.7.10.92, length 46
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    7.7.10.92.137 > 7.7.10.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Are you setting a default gateway of the client to your firewall on the server side?
Also the arp requests, where are they from the firewall looking for the client or the client
Are the ARP requests visible on the server side? are there answers not sent back?
Mike Smith

Author

Commented:
yes im setting the default gw on the client laptop to the firewall IP on the server side...

the arp  requests coming from the client side seem to make it to the server side but no replies seem to  come in or make it back. I do see lots of other  unrelated  ARP traffic arriving  on the client side from the server side though... mostly who-has requests t hat have nothing to do with the client IPs I configured.
nociSoftware Engineer
Distinguished Expert 2018

Commented:
The ARP's are sent to all systems in the broadcast domain ie. (Extended-)LAN.
ARP is a request for the MAC address belonging to an IP address. In LAN's MAC addresses are the real addresses used.  If no ARP reply is sent by a system, it is impossible to send packet to a target.

Can you do a packet trace on the router? see what happens there?
Mike Smith

Author

Commented:
all stars...

traceroute to 7.7.10.92 (7.7.10.92), 64 hops max, 52 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  *


the arp traffic seems to only be flowing in one direction?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Oh with a packet trace i actualy mean a tcpdump ..... (filtered by the address of the laptop).
Mike Smith

Author

Commented:
I see stuff going out but nothing coming back on the bridged ethernet interface on the server side.

15:30:41.597349 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:41.597704 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:42.349645 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:42.350029 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:43.103517 IP 7.7.10.92.60137 > a.resolvers.level3.net.domain: 41497+ A? DFW01-SVP-ITCM.hydro.local. (44)
15:30:43.122269 IP a.resolvers.level3.net.domain > 7.7.10.92.60137: 41497 NXDomain 0/1/0 (119)
15:30:43.157384 IP 7.7.10.92.61388 > 224.0.0.252.5355: UDP, length 32
15:30:43.157630 IP 7.7.10.92.61388 > 224.0.0.252.5355: UDP, length 32
15:30:43.257393 IP 7.7.10.92.61388 > 224.0.0.252.5355: UDP, length 32
15:30:43.257620 IP 7.7.10.92.61388 > 224.0.0.252.5355: UDP, length 32
15:30:43.461602 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:43.461954 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:44.216558 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:44.216931 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:44.968455 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:44.968803 IP 7.7.10.92.netbios-ns > 7.7.10.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
15:30:45.282428 IP 7.7.10.92.52983 > server25103.teamviewer.com.5938: Flags [P.], seq 3392499253:3392499277, ack 2415783948, win 16331, length 24
15:30:45.365167 IP server25103.teamviewer.com.5938 > 7.7.10.92.52983: Flags [.], ack 24, win 513, length 0
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Now it is possible to make such a trace on your firewall/router?
or copy the switchport where it is connected to another port and trace from there?

Possibly the router/firewall has blocked the address range?
Mike Smith

Author

Commented:
ohh on the router yep one second...

we have some arp requests looking for the default gw, seems like it.

15:48:02.537422 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:04.811820 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:05.535033 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:08.826521 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:09.530179 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:10.533914 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:11.869245 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:12.527171 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.lo                                    caldomain tell 7.7.10.92, length 46
15:48:13.558782 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.localdomain tell 7.7.10.92, length 46
15:48:14.899037 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.localdomain tell 7.7.10.92, length 46
15:48:15.536386 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has stormfw1.localdomain tell 7.7.10.92, length 46


nothing coming back though?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Ok that's an issue on the firewall...
Check it's logs... (it isn't by accident another known address on the firewall??)

Now first see if you  can get something communicating with adding a static arp entry for the firewall  on the laptop.

On linux: arp -s 7.7.10.1 mac:add:ress:of:fire:wall

then the arp request is not needed maybe communication to the outside is now possible, or other log entries are show ..
Mike Smith

Author

Commented:
Sometimes when I do arp-a I can see the arp of the firewall and ping it (but not past it via NAT) and then itll time out, and it doesnt show up in arp,  and then its back...  wierd.
nociSoftware Engineer
Distinguished Expert 2018

Commented:
if there is no ipaddress - mac address mapping nothing will be sent.....
so some arp do get answered..... to side step it just add a static one... and see what goes through the firewall.
Something should show up in logs, or there are filter rules or routing rules depending on source address that prevent traffic from going through..., or NAT rules that don't include this address somehow.

But without a stable arp you can not look at that.... (not arp -s is a temporary measure, not a fix...) arp resolving should work as well. So look for possible issues on that as well.
Mike Smith

Author

Commented:
I think maybe packet size has something to  do with it, i'm going to try to clamp mss to 1366 again see if it helps....
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Can be if pmtu is on. Clamping down can help. 1280 is a safe value.
Mike Smith

Author

Commented:
I set the fixmss at 1200 and fragment at 1200 and its the same...  can't figure this out.
Scott SilvaNetwork Administrator

Commented:
It might be a good time to cut your losses and start over and use one of the other tunnel types like L2TP or IPSEC...  You have spent a lot of time with this...  You can always save the configs and go back if you want...
Mike Smith

Author

Commented:
yeah... I may need to look into l2tp over ipsec is there a guide for that for pfsense?
Scott SilvaNetwork Administrator

Commented:
The docs site has a LOT of IPSEC stuff... But it was pretty easy when I did it... Just remember the easy parts... A bridge is for parts of the same subnet, and a tunnel is for a different subnet... Tunnels are easier to get going...

 I also just found this with a google search...
http://www.cyberciti.biz/faq/howto-site-to-site-ipsec-vpn-between-cisco-openbsd-router-pfsense/

I haven't used PFSense for this for several years since our company started spending money for hardware, but when I first started the networks here I used it a LOT...

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial