iPhone's VNC-over-VPN fails while loading screen data, but only with MAC OS X vnc

This is a home network with a linksys wrt54g router connecting to my cable modem. It has port 1723 forwarded to my vpn server and pptp passthrough enabled.

I have an XP Pro machine acting as a vpn server. The VPN server is set up to provide vpn ip addreses in the same subnet as my linksys router, but outside the router's pool of dhcp addresses so that there are no conflicts. This server is also running tight vnc server.

I have another XP Pro machine that is also running tight vnc.

My third machine is a MAC OS X 10.5.4 and it has leopard vnc screen sharing enabled.

All three machines can vnc to each other just fine.

My iPhone can either join the lan through the router's wifi AP (and getting a dhcp address) or through the vpn server over 3g (getting an address from the vpn server). When it is using wifi and connected to the AP, it can vnc to all the machines just fine. Also, when it is using 3g outside the network and not connected to vpn, it can vnc to all the servers via my routers public ip if I forward the vnc ports (I don't want to to do this for security reasons, hence the vpn server)

Now, when I vpn into the network, I run into a problem. I can vnc to both my xp machines, but the Mac connection behaves strangely. It connects and begins loading the screen, but the iPhone stops responding after requesting the first framebuffer update. Wireshark shows that the Mac acks the request, and sends an update, followed closely by a handful of unknown vnc messages, after which it begins retransmitting the framebuffer update. I don't know what went wrong.

 It behaves the same way using two entirely different vnc clients, so I don't think that its a failure of the software. Or rather, I think the failure is more about the nature of the connection. It is important to note that I can ping the Mac from the iPhone and the iPhone from the Mac at this point just fine. I did disable bonjour on both the Mac and the XP VPN server (iTunes installed it) as well as muck around with the hosts files on all 3, but I'm not sure what to try to fix.

As expected (I think) all of the packets going to the vpn-connected iphone are being routed to the vpn server's mac address, not the iPhone's mac address. (this is in the Ethernet II section of the packet) but I think this is normal. It seems to work ok for the connect/disconnect portions of the communication anyway. Is there something about the framebuffer updates that would prevent the vpn server from performing the forwarding to the iPhone?

When I cancel the connection (iPhone looks like it is still waiting for that framebuffer update) the cancel packet goes to the Mac and the Mac acks it (albeit with a reset flag), so I dunno...

I installed a different vnc server on the mac and got the same result. Its not the built-in screen sharing software...

I've been hunting for some 3rd party vpn software to try on xp (can't be ssl based, iPhone doesn't support that kind of vpn) Just to mix things up really.

Anyway, this is bit long winded, but If anyone has any suggestions or anything I would love to hear them.

edit: completely rewritten for clarity,simplicity and new info.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

zingiberAuthor Commented:
I have discovered the problem and the solution. I figured I would post back here in case anyone had read the question and was interested. The problem comes down to the MTU settings on the different PCs.

 I had wireshark running on both the VPn interface and the LAN interface on my vpn server, and I could see that all the packets coming from the Mac were being passed along to the iPhone with the exception of the first framebuffer update. It was going to the vpn server, but the vpn server was not passing it along. Nothing in the pack indicated a problem, and there was no apparent explanation.

I also had a wireshark grab of the successful scenario (when the iPhone is not going through the vpn) as well as a grab of the vpn server itself accessing the vnc server. I compared the first framebuffer update packet from the mac in all three and I noticed that the packet size was different when it was talking directly to the vpn server. These are large chunks of data, so they max out the MTU, but the MTU settings on my vpn server's interfaces are different (1300) than that of my Mac and iPhone (1500).

In the two good scenarios, the packets from the Mac are 1500 (to the iphone) and 1300 (to the vpn server) and are transmitted successfully. In the bad scenario, the packets from the Mac are 1500 and are not forwarded. It seems like the Mac knows that the destination (iPhone) can handle 1500, but it doesn't know that the middleman (vpn server) only takes 1300.

Anyways, setting the mtu on the mac to 1300 fixed the problem. Setting the vpn interfaces to 1500 also worked.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Mac OS X

From novice to tech pro — start learning today.