This is a home network with a linksys wrt54g router connecting to my cable modem. It has port 1723 forwarded to my vpn server and pptp passthrough enabled.
I have an XP Pro machine acting as a vpn server. The VPN server is set up to provide vpn ip addreses in the same subnet as my linksys router, but outside the router's pool of dhcp addresses so that there are no conflicts. This server is also running tight vnc server.
I have another XP Pro machine that is also running tight vnc.
My third machine is a MAC OS X 10.5.4 and it has leopard vnc screen sharing enabled.
All three machines can vnc to each other just fine.
My iPhone can either join the lan through the router's wifi AP (and getting a dhcp address) or through the vpn server over 3g (getting an address from the vpn server). When it is using wifi and connected to the AP, it can vnc to all the machines just fine. Also, when it is using 3g outside the network and not connected to vpn, it can vnc to all the servers via my routers public ip if I forward the vnc ports (I don't want to to do this for security reasons, hence the vpn server)
Now, when I vpn into the network, I run into a problem. I can vnc to both my xp machines, but the Mac connection behaves strangely. It connects and begins loading the screen, but the iPhone stops responding after requesting the first framebuffer update. Wireshark shows that the Mac acks the request, and sends an update, followed closely by a handful of unknown vnc messages, after which it begins retransmitting the framebuffer update. I don't know what went wrong.
It behaves the same way using two entirely different vnc clients, so I don't think that its a failure of the software. Or rather, I think the failure is more about the nature of the connection. It is important to note that I can ping the Mac from the iPhone and the iPhone from the Mac at this point just fine. I did disable bonjour on both the Mac and the XP VPN server (iTunes installed it) as well as muck around with the hosts files on all 3, but I'm not sure what to try to fix.
As expected (I think) all of the packets going to the vpn-connected iphone are being routed to the vpn server's mac address, not the iPhone's mac address. (this is in the Ethernet II section of the packet) but I think this is normal. It seems to work ok for the connect/disconnect portions of the communication anyway. Is there something about the framebuffer updates that would prevent the vpn server from performing the forwarding to the iPhone?
When I cancel the connection (iPhone looks like it is still waiting for that framebuffer update) the cancel packet goes to the Mac and the Mac acks it (albeit with a reset flag), so I dunno...
I installed a different vnc server on the mac and got the same result. Its not the built-in screen sharing software...
I've been hunting for some 3rd party vpn software to try on xp (can't be ssl based, iPhone doesn't support that kind of vpn) Just to mix things up really.
Anyway, this is bit long winded, but If anyone has any suggestions or anything I would love to hear them.
edit: completely rewritten for clarity,simplicity and new info.