I've had to repost and I may just jump out of a window. For four days, it has caused major problems.
Site A has a Draytek 2820vn. Within the VPN/Remote Management, I have configured it to dial-out to the router at Site B. Site B also has a Draytek 2820vn and is configured as dial-in. From Site A, I connect to Site B and we can see a connection. However, the packets transmitted from Site A are so high, that it causes the router at Site B to not be able to give access to those that are on the LAN and using it as a gateway. I have even done vice versa, Site B dialing into Site A, still the same thing via the routers. All of the ping rates suddenly go from 18ms to 800ms.
However, what is strange is that if I drop that connection and instead create a VPN from the PC at Site A, to dial into the same router at Site B, then everybody is fine.
Could this be a faulty router? I'm seriously desperate as it's caused the company a lot of hassle now.