I'm having trouble with a Small Business Server 2003 machine intermittently refusing all outside connections. The problem is sporadic and I haven't found anything in the logs that points to a culprit. Everything will be running just fine and then suddenly all outside RDP sessions are killed and the server will refuse connections on port 25 and 443. This brings mail/webmail to a screeching halt.
It appeared to be related to DNS (ie, server could not do reverse lookups on IPs of incoming connections), so I made some changes based on some research. The problem appeared to be fixed, but it keeps coming back.
Changes made so far:
1. Changed maximum DNS packet size from 512 to 1518
SBS 2003 Server
1. Added key: EnableEDNSProbes, Reg_DWORD = 0x00000000
2. Added key: EDNSCacheTimeout, Reg_DWORD = 0x00057e40
I've also verified that no other processes are using any ports needed by IPSec.
The problem still happens and lasts for intermittent amounts of time. I have 3 other environments identical to this one (SBS 2003 behind a PIX firewall) that aren't having this issue.
Any help would be greatly appreciated.