Strange ISA 2006 problem
I have really strange problem that I really don't even know how to explain. Let me first start by explaining our setup. We have ISA 2006 EE as our perimeter firewall. It has 3 NIC connections. There is an external, internal, and DMZ connection. Our AD DNS forwards external resolution requests to ISA which is a cache only DNS that forwards to our ISP DNS for external name resolution. We have been running this setup for over 2 years so I know rules and DNS forwarding is setup properly.
Last month DNS issues just came out of the blue. No one could get to any websites and mail stopped flowing. However our globes showed as having an internet connection, and from ISA our connectivity verifiers were showing as good. However I could not ping our ISP DNS, or internal DNS or machines. All NIC's on ISA showed connected, but I could not communicate via ping with any internal machine, nor could I ping our router. I could RDP to ISA via allowed IP machine (my computer). All of this I am able to do under normal operations so again it is not an issue with setup. A reboot of the ISA 2006 server picks up normal operations. I also know it is not our edge switch or router as I have separate domain and TMG 2010 firewall for development use, and no internet connectivity was lost and I was able to ping our router and ISP DNS from that firewall. It is setup in the same manner as our ISA 2006 and domain, minus the DMZ. Everything is the same including DNS and DNS forwarding and rules. So I am able to determine from this that this strange problem is only happening on our ISA 2006 firewall. All other internal switches and machines have full connectivity, and the ISA internal NIC is connected to the same switch that our AD DC and DNS is on. No tx or rx errors for that port or any port for that matter.
Fast forward about 2 weeks and the problem happens again. And again I check the other domain and firewall and all is well, but from ISA 2006 firewall no clients or servers have internet access. ISA 2006 also has no crazy alerts, and the connectivity verifiers show a "good" connection. Once again a reboot picks up communitarians. Also if I run query to our internal AD DNS I see the normal operation, but there are several closed connections for port 53 DNS until I reboot. I also see a couple of multi cast (224.0.0.22), but they do not really coincide with the outages. They are at several different intervals.
Fast forward about 11 days to yesterday. Again, the same issues arise. However this last time a reboot did not fix this issue. I tried to ping our internal DNS and ISP DNS to no avail. I also could not ping our router to ISP, but again everything on the other TMG firewall was running as normal. So at this point I have 2 NIC connections that were not being used so I set up the external IP and internal IP on them and checked all the appropriate bindings and order. I then rebooted and could actually ping and perform nslookup on internal connections, but still could not ping ISP DNS so no internet still. A third reboot cleared up the issue this time. I am at real loss as to what is going on. In the query for the time frame I see a ton of DNS connections coming from our internal AD DNS to the ISA over port 53, but I assume this is normal as other logs show this to be the case. My first inclination is that there is some sort of DoS attack, but my switches are not getting over loaded and I have AVG network edition on ISA and all internal server. I even have extra malware protection on our internal AD DC/DNS servers. They all update hourly (AVG every 4 hours), and scan every night with nothing showing. If ANYONE has any information or has seen this before I would be forever be in debt to you, and I am also more than willing to pay for professional services in trying to resolve this issue.
Last month DNS issues just came out of the blue. No one could get to any websites and mail stopped flowing. However our globes showed as having an internet connection, and from ISA our connectivity verifiers were showing as good. However I could not ping our ISP DNS, or internal DNS or machines. All NIC's on ISA showed connected, but I could not communicate via ping with any internal machine, nor could I ping our router. I could RDP to ISA via allowed IP machine (my computer). All of this I am able to do under normal operations so again it is not an issue with setup. A reboot of the ISA 2006 server picks up normal operations. I also know it is not our edge switch or router as I have separate domain and TMG 2010 firewall for development use, and no internet connectivity was lost and I was able to ping our router and ISP DNS from that firewall. It is setup in the same manner as our ISA 2006 and domain, minus the DMZ. Everything is the same including DNS and DNS forwarding and rules. So I am able to determine from this that this strange problem is only happening on our ISA 2006 firewall. All other internal switches and machines have full connectivity, and the ISA internal NIC is connected to the same switch that our AD DC and DNS is on. No tx or rx errors for that port or any port for that matter.
Fast forward about 2 weeks and the problem happens again. And again I check the other domain and firewall and all is well, but from ISA 2006 firewall no clients or servers have internet access. ISA 2006 also has no crazy alerts, and the connectivity verifiers show a "good" connection. Once again a reboot picks up communitarians. Also if I run query to our internal AD DNS I see the normal operation, but there are several closed connections for port 53 DNS until I reboot. I also see a couple of multi cast (224.0.0.22), but they do not really coincide with the outages. They are at several different intervals.
Fast forward about 11 days to yesterday. Again, the same issues arise. However this last time a reboot did not fix this issue. I tried to ping our internal DNS and ISP DNS to no avail. I also could not ping our router to ISP, but again everything on the other TMG firewall was running as normal. So at this point I have 2 NIC connections that were not being used so I set up the external IP and internal IP on them and checked all the appropriate bindings and order. I then rebooted and could actually ping and perform nslookup on internal connections, but still could not ping ISP DNS so no internet still. A third reboot cleared up the issue this time. I am at real loss as to what is going on. In the query for the time frame I see a ton of DNS connections coming from our internal AD DNS to the ISA over port 53, but I assume this is normal as other logs show this to be the case. My first inclination is that there is some sort of DoS attack, but my switches are not getting over loaded and I have AVG network edition on ISA and all internal server. I even have extra malware protection on our internal AD DC/DNS servers. They all update hourly (AVG every 4 hours), and scan every night with nothing showing. If ANYONE has any information or has seen this before I would be forever be in debt to you, and I am also more than willing to pay for professional services in trying to resolve this issue.
Sight of an ipconfig /all from the ISA box would also be useful please. Don't be shy about public ip addresses.
Same for a route print please
Same for a route print please
ASKER
Correct. I only have internal DNS in the internal NIC and DMZ NIC. No DNS in external NIC. Sorry I do not clarify that. What I meant by cache DNS is a forwarder. In other words, the internal DNS has a forwarder of the ISA 2006 internal IP. ISA 2006 then has a forwarder to the ISAP DNS in the forwarder tabs of DNS. I did this to keep AD DNS from possible pollution.
On DMZ no internet is possible and also cannot get to websites on DMZ as they go through ISA from internal.
In the real-time logs I see that most all the DNS traffic from the internal AD DNS (10.10.1.4 and 10.10.1.3) closes connections. As I said I have seen the IGMP address 224.0.0.22 get denied by the enterprise rule, but I have seen this when there were no problems also. They also do not necessarily coincide with the outage. The weird thing is on one occasion we were still receiving email, but no internet access. So DNS was working in some aspects. I could also RDP from the allowed RDP console (my workstation) to the ISA 2006, but could not ping it or could not ping internal machines from ISA.
Here is the ipconfig and route print (Sorry for the X's, but I am under strict rules here. Just know they are the external IP's):
Windows IP Configuration
Host Name . . . . . . . . . . . . : gateway
Primary Dns Suffix . . . . . . . : softdocs.com
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : Yes
WINS Proxy Enabled. . . . . . . . : Yes
DNS Suffix Search List. . . . . . : softdocs.com
Ethernet adapter NEW External:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Gigabit VT Quad Port Server Adap
ter #4
Physical Address. . . . . . . . . : 00-1B-21-32-45-25
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : X.X.X.116
Subnet Mask . . . . . . . . . . . : 255.255.255.240
Default Gateway . . . . . . . . . : X.X.X.114
Ethernet adapter softdocs:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Gigabit VT Quad Port Server Adap
ter #2
Physical Address. . . . . . . . . : 00-1B-21-32-45-21
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 10.10.1.1
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :
DNS Servers . . . . . . . . . . . : 10.10.1.4
10.10.1.3
ROUTE PRINT:
IPv4 Route Table
==========================
Interface List
0x1 ..........................
0x10003 ...00 1b 21 32 45 25 ...... Intel(R) Gigabit VT Quad Port Server Adapter
#4
0x10004 ...00 1b 21 32 45 21 ...... Intel(R) Gigabit VT Quad Port Server Adapter
#2
0x10005 ...00 1b 21 32 45 20 ...... Intel(R) Gigabit VT Quad Port Server Adapter
#3
==========================
==========================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 X.X.X.114 X.X.X.116 10
10.10.1.0 255.255.255.0 10.10.1.1 10.10.1.1 10
10.10.1.1 255.255.255.255 127.0.0.1 127.0.0.1 10
10.255.255.255 255.255.255.255 10.10.1.1 10.10.1.1 10
X.X.X.112 255.255.255.240 X.X.X.116 X.X.X.116 10
X.X.X.116 255.255.255.255 127.0.0.1 127.0.0.1 10
24.255.255.255 255.255.255.255 X.X.X.116 X.X.X.116 10
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
172.16.0.0 255.240.0.0 172.16.0.1 172.16.0.1 10
172.16.0.1 255.255.255.255 127.0.0.1 127.0.0.1 10
172.16.255.255 255.255.255.255 172.16.0.1 172.16.0.1 10
224.0.0.0 240.0.0.0 10.10.1.1 10.10.1.1 10
224.0.0.0 240.0.0.0 X.X.X.116 X.X.X.116 10
224.0.0.0 240.0.0.0 172.16.0.1 172.16.0.1 10
255.255.255.255 255.255.255.255 10.10.1.1 10.10.1.1 1
255.255.255.255 255.255.255.255 X.X.X.116 X.X.X.116 1
255.255.255.255 255.255.255.255 172.16.0.1 172.16.0.1 1
Default Gateway: X.X.X.114
==========================
Persistent Routes:
None
On DMZ no internet is possible and also cannot get to websites on DMZ as they go through ISA from internal.
In the real-time logs I see that most all the DNS traffic from the internal AD DNS (10.10.1.4 and 10.10.1.3) closes connections. As I said I have seen the IGMP address 224.0.0.22 get denied by the enterprise rule, but I have seen this when there were no problems also. They also do not necessarily coincide with the outage. The weird thing is on one occasion we were still receiving email, but no internet access. So DNS was working in some aspects. I could also RDP from the allowed RDP console (my workstation) to the ISA 2006, but could not ping it or could not ping internal machines from ISA.
Here is the ipconfig and route print (Sorry for the X's, but I am under strict rules here. Just know they are the external IP's):
Windows IP Configuration
Host Name . . . . . . . . . . . . : gateway
Primary Dns Suffix . . . . . . . : softdocs.com
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : Yes
WINS Proxy Enabled. . . . . . . . : Yes
DNS Suffix Search List. . . . . . : softdocs.com
Ethernet adapter NEW External:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Gigabit VT Quad Port Server Adap
ter #4
Physical Address. . . . . . . . . : 00-1B-21-32-45-25
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : X.X.X.116
Subnet Mask . . . . . . . . . . . : 255.255.255.240
Default Gateway . . . . . . . . . : X.X.X.114
Ethernet adapter softdocs:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Gigabit VT Quad Port Server Adap
ter #2
Physical Address. . . . . . . . . : 00-1B-21-32-45-21
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 10.10.1.1
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :
DNS Servers . . . . . . . . . . . : 10.10.1.4
10.10.1.3
ROUTE PRINT:
IPv4 Route Table
==========================
Interface List
0x1 ..........................
0x10003 ...00 1b 21 32 45 25 ...... Intel(R) Gigabit VT Quad Port Server Adapter
#4
0x10004 ...00 1b 21 32 45 21 ...... Intel(R) Gigabit VT Quad Port Server Adapter
#2
0x10005 ...00 1b 21 32 45 20 ...... Intel(R) Gigabit VT Quad Port Server Adapter
#3
==========================
==========================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 X.X.X.114 X.X.X.116 10
10.10.1.0 255.255.255.0 10.10.1.1 10.10.1.1 10
10.10.1.1 255.255.255.255 127.0.0.1 127.0.0.1 10
10.255.255.255 255.255.255.255 10.10.1.1 10.10.1.1 10
X.X.X.112 255.255.255.240 X.X.X.116 X.X.X.116 10
X.X.X.116 255.255.255.255 127.0.0.1 127.0.0.1 10
24.255.255.255 255.255.255.255 X.X.X.116 X.X.X.116 10
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
172.16.0.0 255.240.0.0 172.16.0.1 172.16.0.1 10
172.16.0.1 255.255.255.255 127.0.0.1 127.0.0.1 10
172.16.255.255 255.255.255.255 172.16.0.1 172.16.0.1 10
224.0.0.0 240.0.0.0 10.10.1.1 10.10.1.1 10
224.0.0.0 240.0.0.0 X.X.X.116 X.X.X.116 10
224.0.0.0 240.0.0.0 172.16.0.1 172.16.0.1 10
255.255.255.255 255.255.255.255 10.10.1.1 10.10.1.1 1
255.255.255.255 255.255.255.255 X.X.X.116 X.X.X.116 1
255.255.255.255 255.255.255.255 172.16.0.1 172.16.0.1 1
Default Gateway: X.X.X.114
==========================
Persistent Routes:
None
224.x.y.z are multicats anyway so no interest in those. Its your system so your call but you would not get DNS infection/corruption from letting the DNS go straight out. You are aware that there were a number of issues with ISA and DNS Cache corrupotion some time back? If the cache goes pear-shaped this will impact on everything except the ISA itself of course.
As my previous response - you have ISA 2006 sp1 deployed?
Have you run the BPA now? http://www.microsoft.com/downloads/en/details.aspx?FamilyID=d22ec2b9-4cd3-4bb6-91ec-0829e5f84063&displaylang=en
Config looks sound but where is the third nic that connects to your dmz/perimeter - can't see it in the ipconfig?
As my previous response - you have ISA 2006 sp1 deployed?
Have you run the BPA now? http://www.microsoft.com/downloads/en/details.aspx?FamilyID=d22ec2b9-4cd3-4bb6-91ec-0829e5f84063&displaylang=en
Config looks sound but where is the third nic that connects to your dmz/perimeter - can't see it in the ipconfig?
ASKER
Yeah, sorry again for the confusion in terms. I guess it's not a true "DMZ", but more of what you called perimeter. The "DMZ" is actually the 172.16.0.0 network listed. How would I check for DNS Cache corruption? I did clear the DNS cache on ISA yesterday. I had not done that previously. Not sure if that will fix a corruption or not. I haven't run the BPA in a while. I am downloading the latest now, but I am leaving for the day in about 10 minutes so it will be later tonight before I have a chance to get it installed and run it. Thanks so much for all your help and quick response!
I'm always here :)
ASKER
Would using the "clear cache" option on the ISA DNS MMC actually clear out any cache issues, or would it actually take using the DNSCACHE tool to clean it out?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I did run the the ISA BPA and got just typical results. It stated that the external NIC did not have DNS configured, but that is correct since you should only have one DNS set on the internal NIC. It was just a warning. There is one strange alert did pop up on Friday, but this was a couple of days after our issue. It states: "Description: ISA Server disconnected a non-TCP connection from 10.10.1.4 because the connection limit for this IP address was exceeded. Larger custom connection limits should be configured for the IP addresses of chained proxy servers and back-to-back ISA Server computers with a NAT relationship." The 10.10.1.4 is one of our internal AD DNS servers. I did set both of our internal AD DNS servers in the custom connections, but that was more than a week before our last outage so it did not help.
I have used the dns cache tool to clear the DNS cache on the ISA 2006 server in case of corruption. I have also performed an ipconfig/flushdns and ipconfig/registerdns to clear up any internal DNS issues. I need to keep this open for at least a few more weeks as the issue although sporadic seems to occur every 2 weeks for the past month and a half. So I can't say for sure whether any of the DNS cache clearing has fixed it or not. Thanks again for your time and help.
I have used the dns cache tool to clear the DNS cache on the ISA 2006 server in case of corruption. I have also performed an ipconfig/flushdns and ipconfig/registerdns to clear up any internal DNS issues. I need to keep this open for at least a few more weeks as the issue although sporadic seems to occur every 2 weeks for the past month and a half. So I can't say for sure whether any of the DNS cache clearing has fixed it or not. Thanks again for your time and help.
No problem - I'll be here
ASKER
I got an email that I have an abandoned question so I thought I would at least update it. So far Keith, your suggestion seems to have cleared up the issue. However I can't cry victory yet because the last outage has only been about 3 1/2 weeks. That is just not enough time for me to say everything is fine yet. Keith can I award you points by accepting your solution, but also keep the discussion open? I just want to keep an interaction on this until I know for sure all is well.
No issue from my end. You need to have returned every three weeks though just to give an update else you get the message.
Happy to wait another week or two then we'll need to close it down or whatever.
As an aside, I am the Zone Advisor for this area so I can keep it open if it is really necessary.
Thanks
Keith
Happy to wait another week or two then we'll need to close it down or whatever.
As an aside, I am the Zone Advisor for this area so I can keep it open if it is really necessary.
Thanks
Keith
Hello,
Just putting in my 2 cents and my gratitude to Keith! Our issue is almost 100% identical to Alan's. I administer an "inherited" ISA 2006 installation and we had the same issue. I follow Keith's instructions regarding the DNS settings and found that our ISA server was set as the forwarder from our DNS servers for any domains that our DNS servers couldn't resolve.
I removed the DNS entries from the ISA server and put our ISP's DNS servers in our forwarders list on the DNS servers and remove the ISA servers from them. Name resolutions actually seem to be faster now. Hopefully we'll know within a week if it completely resolves the issue.
A+ Thanks for the detailed responses!
Just putting in my 2 cents and my gratitude to Keith! Our issue is almost 100% identical to Alan's. I administer an "inherited" ISA 2006 installation and we had the same issue. I follow Keith's instructions regarding the DNS settings and found that our ISA server was set as the forwarder from our DNS servers for any domains that our DNS servers couldn't resolve.
I removed the DNS entries from the ISA server and put our ISP's DNS servers in our forwarders list on the DNS servers and remove the ISA servers from them. Name resolutions actually seem to be faster now. Hopefully we'll know within a week if it completely resolves the issue.
A+ Thanks for the detailed responses!
ASKER
Keith is the man. He followed up very fast with each comment, and also give me consolation in knowing that someone out there can pinpoint these strange issues when they arise. A VERY BIG THANKS TO KEITH!
ASKER
Also to note I left my ISA as DNS forwarder to the ISP for external resolution, but I did clear the DNS cache on the ISA server as well as a ipconfig /flushdns - ipconfig /registerdns commands, and all seems to be well for now. I was having issues every 2-3 weeks until I followed Keith's suggestions. If it happens again down the road I am going to follow the other instructions from Kieth in that I will use my internal DNS to forward to ISP bypassing ISA alltogeher since there is no real danger for pollution either way. Thanks again Keith!
Glad to have been of help :)
Keith.
Keith.
ASKER
Funny how I just stated "if it happens again" and all of the sudden this morning what do you know, DNS on ISA was flaking out. I logged into ISA and could ping our ISP's DNS servers sporadically. It would drop and ping alternatively. So I went straight to our internal AD DNS servers put in our ISP's DNS servers in the forwarders, and dropped the ISA DNS to the bottom. Went back to ISA and could ping our ISP DNS consistently again. I'll keep an eye on this, but looks as if I will be giving up on ISA DNS forwarding. Thanks again!
As per my article - remove it now and have done with it as it should not be there in a supported system.
ASKER
I realize now. I have not actually removed the DNS sertup from ISA, but I am bypassing ISA using internal DNS to forward directly to IP. You say just go ahead and completely remove the DNS from ISA alltogether right? I will be doing that this evening. Is there anything I need to do from the internal DNS side as far as using the DNScache tool, or clearing with ipconfig /flush-registerdns on ISA? Things seem to be going fine now after setting up the forwarder on internal DNS and settign the rule to allow out to external, but I just wanted to get your thoughts on anything else I may need to do. Thanks a bunch!
No sweat.
Make sure the internal nic is bound first.
Remove DNS completely - utterly - totally
Leave the external nic blank in respect to DNS
Internal nic points to one or more internal DNS servers.
Internal DNS servers have forwarders to external DNS ip addresses.
Rules allowing DNS frominternal & localhost TO internal & localhost
Rule allowing dns from internal to external.
Apply, test and Job done :)
Make sure the internal nic is bound first.
Remove DNS completely - utterly - totally
Leave the external nic blank in respect to DNS
Internal nic points to one or more internal DNS servers.
Internal DNS servers have forwarders to external DNS ip addresses.
Rules allowing DNS frominternal & localhost TO internal & localhost
Rule allowing dns from internal to external.
Apply, test and Job done :)
ASKER
Got it. I will be migrating to Forefront TMG next, and will be sure to set it up this wayy from the start this time. Thanks for all your help, and maybe adding more questions when that happens also.
:) The base setup of FTMG is identical to ISA notwithstanding the change from 32-bit to 64-bit. Disabling IPv6 on the FTMG is also a good move.....
This is the EXACT problem we've been having for a very very long time. The solution outlined on 01/28/11 at 11:31AM from Keith worked perfectly!
Thanks for the detailed answers, this is defiantly a "keeper"!
Thanks for the detailed answers, this is defiantly a "keeper"!
I'll assume you have ISA 2006 sp1 deployed?
Don't want your money. In truth I am not interested in your points either as I have a few million to play with in this particular zone. So lets just concentrate on getting your sorted. I warn you now I am in the UK so will only be around for a couple of hours but if you have the time to spend then lets crack on.
What are you seeing in the ISA realtime log when things so screwy (GUI - monitoring - logging - start query)?
What results have you had from the ISA 2006 best practices analyser?
When things go off-wire, are you seeing the same issues for traffic from the perimeter network and the internal? Just the internal?
Keith