1030 and 1058 errors on ISA server and loss of secure channel, possible DNS issue?

I have a client who has a single ISA 2006 server (an ESX virtual machine) set up in single network card configuration. The server repeatedly loses its secure channel to a DC, and has a load of 1030 and 1058 errors in the Application log, plus 5719 errors in the System log. There are clusters of errors every 1-4 days.

The AD domain is named in a public style, let's call it client.com.au. The local network is on a private 10.x.x.x. The ISA server has a local DC as its DNS server.

The following situation occurs sporadically, for a period of an hour or two it seems:
When I try to resolve the AD domain name client.com.au it gives an external IP - the client's public website's IP (which is also a .info address). For all other machines on the internal network, client.com.au resolves to a local DC as expected.

If I temporarily add an entry to hosts on the ISA box (client.com.au and the IP address of a DC)  this allows correct resolution so I can open \\client.com.au but I get access denied on \\client.com.au\netlogon. However if I choose a local DC 'dcname' and connect using its IP address \\10.x.x.x\sysvol then I have no such problems. If I connect using \\dcname.client.com.au\netlogon I am again unable to connect.

The problem does not persist for more than an hour or two, but each time it happens the secure channel dies and users lose their internet connection and hence complain! I'm guessing this might be a config setting somewhere in ISA but my knowledge of this product is fairly minimal.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Amit BhatnagarTechnology Consultant - SecurityCommented:
'\\IP' is NTLM and '\\Server.Domain' is Kerberos so that clears the reason why you are getting an Access Denied (Secure Channel Issue). Now, if the ISA Server is pointing to the DC with DNS installed on the DC, there should be no reason for the ISA to resolve client.com.au to an external name unless
1.) It is pointing to the ISP DNS as well in TCP\IP Properties.
2.) Local DC\DNS does NOT have the Zone created with the name "client.com.au" in it.
3.) Local DC\DNS has a record created in it which is pointing to the public IP of 'client.com.au'.

Secure Channel can be fixed using NTDSUtil or Netdom. It is quite easy. But it definitely seems some misconfiguration at the DNS level. Also, ISA in single NIC. Are you using just as a Proxy Server?
Amit BhatnagarTechnology Consultant - SecurityCommented:
Also, is the name of the Client's website client.com.au or www.client.com.au?
datacomsmtAuthor Commented:
The website is named with a .info A record, there must be a cname with client.com.au (no www.) somewhere too. There is no entry for the external IP in our DNS.

I can sort the secure channel fine using netdom (in fact I have a task that runs every morning and does it just in case there was some sort of incremental degradation that was causing it to drop out).

Yes, ISA is set as proxy only with one NIC, plus they aren't using the firewall client and are instead controlling access on the perimeter firewall. I would change it, I think this client needs a fairly serious overhaul of the core systems.

There is an external DNS server in the networking properties. It is set to second in the list too. I suppose it is conceivable that the first DNS server could be too busy at times to service requests and it tries this one instead. But would it continue to use that DNS server for a couple of hours? I will remove the external server anyway, I can't see what good it is doing when they have plenty of other DCs running DNS.

Why though did adding an entry to hosts and flushing the DNS cache not fix it? Or would that not aid in locating service records?
Redefining Cyber Security w/ AI & Machine Learning

The implications of AI and machine learning in cyber security are massive and constantly growing, creating both efficiencies and new challenges across the board. Join our webinar on Sept. 21st to learn more about leveraging AI and machine learning to protect your business.

Amit BhatnagarTechnology Consultant - SecurityCommented:
1.) Please correct me if I am wrong...client.com.au is the name of your Domain and the website as well. Is this website hosted on the DC itself or a separate Server? If it is a separate Server then we do have a problem.
2.) The reason I asked you about the public IP is because you mentioned that ISA resolves client.com.au to an external IP. This can only happen under two conditions...
a.) ISA is pointing to the external ISP for Name Resolution even if it is second in list.
b.) Internal DC has an external IP binding to the client.com.au record (which you mentioned in not the case).

Yes, external DNS can create a problem cause one a DNS fails, the Client fails over to the second. But it does NOT come back to the first DNS until the second DNS fails which in your case is the ISP. Now, when ISA contacts the ISP DNS for client.com.au, it starts getting the public IP.

Although, Yes it is still not clear why a host file is not helping the name resolution completely.
Please remove the external ISP DNS's IP from the TCP\IP properties of ISA.

Now, I do have a question for you though. You mentioned that client.com.au within the internal DNS is not binding to a public IP. If this is the case, where is this website hosted?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
datacomsmtAuthor Commented:
1) Sorry, I forgot to mention that the company's website is hosted externally. They are a government client and have a shared services agreement with a separate department who hosts their services. The website is client.info but there seems to be a CNAME in the public DNS domain for client.com.au to map to client.info.  (To be honest there is no need for a publically resolvable client.com.au, which is a purely internal naming system for both AD and DNS.)

2) I think it must be the case that during the 'outages' ISA has been forced to go to the ISP DNS server for its resolution, and somehow decides a few hours later to switch back whereupon it is OK again. I have removed the ISP DNS and made sure there are 3 different internal DNS servers listed in case of problems.

I will need to leave this for 5 days to see if it addresses the issue. Thanks for your help so far - I should have been looking at more basic things rather than worrying it was an ISA-specific issue.
Amit BhatnagarTechnology Consultant - SecurityCommented:
Yes, you are correct. If the website name has nothing to do with the Domain name then it should be kept separate. Not with a Cname etc. Using three internal DNS Server will definitely help. Even under best practices for DNS, Microsoft does not recommend using an ISP's DNS Server in TCP\IP properties. It should always be used within Forwarders option in DNS.

As for your last statement, this is how we learn..Don't we?..:) !! please keep me updated. TC.
As I see it, you are experiencing intermittant communications. If it works at all, then your ISA server is not to blame. Intermittant communications are a result of the server switches and router flooding the network.

There are a number of server, switches, and router configurations you might need to look at:

I have been mostly trying to create a fix to Network Load Balancing on a Switched network. I have seen a lot of errors pertaining to NLB on a switched network. All of these settings apply to your situation. But the correct combination to your application is what you want from me.

First thing we should do is get the right information from you:
Do you have dual NICS on the server?
What are the mutliple nics used for (VPN, Network Load Balancing, ???)
Can you resort to one NIC?
If you are not using multiple NICS, are you using service pack 1?
With that said:

The errors I have seen with NICS pertain to the following settings on switches, servers, and routers. The settings are Spanning tree, portfast, Multicast/unicast,  Mode of operation for switches and routers, A faulty service pack (2003 server SP1). If not in the correct combination, any of these will cause NIC flooding and intermittant communications with 2003 server services (like DHCP, DNS, WSUS ect...).
Putting NLB over a switched network into perspective: (You should really read this article)

Preventing NIC flooding caused by NLB:
A little explaination of spanning tree and portfast.
(NOTE: Portfast is necessary for XP clients. XP clients will time out otherwise.)

An Event error usually associated with a Spanning tree portfast problem:
Event ID 5719, spanning tree portfast:
The differences between Unicast and Multicast modes:
(The server requires Unicast to work with dual nics, but the nics should be on two separate layers of a switched network)
2003 server Service Pack 1 has a discrepancy that can even cause a single NIC to be flooded.

Service pack 1 problem usually has problems with DHCP, (if applicable), and is also sometimes associated with Event 333.
The mode of operation has to be the same on cisco switches and routers. An example of this problem is 100mb full duplex on a switch while the router is set to negotiate on auto. I recommend both be put on auto.

The errror you are seeing are a result of a misconfigured network. I can't see your network, so we will have to evaluate what settings pertain to you.
datacomsmtAuthor Commented:
Many thanks, the server seems to be stable now.
Amit BhatnagarTechnology Consultant - SecurityCommented:
Perfect!! Do revert back incase the issue comes back again.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Forefront ISA Server

From novice to tech pro — start learning today.