Solved

ISA 2006 deniesconnections to outside database

Posted on 2007-11-20
21
621 Views
Last Modified: 2008-11-17
Hi, I have an ISA 2006 Enterprise server installed on windows 2003 Ent SP2. Its connected to two interfaces, one representing the internal network and the other the external network.

I have one rule on right now allowing all outbound traffic from the internal network and localhost to go to the external network for all users.

I am trying to connect to a remote oracle database. The connectivity is there most of the time but its unstable, meaning that clients in the internal networks are denied access. I test the reliability of the connection using TNSPING80.EXE (An oracle utility used to test database connectivity).

I am able to run the command from any client on the network. In less than a minute I re-ran the same command about 80 times at which point the ISA blocks all and any consecutive attempt by any client in the network. It will keep denying me and everyone else on the network for about the next 10 minutes.

The funny thing is that the ISA server itself can connect to that database. I have installed the oracle client on the ISA server and i was able to TNSPING out as many times I want.

I tried to play around with the ISA configuration but there isnt much to configure. I am hoping someone has seen a similar behaviour before and can help in resolving the issue.

Thank you.

Regards,

-Amer

0
Comment
Question by:amersharaf
  • 10
  • 8
21 Comments
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
ISA Server is treating your access as an attempted hack / denial of service.  You are creating a large number of packets in quick succession and it infers that it is being attacked.  I don't think you'll have a problem in practice.

You can adjust the attack mitigation settings in the Configuration --> General --> Additional Security Policy section of ISA Server Management.

Hope that helps.
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
Make sure logging is enabled and configure the log to display all fields.  You can often see more in the RESULT CODE column, and there has been a recent update that provides a lot more information when you select a log entry.
0
 

Author Comment

by:amersharaf
Comment Utility
Thanks for the feedback. I have already played around with the mitigation settings, but it had no effect. I had a closer look at the logs and it seems that after sending out over 80 or so TNSPINGS from the client to the remote database, at which point the client cant TNSPING again,  the database i am trying to communicate with is trying to talk back to the client using the global IP and some random port which i assume was initiated by the NATed Client in TNSPING request. As you recall a client in the network can TNSPING only 80 or so times then the client and any other client in the network cant communicate with the remote database server (unless its the ISA server itself) for the next 10 minutes. As per the logs Any attempts to send a TNSPING request from any client to the remote database server will be initiated as allowed outgoing communication but the necessary response from the database server is not given instead the remote database server tried to talk back on a random port which the ISA tread as denied traffic since its not awar of this random port being open.

I cant seem to understand why the remote database server is trying to talk back to the NATed Client. I can still TNSPING the remote database all i want from the ISA server even if the NATed client is being denied. When the TNSPING request comes from the ISA directly there is no talk back request from the remote database as per the logs even if I TNSPING over 300 times.

I tried to add rules to allow the remote database server access to anywhere on any port in the internal network, but that didnt help.

MY assumption is, ISA is having an issue with NATed users where by it cant take more than 80 or so requests to a single remote server. perhaps it has to do with the number of ports NATed to a single remote IP? My other hunch is that the remote database has a smart firewall that can tell a NATed user is behind the request.

Any idea?

Thank you.
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
ISA is fairly clever in its use of ports.  Basic NAT solutions can only support about 50000 NAT'd communications at a time, but ISA is cleverer and links sources to ports.  The problem may be to do with delays.  It sounds like the NAT relationship is breaking down, so that ISA denies the connection because it no longer believes the connection is valid.  This could occur in two ways:

1)  TCP connections break down when the client closes the connection.
2)  UDP connections break down after a certain period of time.

Can you post your TNS configuration file (changing IP addresses and server names for security)?
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
BTW, this NAT issue would explain why ISA doesn't have a problem.
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
This article may also be relevant:

http://support.microsoft.com/kb/927695/en-us
0
 

Author Comment

by:amersharaf
Comment Utility
Hi Steve,

Below is the TNS Names entry

APP =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.50)(PORT = 1571))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = SRVNAME)
       (SERVER = DEDICATED)
    )
  )  

The above TNS is confirmed to work by other branches. Fortuantly they are not using ISA so they are not facing similar issues.

-Thank you  
0
 

Author Comment

by:amersharaf
Comment Utility
I will try the suggested workaround in the KB article you've linked and let you know. Thanks
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
The address you have shown in the TNS Names has a private address (192.168.x.x).  Routers and firewalls will not pass these addresses unless told to, because they are designed to be private.

Where is the remote Oracle server in relation to your users and ISA firewall?
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:amersharaf
Comment Utility
Actually its a global IP i just changed it to 192.168.1.50 for security reasons, sorry for being misleading. So the Database is indeed remote.

I have tried the KB article you suggested but that didnt help. I managed to get ahold of the remote DB admin and he informed me that its most likely an ISA issue. He added that the ISA is creating half TCP connections when coming from NATed users and because of that their firewalls are denying the request after a certain number of requests. He also added out of experiance that this has to do with the use of non-standard ports passing through the ISA and supposedly there is some work around to that which i am still searching for.
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
See http://forums.oracle.com/forums/thread.jspa?messageID=2033750

I think you need to set the USE_SHARED_SOCKET option.  See http://www.fus-goodrich1.com/imdkb/GP00016%20Oracle%20Limited%20Port%20Operation.htm for more.

The typical behaviour of Oracle is to redirect to another port.  And ISA blocks that!  You may also want to allow all dynamic (1025 - 65535) TCP ports access to the Oracle server.  So long as these are outbound rules and IP restricted, it should be fairly secure, but certainly not ideal.

It may work from the ISA Server because there isn't a NAT issue.
0
 

Author Comment

by:amersharaf
Comment Utility
I had to come back again in the evening while the users are off. Anyway I am more convinced now that the issue at hand is entirely an ISA issue. I TNSPINGed from an internal user and as usual the TNS ping eventually fails with usual TNS-12535 error and the ISA logs clearly show that the oracle server is trying to communicate back on each consecutive TNSPING request after the eminant failure. I restarted the Microsoft Firewall server (ISA service) and i was able to TNSPING again! Ofcourse the same problem will result again after 80-90 trials or so.

I though this might shed some light and help limiting the possible causes of the problems.
0
 

Author Comment

by:amersharaf
Comment Utility
This is what happens when an internal client tries to get in touch with the remote oracle database after TNSPING starts to fail, as detailed by the logs

Entry 1: Internal Client trying to TNSPING
===============================

Original Client IP: 133.1.11.3
Server Name: ISASERVERNAME
Transport: TCP
GMT Log time: 11/26/2007 4:56:38 PM
Source Port: 4254
Result Code: 0x0 ERROR_SUCCESS
Log Record Type: Firewall
Log Time: 11/26/2007 8:56:38 PM
Destination IP: 213.42.ORACLE.REMOTEIP
Destination Port: 1571
Protocol: MY-Protocol
Action: Initiated Connection
Rule: -
Client IP: 133.1.11.3
Source Network: Internal
Destination Network: External

Entry 2: Oracle Server trying to connect back
=================================

Original Client IP: 213.42.ORACLE.REMOTEIP
Server Name: ISASERVERNAME
Transport: TCP
GMT Log time: 11/26/2007 8:56:38 PM
Source Port: 1571
Result Code: 0xc0040034 FWX_E_SEQ_ACK_MISMATCH
Log Record Type: Firewall
Log Time: 11/26/2007 8:56:38 PM
Destination IP: 213.42.MY.WANINTERFACE
Destination Port: 16359
Protocol: Unidentified IP Traffic
Action: Denied Connection
Rule: -
Client IP: 213.42.ORACLE.REMOTEIP
Source Network: External
Destination Network: Local Host
0
 

Author Comment

by:amersharaf
Comment Utility
Before the TNSPING starts to fails at the internal clients end connections initiate as in the previous post and end succesfully with the following log entry on the ISA:

Original Client IP: 133.1.11.3
Server Name: ISASERVERNAME
Transport: TCP
GMT Log time: 11/26/2007 5:30:04 PM
Source Port: 4438
Result Code: 0x80074e20 FWX_E_GRACEFUL_SHUTDOWN
Log Record Type: Firewall
Log Time: 11 11/26/2007 9:30:04 PM
Destination IP: 213.42.ORACLE.REMOTEIP
Destination Port: 1571
Protocol: MYCustomerProtocol
Action: Closed Connection
Rule: MYCustomRule
Client IP: 133.1.11.3
Source Network: Internal
Destination Network: External
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
I'm not sure that Oracle is recognising multiple connections correctly.  This is in line with the comment about the shared socket option.

When a client connects to the TNS listener, Oracle tells the client which port to continue discussions on.  I think that it is not differentiating clients except by IP address, and ISA doesn't know who's who, because Oracle is not responding solely on the initiatied connections.

You really need your Oracle DBA to configure a listener using the USE_SHARED_SOCKET option, and do the same on the clients.

See http://download-uk.oracle.com/docs/cd/B14117_01/win.101/b10113/ap_net.htm for Oracle docs on this.
0
 

Author Comment

by:amersharaf
Comment Utility
I will talk to the DBA admin first thing tommorow morning.

While waiting for your post i googled 0xc0040034 FWX_E_SEQ_ACK_MISMATCH and found many useful leads, I have tried them all unfortunatly none worked:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;936594
http://www.experts-exchange.com/Microsoft/Windows_Security/Microsoft_ISA/Q_22579862.html
http://msmvps.com/blogs/thenakedmvp/archive/2007/03/12/broadcom-toe-dell-isa-2004-and-my-headache.aspx
http://www.it-etc.com/2007/10/10/vmware-host-and-guest-cannot-communicate-over-network-shares-on-dell-poweredge-and-broadcom-toe/
http://support.microsoft.com/kb/939455

I am running a DELL 1950 server . I have updated the bios and disabled TOE both on windows and phsyically as well. I have also made all the registrey changes disabling TCPChimney, TCPA, and RSS. I also updated the on board Broadcom NetXtreem II Network Adapters All of which didnt help.

Just wanted to share.
0
 
LVL 19

Expert Comment

by:SteveH_UK
Comment Utility
Sounds like you're doing all the right things, but I do think that the shared socket is a real killer.  What Oracle is doing, virtually no other system does, and it really gets in the way of typical firewall behaviours.
0
 
LVL 19

Accepted Solution

by:
SteveH_UK earned 250 total points
Comment Utility
I am confident that my solution was an accurate one.  Oracle normally creates secondary connections a bit like FTP, and if you don't configure it to do otherwise then you need to make sure the secondary connections are allowed through the firewall.  The Oracle documentation explains the behaviour quite well.

The author doesn't appear to have followed that recommendation.
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

In all versions of ISA Server and the current version of FTMG, the default https protocol uses TCP port 443 and 563 only. This cannot be changed within the ISA or FTMG GUI and must be completed from a Windows cmd prompt on the ISA Server itself. …
In Africa (and potentially where you live…), reliability of ISPs is questionable.  With the increased reliance on e-mail as one of the primary forms of communication, the costs to business are significant based on interuption of ISP Connectivity.  T…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now