• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 714
  • Last Modified:

Problem with FTP server, in a semi-complex network scenerio that involves: ISA2000, ISA2004, AIX, and Nokia IP330 CheckPoint firewall.

Our Current Environment - for the sake of this problem:

- ISA 2000 Server
- AIX 4.3.3 Server - running an FTP daemon
- Nokia IP330 CheckPoint firewall

Our AIX server runs our business's core application, it also runs an FTP daemon.  This server is multihomed, on one NIC is statically assigned for our 38.213.5.x/24 network which is our internal production network, while the other is setup for our 38.213.10.x/24 network which is a separate network, but production none-the-less.

The 38.213.5.10/24 connection is attached to our internal production network - which is secured on the perimeter by an ISA 2000 firewall - this is our ONE pathway to the internet.  While our 38.213.10.10/24 connection is directly attached to a Nokia IP330 CheckPoint firewall appliance.  On this other side of the CheckPoint firewall is a customer's frame connection.  Our customer comes over their frame connection, through the checkpoint firewall, perform FTP puts and FTP gets in a directory on our AIX server.

Right now, we're under the assumption that our AIX server is acting as a bridge.  We've discussed changing the default gateway, but it was ruled out since we don't house an AIX guru.

Here's the problem:

When our ISA 2000 firewall goes down, our client cannot FTP into our server.

Furthermore, we're actively working on an ISA 2000 to ISA 2004 firewall migration and discovered that when we cut-over to our ISA2004 server, our customer is not able to do any FTP puts/gets.  This doesn't make sense when we look at it on the surface since the 38.213.10.x network is not defined on any of our internal production routers.  Looking at this from above, it looks like everything "should just work."

Where should we start to look, and how do we get this to work?

Please let me know if you need any more info.
0
cjb123
Asked:
cjb123
  • 12
  • 11
  • 10
1 Solution
 
Keith AlabasterEnterprise ArchitectCommented:
Lets stick with the ISA2004 approach as this is vastly superior to the ISA2000 approach. Also ISA2000 went out of standard suupport over a year ago.

I need to know the routing process within gthe network please. What are the dataflows?

       Customer Frame                                                                        Internet
                |                                                                                              |
            Nokia IP330                                                                           ISA2004
            38.213.10.0/24                                                                 38.213.5.0/24
                |                                                                                              |
       -------------------------------------------  AIX box  ---------------------------- -- Internal LAN

Based on this, i would expect anything in the 38.213.10.0 network to have its default gateway pointing at the ip address of the AIX box, including the Nokia.
The AIX box should have a default gateway only on its 38.213.5.0 interface pointing at the ISA2004 internal nic ip address.
Everything on the 38.213.5.0 network should have their default gateway also pointed at the ISA internal nic ip address.
ISa must NOT have a default gatewqay entered on its internal nic, only its external nic.

The ISA server needs to know ALL of the ip address that are visible through its internal nic (open gui, configuration - networks - internal (properties) - addresses. This will include (from your post, 38.213.5.0 - 38.213.5.255, 38.213.10.0 - 38.213.10.255, PLUS ALL of the ip addresses that are on the other side of the Nokia that may arrive at the ISA internal nic. Any ip address that is NOT listed here ISA will expect to ONLY arrive through its external interface. If something that is not listed on its intrernal interface arrives, it will drop and report spoof attacks everywhere zand drop traffic.

remember ISA is NOT a router so routing entries need to be set at the Windows OS level.

From a cmd prompt in windows (on the isa), do a route print. If they are not there, you need to add static routes (using the route -p add syntax) for the 38.213.10.0 network giving it the ip address on the 38.213.5.0 network used by the AIX server.
Give a second static route for the customers frame ip addresses and also give this the AIX server address. ISA now knows how to return the traffic back (at least to the AIX server).

On the AIX box, this will also need a static route for the customers frame ip addresses - I am assuming that this is already in place else nothing would ever work.....

The AIX cannot be acting as a bridge as you have two different subnets - I think the term you mean is a gateway. The point here though is, as you say, that ISA should not be involved in the FTP transfer unless the traffic is being sent there first (for authentication maybe) rather than directly to the AIX box. From the customer, if they perform a traceroute to the AIX box, what results do they get back?

ISA2004 has a neat utility (make sure you have SP3 installed). Open the ISA gui, select monitoring - logging - click start query to begin viewing the real time log.
get the customer to try and make an ftp connection - what do you see in the log? Anything from their ip addresses?

As an aside, ISA2004 installs the FTP rule (by default) as read-only. Right-click the ftp rule and select confure ftp - untick the read-only box and apply the settings & policy.

Keith

0
 
cjb123Author Commented:
A couple more pieces of information..

- the default gateway of our AIX box's 38.213.5.10 NIC network is the 38.213.5.254, which is the IP of one of our Cisco router's interfaces, the LAN router sends all data not known to our LAN out the 0.0.0.0 route to our ISA firewall

- the router does not have the 38.213.10.x/24 network defined

- The interface on our Nokia CheckPoint FW which is connected to our AIX box is 38.213.10.250

- On our AIX box's routing table, an entry has been made for our customer's 172.24.x.x network out to the gateway address of 38.213.10.250


We would like the data flow to be simple and look like this

-----------          FTP >                   ------
Cust Net | -------NOKIA----------  | AIX |
-----------        < FTP                     ------


We've also put a a sniffer on our network on a mirrored port of both the AIX box's 38.213.5.10 interface, and the 192.168.0.2 interface of our ISA2000 box.  No traffic was seen from the 38.213.10.x subnet, or our customer's 172.24.x.x network.  All FTP packets sniffed were normal internal LAN FTP traffic.
0
 
Keith AlabasterEnterprise ArchitectCommented:
Thanks - then the question is what is the nokia doing with the traffic? What are the rules between the two networks? Is the Nokia providing NAT for traffic coming from the customers network or are you just routing between the two networks?
0
Improved Protection from Phishing Attacks

WatchGuard DNSWatch reduces malware infections by detecting and blocking malicious DNS requests, improving your ability to protect employees from phishing attacks. Learn more about our newest service included in Total Security Suite today!

 
cjb123Author Commented:
After researching our documentation on the Nokia.. I found out the following:

Rule 1:

Source hosts:
172.24.144.114, 172.24.144.133, 172.24.145.121, 172.24.145.122 (all of these are set with the location as external)

To the destination hosts:
38.213.10.10 (this location is also set to external)

Service:
FTP

Action:
accept

----
Service allowed is FTP
Port 21
no source port range defined
protocol type is URI
----

on the NAT tab, of the workstation properties - "Add Automatic Address Translation Rules" is checked on, with
the translation method being hide
with a public IP address (that is NOT being used - this firewall used to be the company's primary firewall way back in the day)

-------------------------------------

Rule 2:

Source Host:
38.213.10.10

Destination host :
Any

Service:
Any

Action:
Accept

NAT is the same as rule 1

-------------------------------------

Rule 4:

Source
Any

Destination
Any

Service
Any

Action
DROP

--------------------------------------------------

i also saw screen shots for an HTTP server defined as 38.213.5.10 (which I'm not sure is relavent)
0
 
cjb123Author Commented:
Let me throw this into the mix as well - we also have another ISA2004 firewall that we may migrate to from the Nokia.  We're heavily leaning in that direction since we're not proficient in checkpoint & since the Nokia is a bit old.  After reviewing everything you've seen here, would it simply be replicating these rules onto ISA2004?  Would these weird networking issues still persist?
0
 
Keith AlabasterEnterprise ArchitectCommented:
We are moving into different territory here as the obvious answer is' it depends on the rules on the Nokia. Replacing it with an ISA server is easy but you need to know exactly what it was doing  ie what else is it foing beside the ftp if anything?

If it is 'just' the ftp, then the configuration required is only about three lines.....
0
 
cjb123Author Commented:
All the Nokia is supposed to be doing is allowing FTP traffic between our customer's network, and our AIX server.  If we were to replace the Nokia with an ISA 2004 box, it would be doing the same.  We'd like to avoid this weird networking issue where our internal LAN's firewall is somehow a key component of this design.

Back to the original problem, were you alluding to the idea that if NAT is enabled on the Nokia, it could be causing all of our headaches today??
0
 
Keith AlabasterEnterprise ArchitectCommented:
Only if the nat and the rule combinationon the Nokia were passing the ftp traffic to the ISA address for checking. The checkpoint normally has quite good logging and so should be able to identify the ip address being used as the target for the ftp traffic.
Does the checkpoint show the ftp traffic arriving from the client?
If you disconnect the Checkpoint and put a workstation in its place - using the Cjeckpoint internal Ip address, can that workstation ftp to the AIX box?
recoinnect the checkpoint. Disconnect the Frame connection, can you connect a PC the other side of the checkpoint before it hits the csu/dsu? give it an appropropriate IP address - can it ftp to the AIX box?

I agree with you that the diagram shows that the ISA should be irrelevant to the equation.

0
 
gheistCommented:
I'd suggest installing tcpdump/wireshark on involved end machines so that you can find out what happens when nothing works
0
 
cjb123Author Commented:
I'd like to do that except that one endpoint is our customer and the other endpoint is our AIX box (we don't have an on-site AIX guru).  We do have wireshark running on our internal LAN (on a mirrored port of both our AIX box and our ISA2000 box) but we couldn't find anything  that would come close to explaining why this is happening.  Its just driving us nuts as to why having our ISA2000 box down is causing such chaos on a leg of our network, that technically isn't part of our network.  I think keith was on to something when he asked me about NAT, I'm just waiting for a maintenance window so I can try disabling it, unplugging ISA2000 or switching it over with ISA2004 and see what happens.  
0
 
Keith AlabasterEnterprise ArchitectCommented:
We'll be here waiting.... :)
0
 
gheistCommented:
OK this all is simple. Use ftp command line client, connect to AIX box from client site, do ls -l. If you get timeout your firewall is unable to handle FTP. Dont blame me for this.
0
 
Keith AlabasterEnterprise ArchitectCommented:
Gheist - please read the question and the posts again - I think you have missed the point.
0
 
gheistCommented:
Point #1 - AIX never does bridging
Point #2 - route -n print - look where packets to client are routed. Looks like new ISA does not bounce them to checkpoint.
0
 
cjb123Author Commented:
hello gheist,
we've done a route -n print and everything looks "normal"  there is only one statement that refers to the the 38.213.10.10 interface, and that points to the 172.24.0.0/24 subnet - which is to be expected.  We know that new ISA isn't bouncing them to checkpoint, but I guess the big question is why does ISA matter in this?  It should, by design, just work, and all FTP packets from our customer should be able to come in and go out on that leg of our network that is attached directly to them.

we're still waiting on a maintenance window - probably allllll the way on nov.3 -- we're probably going to swap out the Nokia with an ISA2004 box and pay particular attention to NAT
0
 
gheistCommented:
It may have something to do with DNS setup, where AIX stands still for a minute waiting while it gets nothing from ISA's DNS proxy. One solution is to use /etc/netsvc.conf to add /etc/hosts lookup before DNS lookup, but ultimate solution is to add caching DNS server on AIX itself, so that it holds DNS responses while ISA is down.
Lets us check if my assumption is valid:
echo hosts=local,bind >> /etc/netsvc.conf
add an entry in /etc/hosts for client

now switch off ISA for 5 minutes.

If ftp still works it will keep this way.
If not - problem is in other place.

You may make change more permanent by disabling ftpd's use of DNS by adding -n switch into /etc/inetd.conf
Best solution is DNS cache on local system.
Ask if you need help with that.
0
 
Keith AlabasterEnterprise ArchitectCommented:
ISA does not provide any dns services, it has to use internal DNS the same as all other internal systems.

If the Aix box is responding to a call coming in from a known IP address (albeit on another network subnet), why would it need to use DNS to look up the ip address if it already has it?  If the AIX box was initiating the call I could understand it the rationale here.

May be wrong but .......
0
 
gheistCommented:
Rationale is to display practical textual host names in logs. It is older than you and me.
I doubt you have reverse DNS zone for remote organisation inside your internal DNS server
I guess "internal" DNS cache attempts to reach public DNS servers and look up reverse name for connecting IP address. It definitely uses some log files. If not - tcpdump/wireshark.
Local DNS cache is highly recommened on AIX for flexibility. /etc/resolv.conf in AIX is read once at program startup, and if it points to DNS cache you can change DNS servers.

DNS timeout may cause Checkpoint to forget FTP protoccol states and deny access.
0
 
cjb123Author Commented:
gheist,
I'm having trouble following you..

here's what our /etc/netsvc.conf looks like

$ less /etc/netsvc.conf
hosts=local,bind4



---------------------------------------------------
here what our /etc/resolv.conf looks like

$ less /etc/resolv.conf

domain  hallscorp.com
nameserver      192.168.0.100
nameserver      192.168.0.101


----------------------------------------------
What do you recommend we try?  btw.. when we had wireshark running, we did notice some reverse DNS traffic referring to an IP address on our client's network - i didn't think to mention this before but now that DNS has been mentioned, i hope that helps clear things up a bit.
0
 
gheistCommented:
I guessed correctly - reverse (PTR) DNS lookups time out while Windows 95 firewall bluescreens. Confirmed by your report from traffic sniffer.

Write client's ip address into /etc/hosts
Diagnose problem in controllable environment.

Then use either solution:
a) add -n to ftpd's command line in /etc/inetd.conf and refresh -s inetd
b) Instal local DNS cache on AIX
c) populate /etc/hosts with all client ip addresses

Either will cover times while you wait Windows 95 to reboot.
0
 
Keith AlabasterEnterprise ArchitectCommented:
I am at  a loss here - where does Windows 95 bluescreens come into play?  May be i am just too old....
0
 
gheistCommented:
When your Windows 2004 firewall parody bluescreens your internal DNS servers time out responses to AIX's desperate requests for names for IP addresses connecting to it's FTP service. That exceeds whatever defaults are configured into Nokia-rebranded freebsd pc computer, called IP-whatever, so customers never get working FTP session.

0
 
Keith AlabasterEnterprise ArchitectCommented:
What has blue screened though? I'll keep out of this one. My expertise is networking and ISA Server and i am starting to think that this knowledge is not required here anymore.
0
 
gheistCommented:
Lets me explain who connects where and why.
Your user is launching FTP client that connects to your AIX box on TCP port 21
Connection is intercepted by Checkpoint/Nokia appliance, that is willing to wait for AIX servers banner for 30 seconds or so. If AIX's FTP session commences within 30 seconds user logs in and firewall happily passes through FTP data connections to/from AIX server.
At time of reception of TCP connection AIX server asks your 192.168.0.10[01] DNS server for PTR record for client's IP address solely for logging purposes, allowing for response some 45 seconds
Your DNS server then does full DNS query to servers around the internet starting from root servers up until it finds answer or confirmation that there is no domain name for given IP giving some 60 seconds for each response.
In normal network conditions DNS responses come to your DNS server thru ISA instantly, but in case ISA is down they do not come in time, what causes avalanche of reactions:
AIX waits its maximum time before accepting FTP service
Checkpoint drops connection state, because it was too long
And user never gets connected to FTP service.

Simple solution is to make AIX's ftp service to log IP addresses by editing inetd configuration file and restarting inetd superserver (think services.exe)
Another solution is to make address-> name lookup instant by editing /etc/hosts file and adding missing host there
Yet another is establishing DNS cache with reasonable timeouts on AIX, that gives rejects in case of timeouts eg longer than 5seconds.

It is up to you to decide - Do you need FTP client hostname absolutely always, if network conditions are good, or have no use for it.
Please tell me and I will guide you thru editing AIX config files using "vi" editor.
0
 
cjb123Author Commented:
I'm with keith, I'm confused..... There aren't any Windows bluescreens.. ???  

What i've seen on my sniffer was a query from our AIX box to our Win2003 AD/DNS server for:
PRT 121.145.24.172.in-addr.arpa --- this translates to 172.24.145.121 on our customer's network

Gheist - please keep in mind is that this architecture works fine today, its only when our firewall goes down, or when we try to replace our firewall with a newer version of the software is when FTP doesn't work.

could you point me in the direction of some documentation that would help explain the DNS changes you're proposing?

Also, when you're referring to populating our /etc/hosts file, what would we populate this with?  We have the reverse lookup - and thats just about it.  I think adding an entry there would be the easiest and safest solution.
0
 
gheistCommented:
Request for PRT 121.145.24.172.in-addr.arpa goes to internet from your private DNS servers. It is served quickly when it gets to inernet. If it does not get to internet it causes lengthy waits for AIX's FTP service.

I am suggesting solutions to cope with DNS outages you experience when ISA goes down.

Very basic solution is to make AIX to avoid unnecessary DNS lookup thru breaking down infrastructure.

Let's commence to geting AIX's FTP service to avoid DNS lookups and push DNS infrastructure problem far enough away.

As root user do
#  vi /etc/inetd.conf
(in opened editor type
type /ftp<enter>
use h and l to navigate to -l option (both are small eLL)
type 3x<escape>:wq

in command line type
# refresh -s inetd

Now your FTP server does not look for names for customer's IP addresses.
I cannot help with new ISA 2104 not allowing DNS PTR requests, nor do I know how to track ISA problems.

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds2/ftpd.htm
0
 
cjb123Author Commented:
Keith,
I'd ask you to stay on board - I know these are all viable options, but I think your ISA expertise will definitely help out after we break open our maintenance window

Gheist
Question..
When you said..

"Another solution is to make address-> name lookup instant by editing /etc/hosts file and adding missing host there"

That interested me.. could you please be more specific as to how and why this would help?  I'd rather make this change first and than change our FTP daemon if it doesnt work.  We only allow four IP addresses from our client's network onto our AIX server via the nokia leg of AIX box.  Could you be specific as to what entries we would have to make?  I'm very MS centric, so when I think of adding a reverse lookup, I don't think bout adding it into a hosts file.

Thanks
0
 
gheistCommented:
Let us wait for ISA expertise once you are unable to edit file on AIX
0
 
cjb123Author Commented:
Gheist - i agree with your approach on this to tackle the DNS option first.  are there any dangers to changing this conf file?  would it be easy to change the config back if it causes any problems?  also - if we were to edit the /etc/hosts file as you suggested in previous posts - what exactly would you put in this line?  the IP of our client is 172.24.145.121 - we should get the maintenance window this weekend, so i'm excited to try a couple of these suggestions out
0
 
gheistCommented:
Move /etc/resolv.conf to any place
As a result no DNS lookups will occur.
If you do not like that mobve it back
No file editing involved.
0
 
cjb123Author Commented:
Hello:  Sorry for the delay. We will be working this week to finally resolve the issue and will try your advice. Please keep the thread open for now.  Thanks.
0
 
cjb123Author Commented:
Pls. continue to keep this thread open.  GHeist's suggestions have not been tried yet in our network. Thank you.
0
 
Keith AlabasterEnterprise ArchitectCommented:
marked for deletion - no refund
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Protect Your Employees from Wi-Fi Threats

As Wi-Fi growth and popularity continues to climb, not everyone understands the risks that come with connecting to public Wi-Fi or even offering Wi-Fi to employees, visitors and guests. Download the resource kit to make sure your safe wherever business takes you!

  • 12
  • 11
  • 10
Tackle projects and never again get stuck behind a technical roadblock.
Join Now