?
Solved

Problem with FTP server, in a semi-complex network scenerio that involves:  ISA2000, ISA2004, AIX, and Nokia IP330 CheckPoint firewall.

Posted on 2007-10-17
35
Medium Priority
?
692 Views
Last Modified: 2013-11-17
Our Current Environment - for the sake of this problem:

- ISA 2000 Server
- AIX 4.3.3 Server - running an FTP daemon
- Nokia IP330 CheckPoint firewall

Our AIX server runs our business's core application, it also runs an FTP daemon.  This server is multihomed, on one NIC is statically assigned for our 38.213.5.x/24 network which is our internal production network, while the other is setup for our 38.213.10.x/24 network which is a separate network, but production none-the-less.

The 38.213.5.10/24 connection is attached to our internal production network - which is secured on the perimeter by an ISA 2000 firewall - this is our ONE pathway to the internet.  While our 38.213.10.10/24 connection is directly attached to a Nokia IP330 CheckPoint firewall appliance.  On this other side of the CheckPoint firewall is a customer's frame connection.  Our customer comes over their frame connection, through the checkpoint firewall, perform FTP puts and FTP gets in a directory on our AIX server.

Right now, we're under the assumption that our AIX server is acting as a bridge.  We've discussed changing the default gateway, but it was ruled out since we don't house an AIX guru.

Here's the problem:

When our ISA 2000 firewall goes down, our client cannot FTP into our server.

Furthermore, we're actively working on an ISA 2000 to ISA 2004 firewall migration and discovered that when we cut-over to our ISA2004 server, our customer is not able to do any FTP puts/gets.  This doesn't make sense when we look at it on the surface since the 38.213.10.x network is not defined on any of our internal production routers.  Looking at this from above, it looks like everything "should just work."

Where should we start to look, and how do we get this to work?

Please let me know if you need any more info.
0
Comment
Question by:cjb123
  • 12
  • 11
  • 10
33 Comments
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20095861
Lets stick with the ISA2004 approach as this is vastly superior to the ISA2000 approach. Also ISA2000 went out of standard suupport over a year ago.

I need to know the routing process within gthe network please. What are the dataflows?

       Customer Frame                                                                        Internet
                |                                                                                              |
            Nokia IP330                                                                           ISA2004
            38.213.10.0/24                                                                 38.213.5.0/24
                |                                                                                              |
       -------------------------------------------  AIX box  ---------------------------- -- Internal LAN

Based on this, i would expect anything in the 38.213.10.0 network to have its default gateway pointing at the ip address of the AIX box, including the Nokia.
The AIX box should have a default gateway only on its 38.213.5.0 interface pointing at the ISA2004 internal nic ip address.
Everything on the 38.213.5.0 network should have their default gateway also pointed at the ISA internal nic ip address.
ISa must NOT have a default gatewqay entered on its internal nic, only its external nic.

The ISA server needs to know ALL of the ip address that are visible through its internal nic (open gui, configuration - networks - internal (properties) - addresses. This will include (from your post, 38.213.5.0 - 38.213.5.255, 38.213.10.0 - 38.213.10.255, PLUS ALL of the ip addresses that are on the other side of the Nokia that may arrive at the ISA internal nic. Any ip address that is NOT listed here ISA will expect to ONLY arrive through its external interface. If something that is not listed on its intrernal interface arrives, it will drop and report spoof attacks everywhere zand drop traffic.

remember ISA is NOT a router so routing entries need to be set at the Windows OS level.

From a cmd prompt in windows (on the isa), do a route print. If they are not there, you need to add static routes (using the route -p add syntax) for the 38.213.10.0 network giving it the ip address on the 38.213.5.0 network used by the AIX server.
Give a second static route for the customers frame ip addresses and also give this the AIX server address. ISA now knows how to return the traffic back (at least to the AIX server).

On the AIX box, this will also need a static route for the customers frame ip addresses - I am assuming that this is already in place else nothing would ever work.....

The AIX cannot be acting as a bridge as you have two different subnets - I think the term you mean is a gateway. The point here though is, as you say, that ISA should not be involved in the FTP transfer unless the traffic is being sent there first (for authentication maybe) rather than directly to the AIX box. From the customer, if they perform a traceroute to the AIX box, what results do they get back?

ISA2004 has a neat utility (make sure you have SP3 installed). Open the ISA gui, select monitoring - logging - click start query to begin viewing the real time log.
get the customer to try and make an ftp connection - what do you see in the log? Anything from their ip addresses?

As an aside, ISA2004 installs the FTP rule (by default) as read-only. Right-click the ftp rule and select confure ftp - untick the read-only box and apply the settings & policy.

Keith

0
 

Author Comment

by:cjb123
ID: 20096226
A couple more pieces of information..

- the default gateway of our AIX box's 38.213.5.10 NIC network is the 38.213.5.254, which is the IP of one of our Cisco router's interfaces, the LAN router sends all data not known to our LAN out the 0.0.0.0 route to our ISA firewall

- the router does not have the 38.213.10.x/24 network defined

- The interface on our Nokia CheckPoint FW which is connected to our AIX box is 38.213.10.250

- On our AIX box's routing table, an entry has been made for our customer's 172.24.x.x network out to the gateway address of 38.213.10.250


We would like the data flow to be simple and look like this

-----------          FTP >                   ------
Cust Net | -------NOKIA----------  | AIX |
-----------        < FTP                     ------


We've also put a a sniffer on our network on a mirrored port of both the AIX box's 38.213.5.10 interface, and the 192.168.0.2 interface of our ISA2000 box.  No traffic was seen from the 38.213.10.x subnet, or our customer's 172.24.x.x network.  All FTP packets sniffed were normal internal LAN FTP traffic.
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20096520
Thanks - then the question is what is the nokia doing with the traffic? What are the rules between the two networks? Is the Nokia providing NAT for traffic coming from the customers network or are you just routing between the two networks?
0
Automating Your MSP Business

The road to profitability.
Delivering superior services is key to ensuring customer satisfaction and the consequent long-term relationships that enable MSPs to lock in predictable, recurring revenue. What's the best way to deliver superior service? One word: automation.

 

Author Comment

by:cjb123
ID: 20096847
After researching our documentation on the Nokia.. I found out the following:

Rule 1:

Source hosts:
172.24.144.114, 172.24.144.133, 172.24.145.121, 172.24.145.122 (all of these are set with the location as external)

To the destination hosts:
38.213.10.10 (this location is also set to external)

Service:
FTP

Action:
accept

----
Service allowed is FTP
Port 21
no source port range defined
protocol type is URI
----

on the NAT tab, of the workstation properties - "Add Automatic Address Translation Rules" is checked on, with
the translation method being hide
with a public IP address (that is NOT being used - this firewall used to be the company's primary firewall way back in the day)

-------------------------------------

Rule 2:

Source Host:
38.213.10.10

Destination host :
Any

Service:
Any

Action:
Accept

NAT is the same as rule 1

-------------------------------------

Rule 4:

Source
Any

Destination
Any

Service
Any

Action
DROP

--------------------------------------------------

i also saw screen shots for an HTTP server defined as 38.213.5.10 (which I'm not sure is relavent)
0
 

Author Comment

by:cjb123
ID: 20097072
Let me throw this into the mix as well - we also have another ISA2004 firewall that we may migrate to from the Nokia.  We're heavily leaning in that direction since we're not proficient in checkpoint & since the Nokia is a bit old.  After reviewing everything you've seen here, would it simply be replicating these rules onto ISA2004?  Would these weird networking issues still persist?
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20098995
We are moving into different territory here as the obvious answer is' it depends on the rules on the Nokia. Replacing it with an ISA server is easy but you need to know exactly what it was doing  ie what else is it foing beside the ftp if anything?

If it is 'just' the ftp, then the configuration required is only about three lines.....
0
 

Author Comment

by:cjb123
ID: 20100202
All the Nokia is supposed to be doing is allowing FTP traffic between our customer's network, and our AIX server.  If we were to replace the Nokia with an ISA 2004 box, it would be doing the same.  We'd like to avoid this weird networking issue where our internal LAN's firewall is somehow a key component of this design.

Back to the original problem, were you alluding to the idea that if NAT is enabled on the Nokia, it could be causing all of our headaches today??
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20102947
Only if the nat and the rule combinationon the Nokia were passing the ftp traffic to the ISA address for checking. The checkpoint normally has quite good logging and so should be able to identify the ip address being used as the target for the ftp traffic.
Does the checkpoint show the ftp traffic arriving from the client?
If you disconnect the Checkpoint and put a workstation in its place - using the Cjeckpoint internal Ip address, can that workstation ftp to the AIX box?
recoinnect the checkpoint. Disconnect the Frame connection, can you connect a PC the other side of the checkpoint before it hits the csu/dsu? give it an appropropriate IP address - can it ftp to the AIX box?

I agree with you that the diagram shows that the ISA should be irrelevant to the equation.

0
 
LVL 62

Expert Comment

by:gheist
ID: 20112510
I'd suggest installing tcpdump/wireshark on involved end machines so that you can find out what happens when nothing works
0
 

Author Comment

by:cjb123
ID: 20133191
I'd like to do that except that one endpoint is our customer and the other endpoint is our AIX box (we don't have an on-site AIX guru).  We do have wireshark running on our internal LAN (on a mirrored port of both our AIX box and our ISA2000 box) but we couldn't find anything  that would come close to explaining why this is happening.  Its just driving us nuts as to why having our ISA2000 box down is causing such chaos on a leg of our network, that technically isn't part of our network.  I think keith was on to something when he asked me about NAT, I'm just waiting for a maintenance window so I can try disabling it, unplugging ISA2000 or switching it over with ISA2004 and see what happens.  
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20133376
We'll be here waiting.... :)
0
 
LVL 62

Expert Comment

by:gheist
ID: 20136723
OK this all is simple. Use ftp command line client, connect to AIX box from client site, do ls -l. If you get timeout your firewall is unable to handle FTP. Dont blame me for this.
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20136772
Gheist - please read the question and the posts again - I think you have missed the point.
0
 
LVL 62

Expert Comment

by:gheist
ID: 20141257
Point #1 - AIX never does bridging
Point #2 - route -n print - look where packets to client are routed. Looks like new ISA does not bounce them to checkpoint.
0
 

Author Comment

by:cjb123
ID: 20147846
hello gheist,
we've done a route -n print and everything looks "normal"  there is only one statement that refers to the the 38.213.10.10 interface, and that points to the 172.24.0.0/24 subnet - which is to be expected.  We know that new ISA isn't bouncing them to checkpoint, but I guess the big question is why does ISA matter in this?  It should, by design, just work, and all FTP packets from our customer should be able to come in and go out on that leg of our network that is attached directly to them.

we're still waiting on a maintenance window - probably allllll the way on nov.3 -- we're probably going to swap out the Nokia with an ISA2004 box and pay particular attention to NAT
0
 
LVL 62

Expert Comment

by:gheist
ID: 20148566
It may have something to do with DNS setup, where AIX stands still for a minute waiting while it gets nothing from ISA's DNS proxy. One solution is to use /etc/netsvc.conf to add /etc/hosts lookup before DNS lookup, but ultimate solution is to add caching DNS server on AIX itself, so that it holds DNS responses while ISA is down.
Lets us check if my assumption is valid:
echo hosts=local,bind >> /etc/netsvc.conf
add an entry in /etc/hosts for client

now switch off ISA for 5 minutes.

If ftp still works it will keep this way.
If not - problem is in other place.

You may make change more permanent by disabling ftpd's use of DNS by adding -n switch into /etc/inetd.conf
Best solution is DNS cache on local system.
Ask if you need help with that.
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20149457
ISA does not provide any dns services, it has to use internal DNS the same as all other internal systems.

If the Aix box is responding to a call coming in from a known IP address (albeit on another network subnet), why would it need to use DNS to look up the ip address if it already has it?  If the AIX box was initiating the call I could understand it the rationale here.

May be wrong but .......
0
 
LVL 62

Expert Comment

by:gheist
ID: 20149747
Rationale is to display practical textual host names in logs. It is older than you and me.
I doubt you have reverse DNS zone for remote organisation inside your internal DNS server
I guess "internal" DNS cache attempts to reach public DNS servers and look up reverse name for connecting IP address. It definitely uses some log files. If not - tcpdump/wireshark.
Local DNS cache is highly recommened on AIX for flexibility. /etc/resolv.conf in AIX is read once at program startup, and if it points to DNS cache you can change DNS servers.

DNS timeout may cause Checkpoint to forget FTP protoccol states and deny access.
0
 

Author Comment

by:cjb123
ID: 20150482
gheist,
I'm having trouble following you..

here's what our /etc/netsvc.conf looks like

$ less /etc/netsvc.conf
hosts=local,bind4



---------------------------------------------------
here what our /etc/resolv.conf looks like

$ less /etc/resolv.conf

domain  hallscorp.com
nameserver      192.168.0.100
nameserver      192.168.0.101


----------------------------------------------
What do you recommend we try?  btw.. when we had wireshark running, we did notice some reverse DNS traffic referring to an IP address on our client's network - i didn't think to mention this before but now that DNS has been mentioned, i hope that helps clear things up a bit.
0
 
LVL 62

Expert Comment

by:gheist
ID: 20150586
I guessed correctly - reverse (PTR) DNS lookups time out while Windows 95 firewall bluescreens. Confirmed by your report from traffic sniffer.

Write client's ip address into /etc/hosts
Diagnose problem in controllable environment.

Then use either solution:
a) add -n to ftpd's command line in /etc/inetd.conf and refresh -s inetd
b) Instal local DNS cache on AIX
c) populate /etc/hosts with all client ip addresses

Either will cover times while you wait Windows 95 to reboot.
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20150627
I am at  a loss here - where does Windows 95 bluescreens come into play?  May be i am just too old....
0
 
LVL 62

Expert Comment

by:gheist
ID: 20150916
When your Windows 2004 firewall parody bluescreens your internal DNS servers time out responses to AIX's desperate requests for names for IP addresses connecting to it's FTP service. That exceeds whatever defaults are configured into Nokia-rebranded freebsd pc computer, called IP-whatever, so customers never get working FTP session.

0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 20151009
What has blue screened though? I'll keep out of this one. My expertise is networking and ISA Server and i am starting to think that this knowledge is not required here anymore.
0
 
LVL 62

Expert Comment

by:gheist
ID: 20151153
Lets me explain who connects where and why.
Your user is launching FTP client that connects to your AIX box on TCP port 21
Connection is intercepted by Checkpoint/Nokia appliance, that is willing to wait for AIX servers banner for 30 seconds or so. If AIX's FTP session commences within 30 seconds user logs in and firewall happily passes through FTP data connections to/from AIX server.
At time of reception of TCP connection AIX server asks your 192.168.0.10[01] DNS server for PTR record for client's IP address solely for logging purposes, allowing for response some 45 seconds
Your DNS server then does full DNS query to servers around the internet starting from root servers up until it finds answer or confirmation that there is no domain name for given IP giving some 60 seconds for each response.
In normal network conditions DNS responses come to your DNS server thru ISA instantly, but in case ISA is down they do not come in time, what causes avalanche of reactions:
AIX waits its maximum time before accepting FTP service
Checkpoint drops connection state, because it was too long
And user never gets connected to FTP service.

Simple solution is to make AIX's ftp service to log IP addresses by editing inetd configuration file and restarting inetd superserver (think services.exe)
Another solution is to make address-> name lookup instant by editing /etc/hosts file and adding missing host there
Yet another is establishing DNS cache with reasonable timeouts on AIX, that gives rejects in case of timeouts eg longer than 5seconds.

It is up to you to decide - Do you need FTP client hostname absolutely always, if network conditions are good, or have no use for it.
Please tell me and I will guide you thru editing AIX config files using "vi" editor.
0
 

Author Comment

by:cjb123
ID: 20151223
I'm with keith, I'm confused..... There aren't any Windows bluescreens.. ???  

What i've seen on my sniffer was a query from our AIX box to our Win2003 AD/DNS server for:
PRT 121.145.24.172.in-addr.arpa --- this translates to 172.24.145.121 on our customer's network

Gheist - please keep in mind is that this architecture works fine today, its only when our firewall goes down, or when we try to replace our firewall with a newer version of the software is when FTP doesn't work.

could you point me in the direction of some documentation that would help explain the DNS changes you're proposing?

Also, when you're referring to populating our /etc/hosts file, what would we populate this with?  We have the reverse lookup - and thats just about it.  I think adding an entry there would be the easiest and safest solution.
0
 
LVL 62

Expert Comment

by:gheist
ID: 20151386
Request for PRT 121.145.24.172.in-addr.arpa goes to internet from your private DNS servers. It is served quickly when it gets to inernet. If it does not get to internet it causes lengthy waits for AIX's FTP service.

I am suggesting solutions to cope with DNS outages you experience when ISA goes down.

Very basic solution is to make AIX to avoid unnecessary DNS lookup thru breaking down infrastructure.

Let's commence to geting AIX's FTP service to avoid DNS lookups and push DNS infrastructure problem far enough away.

As root user do
#  vi /etc/inetd.conf
(in opened editor type
type /ftp<enter>
use h and l to navigate to -l option (both are small eLL)
type 3x<escape>:wq

in command line type
# refresh -s inetd

Now your FTP server does not look for names for customer's IP addresses.
I cannot help with new ISA 2104 not allowing DNS PTR requests, nor do I know how to track ISA problems.

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds2/ftpd.htm
0
 

Author Comment

by:cjb123
ID: 20158362
Keith,
I'd ask you to stay on board - I know these are all viable options, but I think your ISA expertise will definitely help out after we break open our maintenance window

Gheist
Question..
When you said..

"Another solution is to make address-> name lookup instant by editing /etc/hosts file and adding missing host there"

That interested me.. could you please be more specific as to how and why this would help?  I'd rather make this change first and than change our FTP daemon if it doesnt work.  We only allow four IP addresses from our client's network onto our AIX server via the nokia leg of AIX box.  Could you be specific as to what entries we would have to make?  I'm very MS centric, so when I think of adding a reverse lookup, I don't think bout adding it into a hosts file.

Thanks
0
 
LVL 62

Expert Comment

by:gheist
ID: 20160849
Let us wait for ISA expertise once you are unable to edit file on AIX
0
 

Author Comment

by:cjb123
ID: 20174256
Gheist - i agree with your approach on this to tackle the DNS option first.  are there any dangers to changing this conf file?  would it be easy to change the config back if it causes any problems?  also - if we were to edit the /etc/hosts file as you suggested in previous posts - what exactly would you put in this line?  the IP of our client is 172.24.145.121 - we should get the maintenance window this weekend, so i'm excited to try a couple of these suggestions out
0
 
LVL 62

Expert Comment

by:gheist
ID: 20181192
Move /etc/resolv.conf to any place
As a result no DNS lookups will occur.
If you do not like that mobve it back
No file editing involved.
0
 

Author Comment

by:cjb123
ID: 20317214
Hello:  Sorry for the delay. We will be working this week to finally resolve the issue and will try your advice. Please keep the thread open for now.  Thanks.
0
 

Author Comment

by:cjb123
ID: 20413948
Pls. continue to keep this thread open.  GHeist's suggestions have not been tried yet in our network. Thank you.
0
 
LVL 51

Accepted Solution

by:
Keith Alabaster earned 1500 total points
ID: 20749097
marked for deletion - no refund
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
So the following errors occurs in 2 ways that I am aware of at this stage, and you receive one of the following error messages: ERROR 1. When trying to save a rule: No Web listener is specified for the Web publishing rule Autodiscovery Publishin…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses
Course of the Month17 days, 4 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question