Client unable to connect to Citrix session through ISA Server, PIX, and routed network to the internet

Does TRACERT from the requestor take you to the correct Citrix server ?
As you server farm is in front of the ISA Server, then surely you can take the ISA Server out of the troubleshooting ?

The citrix server is a internet based system not under our control.  The clients connect to the citrix server by browsing to the website and clicking a link.  This launches the citrix client.  The ISA server is necessary, because that's how the offsite facility gets internet access.  Their internet properties, through IE, are modified to use the ISA server as a Proxy server.

We have been able to get this to work from the subnet that the ISA server resides on.  We set the default gateway of one of the servers in the server farm (DMZ 2) to the inside address of the ISA server and it worked fine.  I can't get this to work on the other side of the PIX firewall.

I believe the problem lies within the PIX, because SecureNAT needs each router in line to default route to the next router.  See the hyperlink above.  The only device in the chain that doesn't do this is the PIX.

The client's default gateway points to the local router interface on the Branch office router.
The Branch office router default routes to the Serial (T1) interface of the 7204 router.
The 7204 router default routes to the DMZ 1 interface of the PIX 525.
The PIX 525 does NOT default route to the ISA server.  Instead the PIX "routes" the traffic for DMZ 2 out its DMZ 2 interface via the CONNECT route (locally connected interface).  Following the rules for SecureNAT, wouldn't you need to point the PIX to default route the DMZ 2 traffic to the ISA Server (next hop address)?

Everything prior to the DMZ 2 interface of the PIX is pointing to the next hop path.  I believe the break is there.

The PIX is a firewall, and not a router, which is possibly where the problem lies in that case, although you can have mutiple static routes and default routes on the PIX so it knows where to send packets ?

eg - 'ip route inside' and 'ip route outside'

This isn't a DNS problem, is it ?  Can you ping the destination Citrix server and resolve the name OK ?  

Again, can you TRACERT to the server in question ?

Does the web page work on the ISA Server itself ?

My answers:

The PIX is a firewall, and not a router, which is possibly where the problem lies in that case, although you can have mutiple static routes and default routes on the PIX so it knows where to send packets ?
ANSWER:    I understand that the PIX isn't a router, but can't the PIX route traffic to any directly connected interface?  I say that because there is a CONNECT route pointing traffic to the DMZ 2 interface.  I am assuming the CONNECT route means its a directly connected interface, similar to a router.  It shows up as connect when I do a show route command on the PIX.  There is no adminstratively entered route in the PIX directing traffic to that subnet.  I tried to enter one and it said it already existed (Besides that, the client can get to the internet with no problems, using the ISA proxy.  If routing wasn't correct, they wouldn't be able to create a connection to the ISA server or any of the servers on the server farm (the ISA and server farm are in the same subnet).

This isn't a DNS problem, is it ?  Can you ping the destination Citrix server and resolve the name OK ?  
ANSWER:    You can browse to the internet via the proxy.  So name resolution shouldn't be a problem.  They are typing in the URL and the systems are using the same DNS server that the proxy is.

Again, can you TRACERT to the server in question ?
ANSWER:    At this point no.  The user cannot tracert across the T1.  I get the local interface hit and the serial (T1) interface, but nothing beyond that.  ICMP is probably being blocked from that point forward.

Does the web page work on the ISA Server itself ?
ANSWER:    Yes, and from another server on the DMZ 2 subnet.


There is one place that this works on our network.  From a system that has internet access from the "inside" interface of the PIX.  These addresses are static translations to the "outside" interface.  From these devices, it works fine.  We have thought about trying to bypass the ISA server with another static translation, but we'd like to get it to work through the ISA.  We were told from the website people that with ISA the connection from ISA needs to be a SecureNAT connection.  I don't think we are making the SecureNAT connection from the client to the ISA.

Moderator, how do I increase points for this question...  I would like to go to 1000.

You can setup routes on the PIX for it to find next hops - does it work if you put a default route in to point to the ISA Server ?  When I say it can't route, it can't send traffic out of the same interface it received it, due to security design.
CONNECT is as you say - a directly connected interface.
The SecureNAT client works by using the ISA server as it's default gateway, and sending all traffic to the ISA server.  The ISA Server firewall then NATs the connection (if applicable) using its firewall service.
The problem here is that there's a PIX in the way, so clients will be using the PIX as the default gateway, right ?
How about putting an access-list in the PIX to allow the Citrix Requestor access to the Internet, instead of going through ISA ?
Citrix and whomever are correct in saying the only way it will work through ISA is to use SecureNAT  - so, why not bypass ISA and just use the PIX ?   Citrix clients work fine without ISA...
SecureNAT is just a layer 3 routing medium.

You can setup routes on the PIX for it to find next hops - does it work if you put a default route in to point to the ISA Server ?
Answer:  I tried to create a route pointing to the ISA Server.  It said that a route already exists to that network.

The SecureNAT client works by using the ISA server as it's default gateway, and sending all traffic to the ISA server.  The ISA Server firewall then NATs the connection (if applicable) using its firewall service.
Answer:  Exactly, We've already proven that we can do this from the DMZ 2 network.  Setting the default gateway on another server in the server farm worked perfectly.

The problem here is that there's a PIX in the way, so clients will be using the PIX as the default gateway, right ?
Answer:  If the clients resided in the DMZ 1 network, they would use the PIX interface 192.168.3.2 for the default gateway.  The problem is they are a couple hops from the PIX.  According to the link about SecureNAT I posted above, you need to set the client's default gateway to the closest router (like any other type of network).  You also need to point the routers default gateway (route) to the next hop interface.  Using this method, I would also have to tell the PIX to default route traffic to the inside interface of the ISA server.  Right now that doesn't work.  The PIX will not let me forward traffic directly to the inside interface of the ISA, because that is a directly connected route.

How about putting an access-list in the PIX to allow the Citrix Requestor access to the Internet, instead of going through ISA ?
Answer:  We already did this and it works.  We used static translation from the DMZ #1 to DMZ #3 (not shown).  DMZ #3 is the main internet connection for us, but it is not maintained by us (meaning Websense access).  The problem is that we need the ISA server to administrate the Websense for a separate set of clients (that need access the DMZ #2 for server applications, internet, etc.) we provide internet access for.  It is important that we maintain the websense to implement changes quickly.


It seems to me that the SecureNAT rules are broken by the PIX, because it is not forwarding traffic directly to the next hop (ISA).  Does this sound reasonable?  Seems to be a design problem.  Would putting a router between the PIX 525 and the ISA server help to solve this problem?

Here's a quick PIX question.  How many default routes can you set?  One or one for each interface?

One default route per interface...

Can you traceroute from the requester to the ISA Server ?  I realise ICMP may be blocked, but in that case, can you unblock it so we can test this ?
Do the routes on the ISA Server point back to the PIX in order to get back to the requester ?
Is it possible the PIX is NATting the requester's IP address and sending the request out to the Internet instead of via the ISA server in the DMZ ?
As I said before, SecureNAT is layer 3 - I don't see a problem getting this to work as long as the routing and any NAT is setup correctly, as packets will traverse multiple router hops unchanged.

I tried tracerouting like you recommended.  I tried to get back to routing 101, and logically step through each device along the way to find out where the break is.  My initial gut feelings were right.  The problem is in the PIX.  The traffic is getting diverted out of the PIX 525 (I am guessing here.  I say this, because the only default route in the PIX points traffic out the OUTSIDE interface)....the wrong way.  If you traceroute, the traffic flows right into the PIX....and it gets forwarded out the "Outside" interface (not shown in the diagram above) or dropped because its not NAT'd.  I recently found this out this weekend, after pouring over the routing tables on the devices along the way.

Part of the problem is that our whole network was setup by outside consultants.  Now, I'm not blaming those guys....but the person that was dealing directly with them knows very little about networking.  So, no one was auditing their work and their design philosophy.  Here is a better diagram detailing what goes into and out of the PIX:


                                             Outside      DMZ #1
                                                       |     |
                                                       |     |
                                                       |     |
                DMZ #3       --------------[PIX 525]----------------   DMZ #2
                                                          |
                                                          |
                                                          |
                                                       Inside

Now, that you can see the whole picture, here is some more information.

The Outside interface is the main company's internet connection.  It connects to a directly connected router
The Inside interface hosts both the main company's systems and the systems that need access to the ISA server.  It is connects to a L3 switch.
The DMZ #3 interface is an internet connection the hosts our VPN traffic.  This connects directly to the VPN Concentrator.
The DMZ #2 interface is the server farm that contains the servers and the inside interface of the ISA server.  As a side note, the outside interface of the ISA is in the DMZ #1 subnet.  Why it is like this is beyond me!  Talk about security.
The DMZ #1 interface connects to the T3 router.  This hosts branch office computers from the main company and users of DMZ #2.

So to wrap up...there are 3 internet connections that the PIX can send traffic to.  I have figured out a couple possible workarounds to the current situation.  But I would have to move the ISA server, and that would only guarantee part of the users of it SecureNAT access.  To give everyone access I would have to move the inside interface users off of that subnet and into a interface on the 7204.



I need to know the following:

I need to route and statically NAT some users on the INSIDE interface to the OUTSIDE interface for their internet connection.  (This currently works)

I need to route and push traffic for some users on the INSIDE interface to the DMZ #2 interface for their internet connection with the ISA server (needs to have SecureNAT access).
(This currently does NOT work)

I need to make sure traffic coming from the DMZ #3 network can get in at least into the inside interface.
(This also works)

I need the DMZ #2 to have internet access through the ISA server to get internet patches, etc.
(This works)

I need to route and push some of the traffic from the DMZ #1 subnet to the INSIDE interface for applications, etc.
(This works)

I need to route and push some of the traffic from the DMZ #1 subnet to the DMZ #2 subnet to use the ISA for internet access.
(This doesn't work)




My questions for you:

Is it possible to route traffic from one interface to two different internet destinations?  This needs to happen for traffic coming in both the DMZ #1 subnet and INSIDE subnet traffic.
If it is...how the heck does the PIX differentiate between them?  Can you group like...IP addresses and forward one group one way and the other the other way?

I know you answered that the PIX can have more than one default gateway.  You have one for each interface.  I saw that somewhere on Cisco's site.  The consultant told me that was impossible.  So you can see his "expert" status on the PIX.  If you can set more than one default route...how does this work when a routing decision is made?  If there are is more than one way to the internet how does it know which way to turn?

In regards to default route and routing on the PIX in general.  I was reading a little about Proxy Arp and I have a couple questions about that.  If you have routers on the outside of every interface wouldn't you be able to set the default route on the PIX to itself, then the PIX would proxy arp out to the connected routers...looking for the network in question?  How would it know which internet connection to choose?


Let me know if what we have going on here is possible.  Simply put we need to be able to segregate internet traffic to certain interfaces on the PIX.

>Is it possible to route traffic from one interface to two different internet destinations?  This needs to happen for traffic >coming in both the DMZ #1 subnet and INSIDE subnet traffic.

Not on the PIX.

>If it is...how the heck does the PIX differentiate between them?  Can you group like...IP addresses and forward one group >one way and the other the other way?

This is called source-based routing.  Routers will do this, but the PIX won't.

>I know you answered that the PIX can have more than one default gateway.  You have one for each interface.  I saw that >somewhere on Cisco's site.  The consultant told me that was impossible.  So you can see his "expert" status on the PIX.  >If you can set more than one default route...how does this work when a routing decision is made?  If there are is more >than one way to the internet how does it know which way to turn?

Good question !  You can have a default route for each interface, but it's up to the NAT rule (inside, outside, dmz etc) as to which interface the traffic leaves on.

>In regards to default route and routing on the PIX in general.  I was reading a little about Proxy Arp and I have a couple >questions about that.  If you have routers on the outside of every interface wouldn't you be able to set the default route >on the PIX to itself, then the PIX would proxy arp out to the connected routers...looking for the network in question?  How >would it know which internet connection to choose?

I don't think proxy ARP is applicable in this case.  It's simply a device issuing a MAC address for an IP address that exists behind that device.  A PIX would never need to do this.

Moving forward....  how about setting up a PAT entry saying anything destined to the ISA server address is NATted ?  eg

global (dmz2) 5 interface
nat (inside) 5 192.168.1.0 255.255.255.0 0 0

This will translate packets AND send them out of the correct interface.

Are you POSITIVE no NAT is already occuring ?  Could you post up the PIX config ?

"Good question !  You can have a default route for each interface, but it's up to the NAT rule (inside, outside, dmz etc) as to which interface the traffic leaves on."

I still don't understand when the PIX makes the routing decision.  Does it work like this:

This is an educated guess...but does the PIX apply routing decisions similar to access-list decisions?  Meaning that if a packet comes inbound to the INSIDE interface, it checks the routing table first (ignoring any default routes on other interfaces, but checking directly connected interfaces??), doesn't see the route, and then finally exhausting all other options...uses the default route that is set on the INSIDE interface??  It would ignore any default routes applied on any other interface, because it only uses the default route from the interface that the packet arrived on.  If this is the case, what happens when there is no default route set on an interface?  If it doesn't make a routing table match does it drop the packet?  Or forward it out some other interface?  If the PIX handles routing on an inbound basis...you would have to specify whatever destination networks you needed...beyond connected interfaces and then set a default route to cover the rest.  Is this right?

The PIX has a ACL that looks like it tells it not to nat some traffic.  The only traffic getting NAT'd in the PIX is the company internet traffic that is statically NAT'd to the OUTSIDE interface.



Let's do this step by step...
There is traffic entering the INSIDE interface for:
   1. Internet destinations out the OUTSIDE interface for statically NAT'd IP addresses (staticly NAT'd for
   2. Internet destinations out the DMZ #2 interface for another group of users to get to the internet using the web proxy and
       SecureNAT
   3. Branch office traffic to specific subnet destinations on the DMZ #1 interface

To make the above possible:
I would define an ACL to statically NAT each address we allow to have internet access with a global outside address.  The ACL would have the inside and outside interfaces listed...source and destination interfaces.
This should be enough to satisfy #1, correct?
It will statically NAT inside addresses to outside global addresses and then forward that traffic out the OUTSIDE interface, correct?
If that is the case, does the router even do a routing table lookup before passing the packet?
Is the NAT translation telling it to go out the OUTSIDE interface?
or
Is the NAT translation saying IF you go outside translate to this address?  Then would it use the routing table lookup...and then use the default route on the OUTSIDE interface to route or the INSIDE interface???

Could you explain how this would work on this interface, with the three requirements listed above?  Not just the commands but the step by step process.  I am the type of person that craves the details.

The reason I am getting confused is there access-lists defined in the PIX that are not active.  I believe the outside consultant was trying to clean this up.  That is why the PIX config is 22 pages long.  I can't post the config, because my boss won't allow that information out.  I guess I really need to know the processes involved in a packet going through the PIX, NATing, routing, etc.  I wish cisco had something on their site that would show the processes and the order of operation for the packet being passed in and out of the PIX.

Traffic cannot leave the PIX on the same interface that it came in, so if a packet does hit the PIX on the inside interface, it will always follow this logic and try and leave on one of the outside interface if it doesn't match up with anything on any of the DMZ interfaces.
NAT is always applied after access lists.

The only meaningful thing on Cisco about this is the establishing connectivity guide:

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/config/bafwcfg.htm

Another useful white paper type thing:

http://www.enterastream.com/whitepapers/cisco/pix/pix-practical-guide.html

If you do a show xlate on the PIX, you will find out whether or not the IP addresses in question have been NATted or not.  Could be a good place to start ?  

What shows up in the PIX's routing table ?

Can you post up parts of your PIX config instead, and just put in fake IP addreses ?  I would like to see all the nat, routing and acl commands.

So the logic goes:

ACL
Routing Decision
NAT (applied when the traffic is switched to the outbound int?)

If the PIX sees a route for the packet then it goes to the next hop address specified in the route, the directly connected interface, or the default route.  Is it run through a routing decision on that information in that order?  What happens if there is a default route set for every interface??  How does it know which one to choose?

To show you the PIX config...I would have to go through like 20 pages of text to sort it all out.  I might be able to do it this weekend, along with the route table.  It will take me some time to look through it, because I didn't setup any of this network.  I am just starting to be allowed to offer my advice.

I have some solutions to the problem in VISIO documents...how could I paste that information here?  I would also like to raise the points on this, but I don't know how.

You said before that the NAT rule determines which interface the traffic leaves on.  How does that work?

First the packet hits the ACL and it is permitted.
Next the packet is matched for NAT rule...if this address, then nat to this interface with this address??
IF that is the case the routing decision is LAST?

1)  Packet hits PIX
2)  ACLs applied
3)  NAT applied
4)  If destination address is on locally attached interface (except the one if came in on) then send out that interface
5)  If not, interrogate routing table, send out packet
6)  If not on routing table, use default route

If routing happened before NAT, the packet would be gone before it could be translated.. !

You could send me the VISIO docs @ tim_holman@hotmail.com if you like ?

I don't think I can help much more without seeing the config and getting this all into context - I'm pretty sure I will see some sort of mismatch, NAT issue or ACL problem that's causing this.

I would also make sure you're running the latest version of PIX software just in case you've hit one of many bugs that screw up simple things like this.

The default route for each interface is used when you use inside, outside, dmz1, dmz2 in your configuration, rather than specific IP addresses.  Plus you do need them if there are hops in between the relevant interface and destination.

I will try to clean up the PIX config, the visios, and send them to you this weekend.  Thanks for all of the help so far.

Tim,

Sorry for the delay.  I haven't had time to go through the PIX configuration yet.  I don't think this will work after reading this at cisco's site:

Only one default route is permitted. This command statement sends any packets destined for the default route, IP address 0.0.0.0 (abbreviated as 0, and 0 for the netmask), to the router 209.165.201.2. The "1" at the end of the command statement indicates that the router is the router closest to the PIX Firewall; that is, one hop away.

I've read in places that the PIX can have one default route per interface....  It seems to only been with early versions of the 5 code.  We are running 6.2 (I believe).  But Cisco's books and their site now tell me that the PIX can only have one default route.  So if the PIX can only have one default route...that means you can only have one way to the internet, correct?  We are trying to have 2 paths to the internet...one for company traffic and one for non-company owned traffic.

This will never work with only one default route.  Is this the answer?



Here's the link to cisco's site that I saw the "only 1 default route" reference:
http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/config/bafwcfg.htm#wp1129432

I recently got the CCSP CSPFA book and it said the same thing.

I would have to agree - I've looked everywhere too, and it seems you can only ever have ONE default route in a PIX.

I still think there's something else up with the config though - overlapping NAT statements, incorrect access lists - that sort of thing ?

I found a link from Cisco that tells me you can have more than one default route, but only if the pix has 2 interfaces.  If the PIX has more than 2 interfaces then only one default route can be set.  It applies only to the outside interface.

http://www.cisco.com/en/US/products/sw/secursw/ps2120/products_configuration_guide_chapter09186a008008c0e2.html

Note that it is version 4.2 of the PIX code.

Just check my logic with this one...

A packet hits the PIX goes through the NAT, ACLs, etc.  The PIX doesn't see the destination network as a connected route (interface), it doesn't see a static route for the destination network, and it finally sends it to the default route.

The problem is I have two internet connections.  I have some traffic hitting the inside interface and needing to go out the outside interface for the internet.  I have other traffic hitting the inside interface and needing to go the internet through the proxy server.  Now this works fine with the web client because the destination is not an internet address.  The destination is the inside address of the proxy.  The problem is you can default route one way.  The securenat client needs the default route path out.

This will never work through the PIX.  I have figured out other ways to get the traffic to work by bypassing the PIX.  It will require a design change, but I can make it work.  I think this is the answer I was looking for.  Unless you have any other suggestions?

Am pretty much out of ideas too.  
In your shoes I would verify all the NAT statements in your config and maybe put some NAT statements in to make traffic is forced out of the correct interface.

Would putting NAT statements in bypass the routing decision?  It still needs to make the routing decision, doesn't it?

I would like to thank you for all of your assistance.  That's saying something, because no one else has attempted to help on this problem.

The destination wouldn't change though.  Sourced based routing wouldn't help either, because the traffic wouldn't always be going one way.

Thanks, for all of your help with this Tim.  I am awarding you the points for helping me come to the conclusion.  I had no experience or training with the PIX prior to this question.  I feel that I have a basic understanding about how it works.  Thanks for the help.