ISA Server keeps dropping TELNET and other TCP/IP application sessions after 66 seconds

Just some suggestions:
The issue, that VPN clients do not loose the connection and internal clients do and ISA is affected (blocking), I assume, that the telnet session runs on the same machine than ISA?
It points me to session keep alive packets through ISA. As an VPN clients acts as internal member of the network (which means, all traffic is passed through), keep alive packets will pass the server. Internal clinets, which will touch the ISA are alos restricted by the ISA roules. That means, that all rules on ISA will be valid for them as long as traffic is routed to or through ISA. Have a look at the ISA protocol trace, what happens to SYN and AKN packets, maybe they are blocked and this is the reason why the telnet server will close the session as the packets never will reach them. Also the fact, that your changes to keep alive settings has no effect, points to this.  

Also have a look to alerts is ISA, if the server interprets keep alive packets or others as spoofing. In that case, the ISA will drop subsequent connection requests for an amount of time. Within the gernal setting for ISA, you have some settings for attack investigation (Flood-Protection) and the thresholds there. You can define a computergroup (i.e. your local subnet) and add this group to the IP-exceptions. You can define normal and exceptional thresholds, which avoids ISA from regognizing internal clients as flood-source, if they send a lot of packages.

Some general remarks:
The telnet server itself will drop the session, if no activity is recognized. have a look here:
 http://technet2.microsoft.com/windowsserver2008/en/library/0259ef70-a70a-442c-9bf3-892a06f9b51f1033.mspx?mfr=true
To avoid the timeout, the client has to send keep alive packets to the server. This can be made by an underlying service as well as from an application (i.e. Internet Explorer).
The session keep alive packets have to pass ISA to take effect.

If your internal clinents are also using the cisco router, keep in mind, that also the cisco router may drop the packets. Usually you can change the settings there. If VPN tunnel through the cisco router and to ISA as endpoint behaves different, as the cisco router does not see the telnet traffic but only the vpn connection. If the cisco is setup to drop telnet sessions but nor vpn traffic, thsi may also affect your problem.
Hopes this helps for futher analysis.

 

The TELNET server is a Unix server and on the client's network (so we have no control of it).  But as VPN clients do not have their telnet sessions dropped then this implies the fault is not with the server.

I will do more tracing and post the results back early next week as I am offiste until then.  Thanks.

I managed to VPN in to the server to try to run a monitor on the SYNC and ASYNC protocol packets, but there are no items that correspond to this in the Protocol list.  How do I run a protocol trace?

I supplied an export of the ISA logs in the ISA format, so should these logs not contain all the relevant data for problem analysis.  If not, what else can I do to capture the data that is needed?

There is no limit to the amount of packets per second at the moment.

What I do see in Alerts is a "Configuration Alert" - see attached screen prints in file.
Configuration-Alerts.zip

Ups, I guess you have setup (or taken over) the default settings, which are usually created during setup of ISA 2004. If you go to ISA - Configuration - Networks, you have the definitions of the network sets there. The Internal definition should only contain the IP ranges of your internal network. The alert above indicates, that there are network ranges defined, which are not really existing. So check, which IP adresses are belonging to your network and which not. The rages are a little bit obscure, as
172.16.9.255-172.16.13.1 is a little bit unusual. Also you have single machines in there.
I would understad 172.16.10.1 - 172.16.12.255 as adress range.

Usually, the ISA has two NICs, on the internal sides are only address ranges, with are behind the ISA server, everything else is external and allowed or denied by firewall rules.

Especially a public address like 134.x.x.x should usually not reside on the internal net, as long as they are not really internally. But this depends a little bit from your general network topology. By the way, regarding of the owner of this address (Uni Ulm), you come from this institution?
But the alert message itself should not target your problem, as long as they are no routes to this (not existing) target adresses. You would get a spoof alert, if this woul dbe relevant.

To run a trace, goto ISA - Monitoring - Logging.
Goto "Edit Filter" on the right pane and there you can define

Log Record type EQUALS Firewall or Web Proxy (default)
Log Time LIVE (default)
Client IP EQUALS
Destination IP EQUALS
Select "Start Query"
This logs all traffic between the client and the server. Maybe you have to switch the settings to see, what the server sends to the client.
On the same page, you find settings for Firewall and web logging. You may also have a look into the log files, where all traffic is logged (as enabled). You may search from the end for "Denied" and see, if your server / client is involved.

Thanks for the replay Bembi,

The 172 network is our clients network and server range that goes over the Cisco router.  The 192 is our internal network.

In regards to the trace, this is exactly what I have done already and extrcted the relevant ISA log entries into the original excel spreadsheet for easier diagnosis.  This spreadsheet shows the difference between VPN clients and internal clients when using telnet and another 3rd party application.

The only difference is not a denied but a "TERMINATE" event that occurs around the 66 seconds timeframe.  In the VPN client access, the terminate only occurs when the telnet session is closed by the user.  Its best to have a look at the time of this premature terminate and then look in the ISA's IIS log file that is also attached.

OK, I see. What I can see is, that you get an error code 0xc0040017 which means FWX_E_TCP_NOT_SYN_PACKET_DROPPED

Read this artice, if it helps
http://blogs.technet.com/softgrid/archive/2008/05/28/softgrid-outlook-exchange-communication-problems-after-installing-the-softgrid-client.aspx
So just try to change temporarily the registry key on a client  to see, if this is an issue. Ussually, this affect only external traffic (over NAT), but keep in mind, that ISA installs a driver into the network stack. This is the reason, why all traffic between the ISA itself and internal computer (which could usually use a direct connection) is touched, if ISA has something to to with the traffic (ISA is either the client or the server). As your telnet server has a different address range than you clients, i assume that ISA is between them?
A second reason may be (ISA denies die Packets for a while and the allows a reconnect) that the flood mitigation settings should be adjusted. Just define a Computerset in ISA with the address range of your internal computers. You can then add this Computerset to the IP_Exception tab within the flood mitigation dialog. There are two limits defined, a very low limit for all adresses and a higher level for all computers, which belongs to the IP-Exception computerlist.
Can you provide a picture, how your network topology looks like? Means, which machines are in which subnets, how the subnets are defined (internal / external)  and how ISA acts between them?

Also have a look on this
http://support.microsoft.com/default.aspx/kb/888042

Hi Bembi,

I have tried all your recommendations without success.  The TELNET and 3rd party app keeps dropping their connections after about a minute.

I have attached a network diagram as requested.
Network-Diagram.jpg

OK, I assume, this is the issue, the Q888042 (see last link) describes. Your external clients will go through ISA, which can handle the keep alive syn packages.
The problem may be produced by your cisco router. All traffic is send to the default gateway of the computer / router. Now it depends on, how your machines are configured.

Your client has as default gatewa the ISA (?). The request is sent to ISA, ISA forwards it to the Cisco and the Cisco forwards it to telnet. The response is send from telnet to cisco, the cisco is regognizing, that is is a internal package and forwards it directly to the client. Now the client sends a keep alive package. I can not see them in ISA, so I assume, it is send to the cisco directly. And then ISA closes the connection .

What are the default gateways on the clients and on the cisco?
Try the following
- set a policy or local setting (if allowed) to bypass traffic for the 172. subnet, where the telnet resides. (Internet options)
- if that do not work, set a static route on the client to forward traffic for the 172. subnet of the telnet server to be forwarded to the cisco (by defualt, the default gateway is used as the 172 address do not belong to the local subweb.)

to see the routing table, just type route print on the command shell.

aswering questions would help