Link to home
Start Free TrialLog in
Avatar of Rikard Micek
Rikard MicekFlag for Croatia

asked on

Forefront TMG reports possible SYN attack and stops accepting new connections globaly

I have a problem on our network with FTMG and trying to understand the problem and FTMG mittigation mechanism.
At our network any user can make Forefront TMG unusable by using uTorrent aplication! FTMG stops accepting new connections and stops routing traffic (both outbound and inbound).

First of all i know that i can block access to torrent sites and disable users in a way that they are unable to install uTorrent (and other tools) and disable the use of torrent tracker sites, and disable the downloads of torrent files.  This is going to reduce the great deal of problems but am i trying to find out what is happening behind the scene and why.
 
Here is what happens: Forefront TMG 2010 SP1, update rollup 1 is installed on Windows Server 2008 Sp2.
There is intrusion detecion and NIS enabled.
SecureNAT client opens uTorrent and loads trackers from torrent file.

Soon after that TMG stops accepting new connections and stops routing traffic (outbound and inbound traffic are both affected) from ALL IP addresses (not just the offendig one). There are LOG entrys that show: Forefront TMG has detected a possible SYN attack and will protect the network accordingly. Short time after that all traffic is gone. FTMG does not accept RDP connections, won't replay to ping, won't route traffic. There is no offending IP address in logs.

I have learned that a "POSSIBLE SYN ATTACK" is mitigated by FTMG mitigation mechanism.
FTMG has a threshold defined for maximum of half-opened connections =  Half the maximum TCP connections per IP address.

On my box this threshold is currently set at 400 so the global number of half-opened connections is 200.

Can someone explain what is happend at that point?
When TMG says "will protect the network accordingly" -how does FTMG protect the network?
Why does it stops accepting new connections globaly for all users, not just from the IP address source with "possible syn attack" activity and what would be the correct flood mitigation setup to stop accepting new connections from the IP address before it "locks" th FTMG completely. The main goal is to isolate/block traffic from user that is performing "possible syn attack" before it blocks access to everyone else.



Avatar of MaurizioSchmidt
MaurizioSchmidt
Flag of Germany image

Hi,

create an execption for these IP Addresses in the Flood Mitigation settings. There is a huge missconfiguration at your TMG Server.
If a SYN Attack happens, TMG should only drop all connections from and to this one client who is starting p2p stuff and not to the whole network on all services.

Avatar of Rikard Micek

ASKER

Yes, it should but it doesn't. It just hangs without allowing any new connections. Established connections work fine.
What means hangs exactly ? if the TMG is setuped to deny all connections from outside, when some syn attack comes in, then in doesnt hang, it just does what you told the TMG to do.
what have you exactly defined in a case of a SYN ATTACK, how should the TMG react to this ? and please tell me, what kind of hardware you use for TMG.
Avatar of voznaj
voznaj

Has this been resolved? I'm having the EXACT same problem!
Avatar of Suliman Abu Kharroub
Me too, have the same exact problem :
https://www.experts-exchange.com/questions/26599376/Forefront-TMG-detected-a-possible-SYN-attack-and-will-protect-the-network-accordingly.html

Did not found the solution anywhere. downgrading the server to ISA 2006 SP1 solved my problem, but for sure with less features...

waiting for someone can help here...
I was trying to work things out here:
https://www.experts-exchange.com/questions/26794869/Is-it-possible-to-monitor-in-realtime-the-current-number-of-half-opened-connections-on-server.html

TWO TOP experts (ketih_alabaster and pwinde)l tryed to help and they did help in a way..

My advice would be:
1. Do not allow utorrent or any other p2p clients
2. Block .torrent through TMG
3. setup Flood mittigation to default settings
4. or eventually disable flood mittigation if nothing else helps..

but real problem still stands: how to identify the local IP address that is generating to much half open connections.
Still trying to find the answer.
I actually know the IP address of the offending machine; it's mine. More concerning is that anyone downloading a torrent has the ability to effectively shutdown entire communications through TMG by doing so. I would think the behavior of TMG would halt traffic to/from the offending address rather than stopping all LAN/WAN traffic. VPN tunnels - everything becomes interrupted.
How can you be sure that this is your computer blocking the entire traffic? I had the same exact problem.  
Did you changed the defaults? Can you provide the numbers for mitigation settings?
I suspect  that it is really important that you do NOT make one this numbers to much larger than defaults.
It was purely accidental I found this. I have VUZE installed on my client machine. When I attempted to download a .torrent, nothing happened except internet for the entire company stopped working. OWA, Email, VPN tunnels – the entire TMG box became non-responsive. Ping requests would time out. Attempted on 3 different instances and Event 15009 Forefront TMG detected a possible SYN attack and will protect the network accordingly was logged each time. Mitigation settings are set to their defaults. It seems to completely locked itself down.
Yes, everything you say happened to our network also.  But it did not locked itself down, it just stopped accepting NEW connections. You can verify that by opening RDP session. Then run p2p client. After some short time TMG will stop accepting new connections nut opened RDP session will still work (you can restart the server).
You are the first one that i met that has the same problems like I do.
Well do you really need the p2p client? Or the stable network?

The main problem is that by this too easy anyone can make DoS attack and make network inoperable.
Just by using p2p client? There must be something else - still looking for solution.
I'm not able to open an RDP while this is occurring, however if I had an RDP session open while download of the .torrent started, the session remains open. Otherwise, there is no way to log onto the server until TMG no longer sees that it is under a SYN attack which and network functionality is somewhat restored. I then reboot for good measure at that point to restore all of the Web services. Most frustrating is that there is nothing 'unique' in our TMG configuration.

I couldn't agree more that network stability is more important that P2P downloads, it's more the principal, as you mentioned, that anyone at any given time can render the network useless by opening a piece of software! I'll post back when I have found another alternative other than disabling P2P.
It looks like we have the same problem and the same ideas. Hope we can be more efficient by working on this problem together.

To share some of the knowledge: an advice from Microsoft premium support senior technician:
THIS DID NOT WORK FOR ME, OR BETTER TO SAY HAVE NOT BEEN ABLE TO VERIFY IF THIS HAVE ANY EFFECT.


As per our discussion, we know that TMG stops accepting new TCP connections after it sees a SYN attack.

SYN attack is to protect TMG server against DDOS attack and it is triggered when we have 1000 half opened connections and TMG has to drop below 200 to start accepting new connections.

SYN Attack thresholds are configurable.
Following are the default settings for SYN attack  – we have 2 settings here

"SynAttackHalfOpenEnable" – default 1000 decimal
"SynAttackHalfOpenDisable" – default 200 decimal

These are reg key values under HKLM\System\CCS\Service\fweng\parameters

So, as soon as global half opened connections goes over 1000 we are not going to start accepting any more connections until the number of half opened connections goes below 200. Existing connections will continue to function OK – it will just be new connections that are affected.

You can consider raising SynAttackHalfOpenEnable to 2000 and SynAttackHalfOpenDisable to 1000 but you should look further at who is opening connections and if there is anything suspicious first.

You should also check Non Paged Pool memory usage as we would not want to increase this and then start to run low on Non Paged Pool which is one of the resources the SYN protection is intended to protect.
Thanks for the info, although it likely won't help. Even after disabling mitigation altogether,the issue is still present.

What type of NICS do you use? I have BCom 5716C - going to try swapping with Intel.
SOLUTION
Avatar of Suliman Abu Kharroub
Suliman Abu Kharroub
Flag of Jordan image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
With the exception of ISA 2004 not being able to handle x64 traffic, it works much better than TMG. I've never had so many issues as I have with this product. I'm not sure ISA 2006 is much of an option. I would really rather avoid updating all my sites with 5 year old technology. TMG simply does not seem to work properly. After spending 20+ hours on the phone with an MS Support Engineer at the highest level, the response was "I don't know what to tell you. I've never seen this behavior before". After some registry key changes - poof, internet access is no more and back to ISA 2004. Great...NEXT!?
ISA 2006 sp1 with GFI web monitor. server is stable and running smoothly.
voznaj,

Did the Microsoft support technician said anything about SynAttackHalfOpenEnable registry settings?
Did give any advice during this 20h phone talk?
Did he said anything about looking into issues?

Btw, i am on intel NIC-s. Intel server board (but have i tried 3 different platforms, one of them was just a PC).
Is ISA 2006 SP1 supported any longer?

The registry key you reference was never mentioned. There has been a LOT of time spent on this. Mostly verifying settings and performing Data captures after which I was told everything was configured properly. The case was originally opened because TMG IPSEC VPN tunnel to PIX 501 would not pass traffic to more than one subnet on the remote end. It has steam rolled into all kinds of other issues including the one we've been going back and forth on here. The network layout is very straight forward:

Clients - Core Switch - Core Router - TMG

The recommendation for the VPN issue was to replace all PIX devices with TMG. This wasn’t really a resolution, but in testing it worked if only downloading specific files didn’t bring TMG to a crashing halt. There is also a known bug with being able to PING the LAN IP of remote TMG servers when connected over VPN. In my opinion, TMG is the VISTA of firewall software. Never have I experienced so much difficulty. The product simply does not work as intended. It was left that his experience tells him it is a network card issue. I’ll test it, although I’m not very optimistic.

Our environment was an ISA 2004 SP3 export to TMG SP1 Rollup 2. I’ll give his recommendation a go and if that doesn’t work, I’ll import to ISA 2006 and export from there to TMG. Possibly there is something TMG doesn’t like about the ISA 2004 import? Anyway, lots more testing to do – I’ll get back to everyone if I find something that works.
UPDATE: I believe the issue has been resolved. After upgrading to TMG with the issue explained above, I rolled back to ISA 2006 SP1; ISA 2006 also randomly stopped answering requests. Ultimately it seems to have been a DNS issue where the backlogged packets was very high and ISA could no longer keep up with processing requests.

The cause: a global deny rule in ISA. You can read about it here: http://blogs.technet.com/b/isablog/archive/2009/01/12/isa-server-2006-stops-answering-requests.aspx

After disabling the rule, ISA has never run more smoothly. I took this to TMG and disabled the rule there; seems to fix the issue on it as well. If you have any of these rules, you can continue to use them, however, you need to perform the steps outlined here: http://support.microsoft.com/kb/891244.

Hope this helps you as well!
Trying to follow advices but i am still able to "brick" my TMG by using torrent clients...
voznaj,

what is the situation with your TMG installation? Did this "solution" of yours covers problem with TMG as i have? Or not? Do you still experience problems?

Did you try to disable Flood Mitigation Feature entirely? Are you able to replicate the problematic behaviour when Flood mitigation is off?
Disabling flood mitigation did not help. The issue still appeared after disabling whenever a P2P client accessed the internet. Do you have any deny rules besides the last default rule? My issue seems to have been resolved after following my comments above. You aren't by chance running DNS from your TMG box? Start a performance monitor and monitor Backlogged Packets. Start your P2P client and report back what the average is of this counter.
Yes i am running DNS server service but only to host public DNS records: My SecureNAT clients are not using this DNS service as a primary DNS (which is hosted on our DC).

I am curious about the fact that the TMG server also hangs when a Flood Mitigation Feature is disabled. All my HALF-OPEN Flood Mitigation assumptions are now questionable and now don't know what to do.
I don't think flood mitigation is your issue; what about any deny rules?
I have a few. But ALL of them are only HTTP/HTTPS traffic type rules and just one is BLOCK SMTP 25.
That sounds like your problem. Can you post the deny rules only? We may be able to get you straightened out today!
Can you clarify the theory behind? What do you think it is happening?
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Test #101

1. Disable all the Deny rules.
2. Disable Flood Mitigation rules.
3. Run p2p client, force an "attack" and wait for the "hang" to occurs.

I will make some tests and report back...
@voznaj
>>"Depending on what your HTTP/HTTPS deny rules are, every time a packet request is sent through ISA/TMG, it is processed by the firewall policy rules in order. When you open a P2P client, it floods the server with requests and if you have a deny HTTP rule setup, it tries to perform a reverse DNS of the request which it will never find. These requests build up and show as backlogged packets."

deny rule could be an issue,but:
ISA tries to reverse lookup names only if you deny traffic based on domain names not IP address.

I have the same exact problem on TMG server and it was happened frequently. In TMG server there was many denies rules. then I downgraded the server to ISA 2006SP1 which is running very well....but 2 weeks ago I had to create denies rules again... and the server does not respond to clients once a day....
No need to disable Flood Mitigation.

Do this:
TEST #1
1. Disable all Deny Rules
2. Run P2P Client

If still fail

TEST #2
1. Disable all Deny Rules
2. Temporarily Disable DNS Server Service
3. Run P2P Client
P2P clients work over HTTP?
It is UDP traffic.
But all my deny rules are limited only to HTTP/HTTPS (TCP) traffic.

Are you saying that it doesn't mater and that EVERY (block) rule is processed completely  even if it does not match the "Protocol type" part of the rule? It would suggest that even if it is a UDP traffic that TMG is is performing a reverse lookup when not necessary (the protocol type section of the rule already defines that this rule is not applicable to this UDP traffic).
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
ASKER CERTIFIED SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
GrdiRuda:"Are you saying that it doesn't mater and that EVERY (block) rule is processed completely  even if it does not match the "Protocol type" part of the rule? It would suggest that even if it is a UDP traffic that TMG is is performing a reverse lookup when not necessary (the protocol type section of the rule already defines that this rule is not applicable to this UDP traffic)."

If you are using domain names and not IP addresses for any deny rules I would say the answer to your question is yes.

From the article: "This policy configuration cased ISA to attempt to perform a reverse lookup for every connection request matching the source network regardless of protocol in order to determine if the IP address matches the DN Set"

I don't think so. Look at the sentence above the one you have pasted here. He is talking about the "custumer rule" that was defined as a "all protocol" rule.

He says (regarding the customer rule from the picture just above this sentences):

This is an access policy that handles “all outbound traffic” and applies to a Domain Name Set as destination. THIS (customer "all protocols" rule)  policy configuration cased ISA to attempt to perform a reverse lookup for every connection request matching the source network regardless of protocol in order to determine if the IP address matches the DN Set.

Don't you agree with me?
I read it differently in that it checks all requests from the source network regardless of Protocol if you are using dn sets.
well OK, but here he states :)
In other words: any rule which use destination Domain name sets should ONLY apply to HTTP/HTTPs protocol (since URL set is already used only for HTTP/HTTPs).  So - protocol definition does mater after all.
Truthfully, I would suggest you try my suggestion to see if it makes a difference.
voznaj, please dont get me wrong i am really grateful for your advices, and already  trying your suggestions.
Thanks for help and advices.

1. preliminary test's i have performed today shows that you could be right. When i disabled the deny rules i could not brick the TMG

but,
2. I searched for FWX_E_NO_BACKLOG_PACKET_DROPPED 0xC0040016  but the logging shows 0 results in last 30 days. I have "bricked" the TMG several times today.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
iz would be nice if you could report here your data - backlogged packets average when using p2p clients...
On average backlogged packets counter should be less than 10. When my TMG/ISA bricked, it was up around 3-4 thousand!
i read currently 32 on average... everyting working with deny rules enabled and p2p clients active.
As more requests come through, I expect it to go much higher before it becomes unresponsive. It's also best practice to not have DNS running on the same box as ISA/TMG. I don't think this is what your issue is, but it's a likely contributor.
I moved the "complex" rules (URLS sets) all the way down just above last "Deny rule".
TMG didn't hang on preliminary tests. Will test further.

I have not yet disabled the reverse DNS lookups but i if necessary will use the script for the TMG (not for ISA) found here:  



http://msdn.microsoft.com/en-us/library/dd447622.aspx

Open in new window

I made all the rule with explicit protocol definitions, specially the global deny rules. I have also patched the OS and TMG with all available updates. I have upgraded memory to 8Gb (100 computers network and some public services on several servers).
Glad to hear testing is going well. Keep us posted!
Good news. We have downloaded a bunch of files thorough torrent protocol (via uTorrent) and TMG haven't stopped responding. We will test this after Easter holidays but thinks look good. I have high hopes that we have resolved this issue!

Hi,

Great news! Its working. I have throughly tested the TMG using tools that have previously caused TMG to stop responding. I have created the setup that's working for me more than fine. It is working great. I didn't have to remove none of the Global deny rules. They have just been "narrowed"  and move as mush as possible to the bottom of the rule list. But that were just the final steps that made it robust and stable. Investigating this problem i have learned a lot.

Thanks to these guys:
ketih_alabaster, 2pwindl, Sulimanw and of course voznaj. This has been a rather complicated issue and all of you helped with your advices and comments. There were no "one perfect solution" but things got sorted out by folowing voznaj's advices. Thanks man.
For the "Accepted solution" i have selected the comments that best describe the final conclusion what the problem really was.
Thanks to these guys:
ketih_alabaster, 2pwindl, Sulimanw and of course voznaj. This has been a rather complicated issue and all of you helped with your advices and comments. There were no "one perfect solution" but things got sorted out by folowing voznaj's advices. Thanks man.
In the end, it's not about points, it's about resolving the issue. I'm glad everyone was able to assist and it has finally gotten straightened out. I hope this thread helps the countless others who experience the same problem.
Hello guys!

We are having the same problems over our network.
We did change the rules order, but it seems that is not working quite effective yet.

Now, we've installed a DNS server on the same machine as TMG is.
Waiting a little for the results.

Anyone has other ideas?

TYVM
Start by disabling ALL of your deny rules. Changing the order only assists after the problem is identified.
you need only to configure deny rules ( which have domain name sets ot URL sets in the TO tab) to be applied only to http/s protocols.... that should solve the problem..
Thanks for the reply guys.

VOZNAJ: We already did that. The problem persists.

SULIMANW: We tryed that too but with no good results.

We've noted that the backlogged packets tends to go insanely up on pic hours, like when employees do the first login of the day (and download emails and check news on internet) and when students do the same thing at nigth.

We also have a certified specialist on ISA TMG and he is crazy about this.
2 weeks on this problem..

Any other thougts?

Thanks!
I would suggest to open a new thread... seems to be a deferent story there :)

Glad to help..
Backlogged Packets is most typically associated with DNS queries. You don't want it running on the same box as TMG. This will cause issues. Have you performed a netstat at the time the average packets are peaking? What is the port range? How many rules do you have? Can you provide screen prints of them? Feel free to PM if you're not comfortable posting.