Solved

Netscaler 10.1 DoS Attack to Backend Web Server

Posted on 2015-01-22
4
883 Views
Last Modified: 2016-10-25
Here is our setup.

Firewall -> Netscaler’s in HA setup -> Backend Windows 2008 Web Servers
.
We had an issue where one of the website hosted by our web servers was getting attacked. Mind you these web servers host many websites but only one was affected. Our Web Admins stated they were seeing thousands of request coming in for web pages that do not exist. Our Security Team noticed in the firewall the attack was not coming from one but from a large number of IP’s not all in the same IP block.

Long story short the attacked was stopped by blocked the IP’s in the firewall. On the Netscaler side during the attacked I noticed thousands of connections to the VIP address for this site which is more than normal. Our web team was reporting they were seeing thousands of passive connections that keep increasing very fast until the server needed to be reboot again.

Queston.
1)      Are there any settings on the Netscaler I should look at to prevent this from happing again?
2)      We are load balancing the site yet the Web Team reported most of the passive connections were focused on one server? How could this be in “Least Connections” load balanced setup. Please note the site responds on http and https.
3)      The java GUI is very slow to get real-time connection information like source IP , destintation IP etc.. Is there a log file I could view?
4)      On a side note what log files should I view to check the overall health and number of open connections that have not timed out?
0
Comment
Question by:compdigit44
  • 2
  • 2
4 Comments
 
LVL 61

Accepted Solution

by:
btan earned 500 total points
ID: 40567903
(1) NS does have DDoS defences and primarily we should also not neglect  looking at L7 too on top of the usual SYN cookie to taper the no of TCP handshake connections (as in any network FW). The IP blacklisting is what they called ACL. Others to consider (http://support.citrix.com/article/CTX131681)

- HTTP DoS Protection (include verify legit browser vs bot based client, can have false positive but mostly catch automated tool). Some consideration in config (http://blogs.citrix.com/2010/10/14/dos-and-ddos-configuration/)
- Rate Limiting (restrict same client IP on thresholding its connection, esp since IP can behind proxy and cant just blacklist)
- Dropping Invalid HTTP Requests (kill off malformed or ill intent packet though legit request can still pass through)

Note also Application Firewall can be enabled on a NetScaler appliance with the purchase of a Platinum license, that can drill into the http besides just looking at rate threshold and connection limit. http://blogs.citrix.com/2014/08/25/getting-your-feet-wet-with-netscaler-appfirewall/


(2) reference the algorithm using same client ip, since it is mostly passive connection, it means that server has responded and the session is "quiet" (small beacon to keep the session alive...) as compared to active connection which is "noisy"- that how i see it and also in its docs some related statement
...it selects the service with the fewest active connections.
When a virtual server uses the least connection method, it considers the waiting connections as belonging to the specific service. Therefore, it does not open new connections to those services.
http://support.citrix.com/proddocs/topic/netscaler-load-balancing-93/ns-lb-customizing-about-leastconnection-con.html


(3), this article touches the various logging possibilities (mainly syslog). http://www.jasonsamuel.com/2014/07/02/mitigating-ddos-and-brute-force-attacks-against-a-citrix-netscaler-access-gateway/
You’ll also need to go to System > Auditing > Settings > Change global auditing settings > check the “User Configurable Log Messages” box for both SYSLOG and NSLOG auditing types. Otherwise you’ll see hits on your Responder policy but no hits on your Auditing Message Action policy:
If you chose to not write to the newnslog and are instead writing to the ns.log, you can use:

tail -f /var/log/ns.log

Just don’t forget about the 7 second delay before the hits appear in your Putty window.

One thing to note, remember the AOL days in the 90’s? Every user was behind the same proxied IP address. Blocking an IP would block pretty much all AOL users.
The default setting for AAA logs is set to save the last 25 log files (circular logging where it will overwrite the oldest) and the size is set to 100 Kilobytes. Not a whole lot. The best practice is not to utilize the device itself for historical logs. It’s really only meant to be used for realtime/neartime logs for troubleshooting purposes. Historical Netscaler AAA logs should be offloaded from the appliance using a syslog server.


(4), I see it as the typical SNMP monitoring, one example using OprMgr
https://msandbu.wordpress.com/2013/04/02/monitoring-netscaler-with-operations-manager-2012/
After you have enabled other Performance Monitors they will appear here as well, this allows you to create a baseline for how connections should be on your box.
This also allows for Operations Manager to generate alarms in case of DDoS attacks.
0
 
LVL 19

Author Comment

by:compdigit44
ID: 40575779
Sorry for the delay in getting back to you.  I have been out sick and place an going through you response in detail later on today.

Thanks Again!!!
0
 
LVL 19

Author Comment

by:compdigit44
ID: 40576335
Currently we have a firewall -> HA Netscaler's -> DMZ Web servers.

Do you suggest places something else in-between the Netscaler's and web servers for added protection?

I as still confused about load balancing of traffic using least connected but still re-reading it to understand it better.

I have been read about Citrix Command Center? Would this be a great way to monitor our Netscaler's???
0
 
LVL 61

Expert Comment

by:btan
ID: 40576568
ActuallNetscalar itself is an application delivery control and already has has those capability (but do check with your sales on if module is active or need to be licenced). We will need Web app FW to better safeguard L7 DoS attempts, in Citrix it is termed as AppFirewall - http://blog.b3rg.nl/netscaler/netscaler-10-1-new-and-updated-features/#appf

monitoring of the box depends on how deep you will want to..even its own console CLI (http://www.jasonsamuel.com/2010/07/09/how-to-monitor-realtime-traffic-stats-accurately-on-a-citrix-netscaler/) will give you information already, but most will want another out of band centralised monitoring oversight systems. It need not be Citrix (again) system like Command Ctr unless you have full of such already running around. There are generic and most based on SNMP like PRTG (http://blogs.lockstepgroup.com/2013/07/how-to-series-citrix-monitoring-made-easy-and-cheap-part-.html), Solarwind, or even pack form MS SCOM (http://www.managementproducts.comtrade.com/management_pack/citrix/netscaler/key_features/Pages/default.aspx) etc..
0

Join & Write a Comment

#Citrix #Citrix Netscaler #HTTP Compression #Load Balance
Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now