Best way to deploy host-based IPS signatures without causing disruption

Posted on 2014-09-16
Last Modified: 2014-10-11
I'm not so concerned with network-based IPS as an entire subnet can
sometimes only have a pair & a signature that disrupts service can be
rolled back rapidly.

However, I have host-based IPS signatures, namely Trendmicro in the
thousands to be deployed into each of the Production VMs & the principal
can't guarantee it won't break the apps or disrupt services.

The signatures have different severity ratings from Critical with Exploits,
Critical with Smart, High, Med, Low so we're dividing them up by different
OSes but still it's a lot.

a) do people generally take a snapshot backups first?  Must say I don't
    have much storage to take snapshots & those snapshots had caused
    datastore to fill up in the past, causing painful disruptions

b) is there a quick way of fallback or rollback in the event we are affected?

c) is it more commonly practised that people deploy more signatures on
    Web servers (ie those facing external or Untrusted zones) & less on,
    say the backend (eg: DB servers) ?

The signatures & the agent sit inside each individual VMs unlike
network-based IPS where network traffic are re-routed to the network
IPS before being passed onto the servers

I'll be checking with Trend as well
Question by:sunhux
  • 3
  • 2
LVL 80

Assisted Solution

by:David Johnson, CD, MVP
David Johnson, CD, MVP earned 170 total points
ID: 40327625
a) do people generally take a snapshot backups first?  Must say I don't
    have much storage to take snapshots & those snapshots had caused
    datastore to fill up in the past, causing painful disruptions
b) is there a quick way of fallback or rollback in the event we are affected?

Your first pressing problem is storage. Increase your storage pool. I would guess that backups are not such a priority due to (a) lack of storage space. I don't think that separating out IPS signatures is a good idea.. getting hit with a low priority intrusion is just as bad as being hit by a high security zero day intrusion.  

Does your backup policy follow the 3-2-1 routine?  3 backups, 2 different types of media, 1 offsite ?   Do you keep a weeks worth of daily's, a months work of weekly's, a years full of monthlies, and a collection of years? If asked to restore bare metal to a particular date and time could you do it? how long would you be offline? Have you tested your backups to ensure that they work?

Any server can be a host to the intrusions.. Database's are a high priority target.
LVL 63

Accepted Solution

btan earned 330 total points
ID: 40327735
for signature, wouldnt it be stating malicious intent as the signature is specifically to detect such content and if really there can be tendency of false positive then the transparent or the signature should be granular such that action specific to those new one can be in Alert state rather than Block state. I am not sure if TC can do that but some WAF appliance can stage rule and signature before block action take place. if the signature is for IPS or AV like, it is better to err on safe side.

application break can only be ascertain in staging and eventually production, there is no 100% warranty by any principal or solution but it should be kep to minimal and is good to know why it break as most of the time is legacy apps required that certain action which TC may deems as positive and probably application exception or whitelisting need to be enforced then..

snapshot is incremental daily and full backup end of the week but varied from practice. nonetheless, in VM snapshot is the last known golden state which is verified to be hardened, clean of misconfig and malicious content. that is just the baseline, the important capture and backup is the data - hence the DB information is something required and cannot be lost unless the risk exposure if known and accepted. you cannot regenerate real time data and data corruption is big issue when dealing with privacy and legacy matters which the company has vouched to safeguard and prevent breach and taint incident...responsibility and accountability is key to ensure data resiliency and privacy

going to TC Deep Security, they have in their practice guide (under " Disaster and Recovery " section) also stated to make sure a regular backup of the Deep Security database is scheduled. And most specially when applying a patch or an upgrade to the software. It is more to ensure  restore or recovery of the database from the same version number as the DS Manager

So in short assuming the signature update is really crashing even if it pass ther internal check as there are real case even for other AV or OS provider, they are not liable as the user has taken the risk knowingly, so they can only advocate best practices when such unforeseen circumstance code is secure and free of bugs..They will always in any upgrade of DS state - please do backup the DS database as it is highly recommended...

other extract to note
Scan Schedule Setting
In addition to scan configurations, there is also an option to set a schedule for all types of scans, including reaItime scan. This can be useful if there is a specific timeframe where you’d like to turn off real-time scanning to improve performance.

- File Server is scheduled to have a backup of all files every day at 2:00-4:00am
- This server will most likely have high activity during this time and whitelisting the 2:00-4:00am timeslot from real-time scan activity would significantly help improve performance for both the backup task and server resource.

- Perform a full manual scan on a server prior to running the actual backup task
- We recommend that weekly scheduled scans are performed on all protected machines.

Author Comment

ID: 40336898
Regional Trend got back to us that they don't recommend more than
350 signatures per VM (though they have about 3700 signatures) as
it will affect the VM's performance (even for a 4 vCPU 16GB RAM VM).

That simplifies things a lot for me, selecting only the critical exploits
to apply.  Odd though is what do we do with the remaining 3450
signatures if one of the vulnerability happen to hit us.

Looks like I have to strengthen it with network-based IPS at the
outer perimeter & can't rely completely on the host (ie VMs)-based
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

LVL 63

Expert Comment

ID: 40339064
indeed defense in depth strategy and divide and conquer since we also do not want to fall into the single pt of failure (as mentioned by the limit of VM processing which can be susceptible to adversaries DoS etc). Whole scheme of approach may be ideal to have the "fight" at the exterior/outer perimeter instead of at the "doorstep" at the target server

Author Comment

ID: 40339400
> can stage rule and signature before block action take place
Yes, Trendmicro's DeepSecurity allows the signatures to be
deployed in 'Detect' mode first & later convert to 'Block' mode.

So I'll have a lot to analyse in Detect mode first before we go
into Block mode
LVL 63

Assisted Solution

btan earned 330 total points
ID: 40339440
that is the challenge for tuning to really have minimal false positive suited to your environment and establish the norm from the anomalous. surely we need to have 'zero' false negative - Trendmicro should already have the known bad detected and probably whitelisting will give you another layer of defense - it always catch up for blacklisting which  we cannot neglect but for long term, this leads to security fatigue as you already understand

Featured Post

Save the day with this special offer from ATEN!

Save 30% on the CV211 using promo code EXPERTS30 now through April 30th. The ATEN CV211 connects a laptop directly to any server allowing you instant access to perform data maintenance and local operations, for quick troubleshooting, updating, service and repair.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many businesses neglect disaster recovery and treat it as an after-thought. I can tell you first hand that data will be lost, hard drives die, servers will be hacked, and careless (or malicious) employees can ruin your data.
Ransomware continues to grow in reach and sophistication, putting data everywhere at risk. Learn how to avoid being caught in its sinister clutches with these 11 key tips.
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…

679 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question