IPS signatures that block web crawling

I got a case where the following network IPS blocks a genuine web crawling application:
" HTTP Embedded Open Type / True Type Font Download "

Currently, it's found that the above network IPS signature blocks the crawling activity.

Can I safely say that such 'crawling' activity is unlikely to be hindered by endpoint IPS
(ie IPS with agent sitting inside the VMs / servers) ?

As we have both network IPS & endpoint based IPS, suppose the crawling still fails
after the network signature is lifted, should I also lift/disable the following endpoint
signatures (pls indicate which of them are likely to hinder ie likely to have same
effect as the above listed network IPS signature) :
1005154 - Adobe Flash Player Remote Code Execution Vulnerability
1005155 - Adobe Flash Player Remote Code Execution Vulnerability (CVE-2012-1535)
1005158 - Restrict Microsoft Office Files With Embedded SWF - 2
1004850 - Identified TTF File/OTF File Download
1004853 - Identified Suspicious Microsoft Office Files With Embedded Font
1004855 - Identified EOT File With Embedded TrueType Font File
1004858 - Identified Suspicious Microsoft Office Files With Embedded Dexter Font
1005250 - Identified Suspicious EOT File With Embedded Dexter Font
1003624 - Embedded OpenType Font Integer Overflow Vulnerability
1003623 - Embedded OpenType Font Heap Overflow Vulnerability

Is there any other way to allow the crawling without compromising (ie
without lifting the signature), say whitelist the source of the known
crawling server?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

btanExec ConsultantCommented:
Q1. Based on this vulnerability, it is targeting, for example, in the font parsing subsystem of the win32.sys driver -- provides an entry point for hackers to take complete control of an unpatched machine without any user action beyond normal browsing or opening a rigged document file.

Specific cases includes
- Malicious fonts (TTF’s) delivered within .eot files hosted on malicious web sites which are rendered in all versions of Internet Explorer by default.
- Malicious office documents e-mailed to victims with social engineering to entice the victim to open the document which contains a malformed embedded font which would then be rendered upon opening the Office document (PowerPoint and Word documents are the most likely attack vectors).

The best protection from likely attacks is for all affected users to download and apply the patch. also at network level, there is still the signature too..e.g. http://tools.cisco.com/security/center/viewAlert.x?alertId=21470

remember the target is the endpoint client/server and not the network device in this case, hence the last defence like HIPS is critical as well ...

Q2. The key words is defense in depth and set up deterence on any form of atatck, adversary also can obfuscate such that the exploit which the signature is looking for is not able to detect. There can be encoding, re-odering or any other form of manipulation to ensure signature breaks and evade the detection to reach the endpoint.

Likewise not all the rigged document with exploit comes from network, there are channel directly into the machine by user carrier or email or password protected attachment that eventually still need to be reside in machine and start their modus operandi. In short, HIPS and NIPS are critical but they are not silver bullet for all attacks and known one if patch is not available...or not pushed down to reach the machine etc...

Q3. If it is authorised source then whitelisting is the best effort to not cause false alarm to ops team. prior informing is required and especially the start to end period to allow these. You should not whitelist ip as long term basis unless necessary and required. Bot and robots and crawler are the form of recon in the cyber kill chain, do not leave this as mild and neglect that.

..of course you can disable directory browsing and other form of recon as layer but it defeat the legit spidering objective. If there are WAF, IPS, NGFW, FW etc, you may want to still let it pass but do track it closely and validate no harm or anomalous activities during the period of sanction such crawling.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sunhuxAuthor Commented:
My netadmin colleague has ack'ed it is possible to whitelist specific IP while still allow the above listed network ips signatures to be enabled.  This helps.  As always thanks for the usual impressive responses
btanExec ConsultantCommented:
thanks for sharing
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Security

From novice to tech pro — start learning today.