IPS signatures that block web crawling

Posted on 2014-11-04
Last Modified: 2014-11-16
I got a case where the following network IPS blocks a genuine web crawling application:
" HTTP Embedded Open Type / True Type Font Download "

Currently, it's found that the above network IPS signature blocks the crawling activity.

Can I safely say that such 'crawling' activity is unlikely to be hindered by endpoint IPS
(ie IPS with agent sitting inside the VMs / servers) ?

As we have both network IPS & endpoint based IPS, suppose the crawling still fails
after the network signature is lifted, should I also lift/disable the following endpoint
signatures (pls indicate which of them are likely to hinder ie likely to have same
effect as the above listed network IPS signature) :
1005154 - Adobe Flash Player Remote Code Execution Vulnerability
1005155 - Adobe Flash Player Remote Code Execution Vulnerability (CVE-2012-1535)
1005158 - Restrict Microsoft Office Files With Embedded SWF - 2
1004850 - Identified TTF File/OTF File Download
1004853 - Identified Suspicious Microsoft Office Files With Embedded Font
1004855 - Identified EOT File With Embedded TrueType Font File
1004858 - Identified Suspicious Microsoft Office Files With Embedded Dexter Font
1005250 - Identified Suspicious EOT File With Embedded Dexter Font
1003624 - Embedded OpenType Font Integer Overflow Vulnerability
1003623 - Embedded OpenType Font Heap Overflow Vulnerability

Is there any other way to allow the crawling without compromising (ie
without lifting the signature), say whitelist the source of the known
crawling server?
Question by:sunhux
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
LVL 64

Accepted Solution

btan earned 500 total points
ID: 40423746
Q1. Based on this vulnerability, it is targeting, for example, in the font parsing subsystem of the win32.sys driver -- provides an entry point for hackers to take complete control of an unpatched machine without any user action beyond normal browsing or opening a rigged document file.

Specific cases includes
- Malicious fonts (TTF’s) delivered within .eot files hosted on malicious web sites which are rendered in all versions of Internet Explorer by default.
- Malicious office documents e-mailed to victims with social engineering to entice the victim to open the document which contains a malformed embedded font which would then be rendered upon opening the Office document (PowerPoint and Word documents are the most likely attack vectors).

The best protection from likely attacks is for all affected users to download and apply the patch. also at network level, there is still the signature too..e.g.

remember the target is the endpoint client/server and not the network device in this case, hence the last defence like HIPS is critical as well ...

Q2. The key words is defense in depth and set up deterence on any form of atatck, adversary also can obfuscate such that the exploit which the signature is looking for is not able to detect. There can be encoding, re-odering or any other form of manipulation to ensure signature breaks and evade the detection to reach the endpoint.

Likewise not all the rigged document with exploit comes from network, there are channel directly into the machine by user carrier or email or password protected attachment that eventually still need to be reside in machine and start their modus operandi. In short, HIPS and NIPS are critical but they are not silver bullet for all attacks and known one if patch is not available...or not pushed down to reach the machine etc...

Q3. If it is authorised source then whitelisting is the best effort to not cause false alarm to ops team. prior informing is required and especially the start to end period to allow these. You should not whitelist ip as long term basis unless necessary and required. Bot and robots and crawler are the form of recon in the cyber kill chain, do not leave this as mild and neglect that.

..of course you can disable directory browsing and other form of recon as layer but it defeat the legit spidering objective. If there are WAF, IPS, NGFW, FW etc, you may want to still let it pass but do track it closely and validate no harm or anomalous activities during the period of sanction such crawling.

Author Comment

ID: 40445602
My netadmin colleague has ack'ed it is possible to whitelist specific IP while still allow the above listed network ips signatures to be enabled.  This helps.  As always thanks for the usual impressive responses
LVL 64

Expert Comment

ID: 40445605
thanks for sharing

Featured Post

Retailers - Is your network secure?

With the prevalence of social media & networking tools, for retailers, reputation is critical. Have you considered the impact your network security could have in your customer's experience? Learn more in our Retail Security Resource Kit Today!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

How to set-up an On Demand, IPSec, Site to SIte, VPN from a Draytek Vigor Router to a Cyberoam UTM Appliance. A concise guide to the settings required on both devices
This story has been written with permission from the scammed victim, a valued client of mine – identity protected by request.
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question