Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1970
  • Last Modified:

ASP.NET Spider/Crawler

We are looking at the possibility of developing a crawler/spider that will be used in conjunction with our own search engine. I know very little about this subject, can anyone help me? I need to research this in great detail but any help would be amazing. Thanks.
0
odonovanpm
Asked:
odonovanpm
3 Solutions
 
wattkeyCommented:
Try this.  I modified this code and it works very well.  It will at least give you some information on how the process works.

http://www.webconcerns.co.uk/aspnet/search/default.asp
0
 
odonovanpmAuthor Commented:
That's great, thanks. But what if I want it to crawl external sites continuously?
0
 
CJ_SCommented:
Been there done that...

Check out the asynchronous HttpWebRequest object. It should provide you with sufficient knowledge of how to download an html page. Create you own queue to download, and your own parser and you're ready to go.
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now