Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

ASP.NET Spider/Crawler

Posted on 2004-10-16
4
Medium Priority
?
1,964 Views
Last Modified: 2008-02-01
We are looking at the possibility of developing a crawler/spider that will be used in conjunction with our own search engine. I know very little about this subject, can anyone help me? I need to research this in great detail but any help would be amazing. Thanks.
0
Comment
Question by:odonovanpm
4 Comments
 

Assisted Solution

by:wattkey
wattkey earned 300 total points
ID: 12335241
Try this.  I modified this code and it works very well.  It will at least give you some information on how the process works.

http://www.webconcerns.co.uk/aspnet/search/default.asp
0
 

Author Comment

by:odonovanpm
ID: 12337053
That's great, thanks. But what if I want it to crawl external sites continuously?
0
 
LVL 6

Accepted Solution

by:
Volkan Vardar earned 400 total points
ID: 12337207
0
 
LVL 22

Assisted Solution

by:CJ_S
CJ_S earned 300 total points
ID: 12338311
Been there done that...

Check out the asynchronous HttpWebRequest object. It should provide you with sufficient knowledge of how to download an html page. Create you own queue to download, and your own parser and you're ready to go.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have developed many web applications with asp & asp.net and to add and use a dropdownlist was always a very simple task, but with the new asp.net, setting the value is a bit tricky and its not similar to the old traditional method. So in this a…
Problem Hi all,    While many today have fast Internet connection, there are many still who do not, or are connecting through devices with a slower connect, so light web pages and fast load times are still popular.    If your ASP.NET page …
This course is ideal for IT System Administrators working with VMware vSphere and its associated products in their company infrastructure. This course teaches you how to install and maintain this virtualization technology to store data, prevent vuln…
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …
Suggested Courses

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question