September 6, 2008 02:16pm pdt

Member Login

Hall of Fame

Yearly

1jpaulino 850,895
2TheLearn... 744,083
3Idle_Mind 450,605
4VBRocks 394,599
5Sancler 373,508
6PaulHews 353,757
7Dhaest 326,704
8Fernando... 326,433
9jaime_oliv... 313,669
10jcoehoorn 260,456
11emoreau 248,669
12webtubbs 240,242
13angelIII 193,312
14vb_jonas 183,591
15rachitkohli 167,704
16GreenGho... 166,611
17mlmcc 144,309
18graye 126,980
19armoghan 113,840
20planocz 98,038
21TimCottee 96,932
22ZachSmith 93,895
23AkisC 89,005
24CodeCruiser 88,690
25philipjonat... 74,550

	[x] Attachment Details

VB.net MultiThreaded Web Page Parsing application - how to parse multiple url's simultaneously

Zones: Microsoft Visual Basic.Net, Visual Studio .NET 2005, .NET Framework 2.0

Tags: vb.net multithreading web requests, Use multithreading application to request multiple url's simultaneously and parse data

Hi Guys,

I am a VB.net newbie and am learning through creating applications I can actually use. Now my application is designed to crawl certain url's and extract needed information from them.

It is working fine in one threaded mode, but when you have to do 17k url's and extract information from 3 key seed url's (yahoo and google) then it needs multithreading.

For the parsing we are currently using webbrowser control but I realize now this is definately not going to work for multithreading.

We are currently using Document.GetAttribute type calls to exctract the information we need (href tags and innerhtml etc from links within the document).

What I need is to be able to say start 10 threads (or enter number of threads to use in main box in gui though I can add that later if we work out the thread part first).

That will read from the dataset containing the url's sequentially, so first thread grabs first url from first row, second will know that the first url has been taken so grab the next one in line, and so on.

Then each thread should be able to get the url and parse the contents of it so we can then extrac html links etc etc. Then write the extracted data back to the table.

So who can help me with setting up a multi url parsing multi threaded code to get moving?

I would be greatly appreciative.

Thanks

Start your free trial to view this solution

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

9.0

Question Stats

Zone: Programming

Question Asked By: marclindsay1

Solution Provided By: DarkoLord

Participating Experts: 2

Solution Grade: A

Translate:

Loading Advertisement...

20080723-EE-VQP-34 / EE_QW_2_20070628