How to build a web crawler?

How exactly does one "build" a web crawler? Are there any recommended tutorials?

I am trying to aggregate product name, quantity in stock, and pricing from a specified website.

That being said, I am not sure where to start. I do have knowledge of programming (mostly in VBA and SQL, but I know my way around other languages, too).
EnigmaMatterAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

LesterClaytonCommented:
I've had to often do this, and there's really two approaches:

Approach 1 - The Lazy Man's approach.


Using a product such as wget to download the entire site as is (or the specific page, if you know the URL and it doesn't change), and then crawl through the resultant .html files to extrac the data you want.

Approach 2 - The Programmers approach.


Write an application that uses an http client and actually make the calls to the web site to retrieve the data you want, and then parse the results.

I have often done Approach 2 using Delphi and the ICS suite which has a nice HTTP(S) client that you can make behave as you see fit.  Delphi also has a suite called Indy which can pretty much do the same thing.  I understand that Delphi is not the most common programming language, but you can do it with other things too - like curl, which is very easily interoperable with many different OS's and languages.

Hopefully, food for thought :)
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
EnigmaMatterAuthor Commented:
Thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.