asked on 7/17/2008

I am looking for an opensource threaded web spider/web crawler/web scraper package for Linux. Any suggestions?

Web Development Linux Programming

Last Comment

Duncan Roe

7/18/2008

Arty K

7/17/2008

http://www.httrack.com/ is like a 'teleport', but open source, multithreaded ...

If you need something different (say if you don't need offline downloading), be more specific.

Arty K

7/17/2008

Say, there is a 'Lucene' engine, that is also a crawler, but used for text indexing/searching: http://lucene.apache.org/java/docs/

Octalys

7/18/2008

ASKER

I need to collect urls with the software.

ASKER CERTIFIED SOLUTION

Duncan Roe

7/18/2008

THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.

View this solution by signing up for a free trial.

Members can start a 7-Day free trial and enjoy unlimited access to the platform.

See Pricing Options

Start Free Trial

Web Development

Web development includes all aspects of presenting content on intranets and the Internet, including delivery development, protocols, languages and standards, server software, browser clients, databases and multimedia generation.

78K

Questions

Followers

Top Experts

Get a personalized solution from industry experts

Ask the experts

"Invaluable tool for our organization"
Josh Regalski
"This is the best money I have ever spent."
@rwheeler23
"Your help has saved me hundreds of hours of internet surfing."
@fblack61
"Just a dang good service!"
@golferski
"Worth every penny."
Steve Hougom
"It’s been a life saver."
Maria Halt
"Best support site I’ve seen!"
Bob Schneider
"I would be lost without them."
@fblack61
"Saved my job multiple times."
Walt Forbes
"I love this site."
Maria Duwe
"Friendly respectful help."
Andrew Kowtalo
"Such top-notch experts."
@magento
"The best money I have ever spent."
@rwheeler23
"Beyond any comprehensible value"
George Morris
"Invaluable tool for our organization"
Josh Regalski
"This is the best money I have ever spent."
@rwheeler23

Read over 600 more reviews

TRUSTED BY