Jack McKenzie
asked on
Service or library for most automated and most productive (fastest, easiest) scraping of different flatshare websites.
I'm trying to scrape available flatshares from certain websites for personal use and I wonder what's the most productive tool that will be flexible enough to scrape all portals and what will be the most automated tool, that can just ask for a URL and in seconds provide a consumable feed, ideally with minimum fees or no fees (downloadable or buildable solution)?
David Favor
You can always use http://phantomjs.org/ + roll your own scrape tool.
ASKER
This is helpful. The solution should avoid hand coding. Do you know a scraping service or tool with AI?
Unsure how AI will assist here. An AI or neural network must be taught to do something. Just doesn't seem to fit your task.
You'll do the following...
1) Scrape your data from sites.
2) Pass data through a tool like html2text or some sort of HTML parser.
3) Do something with the results.
Simple to design.
Time intensive to develop.
Keep in mind, you'll have to keep your developer on staff forever, because each time a Website changes it's format (HTML) your scraper will still work + your parser will have to change... sometimes slightly... sometimes a complete rework...
This is just the nature of how scrape + parse projects work.
Development continues forever.
You'll do the following...
1) Scrape your data from sites.
2) Pass data through a tool like html2text or some sort of HTML parser.
3) Do something with the results.
Simple to design.
Time intensive to develop.
Keep in mind, you'll have to keep your developer on staff forever, because each time a Website changes it's format (HTML) your scraper will still work + your parser will have to change... sometimes slightly... sometimes a complete rework...
This is just the nature of how scrape + parse projects work.
Development continues forever.
ASKER
I know how scrapers work. A scraping service with AI will scrape without coding. Phantom JS is deprecated and so is manual scraping, at least for those who are up to date on information extraction (no pun intended).
In 2018, data can be extracted from a web page without coding anything. Here is an example: https://www.diffbot.com/ This AI-based scraper is not the only one. They have existed for quite some time.
Does anyone know an Open Source library for AI-based scraping, or a free service?
// Note: In expert terms, AI-based scraping is what comes next after the so-called "visual scraping".
In 2018, data can be extracted from a web page without coding anything. Here is an example: https://www.diffbot.com/ This AI-based scraper is not the only one. They have existed for quite some time.
Does anyone know an Open Source library for AI-based scraping, or a free service?
// Note: In expert terms, AI-based scraping is what comes next after the so-called "visual scraping".
This question needs an answer!
Become an EE member today
7 DAY FREE TRIALMembers can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.