How to scrape non-page source data client side on a PC

xenium
xenium used Ask the Experts™
on
hi,

Many web pages contain information that is not stored in the source code for the page, eg pricing info, so it's harder to scrape.

Where is this info stored when loaded to a PC browser, and how can i read it? some kind of cache or cookie? i'm a newbie on the web side.

example Airbnb: https://www.airbnb.co.uk/rooms/.. any room id... ?checkin=20-05-2018&checkout=22-05-2018&guests=2

Background: I'm trying another angle to resolve this other question: https://www.experts-exchange.com/questions/29094633/Web-scrape-macro-VBA.html

Thanks
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Scott FellDeveloper & EE Moderator
Fellow 2018
Most Valuable Expert 2013

Commented:
>  information that is not stored in the source code.... ....Where is this info stored when loaded to a PC browser

If it is not stored in the source code, then it is stored on the server and only available when requested.

Anyway, please review their terms https://www.airbnb.co.uk/terms
use any robots, spider, crawler, scraper or other automated means or processes to access, collect data or other content from or otherwise interact with the Airbnb Platform for any purpose;

It does not matter if you are taking a little at a time or a large amount of data.
Senior Full Stack Developer
Commented:
The short answer is you can't.

This is an explicit anti-scrape measure, which requires that data requests only occur from scripts that are pre-authenticated, served from their own server, and are keyed to the same session. If their scripts are written poorly (they aren't, but some sites are), you can get around this by scraping with Selenium. However client scraping is extremely slow and non-performant. You are also breaking the law and violating their TOS if you scrape them anyways, which could result in a lawsuit or criminal charge. You really have to ask yourself if the data is worth that much to you. It probably isn't.

Author

Commented:
Thanks for the heads up, yes well considering it's my own data I wanted to scrape off them it's not worth that much! I will look into the API route as Scott mentioned on this post this post.
Thanks again
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
I wanted to split points but the site seems to have changed yet again making this seem impossible

Author

Commented:
AutoHotKey might help here
Joe WinogradDeveloper
Fellow 2017
Most Valuable Expert 2018

Commented:
Hi xenium,
Thanks for the reference to my AutoHotkey - Getting Started article...much appreciated! Also, you may find this site to be helpful:
http://the-automator.com/web-scraping-with-autohotkey/
Regards, Joe

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial