How to scrape non-page source data client side on a PC

hi,

Many web pages contain information that is not stored in the source code for the page, eg pricing info, so it's harder to scrape.

Where is this info stored when loaded to a PC browser, and how can i read it? some kind of cache or cookie? i'm a newbie on the web side.

example Airbnb: https://www.airbnb.co.uk/rooms/.. any room id... ?checkin=20-05-2018&checkout=22-05-2018&guests=2

Background: I'm trying another angle to resolve this other question: https://www.experts-exchange.com/questions/29094633/Web-scrape-macro-VBA.html

Thanks
xeniumAsked:
Who is Participating?
 
Brian DayhoffSenior Full Stack DeveloperCommented:
The short answer is you can't.

This is an explicit anti-scrape measure, which requires that data requests only occur from scripts that are pre-authenticated, served from their own server, and are keyed to the same session. If their scripts are written poorly (they aren't, but some sites are), you can get around this by scraping with Selenium. However client scraping is extremely slow and non-performant. You are also breaking the law and violating their TOS if you scrape them anyways, which could result in a lawsuit or criminal charge. You really have to ask yourself if the data is worth that much to you. It probably isn't.
0
 
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
>  information that is not stored in the source code.... ....Where is this info stored when loaded to a PC browser

If it is not stored in the source code, then it is stored on the server and only available when requested.

Anyway, please review their terms https://www.airbnb.co.uk/terms
use any robots, spider, crawler, scraper or other automated means or processes to access, collect data or other content from or otherwise interact with the Airbnb Platform for any purpose;

It does not matter if you are taking a little at a time or a large amount of data.
0
 
xeniumAuthor Commented:
Thanks for the heads up, yes well considering it's my own data I wanted to scrape off them it's not worth that much! I will look into the API route as Scott mentioned on this post this post.
Thanks again
1
 
xeniumAuthor Commented:
I wanted to split points but the site seems to have changed yet again making this seem impossible
1
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.