How to scrape non-page source data client side on a PC

hi,

Many web pages contain information that is not stored in the source code for the page, eg pricing info, so it's harder to scrape.

Where is this info stored when loaded to a PC browser, and how can i read it? some kind of cache or cookie? i'm a newbie on the web side.

example Airbnb: https://www.airbnb.co.uk/rooms/.. any room id... ?checkin=20-05-2018&checkout=22-05-2018&guests=2

Background: I'm trying another angle to resolve this other question: https://www.experts-exchange.com/questions/29094633/Web-scrape-macro-VBA.html

Thanks
xeniumAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
>  information that is not stored in the source code.... ....Where is this info stored when loaded to a PC browser

If it is not stored in the source code, then it is stored on the server and only available when requested.

Anyway, please review their terms https://www.airbnb.co.uk/terms
use any robots, spider, crawler, scraper or other automated means or processes to access, collect data or other content from or otherwise interact with the Airbnb Platform for any purpose;

It does not matter if you are taking a little at a time or a large amount of data.
0
Brian DayhoffSenior Full Stack DeveloperCommented:
The short answer is you can't.

This is an explicit anti-scrape measure, which requires that data requests only occur from scripts that are pre-authenticated, served from their own server, and are keyed to the same session. If their scripts are written poorly (they aren't, but some sites are), you can get around this by scraping with Selenium. However client scraping is extremely slow and non-performant. You are also breaking the law and violating their TOS if you scrape them anyways, which could result in a lawsuit or criminal charge. You really have to ask yourself if the data is worth that much to you. It probably isn't.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
xeniumAuthor Commented:
Thanks for the heads up, yes well considering it's my own data I wanted to scrape off them it's not worth that much! I will look into the API route as Scott mentioned on this post this post.
Thanks again
1
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

xeniumAuthor Commented:
I wanted to split points but the site seems to have changed yet again making this seem impossible
1
xeniumAuthor Commented:
AutoHotKey might help here
0
Joe Winograd, Fellow&MVEDeveloperCommented:
Hi xenium,
Thanks for the reference to my AutoHotkey - Getting Started article...much appreciated! Also, you may find this site to be helpful:
http://the-automator.com/web-scraping-with-autohotkey/
Regards, Joe
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
browser caching

From novice to tech pro — start learning today.