Automated Data Extraction from Web Pages

The number of Exhibitors at GDC 2015 is over 400. There is a list of all of them on this page: . When you click on any one of them, you can determine at what location they have their offices.

It will take too much time, however, to click on each, one-at-a-time, but I want to determine which companies that are exhibitors at GDC 2015 are from California, the Bay Area, or other states in the USA, rather than other countries.

Any ideas how to determine which companies are from California or the Bay Area?

Can you help me to write a script in PHP or Excel VB that will take each link from a list in Excel (or other file format), automatically open each link, scan the page and return company name, country, and state information into a file?
Jerry LOperations ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

You should probably start by contacting the copyright holder UBM Tech who own the database and you could check to see if they can supply it in a usable file format or make an API available.  

If it is your intention to just take the content without the permission of the owners then I would consider that copyright infringement, or even theft of content data which is illegal in most jurisdictions.  As EE has rules against assisting in illegal activity I think no Expert with integrity is going to help with this unless you can explain how it is not copyright infringement.

Jerry LOperations ManagerAuthor Commented:
No, that is not my intention. I am taking information the owners already provide. I even called to ask how I might obtain a ready-made list, but they said it was not available. I am putting their information into a more useful format in which you can associate the company name with the address given on the subsequent page. I am then using it for personal use only, not even posting it or giving it away anywhere, except perhaps to the conference reps who might be able to use it themselves. There is no copyright violation.
Ray PaseurCommented:
It appears that the data you can see on this page is loaded from an API.  The data is not present in the HTML document. This is a technique that web developers deliberately choose to make it difficult to steal their data -- you can't just "scrape" the information out of the web page; you have to know how to interact with the API.  If you ask the owners for the API and they refuse, that should be a pretty clear signal that they do not want you to get this information without paying for it, except through their official web site.

I'll see if I can get a list of the Bay Area companies.  If I can, I'll post it here.
Become a Certified Penetration Testing Engineer

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

Ray PaseurCommented:
I take you at your word that there is no copyright violation.  This is a one-point-in-time analysis, and cannot account for registrations after this morning.

From the Bay Area:

Aarki | Mountain View, California (CA) , 94043
Academy of Art University | San Francisco, California (CA) , 94107
Adya Technologies Inc. | Palo Alto, California (CA) , 94303
Amazon Mobile Ad Network | Palo Alto, California (CA) , 94301
App Annie | San Francisco, California (CA) , 94108
AppLift | San Francisco, California (CA) , 94103
Apportable, Inc. | San Francisco, California (CA) , 94107
AppsFlyer | San Francisco, California (CA) , 94105
Belgian Trade Commission | San Francisco, California (CA) , 94104
Carr & Ferrell LLP | Menlo Park, California (CA) , 94025
Chartboost, Inc. | San Francisco, California (CA) , 94102
Cocos2d-x / Chukong Inc. | Menlo Park, California (CA) , 94025
deltaDNA | San Francisco, California (CA) , 94107
Dolby | San Francisco, California (CA) , 94103
Fresvii | Redwood City, California (CA) , 94063
Google, Inc. | Mountain View, California (CA) , 94043
InMobi | San Francisco, California (CA) , 94107
IRONSOURCE  LTD | San Francisco, California (CA) , 94105
Kongregate | San Francisco, California (CA) , 94111
Last Life Games | El Cerrito, California (CA) , 94530
Loggly Inc | San Francisco, California (CA) , 94104
Manage | San Francisco, California (CA) , 94107
MdotM | San Mateo, California (CA) , 94404
Mego | San Mateo, California (CA) , 94401
Mixamo | San Francisco, California (CA) , 94107
MyDream Interactive | Palo Alto, California (CA) , 94301
Netherfire Entertainment, Inc. | Sunnyvale, California (CA) , 94085
Nod Labs | Mountain View, California (CA) , 94041
O'Clock | San Francisco, California (CA) , 94107
PagerDuty | San Francisco, California (CA) , 94107
Perforce | Alameda, California (CA) , 94501
Personagraph | Sunnyvale, California (CA) , 94085
Pole To Win America | Santa Clara, California (CA) , 95054
Pollen VC | San Francisco, California (CA) , 94104
RadiumOne | San Francisco, California (CA) , 94103
Redis Labs | SANTA CLARA, California (CA) , 95054
ROBLOX | San Mateo, California (CA) , 94401
Samsung Developer Connection | Mountain View, California (CA) , 94043
SEWORKS | San Francisco, California (CA) , 94105
SkyMobi Publishing | San Francisco, California (CA) , 94105
Spellbind Studios | San Francisco, California (CA) , 94103
Sqor Sports | San Francisco, California (CA) , 94111
Stanford University | Palo Alto, California (CA) , 94305
Storm8 | Redwood City, California (CA) , 94065
Supersonic | San Francisco, California (CA) , 94107
Tactical Haptics | Fremont, California (CA) , 94555
Three Flip Studios | San Francisco, California (CA) , 94109
Tipping Goat | Milpitas, California (CA) , 95035
Transifex | Menlo Park, California (CA) , 94025
Unity Technologies | San Francisco, California (CA) , 94133
uSens Inc. | San Jose, California (CA) , 95112
Zendesk, Inc. | San Francisco`, California (CA) , 94118

From elsewhere in Calif:

Airpush | Venice, California (CA) , 90291
Alderac Entertainment Group / Rho Sigma | Ontario, California (CA) , 91761
Blizzard Entertainment, Inc. | Irvine, California (CA) , 92618
BluBox Games | Santa Monica, California (CA) , 90405
EngagedLeadership | Los Angeles, California (CA) , 90210
Esri | Redlands, California (CA) , 92373
Faceware Technologies, Inc. | Sherman Oaks, California (CA) , 91403
FGL | Santa Monica, California (CA) , 90401
Fugitive Games | Santa Monica, California (CA) , 90403
Italian Trade Commission | Los Angeles, California (CA) , 90067
La Cosa Entertainment | Burbank, California (CA) , 91515
Laguna College of Art & Design | Laguna Beach, California (CA) , 92651
Leadbolt | Los Angeles, California (CA) , 90036
Namazu Studios | San Diego, California (CA) , 92108
ProChile | Los Angeles, California (CA) , 90048
Qualcomm | San Diego, California (CA) , 92121
Xsolla | Sherman Oaks, California (CA) , 91403

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jerry LOperations ManagerAuthor Commented:
Thank you for the list. It would also be helpful to complete the list in order to refer to companies in other states and countries. It takes too long to do it by hand. You do not need to do any sorting. I will do that in Excel.
Ray PaseurCommented:
Sorry, this is not really a question that has any answer - it's a requirement for application development.  I understood the objective to be "determine which companies are from California or the Bay Area?" and I've done my best to answer that.  For something more expansive it's probably appropriate to hire a professional developer, since acquiring and reporting this data is a fairly time-consuming (read "expensive") process.  A good developer can give you several views of the data for less than $1,000 in less than a day of work.  But I would be surprised if someone wants to do this for free in a forum like E-E.  We are good at answering technical questions, less so at building end-to-end solutions.

Best of luck with your project, ~Ray
Ray PaseurCommented:
Thanks for the points.  It's a good question, but one with an expansive and expensive workload before it!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.