We help IT Professionals succeed at work.

how to do javascript/ajax web scraping in php (data extraction / parsing)

808 Views
Last Modified: 2013-11-18
Recently I've been hired to do various scrapings but I've come across an unusual website that appears to use some form of AJAX to list its inventory.

Anyway... here's the URL:

http://apps.dealerconnection.com/dealers/bradleyford/usedvehicles?lang=en

Sometimes if Javascript is used you can parse the javascript because it becomes static upon execution, but I believe this uses ajax, so I have no idea what to do at this point. The only thing that I can think to do is maybe try to find the true source of the information, which from my understanding, still has to be some kind of server side script like php or asp... right?

Can anyone find it? I'm lost...
Comment
Watch Question

CERTIFIED EXPERT
Top Expert 2004

Commented:
AJAX could be using a database back-end, so you may not ever be able to access the source.  A good starting point would be to talk to the webmaster to find out if they have an API you can use to access their information.
RobOwner (Aidellio)
CERTIFIED EXPERT
Most Valuable Expert 2015

Commented:
this is a link to the js they use for the AJAX: http://content.dealerconnection.com/vfs/apps/js/yui/connection/connection-min.js

here is an example on that page:

YAHOO.util.Connect.asyncRequest('POST',"http://apps.dealerconnection.com/dealers/bradleyford/emailtofriend", callbackSendEmail);

so yes it does talk to a server side script using AJAX.  you will need to get admin access to the website to get acccess to those files

Author

Commented:
Tagit, if the ajax is posting to a backend script... it must be doing so somewhere in the javascript... correct? Is it possible for me to create a php script to post to the same script?
RobOwner (Aidellio)
CERTIFIED EXPERT
Most Valuable Expert 2015

Commented:
Yes i don't see why not but you would need to pass the same arguments and variables that the scripts is expecting.  give it a try
CERTIFIED EXPERT
Top Expert 2004
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
RobOwner (Aidellio)
CERTIFIED EXPERT
Most Valuable Expert 2015
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.