Link to home
Start Free TrialLog in
Avatar of Travis Hydzik
Travis HydzikFlag for Australia

asked on

PHP Curl to output a url

I need some help grabbing the output of the following url with PHP curl;
link requiring output
The issue is around the link only producing valid data after you click on the following link (no other input is required)
link that must be clicked prior

I can't seem to replicate what is going on with PHP curl.
Avatar of Peos John
Peos John
Flag of Malaysia image

Can you try this code?

 
$URL = 'http://mapimage.net/intramaps80/ApplicationEngine/Search/ComboContents?templateId=06efad38-2204-47c6-9ca1-b642806abad6';


$ch = curl_init();  

curl_setopt($ch,CURLOPT_URL,$URL);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_HEADER, false); 

$output=curl_exec($ch);

curl_close($ch);




print_r($output);

Open in new window

The server encountered an error processing the request.
The mapimage.net web site does not appear to be correctly implemented.  What are we expecting to find here:
http://mapimage.net/intramaps80/
http://mapimage.net/
ASKER CERTIFIED SOLUTION
Avatar of gr8gonzo
gr8gonzo
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
By the way, this type of thing usually tends to be exposed by an official web service. You might want to try contacting the site and ask them if they have an API or a web service. It could result in much faster and more stable results long-term. Web scraping tends to only work until someone at the remote web site makes a change that breaks your script.
Avatar of Travis Hydzik

ASKER

This is the start of the expected output;
<ComboContentResponse xmlns="http://schemas.datacontract.org/2004/07/DMS.IntraMaps.ApplicationEngine" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><header><warning i:nil="true"/><authUser>anonymous</authUser><sessionId>s40e0doj3fkuj5jpswtooxqm</sessionId></header><items><ComboBoxItem><key>Abalone VW</key></ComboBoxItem><ComboBoxItem><key>Abba PL</key></ComboBoxItem><ComboBoxItem><key>Abbeville CIR</key></ComboBoxItem>

Open in new window


Response to Peos John @PeosJohn: that doesn't take into account the fact you need to click on the second link first.
Excellent!

Can I ask.
How did you know the second link was required to set up the session?
It was a bit of trial-and-error. When my script only tried the first link and the the final link, I was still getting the error. I loaded up the first link in my browser and watched the network console to see the various other requests that were sent out. There were a couple that weren't your typical static content requests (images, css, etc...), so I pulled the most promising-looking one and then tested it out and it seemed to make the difference.

Again, official APIs and web services will provide formal documentation about what you would need to do - no guesswork involved. Plus, if they ever add another required step, this script might break and you would have to figure out what to do again. So reach out to the company that hosts this service and ask them if there's an API you can use. It's the difference between sneaking into your neighbor's house to steal a cup of sugar and simply asking your neighbor nicely for the same cup of sugar. When you ask nicely, you'll often get the same result and you won't risk creating a bad relationship.