asked on

How do I get curl to pull back a page that has options on it after a redirect?

I am using bash script with curl in Ubuntu 11.04 to capture a web page. I am new to this and this seems to be a complicated case in that it has a redirect and then options on the page for the data. I can't seem to get the options to change. I have tried both -b to send the cookie and -d to send a setting. Perhaps its the order I am doing this as I also have a -L for the redirect and -A to send as Chrome.

So the page has a drop down for the language and it defaults to a language I can't use. I want to select a different language in that drop down. When I do this on the web page, the URL does not change at all.

noci

Please do a view page as source in a browser.
First check if the page is using java script or not.

If not then curl can be used straight away, but you may need several steps (request/responses) to continue to use it..

If javascript is used you need to work out what it does and emulate it using bash (or perl etc).
That might complicate things. Afaict there is no js interpreter that is usable for such things.

ccbe

ASKER

Thanks for the help.

It is using javascript.

Clearly I'm a newbie, but isn't there a simple way to avoid this complication by using the cookie? The site has my settings when I return in Chrome or FF. I was thinking that because of the redirect I am sending the cookie at the wrong time?

Would it be:

curl -L -b cookie.txt "http:\\www.site.com"

If not, what happens to the sites request for the cookie after the redirect?

Here is an example of the setting I want to change:

<span id="UPB"><div class='prefsbox'>
<select name="ctl00$LDDL" onchange="javascript:setTimeout('__doPostBack(\'ctl00$LDDL\',\'\')', 0)" id="LDDL">
                  <option value="1">English (N. American)</option>
                  <option selected="selected" value="2">Italiano (Italian)</option>
                  <option value="3">Suomi (Finnish)</option>
                  <option value="4">Deutsch (German)</option>
                  <option value="5">Norsk (Norwegian)</option>
                  <option value="6">Português (Portuguese)</option>
                  <option value="7">Español (Spanish)</option>
                  <option value="8">Svenska (Swedish)</option>
</select></div>
      </span>

noci

Well there clearly is form around this, that provides the url,
what does settimeout do (most porbably, start a timer and tun the doPostBack).

Then what does doPostBack do? You realy need to get into the small items... if there is a form post, THAT what's needed to set a variable/cookie.

SOLUTION

parparov

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ccbe

ASKER

nocl: I don't know what any of the pieces do, I didn't put the site together. I am just trying to get some info off of it automatically so I can create a report over time.

parpov: Does this allow me to do what I said above? Again, I am a newbie at this, I looked at the page you gave me, I just don't want to go down another road filled with roadblocks.

Thanks to you both for the help.

ASKER CERTIFIED SOLUTION

noci

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial