Solved

How do I get curl to pull back a page that has options on it after a redirect?

Posted on 2011-09-27
6
443 Views
Last Modified: 2013-11-13
I am using bash script with curl in Ubuntu 11.04 to capture a web page. I am new to this and this seems to be a complicated case in that it has a redirect and then options on the page for the data. I can't seem to get the options to change. I have tried both -b to send the cookie and -d to send a setting. Perhaps its the order I am doing this as I also have a -L for the redirect and -A to send as Chrome.

So the page has a drop down for the language and it defaults to a language I can't use. I want to select a different language in that drop down. When I do this on the web page, the URL does not change at all.
0
Comment
Question by:ccbe
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
6 Comments
 
LVL 40

Expert Comment

by:noci
ID: 36716116
Please do a view page as source in a browser.
First check if the page is using java script or not.

If not then curl can be used straight away, but you may need several steps (request/responses) to continue to use it..

If javascript is used you need to work out what it does and emulate it using bash (or perl etc).
That might complicate things. Afaict there is no js interpreter that is usable for such things.

0
 

Author Comment

by:ccbe
ID: 36717492
Thanks for the help.

It is using javascript.

Clearly I'm a newbie, but isn't there a simple way to avoid this complication by using the cookie? The site has my settings when I return in Chrome or FF. I was thinking that because of the redirect I am sending the cookie at the wrong time?

Would it be:

curl -L -b cookie.txt "http:\\www.site.com"

If not, what happens to the sites request for the cookie after the redirect?

Here is an example  of the setting I want to change:

<span id="UPB"><div class='prefsbox'>
<select name="ctl00$LDDL" onchange="javascript:setTimeout(&#39;__doPostBack(\&#39;ctl00$LDDL\&#39;,\&#39;\&#39;)&#39;, 0)" id="LDDL">
                  <option value="1">English (N. American)</option>
                  <option selected="selected" value="2">Italiano (Italian)</option>
                  <option value="3">Suomi (Finnish)</option>
                  <option value="4">Deutsch (German)</option>
                  <option value="5">Norsk (Norwegian)</option>
                  <option value="6">Portugu&#234;s (Portuguese)</option>
                  <option value="7">Espa&#241;ol (Spanish)</option>
                  <option value="8">Svenska (Swedish)</option>
</select></div>
      </span>


0
 
LVL 40

Expert Comment

by:noci
ID: 36718187
Well there clearly is form around this, that provides the url,
what does settimeout do (most porbably, start a timer and tun the doPostBack).

Then what does doPostBack do?  You realy need to get into the small items... if there is a form post, THAT what's needed to set a variable/cookie.
0
PeopleSoft Has Never Been Easier

PeopleSoft Adoption Made Smooth & Simple!

On-The-Job Training Is made Intuitive & Easy With WalkMe's On-Screen Guidance Tool.  Claim Your Free WalkMe Account Now

 
LVL 9

Assisted Solution

by:parparov
parparov earned 100 total points
ID: 36718325
You may want to use Rhino Shell:
https://developer.mozilla.org/en/Rhino_Shell

for scripting the Javascript commands
0
 

Author Comment

by:ccbe
ID: 36719187
nocl: I don't know what any of the pieces do, I didn't put the site together. I am just trying to get some info off of it automatically so I can create a report over time.

parpov: Does this allow me to do what I said above? Again, I am a newbie at this, I looked at the page you gave me, I just don't want to go down another road filled with roadblocks.

Thanks to you both for the help.
0
 
LVL 40

Accepted Solution

by:
noci earned 400 total points
ID: 36719752
When the selection of the language causes the onSelect to be executed.
what is done is described in __doPostBack()
The code for it is in one of the downloaded css sheets, the html file or just referenced before.

@parpov, interesting tool i have to look into that...

checking out.../compiling ... it seems to work.

It might be usable if it can connect to the remote site, at least the actions of the onSelect can be executed.
But the steps still need to be setup in some order like you would click & select through a page when done.
Curl may be used for cookie management & http queries .., but it needs quite some work...
either way.
There is insufficient info to provide more detailed steps.
0

Featured Post

Resolve Critical IT Incidents Fast

If your data, services or processes become compromised, your organization can suffer damage in just minutes and how fast you communicate during a major IT incident is everything. Learn how to immediately identify incidents & best practices to resolve them quickly and effectively.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Replacing multiple embedded commas in a CSV file with sed 10 44
Saving Web Page as PDF 9 84
ninja forms fields lay next to each other 1 23
wordpress 4.7.4, theme 8 44
Although a lot of people devote their energy toward marketing for specific industries, there are some basic principles that can be applied to any sector imaginable. We’ll look at four steps to take and examine how those steps were put into action fo…
CTAs encourage people to do something specific to show interest in your company, product or service. Keep reading to learn why CTAs should always be thought of as extremely important, albeit small, sections of websites.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question