?
Solved

How do I curl or fopen final javascript source?

Posted on 2009-07-14
20
Medium Priority
?
631 Views
Last Modified: 2012-05-07
I am trying to extract data from a javascript webpage.  Normally for a html page I can use cUrl or fopen and grab the source code then look for and match data I need, but with in those pages there are parts that are javascript and in those parts I can only see javascript code, not the final data like when veiwing from a web browser.

How can I use cUrl and or fopen to view and preg_match through the final Javascript code?
<table>
<tr>
    <td>
<script language="JavaScript" type="text/javascript">
<!--
ctxt_rk_bc = "EC690E";
ctxt_rk_cc = "FFFFFF";
ctxt_rk_lc = "0033FF";
ctxt_rk_tc = "333333";
ctxt_rk_uc = "666666";
// -->
</script>
<script language="JavaScript" src="http://www.sample.com/test.js">
</script>
 
</td>
  </tr>
<table>

Open in new window

0
Comment
Question by:marcelof13
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 6
  • 2
  • +2
20 Comments
 
LVL 2

Expert Comment

by:onethreefour
ID: 24856650
It sounds like your problem is that some of the javascript code is 'hidden' in .js files which are called from 'script src' within the html page... You could use the php split function to parse the URL of the .js file out of the html page.
http://us2.php.net/manual/en/function.split.php

Then use cURL or fopen to pull down the .js file which you could then run your preg_match on.
http://us3.php.net/manual/en/function.preg-match.php

The initial split will be a little dirty, because you will need to split everything off of the front of the javascript src url, then you will need to split off everything after the url... Leaving you with just the URL to the .js file, which you then cURL and then preg_match...
Something like:
$stuff = split('<script language="JavaScript" src="', $wholeHTMLpage);
$stuff2 = split('">',$stuff);
$result = cURL($stuff2);
I would prefer to do this in perl, my php is a bit rusty and the above code is probably not exactly correct for php, but hopefully you get the point...
0
 
LVL 10

Expert Comment

by:racmail2001
ID: 24856873
i can understand something else from your question.

i think that what you are trying to say is:

you whant to grab a page that is generated trough javascript/ajax/jquery and to grab the result of that

i don't think it's posible because javascript it's executed on the client side after all the content it's pulled of the server. with fopen or curl you grab the content of the server (or some of it - not the images, the css, js, etc) and than you shuld run the javascript .

but from php you can't do that
0
 

Author Comment

by:marcelof13
ID: 24857730
hi Racmail2001,
                     There is got to be some way to do that.  If a browser can do it, why cant some small amount of code?

0
Are You Using the Best Web Development Editor?

The worlds of web hosting and web development are constantly evolving. Every year we see design trends change, coding standards adapt and new frameworks/CMS created. With such a quick pace of change it’s easy to get lost trying to keep up.

See if your editor made the list.

 
LVL 2

Accepted Solution

by:
dinu_sorin earned 668 total points
ID: 24857959
marcelof13, if you want to execute the js code, you must use other language. With some language you can use IE api to explore websites. (try delphi, is good) Happy Coding!
0
 

Author Comment

by:marcelof13
ID: 24858814
Come on guys, there is got to be a solution out there.  Going to Delphi cant be the answer, can it?
0
 
LVL 10

Assisted Solution

by:racmail2001
racmail2001 earned 668 total points
ID: 24858852
in php for sure is NOT
0
 
LVL 54

Assisted Solution

by:b0lsc0tt
b0lsc0tt earned 664 total points
ID: 24860883
A web server language (e.g. .NET, PHP) doesn't really offer the answer (as experts have said above).  The closet is Java although you might need some class?/component? to do it.  Also I don't know if the results would work as you want.  I believe when I have heard it discussed the "engine" it has used is the same as Mozilla's so that part should be good and give decent results.  Unfortunately Java as a web language is one that I know the least about so I don't have a lot more info.
The general problem is you don't just need a language because you want the Javascript run and interpreted as the browser would.  That is what makes this not as easy as you may have originally thought.  .NET does support the JScript language, which is basically IE's version of Javascript, so that could be something to try if you use .NET.  This won't be perfect at all though because the languages aren't exactly the same and so the code may fail.  Also the results might not be the same as if you had the Javascript in the browser since it is .NET running the code.  The eval() function does work in JScript though so that is a bonus.
You should figure out exactly what you need from the JS source though?  Also what is the code that makes it?  You might have an easier option if you just need part and can "read" it with your server language, even PHP, and gets its intent.  Basically you aren't running it but you are reading it to make the result.  Any server language could do that but it is definitely limiting.
Let me know if you have a question about this.
bol
0
 
LVL 54

Expert Comment

by:b0lsc0tt
ID: 24861036
You might look at http://j4p5.sourceforge.net/ .  That project seems to be trying to make something for PHP like what you would need.  Notice the limitations though and you should check the results.  I am not aware of how this works or similar projects but there may be others.  IMO they aren't something that will work for all cases though so it really will depend on exactly what you are working with and trying to get.
bol
0
 

Author Comment

by:marcelof13
ID: 25010835
Thanks guys, so far none of the leads have helped me out.  
0
 
LVL 54

Expert Comment

by:b0lsc0tt
ID: 25015418
If the project I suggested won't work then I don't know of anything else for PHP.  If you can use Java (i.e. JSP) then I believe there is an option you can try.  Let me know if you want info on it.
If this has to be in PHP and do this then you could revisit this in a few months or a year.  At that time the project I suggested or another may work for PHP.
bol
0
 

Author Comment

by:marcelof13
ID: 25108084
Please post the information using Java or JSP, thanks.
0
 
LVL 54

Expert Comment

by:b0lsc0tt
ID: 25118252
Java isn't my expertise, I have only done a little work in it, but from what I have read they do have access to a "Javascript engine."  The best info on the topic I have found is at http://java.sun.com/javase/6/docs/technotes/guides/scripting/programmer_guide/index.html .  I haven't even tested the code mentioned there, don't know enough about JSP to know if it can be used in that case, or know if that engine would run the Javascript you need.  However it is at least a start if you can use Java and would seem to do what you need.
Let me know how it helps or if you have a question.
bol
0
 
LVL 54

Expert Comment

by:b0lsc0tt
ID: 25369678
marcelof13,

Although the answer wasn't what you hoped I believe you did get an answer here.  You were told the methods you were using would definitely not work and why.  An option that might work in PHP was presented.  Since that didn't work you were told PHP couldn't do it and given an alternative.  Basically the answer was "you can't" and that can be a valid answer.  If you disagree please comment and explain but I feel this was answered.

By the way since the details of the other language were posted at your request it seems odd that you appear to use them as an excuse to delete this.  Just an observation. :)  I really am sorry there wasn't something in PHP for this (yet) but this definitely wasn't an easy thing you are trying to do.

bol
0
 

Author Comment

by:marcelof13
ID: 25370534
bol,
       I hear you.  Sorry I didn't realize the answer "you can't" was valid, I didn't really know how to end the question and also thought that if no answer was given or an answer that could not really help you then it was considered a not answered.  It doesn't matter to me to give points to people who tried to help.

Thanks for your help, its very much appreciated.

   
0
 
LVL 54

Expert Comment

by:b0lsc0tt
ID: 25370722
marcelof13,

Your welcome!  Is the grade a mistake though?  It is the lowest you can give.  It would normally be used when the answer was incomplete and the experts did not participate well.  Do you feel either is the case?  If the grade was just an accident then let me know what you meant to award.  From your comment it seems you agreed with what I said in my previous comment.  The C grade definitely doesn't fit then.  Otherwise please explain.  If you do have a question about grading on EE or what they mean then let me know.

bol
0
 

Author Comment

by:marcelof13
ID: 25371269
Bol,
     I graded again with the way the answers helped me.  I tried to explain in the comment text field.  Sorry i didnt realize that also effected you guy.  Yes you tried and again I thank you for your help.

Marcelo
0
 

Author Closing Comment

by:marcelof13
ID: 31603589
Was the solution complete:  the answer is really, i dont know.

I dont know if the solution is accurate, I would have to try it and really dont have the time to test.
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article discusses four methods for overlaying images in a container on a web page
This article discusses how to implement server side field validation and display customized error messages to the client.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
Suggested Courses

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question