?
Solved

Read Data Text from an external web page

Posted on 2004-11-05
13
Medium Priority
?
305 Views
Last Modified: 2008-03-06
Hi, I'm developing a website which transfer data to another website (website B). The website B, process that data and gives them status. Each registry has a reference.

If I want to know the status, I can go into that website (no user validation required) put the reference in a query page and submit, and in the next page, the status shows up. I need to make this automatically, I mean, from my website make a submit of that external query page and in the results page, read the values.

I suppose I can do that with XMLHTTPRequest, buy I'm not sure how to, can someone help me ?

Thanks in advance

Regards
0
Comment
Question by:LoThar_71
  • 6
  • 4
  • 3
13 Comments
 
LVL 5

Expert Comment

by:mk_b
ID: 12512465
You can use this to post/get quite useful.

    Dim strValues, strURL
    Dim xmlhttp

    strURL = "http://www.google.com"
    strValues = "q=test"

   set objHttp = Server.CreateObject("Msxml2.ServerXMLHTTP")
   ' set objHttp = Server.CreateObject("Msxml2.ServerXMLHTTP.4.0")
   ' set objHttp = Server.CreateObject("Microsoft.XMLHTTP")
   objHttp.open "get", strURL, false
   objHttp.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
   objHttp.Send strValues


    If objHttp.Status >= 400 And objHttp.Status <= 599 Then
        Response.Write "Error Occurred : " & objHttp.Status & " - " & objHttp.statusText
    Else
        Response.Write objHttp.ResponseText
    End If

    set objHttp = nothing


../mk
0
 

Author Comment

by:LoThar_71
ID: 12514365
That's fine, but that script will only show the "website B". Taking "google.com" as website B, I will need to make a search, read the results and save 1 specific field value (let's say.. quantity of coincidences) in a variable.

I know there are websites wich reads online data from others (stock exchange, price lists, etc.) with the purpose to offer resumed information. They are doing what I need.
0
 
LVL 5

Expert Comment

by:mk_b
ID: 12514417
Well if you know what you are looking for from website B you can put objHttp.ResponseText into a string and then look for what you want in the string and then do some string manipulation??
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:LoThar_71
ID: 12514463
mk you are missing something. Website B is "google.com". My data is in the search results page. Suppose that I'm looking for "imported cars", I have to submit google with that words and get, from the results page, how many coincidences found it.

About string manipulation... if it's the only way... I will do it.

0
 

Author Comment

by:LoThar_71
ID: 12514524
Let me add some data.

When I execute your code, I do not see the results page, I see the main google page.

The sentence - objHttp.Send strValues -, I suppose, is passing the search parameters, isn't it ? If that it's right, something is not working, because only shows the main google page (not even an error).


0
 
LVL 5

Expert Comment

by:mk_b
ID: 12516063
For some reason google is giving me a 403 forbiddien error here are too example of subitting the for the request? you will need to look at the google.com page and see if there are any other paramiters that need to be passes to seach and maybe run a search on "scraping google". the second example is for internic and will give the domin registration details for a site this one works fine? just play around witht the google im sure you will get it working?


1.
<%

    Dim strValues, strURL
    Dim xmlhttp

    strURL = "http://www.google.com/search?"
    strValues = "hl=en&q=test&q=imported cars&btnG=Google+Search"

   set objHttp = Server.CreateObject("Msxml2.ServerXMLHTTP")
   ' set objHttp = Server.CreateObject("Msxml2.ServerXMLHTTP.4.0")
   ' set objHttp = Server.CreateObject("Microsoft.XMLHTTP")
   objHttp.open "get", strURL, false
   objHttp.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
   objHttp.Send strValues


    If objHttp.Status >= 400 And objHttp.Status <= 599 Then
        Response.Write "Error Occurred : " & objHttp.Status & " - " & objHttp.statusText
    Else
        htpStr = objHttp.ResponseText
    End If

    set objHttp = nothing
%>
<% =htpStr %>


<br><br>2.
<%
Set objHttp = Server.CreateObject("MSXML2.ServerXMLHTTP")
objHttp.open "GET", "http://www-whois.internic.net/cgi/whois?whois_nic=www.internetsite.com&type=domain", False
objHttp.SetRequestHeader "Content-type", "application/x-www-form-urlencoded"
objHttp.send str

if (objHttp.status <> 200 ) then
'--HTTP error handling
elseif (objHttp.responseText = "VERIFIED") then
elseif (objHttp.responseText = "INVALID") then
else
'--error
end if

MyString = objHttp.responseText
%>
<% =MyString %>
0
 

Author Comment

by:LoThar_71
ID: 12522364
MK, none of them work. If I try with google, the best what I can get is the main google page, never searchs what I tell it. And the second, always give me a timeout.

Suppose, the target page, for example the search results of Google, works with post instead get. How could I find out the variables needed ?
0
 
LVL 5

Expert Comment

by:mk_b
ID: 12522437
for me the first one work fine and the google one gives me a forbidden error?

have a look at this site(http://www.tss2000.nl/site_pages/index.php?CV=1) and see the google search on the right side. it is a custome search results from google also have a look around goole's site for how you do that? and then you might be able to use that with the script above to get someting working?

i think google might be checking where the request for a search is coming from and if its not from google it wont alow it???
0
 
LVL 18

Accepted Solution

by:
SquareHead earned 1500 total points
ID: 12523917
This works for me:


1st page, "search_google.html"
2nd page, "scrape_google.asp"


Search:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
      <title>Untitled</title>
</head>

<body>
<form action="scrape_google.asp" method="post" name="frm1" id="frm1">

<input type="text" name="txtSearchPhrase"><input type="submit" value="Go!">


</form>


</body>
</html>



Scrape:

<%@ Language = VBScript %>
<%

Response.Buffer = True
Dim objXMLHTTP, xml

' Create an xmlhttp object:
Set xml = Server.CreateObject("Microsoft.XMLHTTP")
' Or, for version 3.0 of XMLHTTP, use:
' Set xml = Server.CreateObject("MSXML2.ServerXMLHTTP")

' Opens the connection to the remote server.
xml.Open "GET", "http://www.google.com/search?hl=en&q=" & request.form("txtSearchPhrase"), False
   
 ' Actually Sends the request and returns the data:
xml.Send

Response.Write xml.responseText
 
  Set xml = Nothing
%>
0
 

Author Comment

by:LoThar_71
ID: 12528312
MK, the website you mention has that google search because they have an arrangement with Google. Probably they have some publicity or something with them. A couple of month ago I wanted to hire that search service (works into your page too) and if I buy a package of publicity they give me that search for free (haha, nothing is free).

Squarehead, your code works fine.

But, suppose google use the POST method instead GET, Do I have a way to find out the variables name to pass it on submit ?

0
 
LVL 18

Expert Comment

by:SquareHead
ID: 12528367
Well, the GET should do the trick, but if you need to POST, check out the in-depth info in this PAQ:

http://www.experts-exchange.com/Web/Web_Languages/XML/Q_21043023.html

0
 
LVL 18

Expert Comment

by:SquareHead
ID: 12529453
I misunderstood your last question...

If Google POSTs values, you may be able to determine the fields being POSTed by viewing the page source, but you won't see any values in there unless they are prepopulated when the page loads. Depending on what you click to submit the form, you may be able to determine values by looking at the code for the clickable item...
0
 

Author Comment

by:LoThar_71
ID: 12530844
Yes, thanks, that's all I need.

0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I recently decide that I needed a way to make my pages scream on the net.   While searching around how I can accomplish this I stumbled across a great article that stated "minimize the server requests." I got to thinking, hey, I use more than one…
I have helped a lot of people on EE with their coding sources and have enjoyed near about every minute of it. Sometimes it can get a little tedious but it is always a challenge and the one thing that I always say is:   The Exchange of informatio…
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …
In a question here at Experts Exchange (https://www.experts-exchange.com/questions/29062564/Adobe-acrobat-reader-DC.html), a member asked how to create a signature in Adobe Acrobat Reader DC (the free Reader product, not the paid, full Acrobat produ…
Suggested Courses
Course of the Month13 days, 15 hours left to enroll

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question