HOW TO SCREEN SCRAPE IFRAME with ASP / ASP.NET / Can't get IFRAME Contents of a non local domain - Thanks

Here's what I've heard so far-

1) I can't get to the HTML contents of an IFRAME that is pointing to a non-local URL such as src=http://www.yahoo.com.

2) I can't use the ActiveXObject("Msxml2.XMLHTTP.4.0") since A) Access is denied and b) a security risk message appears.

so.

Is there a way to screen scrape the contents of an IFRAME using asp.net?

Here's an example Iframe
<body>
<form id="form1">
<IFRAME src="http://www.google.com" name="myIframeId"></IFRAME>
</form>
<body>

Thanks
paygoAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

fritz_the_blankCommented:
Any luck with this?

Function GetHTML(strURL)
      Dim objXMLHTTP, strReturn
      Set objXMLHTTP = Server.CreateObject("MSXML2.ServerXMLHTTP")
      objXMLHTTP.Open "GET", strURL, False
      objXMLHTTP.Send
      strReturn = objXMLHTTP.responseText
      Set objXMLHTTP = Nothing
      GetHTML = strReturn
End Function
0
sciber_dudeCommented:
Two problems:

1) Its a tricky problem if you are trying to scrape a dynamic page that is embedded as an Iframe. coz it would always scrape the original URL i.e. www.google.com and not what has been searched.

2) You need to have MSxml2.XMLHTTP loaded on your server to use it. If not u cant use it.

But here is how u scrape a page.

'create an instance of the XMLHTTP component
Set objXML = Server.CreateObject("MSXML2.ServerXMLHTTP")

'build up the url and store it in strURL variable
strURL = "http://www.google.com"
   
get the strURL
objXML.Open "GET" , strURL , False ,"",""
   
'send the information
objXML.Send
   
'we have no errors
If Err.Number = 0 Then
      'and the url is valid
      If objXML.Status = 200 then
            'store all of the downloaded data in strOpen
            strFileContent = objXML.ResponseText

      Else
            'bad url display a message
            Response.Write "Incorrect URL"
      End if
Else
      'if we do have an error display the description of the error
      Response.Write Err.Description
End If

'clear up
Set objXML = Nothing
0
fritz_the_blankCommented:
So, it sounds like we are on the same page, then?

FtB
0
Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

paygoAuthor Commented:
thx - I'm writing/testing some code now - will keep you posted shortly
0
paygoAuthor Commented:
Hi  I'm not seeing  'ServerXMLHTTP' using intellisense MSXML2.****

do you know where / which version on MSXML*.dll contains this object?
0
fritz_the_blankCommented:
I am not sure, but just for yucks, try my code with a page that you know will work (just about any public page will do) to see if you get a response.
0
paygoAuthor Commented:
Do you have this code in C# Scripting somewhere?

Thanks
0
fritz_the_blankCommented:
I am afraid that I don't. I use this in ASP classic with VBScript.

FtB
0
paygoAuthor Commented:
Thanks Fritz,

I tried this in VB and received the following message.

'Let' and 'Set' assignment statements are no longer supported.

Is there a vb work around?
0
fritz_the_blankCommented:
Did you try this in an ASP page? Just copy and paste the following and save it as testScrape.asp

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<TITLE> New Document </TITLE>
<%
Function GetHTML(strURL)
     Dim objXMLHTTP, strReturn
     Set objXMLHTTP = Server.CreateObject("MSXML2.ServerXMLHTTP")
     objXMLHTTP.Open "GET", strURL, False
     objXMLHTTP.Send
     strReturn = objXMLHTTP.responseText
     Set objXMLHTTP = Nothing
     GetHTML = strReturn
End Function

%>
</HEAD>

<BODY>
<%
=GetHtml("http://www.FairfieldConsulting.com")
%>
</BODY>
</HTML>
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
paygoAuthor Commented:
Thanks Fritz - you answered my question about this.

I going to post a second question regarding the C# scripting.

I am running asp.net and feel confortable with C# from the object side.

Good time for me to learn about C# scripting - can't be too much different than what I already know

Regards

Steve
0
fritz_the_blankCommented:
Best of luck. If I come across a C# example, I'll post it.

FtB
0
sciber_dudeCommented:
Sorry about the double post Ftb,

It took me a while to write the code and then i realized that you had posted earlier.

:) SD
0
fritz_the_blankCommented:
Not a problem--I'll take a look at yours to see what I can learn from it.

FtB
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
ASP

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.