Hello - hoping somebody can help me out here.
PROBLEM BACKGROUND: Created a web-browser with intent to take HTML innertext from HTML tables from websites and save the data to SQL Server. I want to use MSHTML so that I have fine-granular control over the HTML elements. User will browse to a site and hit a download button which will with the HTML delivered to the WebBrowser control
PROBLEM: Cannot seem to use mshtml.HTMLDocument correctly. (I've made a reference to MSHTML)
My mshtml.HTMLDocument is declared as follows: Dim doc As New mshtml.HTMLDocument
I can't cast WebBrowser.Document into mshtml.HTMLDocument and createDocumentFromUrl doesn't seem to work. Think that I'm getting syntax wrong, but haven't found anything out there that fits the bill.
Can anybody out there help me instantiate my mshtml.HTMLDocument correctly? This is rather urgent.
Regards,
A.Wooley
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Doobdave,
Actually, I just figured it out. My code looks like this:
Dim objMSHTML As New mshtml.HTMLDocument
Dim objDocument As mshtml.IHTMLDocument2
Dim ips As IPersistStreamInit
ips = DirectCast(objMSHTML, IPersistStreamInit)
ips.InitNew()
objDocument = objMSHTML.createDocumentFr
Do Until objDocument.readyState = "complete"
Application.DoEvents()
Loop
MsgBox(objDocument.body.in
Dim ObjDocument3 As mshtml.IHTMLDocument3
ObjDocument3 = DirectCast(objDocument, mshtml.IHTMLDocument3)
That plus a lot of cryptic interface declarations at the top that I got off the web.
I appreciate your reply. Now I'm onto the next thing - how to make this app crawl over a website. If you have any clue on that, I'm all ears.
Thanks.
A.Wooley
Hello adwooley, I actually am in the very same boat as you. I also just recently figured out that the Interop interfaces need to be used in order to make the call to createDocument FromUrl. I did want to add a little to what you had here and also show how I would go about iteratign through all fo the anchors on a page.
First, I would recommend having these 4 pages from MSDN easily accessible as you are programming:
http://msdn.microsoft.com/
http://msdn.microsoft.com/
http://msdn.microsoft.com/
http://msdn.microsoft.com/
Those are, respectively, the documentation for mshtml.IHTMLDocument2, mshtml.IHTMLDocument3, mshtml.IHTMLDocument4 and mshtml.IHTMLDocument5. Note that mshtml.IHTMLDocument is not really usable when you are trying to access the DOM; it provides no methods for accessing the DOM (see the documentation on it for additional information).
Also, since all of the interfaces hold references to the same object, you don't need to use DirectCast to make reference assignments, you can just assign them directly.
For browsing to other web pages, you want to use the open() method of mshtml.IHTMLDocument2.
open() can be used a variety of ways, so I have created a function that encapsulates it in order to open a new URL as a document:
Public Function doOpenNewWindow(ByVal URL As String, ByVal WindowName As String, Optional ByVal ReplaceHistory As Boolean = False, Optional ByVal Features As String = "") As Object
'objPrimary is an instance of mshtml.IHTMLDocument2.
'Create new window using supplied parameters.
doOpenNewWindow = objPrimary.open(URL, WindowName, Features.GetFeatureText, ReplaceHistory)
End Function
So a call to this function would look something like this:
clsTest.doOpenNewWindow("h
Do While clsTest.readyState <> "complete"
Application.DoEvents()
Loop
The loop afterwards is important because this is an asynchronous call, so it would return before the document has been loaded. Once you have used that code to navigate to the page of your choice, you could use something like the following function to access all of the anchors in the document:
Public Sub DebugPrintAllAnchors()
'This procedure iterates through all of the anchors in the currently loaded document
'and demonstrates how to read and write to that element, printing the results to the
'Immediate window.
'Variable to hold a mshtml.IHTMLElement object.
Dim elem As mshtml.IHTMLElement
'objPrimary is an instance of mshtml.IHTMLDocument2
'Get a new reference ot the element collection representing all anchors (<A></A>).
Dim objAnchors As mshtml.IHTMLElementCollect
'Iterate through every element in the anchors collection.
For i As Integer = 0 To objAnchors.length - 1
'Assign the anchor element to the element placeholder.
elem = objAnchors.item(i)
'Print the title, tag name, ID and class name to the Immediate window.
Debug.Print("Title = " & elem.title & ", TagName = " & elem.tagName & ", ID = " & elem.id & ", Class Name = " & elem.className)
'Also important: .innerText, .innerHTML, .outerText and .outerHtml
'outerHTML is especially important, as it returns the entire HTML text for that anchor.
'If you wanted to assign an attribute to every link, like explicitly commanding every link to open in a new window,
'you could make the following call:
elem.setAttribute("target"
'Read it back, make sure it changed:
Debug.Print(elem.getAttrib
'Read the entire tag text for this control:
Debug.Print(elem.outerHTML
Debug.Print("")
Next
End Sub
Similarly, if you know the Name of an element that you want to access, you can access that element directly from the .all() element collection. For example, if you wanted to log-in to the Experts Exchange web site programmatically, you could do it like this:
Public Sub ShowHowToLogIn(ByVal username As String, ByVal password As String)
'Once the EE site has been loaded (using mshtml.IHTMLDOcument2.open
'function that I created above), this procedure will log you in using the
'username and password that were provided as parameters.
'objPrimary is an instance of mshtml.IHTMLDocument2
'Gets reference to username input box.
Dim elem1 As mshtml.IHTMLElement = objPrimary.all.item("msuLo
'Assigns the username as the value.
elem1.setAttribute("value"
'Gets reference to the password input box.
Dim elem2 As mshtml.IHTMLElement = objPrimary.all.item("msuPa
'Assigns the password as the value.
elem2.setAttribute("value"
'Gets a reference to the image which serves as the "Log In" button.
Dim elem3 As mshtml.IHTMLElement = objPrimary.all.item("msuLo
'Emulates a click of the image.
elem3.click()
End Sub
The "msuLoginName," "msuPassword," and "msuLoginSubmit" strings I obtained by simply looking @ the page source for the EE front page.
So, I hope that gives you a leg up on how to use the mshtml DOM interfaces.
BeauT,
That's what I was looking for! Thank you so much.
I'm currently putting together a recursive class that will do the following:
Take the opening page, find all the "A hrefs" and loop through each calling the creation of mshtml.IHTMLDocument2. On each loop, it searches for more "A href" and loops through those. So we go deeper and deeper into the site, then out to the next recursive loop.
In the process of testing this right now. If you're interested in the results, let me know.
Very interested. I'm still learnign how to use mshtml myself. How are you using the DOM? In other words, to navigate, are you using a web browser control or an InternetExplorer object? Or some other way?
Also, if you have any questions about anything I put up there, definitely ask away. I know it might not be clear, so I will clarify anything you need.
DOM, Shmom... I'm too new at this to understand your Greek, BeauT. Ha ha ha ha
OK - Here's what I am doing to crawl over a site. I'll try to go slow (the only way I CAN go):
1) Get a starting URL
2) Create an HTML object (what I called clsHtmlObj) and pass in the starting URL
Dim newobj As New clsHtmlObj("http://asnet.a
3) The HTML object is a class that I made, which looks likek this:
==========================
Imports System.Runtime.InteropServ
<ComVisible(True), ComImport(), Guid("7FD52380-4E07-101B-A
InterfaceTypeAttribute(Com
Public Interface IPersistStreamInit
' IPersist interface
Sub GetClassID(ByRef pClassID As Guid)
<PreserveSig()> Function IsDirty() As Integer
<PreserveSig()> Function Load(ByVal pstm As UCOMIStream) As Integer
<PreserveSig()> Function Save(ByVal pstm As UCOMIStream, _
ByVal ByValByValfClearDirty As Boolean) As Integer
<PreserveSig()> Function GetSizeMax(<InAttribute(),
MarshalAs(UnmanagedType.U8
<PreserveSig()> Function InitNew() As Integer
End Interface
Public Class clsHtmlObj
Dim mUrl As String = ""
Dim objMSHTML As New mshtml.HTMLDocument
Dim doc2 As mshtml.IHTMLDocument2
Dim doc3 As mshtml.IHTMLDocument3
Dim ips As IPersistStreamInit
Dim db As New clsDB
Public Sub New(ByVal url As String)
If InStr(url, "http://asnet.autoserver.c
If InStr(url, "java") <> 0 Or InStr(url, "#") <> 0 Then
Exit Sub
Else
db.LinkInsert(url)
frmGrabber.ListBox1.Items.
ips = DirectCast(objMSHTML, IPersistStreamInit)
ips.InitNew()
doc2 = objMSHTML.createDocumentFr
Do Until doc2.readyState = "complete"
Application.DoEvents()
Loop
doc3 = DirectCast(doc2, mshtml.IHTMLDocument3)
Me.LoopThroughHrefs(doc3)
End If
End If
End Sub
Private Sub LoopThroughHrefs(ByVal doc3 As mshtml.IHTMLDocument3)
Dim col As mshtml.IHTMLElementCollect
Dim lnk As mshtml.HTMLAnchorElement
col = doc3.getElementsByTagName(
For Each lnk In col
If db.LinkSelect(lnk.href) Then
'GO TO NEXT LINK
Else
Dim obj As New clsHtmlObj(lnk.href)
End If
Next
End Sub
Protected Overrides Sub Finalize()
MyBase.Finalize()
End Sub
End Class
4) You'll notice that I have a line "Dim db As New clsDB"
This is for another class that I have written for writing to a SQL Server DB. If you notice, I try to skip over links that I've already written to the DB. In the DB, I just have a table called tblLinks and only one nvarchar(200) field to store the URLs that I have traversed.
My clsDB class looks like this:
=========================
Imports System.Data
Imports System.Data.SqlClient
Public Class clsDB
Dim conString As String = ""
Dim con As New SqlConnection
Dim ds As New DataSet
Dim da As New SqlDataAdapter
Dim cmd As New SqlCommand
Public Sub New()
con.ConnectionString = "Data Source=localhost;Initial Catalog=SBAuction;Integrat
End Sub
Public Function LinkSelect(ByVal Link As String) As Boolean
With cmd
.Connection = con
.CommandText = "wuspLinkSEL"
.Parameters.Clear()
.Parameters.AddWithValue("
.CommandType = CommandType.StoredProcedur
End With
da.SelectCommand = cmd
ds.Tables.Clear()
da.Fill(ds)
If ds.Tables(0).Rows.Count > 0 Then
Return True
Else
Return False
End If
End Function
Public Sub LinkInsert(ByVal Link As String)
With cmd
.Connection = con
.CommandText = "wuspLinkINS"
.CommandType = CommandType.StoredProcedur
.Parameters.Clear()
.Parameters.AddWithValue("
Try
con.Open()
cmd.ExecuteNonQuery()
Catch ex As Exception
Finally
con.Close()
End Try
End With
End Sub
Protected Overrides Sub Finalize()
MyBase.Finalize()
cmd.Dispose()
con.Dispose()
ds.Dispose()
da.Dispose()
End Sub
End Class
Ummmm.... any of this make sense? It's not pretty, I'll admit....
If you want me to be a little more descriptive, let me know. Always willing to help a friend.
I'm programming here in Japan, by the way. Where are you, BeauT?
I'm in Vienna, VA, near D.C. It' sfinally getting to be autumn, so I love it.
Anyway, yes, it does make sense to me. Now, are you displaying any of the actual web pages that you call? If you are, what are you using to display them? Also, without a reference to a web browser control or InternetExplorer object, is it still performing correctly?
I ask because the application I am writing appears to only function correctly after I assign a reference to the IE class's Document property (also the case if I decide to use a WebBrowser control instead of the IE object).
'Create a new instance of Internet Explorer.
Dim objIE As New SHDocVw.InternetExplorer
'Navigate to the log-in page; note that the browser is not yet shown...
objIE.Navigate2("http://ww
'Wait until page has finished loading.
'Note that we could also poll until objIE.ReadyState <> SHDocVw.tagREADYSTATE.READ
Do While objIE.ReadyState <> SHDocVw.tagREADYSTATE.READ
Application.DoEvents()
Loop
'Do the same Interop song & dance with the same interfaces declared for the createDocumentFromUrl method.
'HGDocument is a class I created that implements the 4 usable IHTMLDocument interfaces and includes
'custom members that help streamline and codify the underlying mshtml objects.
'In this case, HGDocument.IPersistStreamI
'Interop interfaces are located in my project.
Dim objIPS As HGDocument.IPersistStreamI
objIPS.InitNew()
'clsTest is an instance of a HGDocument object.
'clsTest.DocumentReference
clsTest.DocumentReference = objIPS
'*
'* Do some code behind the scenes....
'*
'Now show the previously declared instance of IE.
objIE.Visible = True
Now, it might be that I have to have a reference to the document of an IE instance or a WebBrowser control because some of the processes I have written interact with rendered elements on the page. Now, my biggest stumbling block is this: assigning an instance of IHTMLDocument2 back to IE or a WebBrowser control.
The .Document property is read-only for both objects. The Webbrowser control does have a DocumentStream property that accepts a Stream object, but I'm having trouble getting the IHTMLDocument2 object to convert to a byte array because of it's COM nature. So, either finding a way to convert the IHTMLDocument2 object to a byte array or simply finding a way to use a stream to assign the document is what I am focussing on now. Any thoughts?
Sure thing Alan. :-) Great name, btw. It's the same as my brother's, same spelling and everything.
Also, I've given up on using DocumentStream. Instead, I'm using DocumentText. It works, but, when I load the HTML into the WebBrowser control, I get numerous script errors. Not sure why.
Alternatively (or additionally), the HTML can be assiged directly to an IE instance, using the IPersist interface once again...
'Create a new instance of Internet Explorer.
Dim objIE As New SHDocVw.InternetExplorer
objIE.Navigate2("about:bla
Dim objIPS As HGDocument.IPersistStreamI
objIPS.InitNew()
Dim strTemp As String = clsTest.documentElement.ou
DirectCast(objIE.Document,
Do While objIE.Busy
Application.DoEvents()
Loop
objIE.Visible = True
Also, you must navigate to some page in your IE instance, even if you are not displaying the browser, so that it will create an instance of it's own document before you can write to the document.
One other incidental note: I've experimented with leaving out the .InitNew() method call after creating the IPersistStreamInit interface and, from what I can tell, it appears to initialize the COM object being referred to foruse in .NET. It's just a guess. If anyone wants to add to that.... ;->
Uhhh... looks like you're beyond me. I actually am not displaying the webpage. My project is as follows:
My company plays middle-man in car web-auctions (we are exporters of cars to Russia, UK, Cyprus, Ireland, Singapore... and on and on). Well, they recently jumped on the web-auction bandwagon, so they need to send out listings of available cars on a daily basis to their customers. I joined the company a couple months ago as they needed somebody to program a cost-analysis web-app for cars going to Pakistan. The web-auction project came up last week, and they're only giving me a couple more days!!!! The task is to web-crawl over an auction site, download certain HTML tables into a database, and also download corresponding images of cars and save them to the hard-drive. What you're doing sounds a little more complicated.
What exactly is your project all about? I've been learning some weird things... maybe I can help with something? (although I doubt it).
Alan
PS - still scanning here and there about assigning an instance of IHTMLDocument2 back to IE. ....Can anybody help here?
Alan, you might want to consider ditching mshtml altogether if you don't need to render the page or interact with it at all. It would be much faster, and give you much more control, to use the System.net namespace, specifically the WebClient class. With it, you can download the source from any page (without waiting for page rendering, history management or any other browser related tasks) in a synchronous or asynchronous matter and can also download the images much more quickly. Also, by sticking with the System.net namespace, you can avoid having to mess with Interop and unmanaged code.
This article shows how to get the source from a URL in a single shot:
http://www.developer.com/n
And this related article shows how to use RegEx (regular expressions) to parse out all links and images:
http://www.developer.com/n
Also, if it is not a requirement for you to save the images as actual files, this article shows an easy way to save images to a DB. Since you are saving your data to a DB anyway, maybe this would be more efficient?
http://www.codeguru.com/vb
Also, to turn a URL into an Image object (for saving to the DB or a file), you can declare your WebClient object WithEvents and then trap the DownloadDataCompleted method like this:
Private Sub client_DownloadDataComplet
'If action was not cancelled...
If Not e.Cancelled Then
'client is the name of the WebClient I have declared.
'imgLastImage is an object of type Image that I have declared elsewhere.
'Create stream from returned data.
Dim objStream As New System.IO.MemoryStream(e.R
'Read data into byte buffer that is then interpreted by the Image object.
imgLastImage = Image.FromStream(objStream
Else
'Return Nothing.
imgLastImage = Nothing
End If
'Toggle busy status.
blnBusy = False
End Sub
This would be used in conjunction with an asynchronous call from the WebClient class.
Unless you have restrictions preventing you from using these classes, I recommend them, based on what I understand of your project. if you need further help, let me know.
And my project is actually progressing ok, I just hate not knowing how to do something, like use the DocumentStream object with mshtml objects. ;->
Although, if you (or anyone) know of any way to completely divorce the IHTMLDocument2 interface from an instance of IE or an instance of a WebBrowser control and then assign it when I choose to do so, that would be great. As it is, in order to interact with elements of the DOM, I have to create a document in either a WebBrowser control or an instance of IE and then assign it to my custom document handler. It's not a big deal, but I hate having to create additional objects in memory that may never be directly needed.
Business Accounts
Answer for Membership
by: jaime_olivaresPosted on 2006-09-28 at 22:35:42ID: 17625326
You can do:
Dim browser as new WebBrowser
browser.DocumentText = ""; ' Initializes the browser control
Dim doc as HtmlDocument = browser.Document
Now doc points to a valid htmldocument
For more information have a look to: http://msdn2.microsoft.com