I would appreciate some advice on how to correctly exploit MSHTML from within a C# application (without the use of a WebBrowser control).
Specifically, I am trying to use MSHTML to load a sequence of web pages that will be programmatically scraped. I understand that it is normal to use a Windows Forms or WPF WebBrowser control with MSHTML, but this is not possible in my case because the application I am developing is a class library and (as far as I can tell) the WebBrowser controls cannot operate without a window handle assigned by a hosting form.
Therefore, the approach I have adopted is the one described here:
I have proved that MSHTML is capable of issuing its own requests in this configuration, because I have succeeded in loading a sequence of pages by successive calls to the Navigate() method, but I can't find any way to get it to navigate from within a loaded document.
Any ideas would be very much appreciated.