Go Premium for a chance to win a PS4. Enter to Win


Extract text from webpage

Posted on 2007-12-07
Medium Priority
Last Modified: 2012-02-15

 I have a webpage that I need to extract text from (the url of the page ends in .jsp if it makes a difference?) The text is in a table and I want to be able to extract all the text on the page to an Excel/.txt file so I can work on the file later. I do not want the HTM code of the page, just the text that is displayed. Is this possible?

Many thanks,

Question by:wildarmsdave
LVL 23

Assisted Solution

by:Ashish Patel
Ashish Patel earned 200 total points
ID: 20426532
No not straightly. First you will have to get teh HTM code of the page and then parse everything. You cannot get just the text out of that, you will have to write a function to remove all html tags and stuff. Search google for removing the HTML tags from text.

Accepted Solution

tsmanyam earned 1000 total points
ID: 20426630
For this you need to understand the difference between static content and dynamic content. A static page does not change across visits, it just shows the same page all the time. A dynamic page has server-side-scripting that is executed everytime you ask for the page. The rendered/output content for you may just be the same HTML, but because it may come from other server components like a backend database or server side objects/components IN RUNTIME, you may not such a file and have its output read. The JSP filetype is one such type used to deploy server side dynamic content which can ultimately be uotputted as html and shown on webpages. Depending on what the server-side JSP is intended to do, I’ll say Yes/No for your question. Say, if that jsp page is supposed to show a account summary in a tabular form, it basically needs a page prior to it whch asked for username and password etc. And the user profile for THAT USER will be shown in this page. If you just take a jsp and try to decipher it, it would not know WHICH USER details to show to you.

So, assuming that you want to read/copy/capture that static content portion of a dynamic page like jsp just for the purpose of having a pre-formatted text that the junk tags etc, you’d have 2 ways.

1. Rename the .jsp file as a .htm file and open in IE. This will show a partly neat file (because it cant get the dynamic content into its placeholders), but should show all the html portios neatly configured as per their tax direction.

2. If you are using a code editor for JSP (which may need a project to be configured with additional resources), you can see a “preview” or “test run” of the jsp in the IDE itself which gives perfect output for your copy/paste. But remember that even that IDE would ask for test inputs that trigger the dynamic content.
LVL 38

Assisted Solution

PaulHews earned 800 total points
ID: 20427598
Here's a simple sample to give you the idea.  You will need references to:
Microsoft VBScript Regular Expressions 5.5
Microsoft WinHTTP Services 5.1
Option Explicit
Private Sub Command1_Click()
    Dim HTTP As WinHttpRequest
    Set HTTP = New WinHttpRequest
    Dim strBody As String
    'Retrieve the HTML from the page'
    HTTP.Open "open", "http://www.theweathernetwork.com/weather/CAQC0363", False
    strBody = HTTP.ResponseText
    'Eliminate the HTML tags'
    Dim RegEx As RegExp
    Set RegEx = New RegExp
    RegEx.Pattern = "<[^>]*>"
    RegEx.Global = True
    strBody = RegEx.Replace(strBody, "")
    'Save the cleaned text to a file'
    Dim hFile As Integer
    hFile = FreeFile
    Open "C:\pagetext.txt" For Output As #hFile
    Print #hFile, strBody
    Close #hFile
End Sub

Open in new window


Author Closing Comment

ID: 31413371
Thanks guys. There were some good info in the answers so I've decided to split the points. As the web page was a static page, I saved the page in (.xls) format which opened up perfect in Excel and enabled me to filter out the information I needed.

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Most everyone who has done any programming in VB6 knows that you can do something in code like Debug.Print MyVar and that when the program runs from the IDE, the value of MyVar will be displayed in the Immediate Window. Less well known is Debug.Asse…
I was working on a PowerPoint add-in the other day and a client asked me "can you implement a feature which processes a chart when it's pasted into a slide from another deck?". It got me wondering how to hook into built-in ribbon events in Office.
As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question