• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1325
  • Last Modified:

Extract text from webpage


 I have a webpage that I need to extract text from (the url of the page ends in .jsp if it makes a difference?) The text is in a table and I want to be able to extract all the text on the page to an Excel/.txt file so I can work on the file later. I do not want the HTM code of the page, just the text that is displayed. Is this possible?

Many thanks,

3 Solutions
Ashish PatelCommented:
No not straightly. First you will have to get teh HTM code of the page and then parse everything. You cannot get just the text out of that, you will have to write a function to remove all html tags and stuff. Search google for removing the HTML tags from text.
For this you need to understand the difference between static content and dynamic content. A static page does not change across visits, it just shows the same page all the time. A dynamic page has server-side-scripting that is executed everytime you ask for the page. The rendered/output content for you may just be the same HTML, but because it may come from other server components like a backend database or server side objects/components IN RUNTIME, you may not such a file and have its output read. The JSP filetype is one such type used to deploy server side dynamic content which can ultimately be uotputted as html and shown on webpages. Depending on what the server-side JSP is intended to do, I’ll say Yes/No for your question. Say, if that jsp page is supposed to show a account summary in a tabular form, it basically needs a page prior to it whch asked for username and password etc. And the user profile for THAT USER will be shown in this page. If you just take a jsp and try to decipher it, it would not know WHICH USER details to show to you.

So, assuming that you want to read/copy/capture that static content portion of a dynamic page like jsp just for the purpose of having a pre-formatted text that the junk tags etc, you’d have 2 ways.

1. Rename the .jsp file as a .htm file and open in IE. This will show a partly neat file (because it cant get the dynamic content into its placeholders), but should show all the html portios neatly configured as per their tax direction.

2. If you are using a code editor for JSP (which may need a project to be configured with additional resources), you can see a “preview” or “test run” of the jsp in the IDE itself which gives perfect output for your copy/paste. But remember that even that IDE would ask for test inputs that trigger the dynamic content.
Here's a simple sample to give you the idea.  You will need references to:
Microsoft VBScript Regular Expressions 5.5
Microsoft WinHTTP Services 5.1
Option Explicit
Private Sub Command1_Click()
    Dim HTTP As WinHttpRequest
    Set HTTP = New WinHttpRequest
    Dim strBody As String
    'Retrieve the HTML from the page'
    HTTP.Open "open", "http://www.theweathernetwork.com/weather/CAQC0363", False
    strBody = HTTP.ResponseText
    'Eliminate the HTML tags'
    Dim RegEx As RegExp
    Set RegEx = New RegExp
    RegEx.Pattern = "<[^>]*>"
    RegEx.Global = True
    strBody = RegEx.Replace(strBody, "")
    'Save the cleaned text to a file'
    Dim hFile As Integer
    hFile = FreeFile
    Open "C:\pagetext.txt" For Output As #hFile
    Print #hFile, strBody
    Close #hFile
End Sub

Open in new window

wildarmsdaveAuthor Commented:
Thanks guys. There were some good info in the answers so I've decided to split the points. As the web page was a static page, I saved the page in (.xls) format which opened up perfect in Excel and enabled me to filter out the information I needed.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now