• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 279
  • Last Modified:

Regular Expression HELP

I am trying to get the Title and Meta data from a site.

To get the data I am using XMLHTTP.

I am then using a Regular Expression to extract the data.

function stripHTMLTags(strPattern, strText)

dim re

set re = new RegExp
      re.pattern = strPattern
      re.ignorecase = true
      re.global = true
           
      Set Matches = re.Execute(strText)
      for each match in matches
           str2 = str2 & Match.value
             response.write server.HTMLEncode(str2)
        next

end function

I can extract the title with:

response.write "<br>" & stripHTMLTags("<title>(.*)?\<\/title>", xmlHTTP.responseText)

This works on some sites, and not on others.

Also Meta data that is proving a little more difficult.  On some pages it works, and others not.

If I use:

response.write "<br>" & stripHTMLTags("<head>(.*)?\<\/head>", xmlHTTP.responseText)

...to extract the entire header block I get nothing.

Also, the following will produce results with some pages, and nothing with others:

response.write "<br>" & stripHTMLTags("<meta(.*)?\>", xmlHTTP.responseText)

I had wanted to grab the <head>...</head> and assign it to a variable so that I don't have to check the entire page code each time I look for data - making the script quicker.  Basically then I could replace the xmlhttp.responsetext with the variable.

Anyway, any idea why the <head> part produces nothing, and the rest work intermittently?

0
russoffl
Asked:
russoffl
1 Solution
 
russofflAuthor Commented:
I am starting to think that line breaks are causing the problem.  Any comments, or solutions to keep reading past these?
0
 
TTomCommented:
Are you sure that the pages you are trying this on have these tags?

People frequently design web pages and omit the <title> and other <meta> tags.

Tom
0
 
russofflAuthor Commented:
Yeah - I used View Source to confirm the tags.  That is where I noticed (in WordPad) that the start of the tag was on one line, and the closing tag on the next - with a crlf.

0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

 
aponcealbuerneCommented:
"." Matches any single character except a newline character.

I think you could use something like:


response.write "<br>" & stripHTMLTags("<title>(.|$*)?\<\/title>", xmlHTTP.responseText)

hope helps
0
 
clockwatcherCommented:
I must be tired because I know this isn't the best way to do this, but it should work:

  <title>((\s|.)*)?</title>


I can't believe that VBScript doesn't have a multiline setting somewhere.  I think that the 5.5 script engine may.  You might try:

function stripHTMLTags(strPattern, strText)

   dim re

   set re = new RegExp
   re.pattern = strPattern
   re.ignorecase = true
   re.global = true
   re.multiline = true
       
   Set Matches = re.Execute(strText)
   for each match in matches
      str2 = str2 & Match.value
      response.write server.HTMLEncode(str2)
   next

end function

With your <title>(.*)?</title> regexp.
0
 
russofflAuthor Commented:
Ver. 5 doesn't support re.multiline = true.

I tried the script without it anyway, and it just hung - as did the 2 dozen other combinations - or it could'nt find a pattern.

Just about ready to give up on this option!
0
 
ASPGuruCommented:
1. you don't need to escape "/" or ">"

2. quantifiers like * are greedy, so keep care... ("<meta(.*)>" will gwt you all bwtween "<meta" and ">" of "</html>")

first:
  re.ignorecase = true
  re.global = true
  re.multiline = true

then:
for title:
"<title>([\s\S]*)</title>"


for head:
"<head>([\s\S]*)</head>"

for what you meant with this: "<meta(.*)?\>":
"<meta([^>]*)>"


the last will produce more than one match if the page has more than one metatags, so handle this...
0
 
russofflAuthor Commented:
Just as you said the ">" was the last in the </html>.

For anyone who may be working on a similar solution, I used <head>([\s\S]*)</head> to grab the <head>...</head>, and then assigned this to a variable.

I then replaced all chr(10) and chr(13) with "", and then replaced all ">" with ">" & chr(10) & chr(13) to force each tag onto a seperate line.  The only problem was <title>
...</title>, so I re-replaced <title> & chr(10) & chr(13) with <title>, forcing it all back onto one line.

Not sure what the impact would be for javascripts and css stuff in the header, but since I am only grabbing meta description, meta keywords, and title I don't have a problem (so far) with the code.

May not be tidy, but it works for my purpose for now.

Just wish people wrote "tidy" HTML.
0
 
ASPGuruCommented:
wouldn't it be an option for just to replace all vbCrLf(=chr(10)&chr(13) btw...) with a space(" ") before and then run normal reg. expressions... to get everything you want...
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: SQL Server Core 2016

This course will introduce you to SQL Server Core 2016, as well as teach you about SSMS, data tools, installation, server configuration, using Management Studio, and writing and executing queries.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now