Trevor013097
asked on
pc here are your points
Here are those points that I owe you plus the extra points for the continuation of the previous question.
Question:
to try and put together the last part of my original question, the best match retrieval.
More specifically a ranking system based on relevancy (number of counts) and a way of
searching with case sensitivity on or off and how to use some sort of Boolean logic.
Please collect your points pc and then we can continue the enhancements to the scripts.
No other experts except pc need answer this question please.
Question:
to try and put together the last part of my original question, the best match retrieval.
More specifically a ranking system based on relevancy (number of counts) and a way of
searching with case sensitivity on or off and how to use some sort of Boolean logic.
Please collect your points pc and then we can continue the enhancements to the scripts.
No other experts except pc need answer this question please.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi pc,
Sorry it has taken so long to get back to you but have had a few probolems the last two weeks. My internet gateway machine decided to crash completely and then the backup tapes would not read and it has taken me two weeks to get back to where I was two weeks ago. Oh well such is life.
Have tried your suggested soultions and they work a treat, I now have case insensitivity and the results ranked by Hit count.
Next thing is the boolean logic.
Okay the Alta Vista style boolean is what I am after and no more complicated than that.
Now what we need to do is to create an array of the terms sent to the query_index.pl and then split this array into the individual terms and also the boolean operators. Then we can evaluate from left to right.
so we would need something like:-
@terms = split(/\s+/, $FORM{'terms'});
if ($boolean eq 'AND') {
for each $term(@terms) {
### search the database
similarly for the other operators.
I am unsure how to code this in though, any ideas?
below is what I have so far for the query_index.pl:-
#!/bin/perl5
# This program queries an index created by mkindex for a single word
# and prints all hits to STDOUT.
# Get the input
read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
# Split the name-value pairs
@pairs = split(/&/, $buffer);
foreach $pair (@pairs) {
($name, $value) = split(/=/, $pair);
$value =~ tr/+/ /;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9]
$FORM{$name} = $value;
}
use SDBM_File;
print "Content-type: text/html\n\n";
$indexfile="/docs/www.pcmaritime.co.uk/search/data/index";
$descfile="/docs/www.pcmaritime.co.uk/search/data/descr";
$urlfile="/docs/www.pcmaritime.co.uk/search/data/urls";
# We keep our index in an associative array / dbm-file
dbmopen(%docs,$indexfile,0
dbmopen(%desc,$descfile,06
dbmopen(%urls,$urlfile,064
if( ! (defined( $docs{$FORM{'terms'}} ) ) ){
$FORM{'terms'}=~tr/A-Z/a-z
print "$terms is not in the database!\n";
}
else {
foreach $document (sort sortByHitCount split(/:/,$docs{$FORM{'ter
($urlid,$hits) = split(/,/,$document);
($title,$description) = split( /::/, $desc{$urlid}, 2);
print "Url-ID: $urlid<br>\n";
print "Document: $document<br>\n";
print "Desc: $desc{$urlid}<br>\n";
print "Title: $title<br>\n";
print "$hits found in <a href=\"".$urls{$urlid}."\"
}
}
sub sortByHitCount {
local($ua,$ca)=split(/,/,$
local($ub,$cb)=split(/,/,$
#suppress a warning...
$ua = $ub;
return $cb <=> $ca;
}
dbmclose(%urls);
dbmclose(%docs);
dbmclose(%desc);
You can see the current version working at:-
http://www.pcmaritime.co.uk/search/search2.htm
this provides you with the simple graphical front end.
Thanks in advance
Sorry it has taken so long to get back to you but have had a few probolems the last two weeks. My internet gateway machine decided to crash completely and then the backup tapes would not read and it has taken me two weeks to get back to where I was two weeks ago. Oh well such is life.
Have tried your suggested soultions and they work a treat, I now have case insensitivity and the results ranked by Hit count.
Next thing is the boolean logic.
Okay the Alta Vista style boolean is what I am after and no more complicated than that.
Now what we need to do is to create an array of the terms sent to the query_index.pl and then split this array into the individual terms and also the boolean operators. Then we can evaluate from left to right.
so we would need something like:-
@terms = split(/\s+/, $FORM{'terms'});
if ($boolean eq 'AND') {
for each $term(@terms) {
### search the database
similarly for the other operators.
I am unsure how to code this in though, any ideas?
below is what I have so far for the query_index.pl:-
#!/bin/perl5
# This program queries an index created by mkindex for a single word
# and prints all hits to STDOUT.
# Get the input
read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
# Split the name-value pairs
@pairs = split(/&/, $buffer);
foreach $pair (@pairs) {
($name, $value) = split(/=/, $pair);
$value =~ tr/+/ /;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9]
$FORM{$name} = $value;
}
use SDBM_File;
print "Content-type: text/html\n\n";
$indexfile="/docs/www.pcmaritime.co.uk/search/data/index";
$descfile="/docs/www.pcmaritime.co.uk/search/data/descr";
$urlfile="/docs/www.pcmaritime.co.uk/search/data/urls";
# We keep our index in an associative array / dbm-file
dbmopen(%docs,$indexfile,0
dbmopen(%desc,$descfile,06
dbmopen(%urls,$urlfile,064
if( ! (defined( $docs{$FORM{'terms'}} ) ) ){
$FORM{'terms'}=~tr/A-Z/a-z
print "$terms is not in the database!\n";
}
else {
foreach $document (sort sortByHitCount split(/:/,$docs{$FORM{'ter
($urlid,$hits) = split(/,/,$document);
($title,$description) = split( /::/, $desc{$urlid}, 2);
print "Url-ID: $urlid<br>\n";
print "Document: $document<br>\n";
print "Desc: $desc{$urlid}<br>\n";
print "Title: $title<br>\n";
print "$hits found in <a href=\"".$urls{$urlid}."\"
}
}
sub sortByHitCount {
local($ua,$ca)=split(/,/,$
local($ub,$cb)=split(/,/,$
#suppress a warning...
$ua = $ub;
return $cb <=> $ca;
}
dbmclose(%urls);
dbmclose(%docs);
dbmclose(%desc);
You can see the current version working at:-
http://www.pcmaritime.co.uk/search/search2.htm
this provides you with the simple graphical front end.
Thanks in advance
ASKER
Why my comment got posted twice I do not know. I only clicked submit once, must be a problem at EE somewhere, perhaps it got stuck in the queue.
ASKER
I have reorganised our site and the current working version has now moved to:http://www.pcmaritime.co.uk/leisure/search/search2.htm
Hi Trevor,
it has taken me about two weeks to get started with the rest, and only about two hours to actually do it... looks like my motivation has dropped a little since I received that 'Wizard level' T-shirt... :-)))
Well, here is an alpha-version of the boolean queryindex.pl:
#!/usr/local/bin/perl -w
#
# Version: 1.1
#
# This program queries an index created by mkindex for a single word
# and prints all hits to STDOUT.
#
# Changes since last version (first working version):
#
# Requires three DB files instead of one (see mkindex.pl).
#
# Output is more verbose.
use SDBM_File;
print "Content-type: text/html\n\n";
$indexfile="/home/conrad/t
$descfile ="/home/conrad/tmp/test/de
$urlfile ="/home/conrad/tmp/test/ur
# We keep our index in an associative array / dbm-file
dbmopen(%docs,$indexfile,0
dbmopen(%desc,$descfile,06
dbmopen(%urls,$urlfile,064
$ENV{"QUERY_STRING"} =~ tr/A-Z/a-z/;
@args = split(/\s+/, $ENV{"QUERY_STRING"});
%A = ();
$first = 1;
foreach $word (@args) {
$key = substr($word,0,1);
if( $key eq "+" ) { $func = "land"; $word = substr($word,1); }
elsif( $key eq "-" ) { $func = "remove"; $word = substr($word,1); }
else { $func = "merge"; }
if( $first != 0 ) {
$func = "merge";
$first = 0;
}
%B = finddocs($word);
@B = keys %B;
print "$word: ".($#B+1)."<br>\n";
# print "A = ".k2s(%A)."\n";
# print "B = ".k2s(%B)."\n";
eval "\%A = $func";
# print "$func(A,B) = ".k2s(%A)."\n";
}
foreach $urlid (sort { $A{$b} <=> $A{$a} } keys %A)
{
($title,$description) = split( /::/, $desc{$urlid}, 2 );
print "$A{$urlid} found in <a href=\"".$urls{$urlid}."\"
ption<p>\n";
}
dbmclose(%urls);
dbmclose(%docs);
dbmclose(%desc);
exit 0;
sub finddocs
{
local ($w)=@_;
local (%B);
if( ! defined( $docs{$w} ) ) {
return ();
}
%B = ();
foreach $document (split(/:/,$docs{$w})) {
($urlid,$hits) = split(/,/,$document);
$B{$urlid} = $hits;
}
return %B;
}
sub merge
# Merge two hashes A and B, i. e. for any element (a,c1) in A:
# if there is (a,c2) in B, substitute (a,c1+c2) for (a,c2) in B
# else add (a,c1) to B.
{
local (%C, $k, $v);
%C = %B;
while( ($k,$v) = each %A )
{
if( ! exists $C{$k} ) {
$C{$k} = $v;
} else {
$C{$k} += $v;
}
}
return %C;
}
sub remove
# Remove all elements which appear in B from A
{
local (%C, $k);
%C = %A;
foreach $k (keys %B) {
delete $C{$k};
}
return %C;
}
sub land
# Create a list containing all elements a with (a,c1) in A and (a,c2) in B
# and none else.
{
local (%C, $k, $v);
%C = ();
while( ($k,$v) = each %A ) {
if( exists $B{$k} ) {
$C{$k} = $B{$k} + $v;
}
}
return %C;
}
#sub k2s
#{
#local (%A)=(@_);
#local ($res);
#
# $res = "";
# foreach $key (keys %A) {
# $res .= ", $key";
# }
# return $res;
#}
I have made a few tests with the boolean stuff which seem to indicate the functions work properly. If they don't it would be helpful if you removed the comments from the 'k2s' subroutine, and from the print statements where it is invoked.
Bye,
Peter
it has taken me about two weeks to get started with the rest, and only about two hours to actually do it... looks like my motivation has dropped a little since I received that 'Wizard level' T-shirt... :-)))
Well, here is an alpha-version of the boolean queryindex.pl:
#!/usr/local/bin/perl -w
#
# Version: 1.1
#
# This program queries an index created by mkindex for a single word
# and prints all hits to STDOUT.
#
# Changes since last version (first working version):
#
# Requires three DB files instead of one (see mkindex.pl).
#
# Output is more verbose.
use SDBM_File;
print "Content-type: text/html\n\n";
$indexfile="/home/conrad/t
$descfile ="/home/conrad/tmp/test/de
$urlfile ="/home/conrad/tmp/test/ur
# We keep our index in an associative array / dbm-file
dbmopen(%docs,$indexfile,0
dbmopen(%desc,$descfile,06
dbmopen(%urls,$urlfile,064
$ENV{"QUERY_STRING"} =~ tr/A-Z/a-z/;
@args = split(/\s+/, $ENV{"QUERY_STRING"});
%A = ();
$first = 1;
foreach $word (@args) {
$key = substr($word,0,1);
if( $key eq "+" ) { $func = "land"; $word = substr($word,1); }
elsif( $key eq "-" ) { $func = "remove"; $word = substr($word,1); }
else { $func = "merge"; }
if( $first != 0 ) {
$func = "merge";
$first = 0;
}
%B = finddocs($word);
@B = keys %B;
print "$word: ".($#B+1)."<br>\n";
# print "A = ".k2s(%A)."\n";
# print "B = ".k2s(%B)."\n";
eval "\%A = $func";
# print "$func(A,B) = ".k2s(%A)."\n";
}
foreach $urlid (sort { $A{$b} <=> $A{$a} } keys %A)
{
($title,$description) = split( /::/, $desc{$urlid}, 2 );
print "$A{$urlid} found in <a href=\"".$urls{$urlid}."\"
ption<p>\n";
}
dbmclose(%urls);
dbmclose(%docs);
dbmclose(%desc);
exit 0;
sub finddocs
{
local ($w)=@_;
local (%B);
if( ! defined( $docs{$w} ) ) {
return ();
}
%B = ();
foreach $document (split(/:/,$docs{$w})) {
($urlid,$hits) = split(/,/,$document);
$B{$urlid} = $hits;
}
return %B;
}
sub merge
# Merge two hashes A and B, i. e. for any element (a,c1) in A:
# if there is (a,c2) in B, substitute (a,c1+c2) for (a,c2) in B
# else add (a,c1) to B.
{
local (%C, $k, $v);
%C = %B;
while( ($k,$v) = each %A )
{
if( ! exists $C{$k} ) {
$C{$k} = $v;
} else {
$C{$k} += $v;
}
}
return %C;
}
sub remove
# Remove all elements which appear in B from A
{
local (%C, $k);
%C = %A;
foreach $k (keys %B) {
delete $C{$k};
}
return %C;
}
sub land
# Create a list containing all elements a with (a,c1) in A and (a,c2) in B
# and none else.
{
local (%C, $k, $v);
%C = ();
while( ($k,$v) = each %A ) {
if( exists $B{$k} ) {
$C{$k} = $B{$k} + $v;
}
}
return %C;
}
#sub k2s
#{
#local (%A)=(@_);
#local ($res);
#
# $res = "";
# foreach $key (keys %A) {
# $res .= ", $key";
# }
# return $res;
#}
I have made a few tests with the boolean stuff which seem to indicate the functions work properly. If they don't it would be helpful if you removed the comments from the 'k2s' subroutine, and from the print statements where it is invoked.
Bye,
Peter
Oops, there's a bad word wrap in the foreach $urlid part...
ASKER
Works fine when performing a simple search, ie. just one search term. However I cannot seem to get it to search on multiple terms. What are the boolean operators, I tried + - but with no effect. Two keywords which I know appear on the same page are navmaster and arcs but when searched for together it retrieves a match of 0.
It is on the site if you want to try it.
At the moment it is without a front end so it requires a command line argument. I have tried the following:
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?navmaster
and
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?navmaster+arcs
no wrap should be there.
Any ideas?
It is on the site if you want to try it.
At the moment it is without a front end so it requires a command line argument. I have tried the following:
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?navmaster
and
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?navmaster+arcs
no wrap should be there.
Any ideas?
ASKER
I have removed the comments from the k2 subroutine and also from the additional print statements and have uploaded a new file to our server. There are now two versions up there:
The script with comments in:-
http://www.pcmaritime.co.uk/cgi-bin/new.pl?search-term
and the script without the comments in:
http://www.pcmaritime.co.uk/cgi-bin/new2.pl?search-term
The script with comments in:-
http://www.pcmaritime.co.uk/cgi-bin/new.pl?search-term
and the script without the comments in:
http://www.pcmaritime.co.uk/cgi-bin/new2.pl?search-term
I forgot special characters are URLencoded. I. e. all spaces are replaced with '+' and other special characters with '%xy', where 'xy' is a hex code.
Before the '@args=split...' line insert the following statements:
$ENV{"QUERY_STRING"} =~ s/\+/ /g;
$ENV{"QUERY_STRING"} =~ s/%([A-Fa-f0-9]{2})/pack("
For testing you'd best use a form, so the browser will handle the encoding. If you want to enter the arguments in the URL, replace all spaces in the request with '+' and all pluses with '%2b':
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?search-term1+%2bsearch-term2
Oh, and of course, the name of the INPUT-field will also be included, so add the line
$ENV{"QUERY_STRING"} =~ s/^[^=]*=//;
before the other two. And use this in your URL:
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?terms=search-term1+%2bsearch-term2
Before the '@args=split...' line insert the following statements:
$ENV{"QUERY_STRING"} =~ s/\+/ /g;
$ENV{"QUERY_STRING"} =~ s/%([A-Fa-f0-9]{2})/pack("
For testing you'd best use a form, so the browser will handle the encoding. If you want to enter the arguments in the URL, replace all spaces in the request with '+' and all pluses with '%2b':
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?search-term1+%2bsearch-term2
Oh, and of course, the name of the INPUT-field will also be included, so add the line
$ENV{"QUERY_STRING"} =~ s/^[^=]*=//;
before the other two. And use this in your URL:
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new.pl?terms=search-term1+%2bsearch-term2
ASKER
I am sure I have inserted the line correctly but no joy. I have tried both a form front end version and a command line version.
form:-
http://www.pcmaritime.co.uk/leisure/search/search2.htm
command argument:-
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new_pc.pl?search-term
neither version appear to work. Any ideas?
maybe I should e-mail you my current scripts?
form:-
http://www.pcmaritime.co.uk/leisure/search/search2.htm
command argument:-
http://www.pcmaritime.co.uk/cgi-bin/pcmweb/new_pc.pl?search-term
neither version appear to work. Any ideas?
maybe I should e-mail you my current scripts?
Yes, please send the code per email. I don't see what's going wrong there.
ASKER
Excellent Peter,
Works an absolute treat. It was the form parsing that was causing the problems, not the method as it was correct but the extra unneccessary lines which once removed worked fine.
The boolean works great and now my next job is designing the front end and return HTML. I will e-mail you the URL when it is up and running and may request your help later on as I want to make some refinements with the returned HTML based on a query, such as matched word highlighting. That can be looked at later though.
For now thank you very much for all your help.
Works an absolute treat. It was the form parsing that was causing the problems, not the method as it was correct but the extra unneccessary lines which once removed worked fine.
The boolean works great and now my next job is designing the front end and return HTML. I will e-mail you the URL when it is up and running and may request your help later on as I want to make some refinements with the returned HTML based on a query, such as matched word highlighting. That can be looked at later though.
For now thank you very much for all your help.
Great. Thanks for the points! It's been a pleasure to work with you. :-)
ASKER
Sorry it has taken so long to get back to you but have had a few probolems the last two weeks. My internet gateway machine decided to crash completely and then the backup tapes would not read and it has taken me two weeks to get back to where I was two weeks ago. Oh well such is life.
Have tried your suggested soultions and they work a treat, I now have case insensitivity and the results ranked by Hit count.
Next thing is the boolean logic.
Okay the Alta Vista style boolean is what I am after and no more complicated than that.
Now what we need to do is to create an array of the terms sent to the query_index.pl and then split this array into the individual terms and also the boolean operators. Then we can evaluate from left to right.
so we would need something like:-
@terms = split(/\s+/, $FORM{'terms'});
if ($boolean eq 'AND') {
for each $term(@terms) {
### search the database
similarly for the other operators.
I am unsure how to code this in though, any ideas?
below is what I have so far for the query_index.pl:-
#!/bin/perl5
# This program queries an index created by mkindex for a single word
# and prints all hits to STDOUT.
# Get the input
read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
# Split the name-value pairs
@pairs = split(/&/, $buffer);
foreach $pair (@pairs) {
($name, $value) = split(/=/, $pair);
$value =~ tr/+/ /;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9]
$FORM{$name} = $value;
}
use SDBM_File;
print "Content-type: text/html\n\n";
$indexfile="/docs/www.pcmaritime.co.uk/search/data/index";
$descfile="/docs/www.pcmaritime.co.uk/search/data/descr";
$urlfile="/docs/www.pcmaritime.co.uk/search/data/urls";
# We keep our index in an associative array / dbm-file
dbmopen(%docs,$indexfile,0
dbmopen(%desc,$descfile,06
dbmopen(%urls,$urlfile,064
if( ! (defined( $docs{$FORM{'terms'}} ) ) ){
$FORM{'terms'}=~tr/A-Z/a-z
print "$terms is not in the database!\n";
}
else {
foreach $document (sort sortByHitCount split(/:/,$docs{$FORM{'ter
($urlid,$hits) = split(/,/,$document);
($title,$description) = split( /::/, $desc{$urlid}, 2);
print "Url-ID: $urlid<br>\n";
print "Document: $document<br>\n";
print "Desc: $desc{$urlid}<br>\n";
print "Title: $title<br>\n";
print "$hits found in <a href=\"".$urls{$urlid}."\"
}
}
sub sortByHitCount {
local($ua,$ca)=split(/,/,$
local($ub,$cb)=split(/,/,$
#suppress a warning...
$ua = $ub;
return $cb <=> $ca;
}
dbmclose(%urls);
dbmclose(%docs);
dbmclose(%desc);
You can see the current version working at:-
http://www.pcmaritime.co.uk/search/search2.htm
this provides you with the simple graphical front end.
Thanks in advance