Solved

Creating and Searching a Database in PERL

Posted on 1997-09-30
61
322 Views
Last Modified: 2013-12-25
I wish to create a database and then be able to search it from a web browser.

I have a list of navigational charts which are currently laid out in 4 columns.

1st col = chart CD # (1 to 11)
2nd col = chart #
3rd col = chart scale
4th col = chart description

The fields are comma delimited and each record appears on one line.

All this information is in a text file and I want to scan the text file and create my database.  I need to then be able to search the database for a chart number a chart scale or a chart description and return all matches by record.

So if I search for chart number 30 if will return the full record.

ie. cd, chart num, scale, description

if there is more than one match it will display all matches.

I know this is alot to ask hence the points are high.

0
Comment
Question by:Trevor013097
  • 33
  • 22
  • 6
61 Comments
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
BTW,

This script will be running on a UNIX Apache server using PERL 5.003.

I do not have access to anything other than the standard PERL distribution.  ie. no other modules.

I cannot run PERL scripts directly from the command line so all scripts must be run from the web browser.


0
 
LVL 2

Expert Comment

by:dmethvin
Comment Utility
For databases of a few hundred entries, DBM works pretty well. Here's a solution using DBM. First is a program to take the text files and put it into the DBM databases. I've used the chart number as a primary key for the database, with scale and description as secondary keys that refer back to the chart primary key. I've also assumed that scale and description are unique. If they're not (that is, multiple entries can have the same scale and/or description) you'd need to set up the index entries to handle multiple chart numbers. You could do this easily by appending them separated by commas and then using split.

# Put the text data into the DBM database

dbmopen(%chart, "chart", 0644);
dbmopen(%scale_index, "scale", 0644);
dbmopen(%desc_index, "desc", 0644);

open(CSV,"charts.txt") or die;
while ( <CSV> ) {
  chop;
  my ($cd,$num,$scale,$desc) = split(/,/);
  last if !$desc;
  $chart{$num} = $_;
  $scale_index{$scale} = $num;
  $desc_index{$desc} = $num;
  print;
}

dbmclose(%chart);
dbmclose(%scale_index);
dbmclose(%desc_index);


OK, here's the program to retrieve data out of the database. I just set up this test harness to get data from stdin and try out the indices. They worked for me.


dbmopen(%chart, "chart", 0644);
dbmopen(%scale_index, "scale", 0644);
dbmopen(%desc_index, "desc", 0644);

Print "Ask for chart, scale, or desc\n";

while (<>) {
  my ($which, $data) = split;
  if ( $which eq "chart" ) {
    print "chart{$data} = $chart{$data}\n";
  } elsif ( $which eq "scale" ) {
    my $chart = $scale_index{$data};
    print "scale{$data} = chart $chart\n";
    print "chart{$data} = $chart{$chart}\n";
  } elsif ( $which eq "desc" ) {
    my $chart = $desc_index{$data};
    print "scale{$data} = chart $chart\n";
    print "chart{$data} = $chart{$chart}\n";
  } else {
    print "huh?\n";
  }
}

dbmclose(%chart);
dbmclose(%scale_index);
dbmclose(%desc_index);


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
I'm in the process of trying to implement your solution but having a few troubles but think they are mine.

In the meantime I need the scripts to be accessible via a web page as I will not be accessing them from a command line so the ability to accept data from a form would be required.

Currently your script was throwing up problems with:-

my ($cd,$num,$scale,$desc) = split(/,/);

and also occurences of my in the searching.

simply removing the my allowed the DB's to be constructed but unable to query them at present.

Also the scale and description of the chart is not unique so additional indexes would be required.  Unfortunately my PERL is not as good as my JavaScript and it would take me alot of trial and error to get this implemented so I would appreciate a modified script to reflect changes.

I have a present got the following for my search script, within which I have included some crude Form parsing to get the values search-term and search-field.

Have modified DB file openers slighlty.

#!/bin/perl
# Get the input
   read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
# Split the name-value pairs
   @pairs = split(/&/, $buffer);
   foreach $pair (@pairs) {
      ($name, $value) = split(/=/, $pair);
      $value =~ tr/+/ /;
      $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
      $FORM{$name} = $value;
   }

print "Content-type: text/html\n\n";

$chartfile="/docs/www.pcmaritime.co.uk/charts/arcs/data/chart";
$scalefile="/docs/www.pcmaritime.co.uk/charts/arcs/data/scale";
$descfile ="/docs/www.pcmaritime.co.uk/charts/arcs/data/desc";

# We keep our index in an associative array / dbm-file

dbmopen(%chart,$chartfile, 0644) || die("Can't open $chartfile\n");
dbmopen(%scale_index,$scalefile,0644) || die("Can't open $scalefile\n");
dbmopen(%desc_index,$descfile,0644) || die("Can't open $descfile\n");

print "Ask for chart, scale, or desc\n";

$FORM{'terms'} =~ tr/A-Z/a-z/;

$data = $FORM{'terms'};
$which = $FORM{'which'};

while (<>) {
      ($which, $data) = split;
      if ( $which eq "chart" ) {
      print "chart{$data} = $chart{$data}\n";
      } elsif ( $which eq "scale" ) {
      $chart = $scale_index{$data};
      print "scale{$data} = chart $chart\n";
      print "chart{$data} = $chart{$chart}\n";
      } elsif ( $which eq "desc" ) {
      $chart = $desc_index{$data};
      print "scale{$data} = chart $chart\n";
      print "chart{$data} = $chart{$chart}\n";
      } else {
      print "huh?\n";
      }
 }

dbmclose(%chart);
dbmclose(%scale_index);
dbmclose(%desc_index);

My current DB builder aswell:-

#!/bin/perl

print "Content-type: text/html\n\n";

$chartfile="/docs/www.pcmaritime.co.uk/charts/arcs/data/chart";
$scalefile="/docs/www.pcmaritime.co.uk/charts/arcs/data/scale";
$descfile ="/docs/www.pcmaritime.co.uk/charts/arcs/data/desc";

# We keep our index in an associative array / dbm-file

dbmopen(%chart,$chartfile, 0644) || die("Can't open $chartfile\n");
dbmopen(%scale_index,$scalefile,0644) || die("Can't open $scalefile\n");
dbmopen(%desc_index,$descfile,0644) || die("Can't open $descfile\n");

open(CSV,"charts.txt") || die("Can't open charts.txt\n");
    print "Scanning Raw ARCS Chart File...<br>";

      while ( <CSV> ) {
            chop;
            ($cd,$num,$scale,$desc) = split(/,/);
            last if !$desc;
            $chart{$num} = $_;
            $scale_index{$scale} = $num;
            $desc_index{$desc} = $num;
            print;
      }

dbmclose(%chart);
dbmclose(%scale_index);
dbmclose(%desc_index);

0
 
LVL 2

Expert Comment

by:dmethvin
Comment Utility
Darn, I'd forgotten about the need to do it all through the browser. I'll fix it up with a CGI test harness and do the non-unique indices.  In the meantime, check your version of Perl; the "my" variables have been around in every version of Perl that I know of.

0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
"my" was introduced in Perl5, if you have Perl4, I'd recommend upgrading.
(This will also make it easier for dmethvin to use all the nice CGI modules.)
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
I am using PERL 5.  If you look at my original question ozo you will see that I stated that I am using PERL 5.003.  Even so my server had errors when running the script.  As also stated in my original question I have not got access to any CGI modules other than those which come with a standard PERL distribution.

What does "my" do anyway? I haven't come across it before.



0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
my error guys, sorry.

I do have PERL 5.003 but I am using #!/bin/perl which on our server defaults to PERL 4 to run 5 I need to have #!/bin/perl5, that'll do the trick.  Sorry for the mistake.


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
One other point and that is there are currently 2,300 records.  Hope that won't affect the DB.

Having created the DB and included all 2,300 charts in it I have 3 files.  1 small file and the other two (chart and scale are 300k and 220k).  Description is only about 30k but I would have thought been the largest with all the descriptions held there, or maybe that will occcur when they are all unique.

Just a few thoughts.


0
 
LVL 2

Expert Comment

by:dmethvin
Comment Utility
OK, here you go. First, define the $dbdir to where you're keeping the files. Then try the following queries from your web browser:

db.pl?cmd=import&data=imeanit
db.pl?cmd=dump&data=dump
db.pl?cmd=chart&data=CHARTNUMBER
db.pl?cmd=desc&data=DESCRIPTION
db.pl?cmd=scale&data=SCALENUMBER

If you import twice, the second set of data will be appended to the first.

====== cut here =======

#!perl

# ------DEBUG-------
my $DEBUG;
BEGIN {
 $DEBUG = 0;
 if ( $DEBUG ) {
    open(STDERR,">&STDOUT");
    print "Content-type: text/plain\n\n-- DEBUG LEVEL $DEBUG --\n";
  }
}
# ------------------

use strict;      # no funny stuff
$^W = 1;      # look for strangeness

my $dbdir = "C:/windows/desktop/experts";

print "Content-type: text/html\n\n";
print "<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>\n";

# parse incoming query
my %form = ();
my $buffer = $ENV{QUERY_STRING} || '';
foreach ( split(/&/, $buffer) ) {
  my ($name, $value) = split(/=/);
  # Remove URL encoding and HTML comments
  $value =~ tr/+/ /;
  $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
  $value =~ s/<!--.*?-->//gs;
  $form{$name} = $value;
}

  # open the databases
  my (%chart, %scale_index, %desc_index);
  dbmopen(%chart, "$dbdir/chart", 0644);
  dbmopen(%scale_index, "$dbdir/scale", 0644);
  dbmopen(%desc_index, "$dbdir/desc", 0644);

  # query like http://wherever.com/cgi-bin/db.pl?cmd=chart&data=14
  my $cmd = $form{cmd} or error("missing cmd in query");
  my $data = $form{data} or error("missing data in query");

  if ( $cmd eq "import" and $data eq "imeanit" ) {
    open(CSV,"$dbdir/charts.txt")
      or error("cannot find charts.txt"), exit(1);
    while ( <CSV> ) {
      chop;
      my ($cd,$num,$scale,$desc) = split(/,/);
      last if !$desc;
      $chart{$num} = $_;
      if ( !defined($scale_index{$scale}) ) {
        $scale_index{$scale} = $num;
      } else{
        $scale_index{$scale} .= ",$num";
      }
      if ( !defined($desc_index{$desc}) ) {
        $desc_index{$desc} = $num;
      } else{
        $desc_index{$desc} .= ",$num";
      }
      print "$_<BR>";
    }
    close(CSV);
  } elsif ( $cmd eq "dump" ) {
    my ($chart,$scale,$desc,$data);
    print "<P>By chart number (chart = data):<BR>\n";
    while ( ($chart,$data) = each(%chart) ) {
       print "$chart = $data<BR>\n";
    }
    print "<P>By scale index (scale = chart numbers):<BR>\n";
    while ( ($scale, $chart) = each(%scale_index) ) {
      print "$scale = $chart<BR>\n";
    }
    print "<P>By desc index (desc = chart numbers):<BR>\n";
    while ( ($desc, $chart) = each(%desc_index) ) {
      print "$desc = $chart<BR>\n";
    }
  } elsif ( $cmd eq "chart" ) {
    print "chart{$data} = $chart{$data}<BR>\n";
  } elsif ( $cmd eq "scale" ) {
    my $charts = $scale_index{$data};
    print "scale{$data} = charts $charts<BR>\n";
    my $chart;
    foreach $chart ( split(/,/,$charts) ) {
      print "chart{$chart} = $chart{$chart}<BR>\n";
    }
  } elsif ( $cmd eq "desc" ) {
    my $charts = $desc_index{$data};
    print "desc{$data} = charts $charts<BR>\n";
    my $chart;
    foreach $chart ( split(/,/,$charts) ) {
      print "chart{$chart} = $chart{$chart}<BR>\n";
    }
  } else {
    error("unknown key type, $cmd");
  }

  dbmclose(%chart);
  dbmclose(%scale_index);
  dbmclose(%desc_index);

print "</BODY></HTML>";


sub error {
  my $msg = shift;
  print "<H1>Error: $msg</H1>";
  '';
}

1;
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay, we are making some progress but I'm getting some weirderrors when actually searching for charts numbers, descriptions or scales.

The import and the entire database dump wors okay but the actual search requests give these errors (the xx- prefix to the perl script name is for debugging):-

Chart Number (xx-arcs_query.pl?cmd=chart&data=30)

Output of script follows:
=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'
argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 79.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 79.
Content-type: text/html

<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
chart{} = <BR>
</BODY></HTML>


Description (xx-arcs_query.pl?cmd=desc&data=plymouth)

Output of script follows:
=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'
argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 90.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 92.
Content-type: text/html

<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
desc{plymouth} = charts <BR>
</BODY></HTML>


Scale (xx-arcs_query.pl?cmd=scale&data=50000)

Output of script follows:
=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'
argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 83.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 85.
Content-type: text/html

<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
scale{50000} = charts <BR>
</BODY></HTML>

From what I can see it is falling over on the string $charts which I assume is the return string of the chart which has been found.

Any ideas?

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay,

I think the problem lies in the declaration of the variables $chart at lines 82 and 89 but I can't understand why there is an error at line 79, as no variables are declared simply the input confirmed.

The variables are declared using the my Expression which (having done some research) is for declaring variables to be local within the enclosing block or subroutine but the declaration here appears to be outside the enclosing block meaning the when called the variable is unintialised (as in the debug).

Any thoughts or am I barking up the wrong tree?


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
There was a bug in 5.003 with scope of my.  It looks like the
    } elsif ( $cmd eq "dump" ) {
        my ($chart,$scale,$desc,$data);
is masking the $data in
    } elsif ( $cmd eq "chart" ) {
        print "chart{$data} = $chart{$data}<BR>\n";
    }
I'd suggest either upgrading to 5.004, or using different variable names in $cmd eq "dump", or using local instead of my
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
ozo I think you are on the right path here.

I commented out the dump section and then tried doing a search on chart number = 30 and it returned the correct entry.  However I then tried a scale and description search and both failed on the same lines as before.

I assume therefore that the $data from charts is now masking the next section.  So I will try changing my to local and see what effect that has.


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay I have renamed the variable $data under the code block "dump" as $data1 and this has enabled the script to proceed to the next stage but then it fails when searching for a description.

I have rearranged the construction of the database as I had mistakenly got the scale and description columns the wrong way round so here is my current listing:-

#!/bin/perl5

                        # ------DEBUG-------
                        my $DEBUG;
                        BEGIN {
                         $DEBUG = 0;
                         if ( $DEBUG ) {
                            open(STDERR,">&STDOUT");
                            print "Content-type: text/plain\n\n-- DEBUG LEVEL $DEBUG --\n";
                          }
                        }
                        # ------------------

                        use strict; # no funny stuff
                        $^W = 1; # look for strangeness

                        my $dbdir = "/docs/www.pcmaritime.co.uk/charts/arcs/data/";

                        print "Content-type: text/html\n\n";
                        print "<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>\n";

                        # parse incoming query
                        my %form = ();
                        my $buffer = $ENV{QUERY_STRING} || '';
                        foreach ( split(/&/, $buffer) ) {
                          my ($name, $value) = split(/=/);
                          # Remove URL encoding and HTML comments
                          $value =~ tr/+/ /;
                          $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
                          $value =~ s/<!--.*?-->//gs;
                          $form{$name} = $value;
                        }

                          # open the databases
                          my (%chart, %scale_index, %desc_index);
                          dbmopen(%chart, "$dbdir/chart", 0644);
                          dbmopen(%scale_index, "$dbdir/scale", 0644);
                          dbmopen(%desc_index, "$dbdir/desc", 0644);

                          # query like http://wherever.com/cgi-bin/db.pl?cmd=chart&data=14
                          my $cmd = $form{cmd} or error("missing cmd in query");
                          my $data = $form{data} or error("missing data in query");

                          if ( $cmd eq "import" and $data eq "imeanit" ) {
                            open(CSV,"$dbdir/charts.txt")
                            or error("cannot find charts.txt"), exit(1);
                            while ( <CSV> ) {
                            chop;
                            my ($cd,$num,$desc,$scale) = split(/,/);
                            last if !$scale;
                            $chart{$num} = $_;
                            if ( !defined($desc_index{$desc}) ) {
                            $desc_index{$desc} = $num;
                            } else{
                            $desc_index{$desc} .= ",$num";
                            }
                            if ( !defined($scale_index{$scale}) ) {
                            $scale_index{$scale} = $num;
                            } else{
                            $scale_index{$scale} .= ",$num";
                            }
                            print "$_<BR>";
                            }
                            close(CSV);
                          } elsif ( $cmd eq "dump" ) {
                            my ($chart,$desc,$scale,$data1);
                            print "<P>By chart number (chart = data):<BR>\n";
                            while ( ($chart,$data1) = each(%chart) ) {
                            print "$chart = $data1<BR>\n";
                            }
                            print "<P>By desc index (desc = chart numbers):<BR>\n";
                            while ( ($desc, $chart) = each(%desc_index) ) {
                            print "$desc = $chart<BR>\n";
                            }
                            print "<P>By scale index (scale = chart numbers):<BR>\n";
                            while ( ($scale, $chart) = each(%scale_index) ) {
                            print "$scale = $chart<BR>\n";
                            }
                          } elsif ( $cmd eq "chart" ) {
                            print "chart{$data} = $chart{$data}<BR>\n";
                          } elsif ( $cmd eq "desc" ) {
                            my $charts = $desc_index{$data};
                            print "desc{$data} = charts $charts<BR>\n";
                            my $chart;
                            foreach $chart ( split(/,/,$charts) ) {
                            print "chart{$chart} = $chart{$chart}<BR>\n";
                            }
                          } elsif ( $cmd eq "scale" ) {
                            my $charts = $scale_index{$data};
                            print "scale{$data} = charts $charts<BR>\n";
                            my $chart;
                            foreach $chart ( split(/,/,$charts) ) {
                            print "chart{$chart} = $chart{$chart}<BR>\n";
                            }
                          } else {
                            error("unknown key type, $cmd");
                          }

                          dbmclose(%chart);
                          dbmclose(%scale_index);
                          dbmclose(%desc_index);

                        print "</BODY></HTML>";


                        sub error {
                          my $msg = shift;
                          print "<H1>Error: $msg</H1>";
                          '';
                        }

                        1;

So I can now dump the entire database to screen successfully and all records match correctly.

I can search by chart number.

When I search by chart scale however, I get only 1 record returned when there should be more.  For instance searching on scale 12500 using:-

http://www.pcmaritime.co.uk/cgi-bin/pcmweb/arcs_query.pl?cmd=scale&data=12500

returns only 1 chart (3194) but chart 30 also has this scale (plus many others).  Analysing the entire dump and when you get down to the scale eq chart section (at the bottom) you notice that each scale has only one chart against it, as if it was still a unique index (which it is not, each scale is on a one to many relationship).

Also some charts (not all) in the database are stored with a comma prefix which I would have thought (as this is a split delimiter) would be removed and not stored.

So, the masking is still occuring and a few inconsistences are appearing.

Thanks in advance for you help.

ozo, if you have any more ideas, then post them and if they help (like the last one did, I'll either:-

a) split the points
b) ask a dummy question for you to answer to pick up some points.

Cheers,

    Trevor
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
BTW,

couldn't get local to work when I subst it for my, any reason or is it me being stupid.

0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Hmm, $scale_index{$data} is a unique index, but its value should be
a comma separated list of chart numbers, as built by
if ( !defined($scale_index{$scale}) ) {
    $scale_index{$scale} = $num;
} else{
    $scale_index{$scale} .= ",$num";
}

I might suspect that $scale or $num may not be being read correctly from
$dbdir/charts.txt
Can you verify the lines from there which would correspond to
$scale_index{12500}
?
Do you have any entrys with $scale=0, which would terminate the
  while ( <CSV> ) {
loop at the
  last if !$scale;
statement?

(BTW, What erors do you get trying to substitute local for my?)
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
There are 4 records with scale equal to 12500 in the first 8 lines, but something I did notice is that the record it is storing is the last occurence of scale 12500.

So out of the first 8 lines the scale 12500 appears on lines 3,5 and 8.

There are no entries with a scale equal to 0.


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
When I substitute my with local and I try and do a search on chart 30 (which incidentally does work with my at the moment) I get the following errors:-


Output of script follows:
=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'
argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Global symbol "charts" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 82.
Global symbol "charts" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 83.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 84.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 85.
Global symbol "charts" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 85.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 86.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 86.
Global symbol "charts" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 89.
Global symbol "charts" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 90.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 91.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 92.
Global symbol "charts" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 92.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 93.
Global symbol "chart" requires explicit package name at /cgi-bin/pcmweb/xx-arcs_query.pl line 93.
Execution of /cgi-bin/pcmweb/xx-arcs_query.pl aborted due to compilation errors.

and as far as I can see they all relate to the variables not having been declared which is strange becuase local should declare them.


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Just a minor query,

why does the script end with 1;

I thought that was only when writing modules.


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
 no strict 'vars';
will turn off those errors.
(use strict 'vars'; is probably a good idea in general,
but just for the purposes of tracking down the current problem,
it may be usefull to turn it off temporarily to allow experimenting)
0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Do you have the same problem with $desc_index{$desc} only getting
the last entry like you do with $scale_index{12500}
0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Ok, I can't reproduce the problem with $scale_index{12500}
Are you sure there are no trailing blanks in charts.txt?
Does cmd=dump show any sign of those other records?
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
At present I cannot do a search on Description as it gives me a server error as mentioned previously.

If I do an entire dump of the database then I can clearly see many records containing the scale 12500.


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Can you see them in the
<P>By scale index (scale = chart numbers):<BR>
section?
Are you sure some of them aren't scale="12500 ", rather than scale="12500"?
0
 
LVL 2

Expert Comment

by:dmethvin
Comment Utility
Sorry, I've been out of the office.

I agree with ozo, the most likely problem is that there are some spaces between the commas in your input data. I hadn't designed the code to be bulletproof, but just a starting point for a full implementation that included an HTML form as a front end. It's hard to design for some of these cases without seeing all the input data, for example.

If you want to trim spaces off the input fields, change

my ($cd,$num,$scale,$desc) = split(/,/);

to

my ($cd,$num,$scale,$desc) = split(/\s*,\s*/);

>Also some charts (not all) in the database are stored
>with a comma prefix which I would have thought
>(as this is a split delimiter) would be removed and
>not stored.

It sounds like the input data may not be squeaky clean.
Are there any of them that have missing fields for the
CD or chart number, for example? Right after that new
split I mentioned above, you could put in a bunch of
tests for validity, such as

$num > 0 and $num < 958 or warn("bad chart num, $num");
$cd > 0 and $cd < 49 or warn("bad cd number, $cd");
$scale > 0 and $scale < 3.13159 or warn("bad scale, $scale");



0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay,

I have added in the improved split which has now meant that the database is constructed okay from the part of the descriptions. Before I was seeing descriptions like

12500 = ,4324

which was obviously wrong, now they are all okay.

But....the scales are still wrong as they show only one record per scale and some of the scales are prefixed with a comma, as the descriptions used to be.

I have looked through the text again and again but cannot find any errors.

If you want to see the text for yourself then it is at:-

http://www.pcmaritime.co.uk/charts/arcs/data/charts.txt


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
It works when I run it on http://www.pcmaritime.co.uk/charts/arcs/data/charts.txt

Maybe you should try
dbmopen(%scale_index, "$dbdir/scale", 0644) or error("scale $!")
;

and removing $dbdir/chart, $dbdir/scale, and $dbdir/desc,
then running cmd=import&data=imeanit again?
Also, try it on a smaller part of charts.txt, and see if you
can isolate the error.
0
 
LVL 2

Expert Comment

by:dmethvin
Comment Utility
It seems to work for me as well using the data at your URL. I'd do what ozo suggests, remove the existing database and re-import. You might also do a diff between what's in the answer here and what you currently have, to see if you changed something else.
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Before I start messing around and changing more lines, could one of you please post your current version of the script to make sure that I have a working version from your side.

I have re-imported the charts.txt and still have problems.

Goto:-

http://www.pcmaritime.co.uk/cgi-bin/pcmweb/arcs_query.pl?cmd=dump&data=dump

When you get down to the scale = chart part you will see that all the scales have only one chart and no scale appears twice or more and some charts have a comma prefix.

If I could compare against one of your working versions it ight help me.


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Well, I used the most recent code you posted, in the comment starting
Okay I have renamed the variable $data under the code block "dump" as $data1 and this

But the symptoms look almost as if, where the code above says
       $scale_index{$scale} .= ",$num";
you instead have
       $scale_index{$scale} = ",$num";

Before we assume there's some problem with the dbmopen
which keeps .= from working, and to resort to something wierd like
       $tmp = $scale_index{$scale};
       $tmp .= ",$num";
       $scale_index{$scale} = $tmp

I'd like to try adding some print statements like
       print "before='$scale_index{$scale}'<br>\n";
       $scale_index{$scale} .= ",$num";
       print "after='$scale_index{$scale}'<br>\n";

And then try importing a charts.txt containing only:
RC1,26,Harbours on the South Coast of Devon,12500
RC1,28,Salcombe Harbour,12500

0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay, I have added the print statements s you suggested ozo, and I have imported the database again but with only the two lines you suggested.  The printout I get on screen looks wrong to me as the second chart is truncated but the original text file is correct.

Here;s the output:-

RC1,26,Harbours on the South Coast of Devon,12500
before='26'
after=',26'
RC1,28,Salcombe Harbour,1250
before='28'
after=',28'

The second chart is missing an extra zero on the scale.

The text file is at the same location as before and is still called "charts.txt".  The old or original text file has been renamed "charts_orig.txt"



0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
I think I got the above by putting the prints in the wrong place.  I added them here:-

if ( !defined($scale_index{$scale}) ) {
$scale_index{$scale} = $num;
} else{
$scale_index{$scale} .= ",$num";
}
print "$_<BR>";
 print "before='$scale_index{$scale}'<br>\n";
 $scale_index{$scale} .= ",$num";
 print "after='$scale_index{$scale}'<br>\n";
}
close(CSV);


but when I put them in place of the current line as I assume you meant I get justthe charts displayed and no print statements.  The code looks like this:-


if ( !defined($scale_index{$scale}) ) {
$scale_index{$scale} = $num;
} else{
 print "before='$scale_index{$scale}'<br>\n";
 $scale_index{$scale} .= ",$num";
 print "after='$scale_index{$scale}'<br>\n";
}
print "$_<BR>";
}
close(CSV);

The result I get is this:-

RC1,26,Harbours on the South Coast of Devon,12500
RC1,28,Salcombe Harbour,1250

Still same problem though, the last zero on the scale is missing.



0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Yes, I did mean for the
  print "before='$scale_index{$scale}'<br>\n";
  $scale_index{$scale} .= ",$num";
  print "after='$scale_index{$scale}'<br>\n";
to replace the current
  $scale_index{$scale} .= ",$num";

the missing 0 on the last scale is probably because there is no
end of line charter termnating the last line in charts.txt
Replacing
    while( <CSV> ){
        chop;
with
    while( <CSV> ){
        chomp;
should guard against that problem,
and allow us to get to the print statements

On the other hand, the other print statements in the wrong place
do seem to be telling us that the .= is not working on $scale_index{$scale}
so maybe you do need to try some kludge like
 $tmp = "$scale_index{$scale},$num";
 $scale_index{$scale} = $tmp;

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay ozo, I have changed the chop line to chomp (BTW what does that do, I haven't come across chomp before).

When I import the Database I now get the following print lines:-

RC1,26,Harbours on the South Coast of Devon,12500
before='26'
after=',28'
RC1,28,Salcombe Harbour,12500


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
With the new print lines in place I tried to do a complete Database import and I got the following results, which seemed a bit odd to me.

RC1,18,Falmouth Inner Harbour,5000
RC1,20,Ile d'Ouessant to Pointe de la Coubre,500000
RC1,26,Harbours on the South Coast of Devon,12500
before='26'
after=',28'
RC1,28,Salcombe Harbour,12500
before=',28'
after=',30'
RC1,30,Plymouth Sound and Approaches,12500
RC1,31,Harbours on the South Coast of Cornwall,6250
RC1,60,Alderney and the Casquets,25000
before=',30'
after=',74'
RC1,74,Puerto de Bilbao,12500
RC1,83,Ports on the South Coasts of Portugal and Spain,60000
RC1,85,Rio Guadalquivir,40000
before='60'
after=',86'
RC1,86,Bahia de Cadiz,25000


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Hi ozo,

I tried your suggestion:-

$tmp = "$scale_index{$scale},$num";
$scale_index{$scale} = $tmp;

but all I got was a server error, so I'll have a closer look this morning and run it through the debugger and try again.

Any thoughts at your end?

dmethvin, are you still there or is it just ozo and I?

0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
What error?
Maybe
my $tmp = "$scale_index{$scale},$num";
$scale_index{$scale} = $tmp;
?

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Morning ozo,

I have added the new line and the database imports perfectly now, the entire DB dump has multiple charts for each scale index.  I can search by chart number successfully and by chart scale but I get aa server error when I search for chart descriptions.

When I run the debugger this is the output:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 85.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 87.
Content-type: text/html<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
desc{Plymouth} = charts <BR></BODY></HTML>

Any ideas?

This is the piece of code it is stumbling at:-

} elsif ( $cmd eq "desc" ) {
my $charts = $desc_index{$data};
print "desc{$data} = charts $charts<BR>\n";  <<<<<<<<< line 85
my $chart;
foreach $chart ( split(/,/,$charts) ) {      <<<<<<<<< line 87
print "chart{$chart} = $chart{$chart}<BR>\n";
}
}

BTW I will be throwing some points your way, thats for sure, about 300, or 400 if you catch me at a good time, and a definate 'A' grade.  Not quite sure where dmethvin has got to mind you.


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
ozo,

you can see the almost working script at:-

www.pcmaritime.co.uk/cgi-bin/pcmweb/arcs_query.pl?cmd=dump&data=dump

the debug output for the description can be found at:-

www.pcmaritime.co.uk/cgi-bin/pcmweb/xx-arcs_query.pl?cmd=desc&data=plymouth

0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
I don't have a lot of time to look at it right now,
but if I were to take a wild guess, I might try using a different
name than $chart in there, just in case it's being confused by
the other my $chart in the earlier block.
Wait, it looks more like $charts is the problem.  (maybe both?)
If you're in the mood to experiment, I might also try seeing if
an extra layer of { } will help to isolate the various my variables.
(or even a dummy statement just before the first my in a block)

BTW, what happens if you query with something like cmd=chart&data=no+such+chart(or_scale_or_desc)?

I'll probably check back in later when I have more time to study it in detail

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
If I search for chart 1 (which does not exist), I get :-

chart{1} =

If I search for scale 1 (which does not exist) or any other scale which does not exist, 2000000 for instance, I get a server error and the debig of gives the following output:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 92.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 94.
Content-type: text/html<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
scale{1} = charts <BR></BODY></HTML>

Fails on the same variables (I presume) as the desc but in the scale code block.

If I search for a description which does not exist, like londonwater I get a server error as when I search for a valid desription:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 85.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 87.
Content-type: text/html<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
desc{londonwater} = charts <BR></BODY></HTML>

Will try some changes and get back.

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay,

I have tried various options and all have provided the same errors.

I have tried renaming the $chart and $charts in one block to $chart1 and $charts1 but same errors or worse wouldn't search on a exisiting chart scale.

Tried adding {} around the scale code block, error.

Tried changing my for local and error. Realised I hadn't changed to 'no strict 'vars'; ' so did that and it worked okay so far as it does with my with the exception that searching on chart number 1 produces a server error.




0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
(it might make it more convenient for everyone working on this to do the
testing on a smaller charts.txt database)

I don't see any chart named 'plymouth' in the dump.
I see
30 = RC1,30,Plymouth Sound and Approaches,12500
1900 = RC1,1900,Whitsand Bay to Yealm Head including Plymouth Sound,25000
1267 = RC1,1267,Falmouth to Plymouth,75000
1967 = RC1,1967,Plymouth Sound,7500
But no 'plymouth'

If we want to find substrings, or case insensitive matches,
that's a different program.
But the program should also be more robust against errors in the query,
and should fail more gracefully when $data is not found.

0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
#this should check for non-existant index entrys
sub showindex($$\%;\%){
        my ($name,$index,$table,$chart) = @_;
        if( !defined($table->{$index}) ){
                error("no such $name\{'$index'}");
        }
        print "$name\{'$index'} = $table->{$index}<BR>\n";
        foreach(split(/,/,$table->{$index})){
                print "chart{$_} = $chart->{$_}<BR>\n";
        }
}
#use by

}elsif( $cmd eq "desc" ){
  showindex("desc",$data,%desc_index,%chart);
}elsif( $cmd eq "scale" ){
  showindex("scale",$data,%scale_index,%chart);
}
#etc

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay ozo I think we have found the error.  Not an error in the program or in the help but in my assumption that the program would handle substring and case insensitive searching (I feature I would ideally like to see implemented and am willing to increase the points for).

Anyway getting back tot he current problem.  Searching on a real desc works but the problem is still providing more elegant notification of desc not found.

I have tried your suggested code but have come up against some problems.  Have a made some attempts to find the problem and have a lot of debug output:-

Inserting your code into the program as follows:-

} elsif ( $cmd eq "chart" ) {
print "chart{$data} = $chart{$data}<BR>\n";

# beginning of new code block (sub at bottom)

}elsif( $cmd eq "desc" ){
showindex("desc",$data,%desc_index,%chart);
}elsif( $cmd eq "scale" ){
showindex("scale",$data,%scale_index,%chart);
# end of new code block

} else {
error("unknown key type, $cmd");
}

dbmclose(%chart);
dbmclose(%scale_index);
dbmclose(%desc_index);

print "</BODY></HTML>";

#this should check for non-existant index entrys
sub showindex($$\%;\%){
 my ($name,$index,$table,$chart) = @_;
 if( !defined($table->{$index}) ){
 error("no such $name\{'$index'}");
 }
 print "$name\{'$index'} = $table->{$index}<BR>\n";
 foreach(split(/,/,$table->{$index})){
 print "chart{$_} = $chart->{$_}<BR>\n";
 }
}
                        sub error {
                          my $msg = shift;
                          print "<H1>Error: $msg</H1>";
                          '';
                        }

                        1;

When I search for 'plymouth' I get an error and this debug:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
syntax error at /cgi-bin/pcmweb/xx-arcs_query.pl line 118, near "sub showindex("
Execution of /cgi-bin/pcmweb/xx-arcs_query.pl aborted due to compilation errors.

So I then modified the sub showindex first line to read:-

 sub showindex {

and this is the debug error I got:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Can't use string ("Plans on the South Coast of Corn") as a HASH ref while "strict refs" in use at /cgi-bin/pcmweb/xx-arcs_query.pl line 121.
Content-type: text/html<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>

Now at this point I turned off strict refs and had 'no strict vars' instead and I got the following debug output:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 124.
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 125.
Content-type: text/html<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
<H1>Error: no such desc{'plymouth'}</H1>desc{'plymouth'} = <BR></BODY></HTML>

Now this output shows that the desc 'plymouth' does not exist but it has problems with some unitialized variables.

Have also created a smaller DB, only 2 records.  One description conatins the word Harbour and another Harbours:-

There complete desc are:-

Harbours on the South Coast of Devon
Salcombe Harbour


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Sorry, I thought you said you were using 5.003, which had prototypes.
(could you add a
print "Perl version: $]<br>";
somewhere to double check?)
Anyway, without prototypes, you'd have to change the call:
showindex("desc",$data,\%desc_index,\%chart);

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
My appologies ozo, I am using PERL 5.001.  Sorry for the cock up.


0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
I have changed the call but am still getting errors:-

Output of script follows:=====================================================
path:  '/cgi-bin/pcmweb/xx-arcs_query.pl'argv[0]:  'xx-arcs_query.pl'
argv[1]:  '<NULL>'
syntax error at /cgi-bin/pcmweb/xx-arcs_query.pl line 118, near "sub showindex("
Execution of /cgi-bin/pcmweb/xx-arcs_query.pl aborted due to compilation errors.


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Ok, I think you would have needed 5.002 for prototypes, so remove the ($$\%;\%)
(which I thought you said you did?)
And call it with an explicit \%desc_index,\%chart

(I wonder if that explains the dbm problems too?  Do you know which DBM_File package you're getting tie'd to?)
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
I have the explicit call in place and am having no joy.

I have absolutely no idea which DBM_File I am using.  I know that the server has only the standard PERL distribution which according to O'Reilly's "Programming Perl" says that SDBM_File is always available, because it's part of the standard perl distribution.

Have used AnyDBM_File successfully for creating a search engine database if that helps.


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Did you remove the ($$\%;\%) prototype?  The
near "sub showindex("
error seems to suggest that you did not, but the earlier
Use of uninitialized value at /cgi-bin/pcmweb/xx-arcs_query.pl line 125.
Content-type: text/html<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>
<H1>Error: no such desc{'plymouth'}</H1>desc{'plymouth'} = <BR></BODY></HTML>
seems to suggest that you did remove the prototype, but didn't put the \ in the call.
0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Maybe I should clarify, in 5.001, there are no  prototypes, so the declaration looks like:
 sub showindex {
which should make an error near "sub showindex(" impossible.
then the call would need to look like
}elsif( $cmd eq "desc" ){
  showindex("desc",$data,\%desc_index,\%chart);
}elsif( $cmd eq "scale" ){
  showindex("scale",$data,\%scale_index,\%chart);
}
to make an explicit hash reference.  (I think 5.001 still had those)
Now if you do that, do you still get errors?
               }
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Okay it works now and graceful errors are returned when scales and descriptions are not found.

Any chance of some help on a substring and case insensitive search.

I will reject dmethvin's answer and award you the points ozo as I don't think he has really earned the 2000 quality points (500 * A).

Perhaps rejecting his answer might wake him up.

I appreciate he did the initial script so if he posts a comment I'll give him a dummy question for 150 points.


0
 
LVL 2

Expert Comment

by:dmethvin
Comment Utility
Trevor, sorry I've been unavailable. The followup to the original answer required a lot more time than I had, but it seems that ozo did a great job helping you debug it. I'd be happy to relinquish the points to him.

I can try to answer your additional questions, but don't have the time to test it out so you may run into some problems I haven't anticipated.

As for the case-insensitive searching, it should be pretty easy. In the import section before the line:

if ( !defined($desc_index{$desc}) ) {

add the line:

 $desc =~ tr/A-Z/a-z/;  # fold all chars to lowercase

Then in the lookup add this line:

} elsif ( $cmd eq "desc" ) {
 $data =~ tr/A-Z/a-z/;  # <--- fold all chars to lowercase
 my $charts = $desc_index{$data};
 print "desc{$data} = charts $charts<BR>\n";

You could extend this to the other indices as well, or enhance it by removing blanks and noise words like "the" and "of". For example, replace the $desc =~ tr/A-Z/a-z/; with $desc = munge_index($desc) and use this:(untested code)

sub munge_index {
  local $_ = shift;

  tr/A-Z/a-z/;  # lowercase
  s/\bof\b//g;  # remove noise words
  s/\bthe\b//g;
  s/\s+//g;     # remove spaces *after* blasting noise words
  $_;
}

Substring searching on the indices, such as trying to find "Altamira" in "Caves of Altamira", takes a few more lines. You'll probably have to just do a loop through the index until you find an entry that matches: (again, untested)

} elsif ( $cmd eq "desc" ) {
 # assume %desc_index was done with munge_index above...
 $data = munge_index($data);
 while ( ($desc, $chart) = each(%desc_index) ) {
  if ( $desc =~ /$data/ ) {
    print "desc{substring of '$data'} = chart $chart<BR>\n";
  }
 }

BTW, this message is getting way too long (78KB!). We'd better put it out of its misery, even if that means creating a new one.
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
Well the case insensitivity works fine, no problems there.  The search on a substring is throwing problems though.  It returns no matches when I search for a substring.

As you (ozo) had written a new routine for searching on desc and scale dmethvin's suggested substring routine was incompatible.  I have tried to merge the two routines but have been having some problems. This is what I have got at the moment for the sub:-

#this should check for non-existant index entrys
sub showindex {
 my ($name,$index,$table,$chart) = @_;
 if( !defined($table->{$index}) ){
 error("no such $name\{'$index'}");
 }
while ( ($table, $chart) = each(%desc_index) ) {
if ( $table =~ /$index/ ) {
 print "desc{substring of '$index'} = chart $chart<BR>\n";
 }
}
 }
}

Any thoughts where it's going wrong.  it doesn't give any server errors, just won't search for the substring.


0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
sub showindexsubstrings {
  my ($name,$index,$myhash,$charthash) = @_;
  my($table,$chart);
  while( ($table, $chart) = each(%{$myhash}) ){
    if( $table =~ /\Q$index\E/i ){
       print "desc{substring of '$index'} = chart $chart<BR>\n";
       print "chart{$chart} = $charthash->{$chart}<BR>\n";
    }
  }
}
call with
showindexsubstrings("scale",$data,\%desc_index,\%chart);
0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
Maybe that
  print "chart{$chart} = $charthash->{$chart}<BR>\n";
would be better written as
  printcharts($chart,$charthash);
where
  sub printcharts{
    my($charts,$hash)=@_;
    foreach( split(/,/,$charts) ){
      if( !defined($hash->{$_}) ){
        print "no such chart{$_}<BR>\n";
      }else{
        print "chart{$_} = $hash->{$_}<BR>\n";
      }    
    }
  }
0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
$table =~ /\Q$index\E/i
is inefficient, I'd probably replace it with
 index(lc($table),lc($index)) >= 0
0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
ozo you are a star, it works great. Thank you very much for all your help and please now submit the final script as an answer and I will grade your answer 'A'. If you want to test it by all means do.  Searching on harbour returns both charts and searching on harbours returns only one chart.  All other search criteria also work absolutely perfectly.

Thank you once again for all your help.


0
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
Comment Utility
#!/bin/perl5
# ------DEBUG-------
my $DEBUG;
BEGIN {
  $DEBUG = 0;
  if( $DEBUG ){
     open(STDERR,">&STDOUT");
     print "Content-type: text/plain\n\n-- DEBUG LEVEL $DEBUG --\n";
  }
}
# ------------------
use AnyDBM_File;

use strict; # no funny stuff
$^W = 1; # look for strangeness
use diagnostics;

my $dbdir = "/docs/www.pcmaritime.co.uk/charts/arcs/data/";

print "Content-type: text/html\n\n";
print "<HTML><HEAD><TITLE>Database example</TITLE></HEAD><BODY>\n";

sub html{
        local($_) = @_;
        s/&/&amp;/g; s/</&lt;/g; s/>/&gt;/g;
        return $_;
}
#sub showindex($$\%;\%){
sub showindex{
        my ($name,$index,$table,$charthash) = @_;
        if( !defined($table->{$index}) ){
                error("no such $name\{$index}"); return;
        }
        print html("$name\{'$index'} = $table->{$index}"),"<BR>\n";
        return unless defined $charthash;
        printcharts($table->{$index},$charthash);
}
sub showindexsubstrings {
   my ($substr,$myhash,$charthash) = @_;
   my($table,$chart);
   $substr = lc($substr);
   while( ($table, $chart) = each(%{$myhash}) ){
     if( index(lc($table),$substr) >= 0 ){
        print html("$table = chart $chart"),"<BR>\n";
        printcharts($chart,$charthash) if( defined($charthash) );
     }
   }
}
sub printcharts{
   my($charts,$hash)=@_;
   foreach(split(/,/,$charts)){
     if( !defined($hash->{$_}) ){
        print html("no such chart{$_}");
     }else{
        print html("chart{$_} = $hash->{$_}");
     }
     print "<BR>\n";
  }
}


# parse incoming query
my %form = ();
my $buffer = $ENV{QUERY_STRING} || shift;
if( defined($ENV{CONTENT_LENGTH}) ){
        local $/=undef; $buffer.=<>;
}
foreach ( split(/&/, $buffer) ) {
   my ($name, $value) = split(/=/);
   # Remove URL encoding and HTML comments
   $value =~ tr/+/ /;
   $value =~ s/%([A-F0-9]{2})/pack("H*",$1)/egi;
   $value =~ s/<!--.*?-->//gs;
   $form{$name} = $value;
}

# open the databases
my (%chart, %scale_index, %desc_index);
dbmopen(%chart, "$dbdir/chart", 0644);
dbmopen(%scale_index, "$dbdir/scale", 0644);
dbmopen(%desc_index, "$dbdir/desc", 0644);

# query like http://wherever.com/cgi-bin/db.pl?cmd=chart&data=14
my $cmd = $form{cmd} or error("missing cmd in query");
my $data = $form{data} or error("missing data in query");

if( $cmd eq "import" and $data eq "imeanit" ){
   open(CSV,"$dbdir/charts.txt") or error("cannot find charts.txt $!"), exit(1);
   while( <CSV> ){
      my $tmp;
      chop;
      my ($cd,$num,$desc,$scale) = split(/,/);
      last if !$scale;
      $chart{$num} = $_;
      if( !defined($tmp = $desc_index{$desc}) ){
          $tmp = $num;
      }else{
          $tmp .= ",$num";
      }
      $desc_index{$desc} = $tmp;
      if( !defined($tmp=$scale_index{$scale}) ){
         $tmp = $num;
      }else{
         $tmp .= ",$num";
      }
      print "$_<BR>\n";
   }
   close(CSV);
}elsif( $cmd eq "dump" ){
   my ($chart,$desc,$scale,$data1);
   print "<P>By chart number (chart = data):<BR>\n";
   showindexsubstrings('',\%chart);
   print "<P>By desc index (desc = chart numbers):<BR>\n";
   showindexsubstrings('',\%desc_index);
   print "<P>By scale index (scale = chart numbers):<BR>\n";
   showindexsubstrings('',\%scale_index);
}elsif( $cmd eq "chart" ){
    printcharts($data,\%chart);
}elsif( $cmd eq "desc" ){
   #showindex("desc",$data,\%desc_index,\%chart);
   print html("grep(/\Q$data\E/,keys(%desc_index)):"),"<BR>\n";
   showindexsubstrings($data,\%desc_index,\%chart);
}elsif( $cmd eq "scale" ){
  showindex("scale",$data,\%scale_index,\%chart);
  #showindexsubstrings("scale",$data,\%scale_index,\%chart);
}else{
  error("unknown key type, $cmd");
}

dbmclose(%chart);
dbmclose(%scale_index);
dbmclose(%desc_index);

print "</BODY></HTML>\n";

sub error {
   my $msg = shift;
   print "<H1>Error: ",html($msg),"</H1>";
   '';
}
1;

0
 
LVL 5

Author Comment

by:Trevor013097
Comment Utility
ozo,

Excellent, it all works a treat.  Thanks for your tidying up in the final answer and the inclusion of a few extra efficiency features.  Works great.  Had to tweak a couple of things becayse you has set the tmp for the scale when importing but had forgotten to set the scale_index to the tmp.

It has been nice working with you ozo, thanks once again for all your help.

0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Introduction This tutorial will give you a fast look what you can do with WhizBase. I expect you already know how to work with HTML at least, and that you understand the basics of the internet and how the internet works. WhizBase is a server-s…
Batch, VBS, and scripts in general are incredibly useful for repetitive tasks.  Some tasks can take a while to complete and it can be annoying to check back only to discover that your script finished 5 minutes ago.  Some scripts may complete nearly …
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now