Solved

XML to MySQL

Posted on 2006-06-29
6
742 Views
Last Modified: 2013-11-19
I have a (large - about 27k entries) xml file that I need to import into mysql.
It’s a dump of the Cisco error message database and I want to import it to use on the syslog server so that when a message comes in, it will display more info w/o having to go to CCO and look it up manually.

Here’s what the xml looks like:

<?xml version="1.0"?>

<cemdb class="simple">
  <date>2006-06-28</date>
  <error name="%">
    <explanation></explanation>
    <action></action>
  </error>
  <error name="%">
    <explanation>The POST has reported an incorrect memory size.</explanation>
    <action>Replace the EAIM.</action>
  </error>
  <error name="%00:00:00:00:00:00">
    <explanation>This is not a valid MAC address, and is probably being generated by non 802.1d
compliant hardware or software in the network. When there is a request to show the hosts the switch
has learned, it must sort the MAC addresses it has learned to reply to the request. The sort algorithm
cannot deal with a MAC address which is all zeroes, so the address will not be included in the table
of hosts which needs to be sorted. This means that even though the switch has learned a host with an
all zero MAC address, it will not show up in any display of hosts the switch has learned.</explanation>
    <action>Track down the source of this bad address.</action>
  </error>
  <error name="%AAA-1-AAA_SESSION_LIMIT_REJECT: aaa request rejected as maximum aaa sessions are in progress.">
    <explanation>The AAA request is rejected because the maximum limit for concurrent AAA sessions
was reached.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system
log, contact your Cisco technical support representative, and provide the representative with the
gathered information.</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical s
upport representative, and provide the representative with the gathered information.</action>
  </error>
  <error name="%AAA-2-AAA_NVRAM_UPGRADE_FAILURE: upgrading of accounting log failed. [chars].">
    <explanation>The accounting log could not be upgraded because of a problem with NVRAM. The
problem is described in the error message.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 2.0(1b).</action>
  </error>
  <error name="%AAA-2-AAA_PROGRAM_EXIT: aaa daemon exiting: [chars].">
    <explanation>The AAA daemon is exiting.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
</cemdb>



And here’s the mysql table:
CREATE TABLE cemdb (
id bigint(20) unsigned NOT NULL auto_increment,
name varchar(128) NOT NULL default '',
error text,
descr text,
action text,
datetime datetime default NULL,
PRIMARY KEY  (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;

Can someone please write a perl script to import this into mysql for me?
0
Comment
Question by:cdukes
  • 3
  • 2
6 Comments
 
LVL 12

Expert Comment

by:stefan73
ID: 17011064
http://www.rentacoder.com ;-)

Try it yourself and ask when you have specific problems.
0
 

Author Comment

by:cdukes
ID: 17011115
I've been trying myself for 4 days now -- that's why I'm here.
If you don't have anything useful to say, please...
0
 
LVL 39

Expert Comment

by:Adam314
ID: 17011221
I'm not sure how big the file is, but if it's small enough to load the entire thing into memory, then you can use XML::Simple to load the file:
http://search.cpan.org/~grantm/XML-Simple-2.14/lib/XML/Simple.pm

Then use DBI to add to the DB
http://search.cpan.org/~timb/DBI-1.51/DBI.pm

Here is some code to get you started reading the XML file...

#!/usr/bin/perl
use strict;
use Data::Dumper;
use XML::Simple;

my $ref = XMLin("/temp_ee/file1.xml");


print "Date=$ref->{date}\n";
foreach my $key (keys %{$ref->{'error'}}){
      print "Name=$key\n";
      print "Explanation=$ref->{error}->{$key}->{explanation}\n";
      print "Action=$ref->{error}->{$key}->{explanation}\n";
      #Any other fields....
}
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 

Author Comment

by:cdukes
ID: 17014413
Here's what I have.
This seems to be working, except that if a duplicate entry is found, it stops.
How can I check to see if $name exists and if it does, just do an update on the record?

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "pass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 1;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");


my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach (@{ $xml_ref->{error}}) {
       my($expl,$act,$name,$errmsg);

   if ($_->{name} =~ /(%.*?):\s?(.*)/) {
       ($name, $errmsg) = ($1,$2);
   } else {
       $name = $_->{name};
       $errmsg = '';
   }

   print "$_->{name}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation}} ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action}} ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
}
0
 
LVL 39

Accepted Solution

by:
Adam314 earned 500 total points
ID: 17017887
Create a query and update statement near where you create your insert statement (I'm not a SQL expert, so the SQL might not be correct - I know for sure the UPDATE is not correct)


my $cemdb_query_sql = "SELECT name FROM cemdb WHERE (name='?')";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET name='?' WHERE name='?'";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);




Then where you have this:
  $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
Do this:
  $sth_query->execute($name);
  $ary_ref  = $sth->fetchall_arrayref;
  if( $#{$ary_ref}>=0) {
      #existing item found, use update
      $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }
  else {
    #no items found, insert
    $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }




0
 

Author Comment

by:cdukes
ID: 17018575
Had to change it a little but this helped immensley, thanks Adam!

Here's the final code in case anyone else ever needs it:

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "somepass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 0;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");

my $cemdb_query_sql = "SELECT id FROM cemdb WHERE name=?";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET error=?, descr=?, action=? WHERE name=?";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach ( @{ $xml_ref->{error} } ) {
       my($expl,$act,$name,$errmsg);

       $name = $_->{name};
   print "$_->{name}\n" if ($DEBUG);

       $errmsg = $_->{message};
   print "$_->{message}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation} } ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action} } ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   #$sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
   $sth_query->execute($name) or die("$DBI::errstr");
   my $ary_ref  = $sth_query->fetchall_arrayref;
   if( $ary_ref >=0) {
           #existing item found, use update
           $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Updating $name\n"
   }
   else {
           #no items found, insert
           $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Inserting $name\n"
   }

}
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Introduction Knockoutjs (Knockout) is a JavaScript framework (Model View ViewModel or MVVM framework).   The main ideology behind Knockout is to control from JavaScript how a page looks whilst creating an engaging user experience in the least …
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
The viewer will learn the basics of jQuery including how to code hide show and toggles. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery…

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question