Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 787
  • Last Modified:

XML to MySQL

I have a (large - about 27k entries) xml file that I need to import into mysql.
It’s a dump of the Cisco error message database and I want to import it to use on the syslog server so that when a message comes in, it will display more info w/o having to go to CCO and look it up manually.

Here’s what the xml looks like:

<?xml version="1.0"?>

<cemdb class="simple">
  <date>2006-06-28</date>
  <error name="%">
    <explanation></explanation>
    <action></action>
  </error>
  <error name="%">
    <explanation>The POST has reported an incorrect memory size.</explanation>
    <action>Replace the EAIM.</action>
  </error>
  <error name="%00:00:00:00:00:00">
    <explanation>This is not a valid MAC address, and is probably being generated by non 802.1d
compliant hardware or software in the network. When there is a request to show the hosts the switch
has learned, it must sort the MAC addresses it has learned to reply to the request. The sort algorithm
cannot deal with a MAC address which is all zeroes, so the address will not be included in the table
of hosts which needs to be sorted. This means that even though the switch has learned a host with an
all zero MAC address, it will not show up in any display of hosts the switch has learned.</explanation>
    <action>Track down the source of this bad address.</action>
  </error>
  <error name="%AAA-1-AAA_SESSION_LIMIT_REJECT: aaa request rejected as maximum aaa sessions are in progress.">
    <explanation>The AAA request is rejected because the maximum limit for concurrent AAA sessions
was reached.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system
log, contact your Cisco technical support representative, and provide the representative with the
gathered information.</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical s
upport representative, and provide the representative with the gathered information.</action>
  </error>
  <error name="%AAA-2-AAA_NVRAM_UPGRADE_FAILURE: upgrading of accounting log failed. [chars].">
    <explanation>The accounting log could not be upgraded because of a problem with NVRAM. The
problem is described in the error message.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 2.0(1b).</action>
  </error>
  <error name="%AAA-2-AAA_PROGRAM_EXIT: aaa daemon exiting: [chars].">
    <explanation>The AAA daemon is exiting.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
</cemdb>



And here’s the mysql table:
CREATE TABLE cemdb (
id bigint(20) unsigned NOT NULL auto_increment,
name varchar(128) NOT NULL default '',
error text,
descr text,
action text,
datetime datetime default NULL,
PRIMARY KEY  (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;

Can someone please write a perl script to import this into mysql for me?
0
cdukes
Asked:
cdukes
  • 3
  • 2
1 Solution
 
stefan73Commented:
http://www.rentacoder.com ;-)

Try it yourself and ask when you have specific problems.
0
 
cdukesAuthor Commented:
I've been trying myself for 4 days now -- that's why I'm here.
If you don't have anything useful to say, please...
0
 
Adam314Commented:
I'm not sure how big the file is, but if it's small enough to load the entire thing into memory, then you can use XML::Simple to load the file:
http://search.cpan.org/~grantm/XML-Simple-2.14/lib/XML/Simple.pm

Then use DBI to add to the DB
http://search.cpan.org/~timb/DBI-1.51/DBI.pm

Here is some code to get you started reading the XML file...

#!/usr/bin/perl
use strict;
use Data::Dumper;
use XML::Simple;

my $ref = XMLin("/temp_ee/file1.xml");


print "Date=$ref->{date}\n";
foreach my $key (keys %{$ref->{'error'}}){
      print "Name=$key\n";
      print "Explanation=$ref->{error}->{$key}->{explanation}\n";
      print "Action=$ref->{error}->{$key}->{explanation}\n";
      #Any other fields....
}
0
The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

 
cdukesAuthor Commented:
Here's what I have.
This seems to be working, except that if a duplicate entry is found, it stops.
How can I check to see if $name exists and if it does, just do an update on the record?

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "pass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 1;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");


my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach (@{ $xml_ref->{error}}) {
       my($expl,$act,$name,$errmsg);

   if ($_->{name} =~ /(%.*?):\s?(.*)/) {
       ($name, $errmsg) = ($1,$2);
   } else {
       $name = $_->{name};
       $errmsg = '';
   }

   print "$_->{name}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation}} ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action}} ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
}
0
 
Adam314Commented:
Create a query and update statement near where you create your insert statement (I'm not a SQL expert, so the SQL might not be correct - I know for sure the UPDATE is not correct)


my $cemdb_query_sql = "SELECT name FROM cemdb WHERE (name='?')";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET name='?' WHERE name='?'";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);




Then where you have this:
  $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
Do this:
  $sth_query->execute($name);
  $ary_ref  = $sth->fetchall_arrayref;
  if( $#{$ary_ref}>=0) {
      #existing item found, use update
      $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }
  else {
    #no items found, insert
    $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }




0
 
cdukesAuthor Commented:
Had to change it a little but this helped immensley, thanks Adam!

Here's the final code in case anyone else ever needs it:

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "somepass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 0;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");

my $cemdb_query_sql = "SELECT id FROM cemdb WHERE name=?";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET error=?, descr=?, action=? WHERE name=?";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach ( @{ $xml_ref->{error} } ) {
       my($expl,$act,$name,$errmsg);

       $name = $_->{name};
   print "$_->{name}\n" if ($DEBUG);

       $errmsg = $_->{message};
   print "$_->{message}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation} } ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action} } ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   #$sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
   $sth_query->execute($name) or die("$DBI::errstr");
   my $ary_ref  = $sth_query->fetchall_arrayref;
   if( $ary_ref >=0) {
           #existing item found, use update
           $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Updating $name\n"
   }
   else {
           #no items found, insert
           $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Inserting $name\n"
   }

}
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now