?
Solved

XML to MySQL

Posted on 2006-06-29
6
Medium Priority
?
759 Views
Last Modified: 2013-11-19
I have a (large - about 27k entries) xml file that I need to import into mysql.
It’s a dump of the Cisco error message database and I want to import it to use on the syslog server so that when a message comes in, it will display more info w/o having to go to CCO and look it up manually.

Here’s what the xml looks like:

<?xml version="1.0"?>

<cemdb class="simple">
  <date>2006-06-28</date>
  <error name="%">
    <explanation></explanation>
    <action></action>
  </error>
  <error name="%">
    <explanation>The POST has reported an incorrect memory size.</explanation>
    <action>Replace the EAIM.</action>
  </error>
  <error name="%00:00:00:00:00:00">
    <explanation>This is not a valid MAC address, and is probably being generated by non 802.1d
compliant hardware or software in the network. When there is a request to show the hosts the switch
has learned, it must sort the MAC addresses it has learned to reply to the request. The sort algorithm
cannot deal with a MAC address which is all zeroes, so the address will not be included in the table
of hosts which needs to be sorted. This means that even though the switch has learned a host with an
all zero MAC address, it will not show up in any display of hosts the switch has learned.</explanation>
    <action>Track down the source of this bad address.</action>
  </error>
  <error name="%AAA-1-AAA_SESSION_LIMIT_REJECT: aaa request rejected as maximum aaa sessions are in progress.">
    <explanation>The AAA request is rejected because the maximum limit for concurrent AAA sessions
was reached.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system
log, contact your Cisco technical support representative, and provide the representative with the
gathered information.</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical s
upport representative, and provide the representative with the gathered information.</action>
  </error>
  <error name="%AAA-2-AAA_NVRAM_UPGRADE_FAILURE: upgrading of accounting log failed. [chars].">
    <explanation>The accounting log could not be upgraded because of a problem with NVRAM. The
problem is described in the error message.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 2.0(1b).</action>
  </error>
  <error name="%AAA-2-AAA_PROGRAM_EXIT: aaa daemon exiting: [chars].">
    <explanation>The AAA daemon is exiting.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
</cemdb>



And here’s the mysql table:
CREATE TABLE cemdb (
id bigint(20) unsigned NOT NULL auto_increment,
name varchar(128) NOT NULL default '',
error text,
descr text,
action text,
datetime datetime default NULL,
PRIMARY KEY  (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;

Can someone please write a perl script to import this into mysql for me?
0
Comment
Question by:cdukes
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
6 Comments
 
LVL 12

Expert Comment

by:stefan73
ID: 17011064
http://www.rentacoder.com ;-)

Try it yourself and ask when you have specific problems.
0
 

Author Comment

by:cdukes
ID: 17011115
I've been trying myself for 4 days now -- that's why I'm here.
If you don't have anything useful to say, please...
0
 
LVL 39

Expert Comment

by:Adam314
ID: 17011221
I'm not sure how big the file is, but if it's small enough to load the entire thing into memory, then you can use XML::Simple to load the file:
http://search.cpan.org/~grantm/XML-Simple-2.14/lib/XML/Simple.pm

Then use DBI to add to the DB
http://search.cpan.org/~timb/DBI-1.51/DBI.pm

Here is some code to get you started reading the XML file...

#!/usr/bin/perl
use strict;
use Data::Dumper;
use XML::Simple;

my $ref = XMLin("/temp_ee/file1.xml");


print "Date=$ref->{date}\n";
foreach my $key (keys %{$ref->{'error'}}){
      print "Name=$key\n";
      print "Explanation=$ref->{error}->{$key}->{explanation}\n";
      print "Action=$ref->{error}->{$key}->{explanation}\n";
      #Any other fields....
}
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:cdukes
ID: 17014413
Here's what I have.
This seems to be working, except that if a duplicate entry is found, it stops.
How can I check to see if $name exists and if it does, just do an update on the record?

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "pass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 1;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");


my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach (@{ $xml_ref->{error}}) {
       my($expl,$act,$name,$errmsg);

   if ($_->{name} =~ /(%.*?):\s?(.*)/) {
       ($name, $errmsg) = ($1,$2);
   } else {
       $name = $_->{name};
       $errmsg = '';
   }

   print "$_->{name}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation}} ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action}} ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
}
0
 
LVL 39

Accepted Solution

by:
Adam314 earned 1500 total points
ID: 17017887
Create a query and update statement near where you create your insert statement (I'm not a SQL expert, so the SQL might not be correct - I know for sure the UPDATE is not correct)


my $cemdb_query_sql = "SELECT name FROM cemdb WHERE (name='?')";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET name='?' WHERE name='?'";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);




Then where you have this:
  $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
Do this:
  $sth_query->execute($name);
  $ary_ref  = $sth->fetchall_arrayref;
  if( $#{$ary_ref}>=0) {
      #existing item found, use update
      $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }
  else {
    #no items found, insert
    $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }




0
 

Author Comment

by:cdukes
ID: 17018575
Had to change it a little but this helped immensley, thanks Adam!

Here's the final code in case anyone else ever needs it:

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "somepass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 0;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");

my $cemdb_query_sql = "SELECT id FROM cemdb WHERE name=?";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET error=?, descr=?, action=? WHERE name=?";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach ( @{ $xml_ref->{error} } ) {
       my($expl,$act,$name,$errmsg);

       $name = $_->{name};
   print "$_->{name}\n" if ($DEBUG);

       $errmsg = $_->{message};
   print "$_->{message}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation} } ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action} } ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   #$sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
   $sth_query->execute($name) or die("$DBI::errstr");
   my $ary_ref  = $sth_query->fetchall_arrayref;
   if( $ary_ref >=0) {
           #existing item found, use update
           $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Updating $name\n"
   }
   else {
           #no items found, insert
           $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Inserting $name\n"
   }

}
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Preface In the first article: A Better Website Login System (http://www.experts-exchange.com/A_2902.html) I introduced the EE Collaborative Login System and its intended purpose. In this article I will discuss some of the design consideratio…
Introduction Knockoutjs (Knockout) is a JavaScript framework (Model View ViewModel or MVVM framework).   The main ideology behind Knockout is to control from JavaScript how a page looks whilst creating an engaging user experience in the least …
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:
The viewer will learn how to dynamically set the form action using jQuery.
Suggested Courses

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question