Solved

XML to MySQL

Posted on 2006-06-29
6
739 Views
Last Modified: 2013-11-19
I have a (large - about 27k entries) xml file that I need to import into mysql.
It’s a dump of the Cisco error message database and I want to import it to use on the syslog server so that when a message comes in, it will display more info w/o having to go to CCO and look it up manually.

Here’s what the xml looks like:

<?xml version="1.0"?>

<cemdb class="simple">
  <date>2006-06-28</date>
  <error name="%">
    <explanation></explanation>
    <action></action>
  </error>
  <error name="%">
    <explanation>The POST has reported an incorrect memory size.</explanation>
    <action>Replace the EAIM.</action>
  </error>
  <error name="%00:00:00:00:00:00">
    <explanation>This is not a valid MAC address, and is probably being generated by non 802.1d
compliant hardware or software in the network. When there is a request to show the hosts the switch
has learned, it must sort the MAC addresses it has learned to reply to the request. The sort algorithm
cannot deal with a MAC address which is all zeroes, so the address will not be included in the table
of hosts which needs to be sorted. This means that even though the switch has learned a host with an
all zero MAC address, it will not show up in any display of hosts the switch has learned.</explanation>
    <action>Track down the source of this bad address.</action>
  </error>
  <error name="%AAA-1-AAA_SESSION_LIMIT_REJECT: aaa request rejected as maximum aaa sessions are in progress.">
    <explanation>The AAA request is rejected because the maximum limit for concurrent AAA sessions
was reached.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system
log, contact your Cisco technical support representative, and provide the representative with the
gathered information.</action>
  </error>
  <error name="%AAA-2-AAAMULTILINKERROR: [chars]([hex]): Acct db for Id [hex] absent">
    <explanation>An AAA internal error has occurred.</explanation>
    <action>Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical s
upport representative, and provide the representative with the gathered information.</action>
  </error>
  <error name="%AAA-2-AAA_NVRAM_UPGRADE_FAILURE: upgrading of accounting log failed. [chars].">
    <explanation>The accounting log could not be upgraded because of a problem with NVRAM. The
problem is described in the error message.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 2.0(1b).</action>
  </error>
  <error name="%AAA-2-AAA_PROGRAM_EXIT: aaa daemon exiting: [chars].">
    <explanation>The AAA daemon is exiting.</explanation>
    <action>If you purchased Cisco support through a Cisco reseller, contact the reseller
directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support.
Introduced  Cisco MDS SAN-OS Release 1.3(1).</action>
  </error>
</cemdb>



And here’s the mysql table:
CREATE TABLE cemdb (
id bigint(20) unsigned NOT NULL auto_increment,
name varchar(128) NOT NULL default '',
error text,
descr text,
action text,
datetime datetime default NULL,
PRIMARY KEY  (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;

Can someone please write a perl script to import this into mysql for me?
0
Comment
Question by:cdukes
  • 3
  • 2
6 Comments
 
LVL 12

Expert Comment

by:stefan73
Comment Utility
http://www.rentacoder.com ;-)

Try it yourself and ask when you have specific problems.
0
 

Author Comment

by:cdukes
Comment Utility
I've been trying myself for 4 days now -- that's why I'm here.
If you don't have anything useful to say, please...
0
 
LVL 39

Expert Comment

by:Adam314
Comment Utility
I'm not sure how big the file is, but if it's small enough to load the entire thing into memory, then you can use XML::Simple to load the file:
http://search.cpan.org/~grantm/XML-Simple-2.14/lib/XML/Simple.pm

Then use DBI to add to the DB
http://search.cpan.org/~timb/DBI-1.51/DBI.pm

Here is some code to get you started reading the XML file...

#!/usr/bin/perl
use strict;
use Data::Dumper;
use XML::Simple;

my $ref = XMLin("/temp_ee/file1.xml");


print "Date=$ref->{date}\n";
foreach my $key (keys %{$ref->{'error'}}){
      print "Name=$key\n";
      print "Explanation=$ref->{error}->{$key}->{explanation}\n";
      print "Action=$ref->{error}->{$key}->{explanation}\n";
      #Any other fields....
}
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 

Author Comment

by:cdukes
Comment Utility
Here's what I have.
This seems to be working, except that if a duplicate entry is found, it stops.
How can I check to see if $name exists and if it does, just do an update on the record?

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "pass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 1;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");


my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach (@{ $xml_ref->{error}}) {
       my($expl,$act,$name,$errmsg);

   if ($_->{name} =~ /(%.*?):\s?(.*)/) {
       ($name, $errmsg) = ($1,$2);
   } else {
       $name = $_->{name};
       $errmsg = '';
   }

   print "$_->{name}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation}} ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action}} ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
}
0
 
LVL 39

Accepted Solution

by:
Adam314 earned 500 total points
Comment Utility
Create a query and update statement near where you create your insert statement (I'm not a SQL expert, so the SQL might not be correct - I know for sure the UPDATE is not correct)


my $cemdb_query_sql = "SELECT name FROM cemdb WHERE (name='?')";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET name='?' WHERE name='?'";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);




Then where you have this:
  $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
Do this:
  $sth_query->execute($name);
  $ary_ref  = $sth->fetchall_arrayref;
  if( $#{$ary_ref}>=0) {
      #existing item found, use update
      $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }
  else {
    #no items found, insert
    $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
  }




0
 

Author Comment

by:cdukes
Comment Utility
Had to change it a little but this helped immensley, thanks Adam!

Here's the final code in case anyone else ever needs it:

#!/usr/bin/perl -w

use strict;
use lib ".";
use XML::Simple qw( :strict );
use DBI;
use Data::Dumper;

sub MY_DB_NAME () { "log" }
sub MY_DB_USER () { "root" }
sub MY_DB_PASS () { "somepass" }

my $xml_file = 'emd.xml';
my $xml_ref = XMLin("$xml_file", ForceArray => 1,  KeyAttr => []);

my $DEBUG = 0;
my $dbh = DBI->connect('dbi:mysql:'.MY_DB_NAME,MY_DB_USER,MY_DB_PASS)
       or die("Couldn't connect to mySQL ", MY_DB_NAME ," : $DBI::errstr");

my $cemdb_query_sql = "SELECT id FROM cemdb WHERE name=?";
my $sth_query = $dbh->prepare($cemdb_query_sql);

my $cemdb_update_sql = "UPDATE cemdb SET error=?, descr=?, action=? WHERE name=?";
my $sth_update = $dbh->prepare($cemdb_update_sql);

my $cemdb_ins_sql = "INSERT INTO cemdb (name, error, descr, action) VALUES (?,?,?,?)";
my $sth = $dbh->prepare($cemdb_ins_sql);

foreach ( @{ $xml_ref->{error} } ) {
       my($expl,$act,$name,$errmsg);

       $name = $_->{name};
   print "$_->{name}\n" if ($DEBUG);

       $errmsg = $_->{message};
   print "$_->{message}\n" if ($DEBUG);

   if ((ref $_->{explanation}) eq "ARRAY") {
       foreach my $line ( @{ $_->{explanation} } ) {
           $expl .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $expl = $_->{explanation};
       print "$_->{explanation}\n" if ($DEBUG);
   }

   if ((ref $_->{action}) eq "ARRAY") {
       foreach my $line ( @{ $_->{action} } ) {
           $act .= $line;
           print "$line\n" if ($DEBUG);
       }
   } else {
       $act = $_->{action};
       print "$_->{action}\n" if ($DEBUG);
   }

   #$sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
   $sth_query->execute($name) or die("$DBI::errstr");
   my $ary_ref  = $sth_query->fetchall_arrayref;
   if( $ary_ref >=0) {
           #existing item found, use update
           $sth_update->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Updating $name\n"
   }
   else {
           #no items found, insert
           $sth->execute($name, $errmsg, $expl, $act) or die("$DBI::errstr");
           print "Inserting $name\n"
   }

}
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
PHP parse error 7 51
Simple Text Web Page rendering differently on IE versus Chrome 3 69
wordpress limitations 4 91
Microsoft Edge 9 79
I will show you how to create a ASP.NET Captcha control without using any HTTP HANDELRS or what so ever. you can easily plug it into your web pages. For Example a = 2 + 3 (where 2 and 3 are 2 random numbers) Session("Answer") = 5 then we…
JavaScript has plenty of pieces of code people often just copy/paste from somewhere but never quite fully understand. Self-Executing functions are just one good example that I'll try to demystify here.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
HTML5 has deprecated a few of the older ways of showing media as well as offering up a new way to create games and animations. Audio, video, and canvas are just a few of the adjustments made between XHTML and HTML5. As we learned in our last micr…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now