Solved

linux system crash involving perl mysql

Posted on 2011-03-11
5
438 Views
Last Modified: 2013-12-16
Hello

I was running a perl program on a linux virtual machine (virtual box) and it has caused the whole VM to crash. I was able to close the machine but I would like to try and find out what just happened. A perl program that writes to mysql was running.

I know nothing about linux admin.  the /var/log/mysql/error.log is below.

I was running this perl program because a particular set of input data made another person's machine crash while running the same perl code. The input data does not appear any different than other data which has been through the program many times before. I do not have a copy of the other persons error log.

Mysql is running fine and i can look at the data that was already written to mysql before the crash.

thanks
110311 20:58:09 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110311 20:58:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110311 20:58:23  InnoDB: Started; log sequence number 36 2238638524
110311 20:58:24 [Note] Event Scheduler: Loaded 0 events
110311 20:58:24 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.49-1ubuntu8.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
110311 20:59:24 [ERROR] /usr/sbin/mysqld: Table './annotationDB/harry' is marked as crashed and should be repaired
110311 20:59:24 [Warning] Checking table:   './annotationDB/harry'
110311 20:59:24 [Warning] Recovering table: './annotationDB/harry'
error.log (END)

Open in new window

0
Comment
Question by:andieje
5 Comments
 
LVL 3

Expert Comment

by:tearman
ID: 35112789
Good place to start is to run strace while you're executing the perl program to see what the kernel is doing at that point in time.  Wish we had dtrace on Linux, but not available.

Otherwise, check /var/logs, especially the dmesg log for anything interesting.
0
 
LVL 7

Expert Comment

by:droyden
ID: 35113082
What was the perl script attepting to do?
0
 

Author Comment

by:andieje
ID: 35113308
i dont know the point at which it crashed and now the perl code runs through perfectly but i would like to find out why 2 machines crashed

what might be interesting in dmesg as it all looks the same to me?

the perl program was probabily trying to update a record into the db when it crashed but i'm not 100%. the perl program had created some tables, added some records before it crashed and was due to update the records with some data from an external program. the external program had successfully completed before the perl crashed
0
 
LVL 5

Accepted Solution

by:
paulqna earned 500 total points
ID: 35133229
Messages around the time of the crash in these logs could be interesting:

/var/log/syslog*
/var/log/messages*

So that would be anything before a line like:

Mar 12 21:10:49 hostname syslogd version: restart

Open in new window

0
 
LVL 1

Expert Comment

by:jmreidy
ID: 35195845
Check the logs.

If the script was doing inserts and a disk filled up - the system shouldn't crash - the query would just be stopped until the problem was fixed. However I guess a full disk might cause a problem.

Apart from that mysql really does network I/O and disk I/O - with file locking.

I can see 2 causes of a crash:
Some sort of kernel bug - to do with file handling or locking.
or
A hardware problem - bad memory or disk that could be a coincidence or was triggered by the query.

Software/kernel errors should be logged - hardware errors - maybe not.
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question