Solved

linux system crash involving perl mysql

Posted on 2011-03-11
5
431 Views
Last Modified: 2013-12-16
Hello

I was running a perl program on a linux virtual machine (virtual box) and it has caused the whole VM to crash. I was able to close the machine but I would like to try and find out what just happened. A perl program that writes to mysql was running.

I know nothing about linux admin.  the /var/log/mysql/error.log is below.

I was running this perl program because a particular set of input data made another person's machine crash while running the same perl code. The input data does not appear any different than other data which has been through the program many times before. I do not have a copy of the other persons error log.

Mysql is running fine and i can look at the data that was already written to mysql before the crash.

thanks
110311 20:58:09 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110311 20:58:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110311 20:58:23  InnoDB: Started; log sequence number 36 2238638524
110311 20:58:24 [Note] Event Scheduler: Loaded 0 events
110311 20:58:24 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.49-1ubuntu8.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
110311 20:59:24 [ERROR] /usr/sbin/mysqld: Table './annotationDB/harry' is marked as crashed and should be repaired
110311 20:59:24 [Warning] Checking table:   './annotationDB/harry'
110311 20:59:24 [Warning] Recovering table: './annotationDB/harry'
error.log (END)

Open in new window

0
Comment
Question by:andieje
5 Comments
 
LVL 3

Expert Comment

by:tearman
ID: 35112789
Good place to start is to run strace while you're executing the perl program to see what the kernel is doing at that point in time.  Wish we had dtrace on Linux, but not available.

Otherwise, check /var/logs, especially the dmesg log for anything interesting.
0
 
LVL 7

Expert Comment

by:droyden
ID: 35113082
What was the perl script attepting to do?
0
 

Author Comment

by:andieje
ID: 35113308
i dont know the point at which it crashed and now the perl code runs through perfectly but i would like to find out why 2 machines crashed

what might be interesting in dmesg as it all looks the same to me?

the perl program was probabily trying to update a record into the db when it crashed but i'm not 100%. the perl program had created some tables, added some records before it crashed and was due to update the records with some data from an external program. the external program had successfully completed before the perl crashed
0
 
LVL 5

Accepted Solution

by:
paulqna earned 500 total points
ID: 35133229
Messages around the time of the crash in these logs could be interesting:

/var/log/syslog*
/var/log/messages*

So that would be anything before a line like:

Mar 12 21:10:49 hostname syslogd version: restart

Open in new window

0
 
LVL 1

Expert Comment

by:jmreidy
ID: 35195845
Check the logs.

If the script was doing inserts and a disk filled up - the system shouldn't crash - the query would just be stopped until the problem was fixed. However I guess a full disk might cause a problem.

Apart from that mysql really does network I/O and disk I/O - with file locking.

I can see 2 causes of a crash:
Some sort of kernel bug - to do with file handling or locking.
or
A hardware problem - bad memory or disk that could be a coincidence or was triggered by the query.

Software/kernel errors should be logged - hardware errors - maybe not.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

How many times have you wanted to quickly do the same thing to a list but found yourself typing it again and again? I first figured out a small time saver with the up arrow to recall the last command but that can only get you so far if you have a bi…
Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now