projects
asked on
zimbra - mysql won't start
I accidentally deleted a few logs in the /opt/zimbra/log directory, but didn't think I deleted anything which would prevent the server from starting, mostly just *.gz log files.
Yet, since that time, and having changed public IP of the DNS record, I get the following when trying to start the server.
Because of this, some of the zimbra services won't start and so the services will not run correctly.
Am at a loss, forums are pretty useless, filled with questions going unanswered so, hoping someone here can help.
Yet, since that time, and having changed public IP of the DNS record, I get the following when trying to start the server.
151030 21:22:58 mysqld_safe mysqld from pid file /opt/zimbra/db/mysql.pid ended
151030 21:24:59 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
151030 21:24:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages
151030 21:24:59 [Note] InnoDB: The InnoDB memory heap is disabled
151030 21:24:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
151030 21:24:59 [Note] InnoDB: Memory barrier is not used
151030 21:24:59 [Note] InnoDB: Compressed tables use zlib 1.2.3
151030 21:24:59 [Note] InnoDB: Using Linux native AIO
151030 21:24:59 [Note] InnoDB: Not using CPU crc32 instructions
151030 21:24:59 [Note] InnoDB: Initializing buffer pool, size = 1.7G
151030 21:24:59 [Note] InnoDB: Completed initialization of buffer pool
151030 21:24:59 [ERROR] InnoDB: Log file ./ib_logfile0 size 524287981 is not a multiple of innodb_page_size
151030 21:24:59 [ERROR] Plugin 'InnoDB' init function returned error.
151030 21:24:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151030 21:24:59 [Note] Plugin 'FEEDBACK' is disabled.
151030 21:24:59 [ERROR] Unknown/unsupported storage engine: InnoDB
151030 21:24:59 [ERROR] Aborting
151030 21:24:59 [Note] /opt/zimbra/mariadb/bin/mysqld: Shutdown completeBecause of this, some of the zimbra services won't start and so the services will not run correctly.
Am at a loss, forums are pretty useless, filled with questions going unanswered so, hoping someone here can help.
ASKER
I moved the two from /opt/zimbra/db/data and restarted the services. Now mysql seems to have started but...
Watching the log for mysql and it almost looks like mysql is stuck in some loop.
$ zmcontrol status
Connect: Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Host mail.domain.com
amavis Running
antispam Running
antivirus Running
ldap Stopped
logger Running
mailbox Stopped
mysql.server is not running.
zmmailboxdctl is not running.
mta Running
opendkim Running
service webapp Stopped
mysql.server is not running.
zmmailboxdctl is not running.
snmp Running
spell Running
stats Running
zimbra webapp Stopped
mysql.server is not running.
zmmailboxdctl is not running.
zimbraAdmin webapp Stopped
mysql.server is not running.
zmmailboxdctl is not running.
zimlet webapp Stopped
mysql.server is not running.
zmmailboxdctl is not running.
zmconfigd RunningWatching the log for mysql and it almost looks like mysql is stuck in some loop.
InnoDB: End of page dump
2015-10-31 07:21:13 7f9e05b8c700 InnoDB: uncompressed page, stored checksum in field1 3760564224, calculated checksums for field1: crc32 3869179214, innodb 2910242503, none 3735928559, stored checksum in field2 769150599, calculated checksums for field2: crc32 3869179214, innodb 1292878112, none 3735928559, page LSN 135 1877614149, low 4 bytes of LSN at page end 1877614273, page number (if stored to page already) 81152, space id (if created with >= MySQL-4.1.1 and stored already) 230144
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 317.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
2015-10-31 07:21:13 7f9e05b8c700 InnoDB: Assertion failure in thread 140316677555968 in file buf0buf.cc line 4408
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
151031 7:21:13 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.0.15-MariaDB-log
key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=0
max_threads=112
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362747 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/opt/zimbra/mariadb/bin/mysqld(my_print_stacktrace+0x2e)[0xb7959e]
/opt/zimbra/mariadb/bin/mysqld(handle_fatal_signal+0x40b)[0x723e0b]
/lib64/libpthread.so.0[0x30c500f790]
/lib64/libc.so.6(gsignal+0x35)[0x30c4c32625]
/lib64/libc.so.6(abort+0x175)[0x30c4c33e05]
/opt/zimbra/mariadb/bin/mysqld[0x9e41ab]
/opt/zimbra/mariadb/bin/mysqld[0x9f8388]
/opt/zimbra/mariadb/bin/mysqld[0x9f8a13]
/opt/zimbra/mariadb/bin/mysqld[0x9e5ef7]
/opt/zimbra/mariadb/bin/mysqld[0x9cef0d]
/opt/zimbra/mariadb/bin/mysqld[0x96f325]
/opt/zimbra/mariadb/bin/mysqld[0x96d148]
/opt/zimbra/mariadb/bin/mysqld[0x9cf483]
/opt/zimbra/mariadb/bin/mysqld[0x96f1da]
/opt/zimbra/mariadb/bin/mysqld[0x96d874]
/opt/zimbra/mariadb/bin/mysqld[0x96e237]
/opt/zimbra/mariadb/bin/mysqld[0x96e7c3]
/opt/zimbra/mariadb/bin/mysqld[0x93b840]
/opt/zimbra/mariadb/bin/mysqld[0x99790b]
/opt/zimbra/mariadb/bin/mysqld[0x987d9e]
/lib64/libpthread.so.0[0x30c5007a51]
/lib64/libc.so.6(clone+0x6d)[0x30c4ce893d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
151031 07:21:13 mysqld_safe Number of processes running now: 0
151031 07:21:13 mysqld_safe mysqld restarted
151031 7:21:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages
151031 7:21:13 [Note] InnoDB: The InnoDB memory heap is disabled
151031 7:21:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
151031 7:21:13 [Note] InnoDB: Memory barrier is not used
151031 7:21:13 [Note] InnoDB: Compressed tables use zlib 1.2.3
151031 7:21:13 [Note] InnoDB: Using Linux native AIO
151031 7:21:13 [Note] InnoDB: Not using CPU crc32 instructions
151031 7:21:13 [Note] InnoDB: Initializing buffer pool, size = 1.7G
151031 7:21:13 [Note] InnoDB: Completed initialization of buffer pool
151031 7:21:13 [Note] InnoDB: Highest supported file format is Barracuda.
151031 7:21:13 [Note] InnoDB: Log scan progressed past the checkpoint lsn 2273647116
151031 7:21:13 [Note] InnoDB: Database was not shutdown normally!
151031 7:21:13 [Note] InnoDB: Starting crash recovery.
151031 7:21:13 [Note] InnoDB: Reading tablespace information from the .ibd files...
151031 7:21:13 [Note] InnoDB: Restoring possible half-written data pages
151031 7:21:13 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2273647176ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I was lazy and don't have one since Oct. Is there some way of finding out which table it might be? Maybe it's one that is relatively static where I could replace just that.
Try starting mysql/mariadb with "innodb_force_recovery = 1".
This should allow you to export your data to a file. If the export is successful, you can delete the whole database and import from the file you created.
Reference: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
This should allow you to export your data to a file. If the export is successful, you can delete the whole database and import from the file you created.
Reference: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
ASKER
That didn't work so am now restoring the backed up vm.
That server will have the previous public IP and again, I'm having a hard time finding the required information on what needs to be changed in order to make the change without destroying the server again.
That server will have the previous public IP and again, I'm having a hard time finding the required information on what needs to be changed in order to make the change without destroying the server again.
ASKER
I changed the localnets IP to the new public IP and see this.
Oct 31 11:49:49 uc postfix/smtpd[11428]: warning: non-null host address bits in "xx.xx.xx.44/24", perhaps you should use "xx.xx.xx.0/24" instead
I only have one static IP at tihs particular location and I don't really want to allow the entire /24 class to have free access.
Oct 31 11:49:49 uc postfix/smtpd[11428]: warning: non-null host address bits in "xx.xx.xx.44/24", perhaps you should use "xx.xx.xx.0/24" instead
I only have one static IP at tihs particular location and I don't really want to allow the entire /24 class to have free access.
xx.xx.xx.44/24 is incorrect.
It still means xx.xx.xx.1-xx.xx.xx.255, just incorrect notation.
You need xx.xx.xx.44/32 to only allow 1 IP.
It still means xx.xx.xx.1-xx.xx.xx.255, just incorrect notation.
You need xx.xx.xx.44/32 to only allow 1 IP.
ASKER
That's what I had at first and it was complaining. I've changed it again and now it seems to be working.
So, that's all I need to change is mynetworks. The server is behind a firewall so is using a private IP.
So, that's all I need to change is mynetworks. The server is behind a firewall so is using a private IP.
Delete those log files.
Restart MySQL.
Restart Zimbra.
Check if you lost something.
HTH,
Dan