zimbra - mysql won't start

I accidentally deleted a few logs in the /opt/zimbra/log directory, but didn't think I deleted anything which would prevent the server from starting, mostly just *.gz log files.

Yet, since that time, and having changed public IP of the DNS record, I get the following when trying to start the server.

151030 21:22:58 mysqld_safe mysqld from pid file /opt/zimbra/db/mysql.pid ended
151030 21:24:59 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
151030 21:24:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages
151030 21:24:59 [Note] InnoDB: The InnoDB memory heap is disabled
151030 21:24:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
151030 21:24:59 [Note] InnoDB: Memory barrier is not used
151030 21:24:59 [Note] InnoDB: Compressed tables use zlib 1.2.3
151030 21:24:59 [Note] InnoDB: Using Linux native AIO
151030 21:24:59 [Note] InnoDB: Not using CPU crc32 instructions
151030 21:24:59 [Note] InnoDB: Initializing buffer pool, size = 1.7G
151030 21:24:59 [Note] InnoDB: Completed initialization of buffer pool
151030 21:24:59 [ERROR] InnoDB: Log file ./ib_logfile0 size 524287981 is not a multiple of innodb_page_size
151030 21:24:59 [ERROR] Plugin 'InnoDB' init function returned error.
151030 21:24:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151030 21:24:59 [Note] Plugin 'FEEDBACK' is disabled.
151030 21:24:59 [ERROR] Unknown/unsupported storage engine: InnoDB
151030 21:24:59 [ERROR] Aborting

151030 21:24:59 [Note] /opt/zimbra/mariadb/bin/mysqld: Shutdown complete

Open in new window

Because of this, some of the zimbra services won't start and so the services will not run correctly.

Am at a loss, forums are pretty useless, filled with questions going unanswered so, hoping someone here can help.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Dan CraciunIT ConsultantCommented:
Backup any ib_logfile0, ib_logfile1, ib_logfileX you find.

Delete those log files.

Restart MySQL.

Restart Zimbra.

Check if you lost something.

projectsAuthor Commented:
I moved the two from /opt/zimbra/db/data and restarted the services. Now mysql seems to have started but...

$ zmcontrol status
Connect: Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Host mail.domain.com
        amavis                  Running
        antispam                Running
        antivirus               Running
        ldap                    Stopped
        logger                  Running
        mailbox                 Stopped
                mysql.server is not running.
                zmmailboxdctl is not running.
        mta                     Running
        opendkim                Running
        service webapp          Stopped
                mysql.server is not running.
                zmmailboxdctl is not running.
        snmp                    Running
        spell                   Running
        stats                   Running
        zimbra webapp           Stopped
                mysql.server is not running.
                zmmailboxdctl is not running.
        zimbraAdmin webapp      Stopped
                mysql.server is not running.
                zmmailboxdctl is not running.
        zimlet webapp           Stopped
                mysql.server is not running.
                zmmailboxdctl is not running.
        zmconfigd               Running

Open in new window

Watching the log for mysql and it almost looks like mysql is stuck in some loop.

InnoDB: End of page dump
2015-10-31 07:21:13 7f9e05b8c700 InnoDB: uncompressed page, stored checksum in field1 3760564224, calculated checksums for field1: crc32 3869179214, innodb 2910242503, none 3735928559, stored checksum in field2 769150599, calculated checksums for field2: crc32 3869179214, innodb 1292878112, none 3735928559, page LSN 135 1877614149, low 4 bytes of LSN at page end 1877614273, page number (if stored to page already) 81152, space id (if created with >= MySQL-4.1.1 and stored already) 230144
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 317.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
2015-10-31 07:21:13 7f9e05b8c700  InnoDB: Assertion failure in thread 140316677555968 in file buf0buf.cc line 4408
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
151031  7:21:13 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.15-MariaDB-log
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362747 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
151031 07:21:13 mysqld_safe Number of processes running now: 0
151031 07:21:13 mysqld_safe mysqld restarted
151031  7:21:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages
151031  7:21:13 [Note] InnoDB: The InnoDB memory heap is disabled
151031  7:21:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
151031  7:21:13 [Note] InnoDB: Memory barrier is not used
151031  7:21:13 [Note] InnoDB: Compressed tables use zlib 1.2.3
151031  7:21:13 [Note] InnoDB: Using Linux native AIO
151031  7:21:13 [Note] InnoDB: Not using CPU crc32 instructions
151031  7:21:13 [Note] InnoDB: Initializing buffer pool, size = 1.7G
151031  7:21:13 [Note] InnoDB: Completed initialization of buffer pool
151031  7:21:13 [Note] InnoDB: Highest supported file format is Barracuda.
151031  7:21:13 [Note] InnoDB: Log scan progressed past the checkpoint lsn 2273647116
151031  7:21:13 [Note] InnoDB: Database was not shutdown normally!
151031  7:21:13 [Note] InnoDB: Starting crash recovery.
151031  7:21:13 [Note] InnoDB: Reading tablespace information from the .ibd files...
151031  7:21:13 [Note] InnoDB: Restoring possible half-written data pages
151031  7:21:13 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2273647176

Open in new window

Dan CraciunIT ConsultantCommented:
Looks like it's time to restore from the latest backup. At least one table is corrupt.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

projectsAuthor Commented:
I was lazy and don't have one since Oct. Is there some way of finding out which table it might be? Maybe it's one that is relatively static where I could replace just that.
Dan CraciunIT ConsultantCommented:
Try starting mysql/mariadb with "innodb_force_recovery = 1".

This should allow you to export your data to a file. If the export is successful, you can delete the whole database and import from the file you created.

Reference: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
projectsAuthor Commented:
That didn't work so am now restoring the backed up vm.

That server will have the previous public IP and again, I'm having a hard time finding the required information on what needs to be changed in order to make the change without destroying the server again.
projectsAuthor Commented:
I changed the localnets IP to the new public IP and see this.

Oct 31 11:49:49 uc postfix/smtpd[11428]: warning: non-null host address bits in "xx.xx.xx.44/24", perhaps you should use "xx.xx.xx.0/24" instead

I only have one static IP at tihs particular location and I don't really want to allow the entire /24 class to have free access.
Dan CraciunIT ConsultantCommented:
xx.xx.xx.44/24 is incorrect.
It still means xx.xx.xx.1-xx.xx.xx.255, just incorrect notation.

You need xx.xx.xx.44/32 to only allow 1 IP.
projectsAuthor Commented:
That's what I had at first and it was complaining. I've changed it again and now it seems to be working.

So, that's all I need to change is mynetworks. The server is behind a firewall so is using a private IP.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.