MySQL Percona Cluster: WSREP Error "Failed to initialize backend"

Posted on 2012-08-16
Last Modified: 2012-08-26
Hello everyone -
I am dealing with another DBA's installation of Percona Cluster this afternoon - and he's not available.
I have searched the forums here and I have failed to find a post regarding the problem I am seeing.
The sysadmin restarted NODE01 and MySQL (Percona Server) is not starting due to:
120816 13:11:33 WSREP: gcs/src/gcs_backend.c:gcs_backend_init():87: Invalid backend URI: 0
 120816 13:11:33 WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to initialize backend using '0': -22 (Invalid argument)
 120816 13:11:33 WSREP: gcs/src/gcs.c:gcs_open():1290: Failed to open channel 'c_l' at '0': -22 (Invalid argument)
 120816 13:11:33 WSREP: gcs connect failed: Invalid argument
 120816 13:11:33 WSREP: wsrep::connect() failed: 6
 120816 13:11:33 Aborting

That's from the NODE1 error log.
Any idea at all what's going on here?
The system has been previously up and running in a four node arrangement. The other three nodes have MySQL up and running.
Thanks in advance for ANY advice, thoughts or suggestions you may be able to provide.
/David C.
Question by:learningtechnologies
    LVL 2

    Author Comment

    Additional information that may help:

     uname -a
    Linux hsdb01 3.2.0-24-virtual #37-Ubuntu SMP Wed Apr 25 10:17:19 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
    free -m
                 total       used       free     shared    buffers     cached
    Mem:         16050        605      15444          0         30        155
    -/+ buffers/cache:        419      15630
    Swap:         3317          0       3317

    Open in new window


    /David C.
    LVL 24

    Accepted Solution

    Not too many people know about the Percona Cluster here.  I haven't used it yet myself though I plan to try it out on test machines.   May I suggest you also post at the Percona Users Google Group?  See!forum/percona-discussion
    LVL 2

    Author Comment

    Thank you for that excellent suggestion!

    I have posted at the Percona community forum.

    I have the feeling I'm going to be a Percona Cluster expert by the time this is all over.

    Thanks -

    /David C.
    LVL 24

    Assisted Solution

    Teach me!  Teach me! hehehe
    LVL 2

    Author Comment

    Well ... I ~have~ been wondering about when I could write it all up because the next three days are full for me ... but I did figure it out after about six straight hours of working on it.
    So ... more info will be on its way eventually.

    I appreciate everyone's willingness to help.

    /David C.
    LVL 2

    Assisted Solution

    What I did to get the node rejoined to the cluster was to:

    1) Copy the running process values from the ps aux | grep ^mysql command
    2) Modify the running process values from using the network where none of the nodes were listening to the network where they were listening

    Original process values shown by ps aux | grep ^mysql:

    /usr/local/mysql/bin/mysqld_safe --basedir=/usr/local/mysql --datadir=/data/cluster-data --plugin-dir=/usr/local/mysql/lib/mysql/plugin --user=mysql --log-error=/data/cluster-data/pcdb01.err --pid-file=/data/cluster-data/ --wsrep_cluster_address=gcomm://

    Values passed to system as start up command that resulted in the node rejoining the cluster:

    /usr/local/mysql/bin/mysqld_safe --basedir=/usr/local/mysql --datadir=/data/cluster-data --plugin-dir=/usr/local/mysql/lib/mysql/plugin --user=mysql --log-error=/data/cluster-data/pcdb01.err --pid-file=/data/cluster-data/ --wsrep_cluster_address=gcomm://

    What happened was that the maintenance performed by the sysadmin which resulted in needing to reboot the system was to add an additional network to eth0.

    Because none of the other nodes in the cluster had that network configured they were not able to communicate with that node.

    That node was using the last network added to eth0 as its default network route for all traffic.

    So - Percona Cluster (tertiary party MySQL product) relies heavily on Galera Cluster (quaternary party) which in turn relies on WSRep (quinary! party) for the communications layer between the nodes. (From what I am told, this is actually fairly common practice.)

    Once I had all that teased apart and understood the error message more accurately, I found the following page to be VERY helpful:

    especially this section on joing a new node to the cluster - because rejoining an existing node works the same way:

    I tried using the network address in the running process list above ( - but that failed with a 'cluster not found' error.

    At this point I new that I needed to find which address one of the other nodes was running on.

    A quick ps aux | grep ^mysql on node 2 showed that it was participating on the address, not the address.

    Once I passed the correct network address value to node 1 for start up, it came up right away and began using rsync to catch itself up with the other nodes on the cluster.

    I did not have to specify the port 4567 because that is the default port for Percona Cluster.

    I think that's about it.

    Any questions?

    /David C.
    LVL 2

    Author Closing Comment

    I figured it out on my own. I thought that the suggestion to visit Percona's google group was excellent.  I had not thought of that.

    Featured Post

    Maximize Your Threat Intelligence Reporting

    Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

    Join & Write a Comment

    More Fun with XML and MySQL – Parsing Delimited String with a Single SQL Statement Are you ready for another of my SQL tidbits?  Hopefully so, as in this adventure, I will be covering a topic that comes up a lot which is parsing a comma (or other…
    This guide whil teach how to setup live replication (database mirroring) on 2 servers for backup or other purposes. In our example situation we have this network schema (see atachment). We need to replicate EVERY executed SQL query on server 1 to…
    This video discusses moving either the default database or any database to a new volume.
    Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

    734 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    19 Experts available now in Live!

    Get 1:1 Help Now