Redundant IMAP Servers using Debian

So, I have a new task given to me by my boss.  We have a Debian 6 server set up with Dovecot IMAP and Postfix for the MTA.  Right now, everything is configured on one physical server.

We now need to implement redundancy for this email server.  What I was thinking was setting up a second identical server by cloning it.  And in order for the two servers to access the same mail queues, I would point the mail queues to a third server that would simply be a ton of storage.

So that would require three physical machines: the two imap servers and the shared storage server.  

Basically, since having to set up redundant ANYTHING in Linux is new to me, I'm really just looking for any ideas on how to best do this.  Any suggestions?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

That sounds dangerous, having 2 MTA's accessing the same queue as if it were their own, not to mention anything else that needs to access it such as Dovecot or SpamAssassin etc.  Not saying it's impossible but off hand it strikes me as, well, potentially dangerous.

What part of that infrastructure are you trying to make redundant?  What exactly are you trying to achieve?  Any more environment related info is always helpful, in the meantime I'll brainstorm a solution.  If any other experts beat me to it then all good, I'm a bit distracted at this moment.
sedberg1Author Commented:
I'm trying to create a scenario so that if the first IMAP server goes down, a second one would be ready to go.  I figured I could simply create two MX records, and (with higher MX number).  And if the two IMAP servers shared the same queue, I wouldn't have to worry about replicating any mail queues.  I have no problem creating two user accounts for any new email addresses so that imap1 and imap2 would have the same list of user accounts.  The reason I thought about the shared storage for mail queues would be to not worry about replicating mail queues.
sedberg1Author Commented:
We're a pure Windows environment (aside from the Debian server), so the Linux is basically isolated from everything else.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

What you need is a common storage and configure postfix to use Maildir/ format.
The benefit of Maildir is that each message is stored as a single ole versus a mbox format where each message is appended into a single file.
The storage has to be an NFS share.
Not too familiar with Debian on whether you have an option such Andrew fs which could handle the replication if a single storage server is an issue. You use AFS to replicate data I.e. emails that are delivered or removed from one are then copied to or deleted from the other.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sedberg1Author Commented:
So, basically, the Maildir directories I'm using now would be transferred over to the shared storage server instead.  I'd mount an NFS export from the shared storage server onto imap1 and set up users to use that NFS export as their home directory, which would then make their Maildir folders exist on that export?  Or, instead of NFS, I might be able to use AFS.  Have to look into AFS since this is the first I"m hearing about that.

What about having both imap1 and imap2 use that NFS export?  Any issues with this?
There are no issues since each message is stored as an individual file and the naming convention for Maildir is such that there is no way that they will have the same name.
The filename is made up of unixtimestamp.PID.hostname hostname is that of the mail server.
The only issue is that with a common storage your single point of failure is the storage server.
Having a redundant higher class server for storage (dual power supply, dual nics, RAIDEd os,storage) while the mailserver, head units could be of a lower range.
Gmail does this.
sedberg1Author Commented:
How would the email clients work with this?  For DNS,:

mx priority 10
mx priority 20

That's no problem, but when the client connects to get email, what would he connect to in order to not have any downtime or not reconfigure their email clients?
You could do it that way or define
While the true names are imap1 and imap2, you would then create a separate that points to both IPs and this is the hostname to which the iMap clients will connect to. Similarly if you want to provide pop, or the secure versions of the two protocols.
Better off to have a production IP address that can be bound to the operating mail server.
When you wish to fail over you unbind from the box you want offline ( if it's still there) and then bind the IP address to the new failover box.
The problem with a hot stand by is that unless it is monitored, one would find out that it is not working exactly at the time when it is needed.
If the option is there to have both running at the same time and processing the incoming emails as well as responding to user requests.
I'm more familiar with centos/redhat cluster but that might be a consideration to cluster the two head units and have two separate services one is a floating ip that will deal with the iMap service and the other will deal with the incoming mail. You might be able to configure so long as both servers are up the active node will have only one of the functions.
Outside the clustering, the common storage NSF share makes the setup scalable I.e. if the influx increases it is straight forward to add another head unit to take the additional load of handling email processing.

Your question is fairly specific on what you are looking for.
Depending on your available resources, you could setup a pair of clustered servers that will distribute the requests across a set of servers. I.e. the two front will act as load balancers. Etc.................
sedberg1Author Commented:
Well, to throw a wrench into the mix, management wants to put the backup IMAP server at our satellite office.  There's a permanent connection to that satellite via MPLS which has over the past three years gone down only once and for less than two hours on a weekend.  

I liked the idea of doing the IP redirect once IMAP1 went down.  We have monitoring in place, so I'd be able to fail that over.  But now with putting IMAP2 in the second location, I can no longer share storage between these servers and will have to use rsync or imapsync I suppose to get the mailboxes the same on both servers.

I don't know how to handle connecting the clients to the servers now though.  If I set up IMAP.MYDOMAIN.COM to use round-robin DNS to connect to IMAP1 and IMAP2, my clients could be connecting to either the IMAP1 server at HQ or IMAP2 at the satellite at any given time.  But due to bandwidth issues, I want to force them to connect to IMAP1 and then only to IMAP2 if IMAP1 is down.  HQ has 50Mbps down/20 Mbps up.  Satellite has two bonded T1s.
In this scenario, rsync, or Andrew FS might be what is needed.

The issue deals with bandwidth consumption for the replication of data the other major point is how the "message deletions" are handled.
imap1 to imap2 and impa2 to imap1.
When a message is deleted on one, it needs to be deleted on the other.

In the case you outline, you would need to use a script that will check the status of the primary IMAP server to make sure it is listening while at the same time monitors the secondary and generates alerts if the secondary is not accessible (this is to make sure the secondary is there in the event the primary fails).
Once the primary fails, the script will generate a DNS update event to add IN A Ip_of_secondary
while at the same time issue a delete for IN A IP_of_primary

This requires that zone can be dynamically update (from a preauthorized location)
One way to do this is to have the public DNS configured as secondary while your internal DNS servers are the primary.

The other issue is that the testing process can not be only on the HQ side in the event HQ looses its connection, there will be nothing that would update the DNS.


another option if your have SANs, the SANs can replicate data at the lower media level.
Have a look at NGINX, which is an IMAP load balancer.
It's also a web server and reverse proxy as well,
but it might help you out.

sedberg1Author Commented:
So, I started testing with rsync.  I set up a test domain on both imap1 and imap2.  And set up a test user named testuser1.  (I'm not one for creativity unfortunately)  When I ran this:

rsync -rve ssh /home/testingimap/homes/testuser1 imap2:/home/testingimap/homes/testuser1

New messages were synched up fine from imap1 to imap2.  But then I deleted the messages in the testuser1 inbox on imap2 and reran the script.  The messages are still there in the imap2 inbox.  How can I get the mailboxes to actually duplicate?
Usually you have to run rsync on both to get each to push the added messages.
This why the deleted messages are a difficult thing that needs to be managed.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.