Solved

Redundant IMAP Servers using Debian

Posted on 2012-03-27
16
607 Views
Last Modified: 2012-04-13
So, I have a new task given to me by my boss.  We have a Debian 6 server set up with Dovecot IMAP and Postfix for the MTA.  Right now, everything is configured on one physical server.

We now need to implement redundancy for this email server.  What I was thinking was setting up a second identical server by cloning it.  And in order for the two servers to access the same mail queues, I would point the mail queues to a third server that would simply be a ton of storage.

So that would require three physical machines: the two imap servers and the shared storage server.  

Basically, since having to set up redundant ANYTHING in Linux is new to me, I'm really just looking for any ideas on how to best do this.  Any suggestions?
0
Comment
Question by:sedberg1
  • 6
  • 6
  • 3
  • +1
16 Comments
 
LVL 21

Expert Comment

by:Papertrip
Comment Utility
That sounds dangerous, having 2 MTA's accessing the same queue as if it were their own, not to mention anything else that needs to access it such as Dovecot or SpamAssassin etc.  Not saying it's impossible but off hand it strikes me as, well, potentially dangerous.

What part of that infrastructure are you trying to make redundant?  What exactly are you trying to achieve?  Any more environment related info is always helpful, in the meantime I'll brainstorm a solution.  If any other experts beat me to it then all good, I'm a bit distracted at this moment.
0
 

Author Comment

by:sedberg1
Comment Utility
I'm trying to create a scenario so that if the first IMAP server goes down, a second one would be ready to go.  I figured I could simply create two MX records, imap1.mydomain.com and imap2.mydomain.com (with higher MX number).  And if the two IMAP servers shared the same queue, I wouldn't have to worry about replicating any mail queues.  I have no problem creating two user accounts for any new email addresses so that imap1 and imap2 would have the same list of user accounts.  The reason I thought about the shared storage for mail queues would be to not worry about replicating mail queues.
0
 

Author Comment

by:sedberg1
Comment Utility
We're a pure Windows environment (aside from the Debian server), so the Linux is basically isolated from everything else.
0
 
LVL 76

Accepted Solution

by:
arnold earned 375 total points
Comment Utility
What you need is a common storage and configure postfix to use Maildir/ format.
http://wiki.dovecot.org/MailboxFormat/Maildir
http://www.postfix.org/postconf.5.html
The benefit of Maildir is that each message is stored as a single ole versus a mbox format where each message is appended into a single file.
The storage has to be an NFS share.
Not too familiar with Debian on whether you have an option such Andrew fs which could handle the replication if a single storage server is an issue. You use AFS to replicate data I.e. emails that are delivered or removed from one are then copied to or deleted from the other.
0
 

Author Comment

by:sedberg1
Comment Utility
So, basically, the Maildir directories I'm using now would be transferred over to the shared storage server instead.  I'd mount an NFS export from the shared storage server onto imap1 and set up users to use that NFS export as their home directory, which would then make their Maildir folders exist on that export?  Or, instead of NFS, I might be able to use AFS.  Have to look into AFS since this is the first I"m hearing about that.

What about having both imap1 and imap2 use that NFS export?  Any issues with this?
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
There are no issues since each message is stored as an individual file and the naming convention for Maildir is such that there is no way that they will have the same name.
The filename is made up of unixtimestamp.PID.hostname hostname is that of the mail server.
The only issue is that with a common storage your single point of failure is the storage server.
Having a redundant higher class server for storage (dual power supply, dual nics, RAIDEd os,storage) while the mailserver, head units could be of a lower range.
0
 
LVL 19

Expert Comment

by:bevhost
Comment Utility
Gmail does this.
0
 

Author Comment

by:sedberg1
Comment Utility
How would the email clients work with this?  For DNS,:

mx priority 10 imap1.mydomain.com
mx priority 20 imap2.mydomain.com

That's no problem, but when the client connects to get email, what would he connect to in order to not have any downtime or not reconfigure their email clients?
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 76

Assisted Solution

by:arnold
arnold earned 375 total points
Comment Utility
You could do it that way or define
While the true names are imap1 and imap2, you would then create a separate imap.domain.com that points to both IPs and this is the hostname to which the iMap clients will connect to. Similarly if you want to provide pop, or the secure versions of the two protocols.
0
 
LVL 19

Expert Comment

by:bevhost
Comment Utility
Better off to have a production IP address that can be bound to the operating mail server.
When you wish to fail over you unbind from the box you want offline ( if it's still there) and then bind the IP address to the new failover box.
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
The problem with a hot stand by is that unless it is monitored, one would find out that it is not working exactly at the time when it is needed.
If the option is there to have both running at the same time and processing the incoming emails as well as responding to user requests.
I'm more familiar with centos/redhat cluster but that might be a consideration to cluster the two head units and have two separate services one is a floating ip that will deal with the iMap service and the other will deal with the incoming mail. You might be able to configure so long as both servers are up the active node will have only one of the functions.
Outside the clustering, the common storage NSF share makes the setup scalable I.e. if the influx increases it is straight forward to add another head unit to take the additional load of handling email processing.

Your question is fairly specific on what you are looking for.
Depending on your available resources, you could setup a pair of clustered servers that will distribute the requests across a set of servers. I.e. the two front will act as load balancers. Etc.................
0
 

Author Comment

by:sedberg1
Comment Utility
Well, to throw a wrench into the mix, management wants to put the backup IMAP server at our satellite office.  There's a permanent connection to that satellite via MPLS which has over the past three years gone down only once and for less than two hours on a weekend.  

I liked the idea of doing the IP redirect once IMAP1 went down.  We have monitoring in place, so I'd be able to fail that over.  But now with putting IMAP2 in the second location, I can no longer share storage between these servers and will have to use rsync or imapsync I suppose to get the mailboxes the same on both servers.

I don't know how to handle connecting the clients to the servers now though.  If I set up IMAP.MYDOMAIN.COM to use round-robin DNS to connect to IMAP1 and IMAP2, my clients could be connecting to either the IMAP1 server at HQ or IMAP2 at the satellite at any given time.  But due to bandwidth issues, I want to force them to connect to IMAP1 and then only to IMAP2 if IMAP1 is down.  HQ has 50Mbps down/20 Mbps up.  Satellite has two bonded T1s.
0
 
LVL 76

Assisted Solution

by:arnold
arnold earned 375 total points
Comment Utility
In this scenario, rsync, or Andrew FS might be what is needed.

The issue deals with bandwidth consumption for the replication of data the other major point is how the "message deletions" are handled.
imap1 to imap2 and impa2 to imap1.
When a message is deleted on one, it needs to be deleted on the other.

In the case you outline, you would need to use a script that will check the status of the primary IMAP server to make sure it is listening while at the same time monitors the secondary and generates alerts if the secondary is not accessible (this is to make sure the secondary is there in the event the primary fails).
Once the primary fails, the script will generate a DNS update event to add imap.mydomain.com IN A Ip_of_secondary
while at the same time issue a delete for
imap.mydomain.com IN A IP_of_primary

This requires that zone mydomain.com can be dynamically update (from a preauthorized location)
One way to do this is to have the public DNS configured as secondary while your internal DNS servers are the primary.

The other issue is that the testing process can not be only on the HQ side in the event HQ looses its connection, there will be nothing that would update the DNS.

etc.

another option if your have SANs, the SANs can replicate data at the lower media level.
0
 
LVL 19

Assisted Solution

by:bevhost
bevhost earned 125 total points
Comment Utility
Have a look at NGINX, which is an IMAP load balancer.
It's also a web server and reverse proxy as well,
but it might help you out.

see
http://wiki.nginx.org/MailCoreModule
0
 

Author Comment

by:sedberg1
Comment Utility
So, I started testing with rsync.  I set up a test domain testingimap.com on both imap1 and imap2.  And set up a test user named testuser1.  (I'm not one for creativity unfortunately)  When I ran this:

rsync -rve ssh /home/testingimap/homes/testuser1 imap2:/home/testingimap/homes/testuser1

New messages were synched up fine from imap1 to imap2.  But then I deleted the messages in the testuser1 inbox on imap2 and reran the script.  The messages are still there in the imap2 inbox.  How can I get the mailboxes to actually duplicate?
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
Usually you have to run rsync on both to get each to push the added messages.
http://abbeyworkshop.com/howto/unix/nix_rsync/index.html
This why the deleted messages are a difficult thing that needs to be managed.
http://abbeyworkshop.com/howto/unix/nix_rsync/index.html

http://www.linuxquestions.org/linux/answers/Networking/Using_rsync_to_mirror_data_between_servers
0

Featured Post

Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

Join & Write a Comment

Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now