[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

How to create a scalable email solution

Posted on 2004-11-04
11
Medium Priority
?
365 Views
Last Modified: 2010-03-18
In the article http://www.horde.org/papers/Scalable_webmail_HOWTO.html, there is a brief description for creating a scalable webmail solution.  I am looking for a very specific implementation of this that at the moment does not need to have webmail as part of it although if it does it would be nice.  Also (sorry to complicate this but the solution will in the end entail having a scalable email AND file storage system).

Currently
A single machine hosts email and file storage for users.  By creating a user, he/she can immediately send/receive emails as well as put/get files.  The day is fast approaching where this machine will no longer suffice.  We will most likely need more than one email server and more than one file server.  

Planning for the future
We are considering a solution involving an LDAP server to centralize the following:
1. user creation (by registering a user in the LDAP server he/she is immediately registered in the email and file storage system)
2. sending and receiving email
3. puting and getting files(forget this entirely if it can be completely removed from the email solution)

How would such a LDAP central system work?
1. How do you create a user such that by registering to the LDAP server your email account is created on serverx.mail.xyz and your file storage is created on server23.file.xzy?  Consider the following:
  a. authentication
  b. load balancing(create email account on a server that is available)
  c. load balancing(create file storage account on a server that is available)

2. How do you make it so when a user request to send email is authenticated by the system and sent out?
using the maildrop attribute and mailacceptinggeneralid attribute
3. How do you make it so when an email is sent to the user it is forwarded to the correct mail server?
using the maildrop attribute and mailacceptinggeneralid attribute

4. How do you make it so when a user request to retrieve email is forwarded to the correct mail server?

Please answer with very specific instructions.   Articles, books and other resources are highly welcome.
0
Comment
Question by:bisonfur37
  • 5
  • 3
  • 3
11 Comments
 
LVL 36

Expert Comment

by:grblades
ID: 12502717
Hi bisonfur37,
Personally the way I would do it is to have a single machine holding the IMAP store with a hugh RAID5 disk array.
Another server(2) as the file server.
Multiple email servers for the transfering of inbound and outbound email with virus scanning etc...

You could have a central LDAP database as the central authentication server. You could have some other servers as backups which will reduce the load on the central server. LDAP has built in support for slave servers.
0
 
LVL 3

Expert Comment

by:mac_
ID: 12503839
I've done a similar setup using MySQL as a backend (and using the MySQL replication functions for better scalability and reliability), and a NFS shared storage space (it was a NetApp, but other solutions would do).

There is no easy way to do that kind of things without a rock-solid shared storage space anyways.

A good starting point (for the mail part at least) is there :
- Postfix + Courier-IMAP : http://kirb.insanegenius.net/postfix.html. They don't have to run on the same server.
- Webmail : you'd better use a webmail system that acts as an IMAP client, so it's way less worries and it's quite independent (and it can run on another server as well if needed one day)

For the storage part, all depends on the protocols you're using (SMB, FTP, ...?). It should be relatively easy to find a daemon that can use MySQL as an authentication resource anyways.

The choice of MySQL over LDAP is simply because it's easier to find people who can use MySQL than people who can use LDAP... And because it's easier to run stats, massive updates or complex queries with a SQL DB (that's what I think anyways). There is no big difference in speed or in scalability in the 2 solutions.
0
 
LVL 36

Expert Comment

by:grblades
ID: 12504714
I agree with mac_ with regard to using Postfix nd Courier-IMAP. However I would still use LDAP if you intend to run the Samba windows file sharing software since it will integrate with LDAP but not SQL.
0
Fill in the form and get your FREE NFR key NOW!

Veeam is happy to provide a FREE NFR server license to certified engineers, trainers, and bloggers.  It allows for the non‑production use of Veeam Agent for Microsoft Windows. This license is valid for five workstations and two servers.

 
LVL 2

Author Comment

by:bisonfur37
ID: 12569075
Thank you for the responses.  How would someone recommend creating a group of email servers?  Have a single 'master email server' that forwards requests to email1...emailn servers or just a group of email1...emailn servers.  A huge RAID array just won't cut it.  At some point one computer will not do it so I need to know what people THAT HAVE ALREADY DONE THIS recommend.  Basically, I need to know how people have built scalable email server clusters.

Thanks
0
 
LVL 36

Expert Comment

by:grblades
ID: 12569206
How many users are you thinking of?

Normally you would have one or more servers accepting mail and performing address validation, spam filtering and virus scanning. If you have more than 1 machine you can either have them all listed in the DNS MX records so they get chosen randomly or you can use a load balancer.

Assuming your domain is company.com you can have multiple servers accepting mail for server1.company.com, server2.company.com, server3.company.com. On the mail servers accepting email there is an alias file so it redirects mail for particular users onto the correct destination server.

Each user is configured on a particular server so that way you split the load of users accessing email across servers.
Email being received is balanced between one or more servers dedicated for that purpose.
You have one or more (as a backup)  LDAP servers for user authentication and to store the alias information about who is on which server.

This will give you a fully scalable system even down to the authentication servers which will be lightly utilised compared to the other servers.
0
 
LVL 3

Expert Comment

by:mac_
ID: 12569209
No master server, that's a scalability problem as well as a SPOF (Single Point Of Failure).

SMTP offers a built-in backup/failover solution at the DNS level, so you can easily use it.

On the IMAP/POP side, you can use a simple Round-Robin DNS system, or go for a load-balancer depending on your real needs and the money you have :)
0
 
LVL 2

Author Comment

by:bisonfur37
ID: 12670261
grblades and mac,

Thank you both for the response.  I would like to get a bit more information from you before splitting points right down the middle.  

1.  Imagine we have three Postfix-SMTP servers in use and chose to have the traffic balanced through the DNS MX records.  Would the setup be as below?  Can you assign an MX0 value to multiple entries?  Otherwise, how would you do it to spread the traffic EVENLY among the servers?  As far as I am concerned each hostname needs to have a different MX value but these are not the same hostnames although they would be serving the same domain.
@                        A         255.2555.255.255
@                        MX0      server1.domain.com
@                        MX0     server2.domain.com
@                        MX0     server3.domain.com

2. Assuming that 1. works as needed, and that each server does an LDAP look up for incoming email so they are routed to the right server, how are the outgoing emails managed?

3. How do we setup the the Webmail interface?
What I am trying to get at is the following:
Imagine we have an apache server running squirrelmail for webmail.  Squirrelmail allows you to have only ONE entry for the SMTP and IMAP servers.  So actually this raises two questions.
a. Is there a scheme for IMAP that functions like SMTP?
b. This will most likely have to be an addition to Squirrelmail, but do you see what I mention about the single entry for SMTP/IMAP configuration as a problem?  Do you know of any other webmail clients that would be more flexible?

Thank you all for your time.
0
 
LVL 3

Assisted Solution

by:mac_
mac_ earned 750 total points
ID: 12670369
1. I would do :

     A     server1.domain.com.
     A     server2.domain.com.
     A     server3.domain.com.
     MX 10 server1.domain.com.
     MX 10 server1.domain.com.
     MX 10 server1.domain.com.

(the round-robin on the A record is slightly overkill, but I would definitely not use 255.255.255.255)

2. Usually outgoing e-mails are less of a problem because the ACL are much easier to deal with (quite often a range of IP) and because there is no local shared storage, so it's just a matter of stacking enough boxes and playing round-robin to share the load (for example). More complex settings are always possible and sometimes needed.

3. a. - DNS round-robin is a cheap way to go, but for your point of view a Webmail is just like a regular IMAP client (except that you provide the interface).
b. It's not an issue. I assume your main goal is to be flexible for regular IMAP client, which are usually not capable of anything really clever in terms of choosing the right server. If your solution works for Eudora/Mozilla/Whatever, then it will work for your webmail as well.
0
 
LVL 36

Accepted Solution

by:
grblades earned 750 total points
ID: 12672883
1. Agree with mac_

2. You can have a single server designed to accept outgoing mail from internal users or you can use the server which holds the users mailbox accept mail from them. Outgoing mails should be easy by comparison.

3a no but you can work around it :)
3b You can install multiple copies of squirrelmail on a machine as it is just PHP software you install in a particular directory. All you need to do is have multiple installations with each user using a particular one for their account.
You could have a front screen which first asks them for their username and then looks up in LDAP to find there server and then redirects the browser to the installation of squirrelmail configured for their server.
It may sound messy but you could have one central installation of squirrelmail and then in the other directories have lots of symbolic links to the master files. Only the configuration file will be individual for each installation. This will make upgrades easier etc...
0
 
LVL 2

Author Comment

by:bisonfur37
ID: 12720949
Sorry about the 255.255.255.255 entry for the Aname.  I meat to throw there any valid IP.  
As a matter of fact, what do you mean by "the round-robin on the A record is slightly overkill."
Also, you posted the following DNS records

     A     server1.domain.com.
     A     server2.domain.com.
     A     server3.domain.com.
     MX 10 server1.domain.com.
     MX 10 server1.domain.com.
     MX 10 server1.domain.com.

Wouldn't the MX records be server1 through server3?
0
 
LVL 36

Expert Comment

by:grblades
ID: 12721096
Yes the MX should be server1 through server3.
0

Featured Post

NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Look below the covers at a subform control , and the form that is inside it. Explore properties and see how easy it is to aggregate, get statistics, and synchronize results for your data. A Microsoft Access subform is used to show relevant calcul…

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question