14th April 2006
|
#1 (permalink)
|
Registered User (15) Welcome aboard!
Join Date: Mar 2006
Posts: 12
|
Mail relaying, spam and antivirus
Hi,
I'm looking to build some new mail relays to replace our aging sendmail set-up, and would like to consider an alternative MTA such as Postfix or Exim.
We need the following features:
Optional SMTP authentication (against LDAP), otherwise restrict relaying to local domains.
DNSBL
sender verification callbacks and greylisting
spam filtering
virus scanning
We currently use sendmail with SASL, milter-sender, milter-amavis, amavisd, SpamAssasin and ClamAV to acheive all this (minus LDAP), however, I'm not entirely happy with the reliability of this solution.
There seems to be a fair few options for open source software out there that can do this kind of stuff now, and a bewildering array of ways to put them together. My biggest priority is reliability, but clarity of configuration and overall performance are also important.
On the later point, I've seen various reports of some MTA's performing better than others, but nothing that takes into account the whole system including spam and AV filtering. Since Spam and AV filtering will be most of the work done by these servers, it is perhaps more important how efficiently the MTA interfaces to the Spam and AV filtering than the raw speed of the MTA.
So what combinations of software do you folks use and/or recommend?
Cheers,
Ben.
__________________
Registered User
|
|
|
14th April 2006
|
#2 (permalink)
|
Trusted User (569) Platinum User
Join Date: Mar 2004
Age: 24
Posts: 544
|
Exim + EximSA + ClamAV + SpamAssassin + MySQL (Config+MessageStore)
__________________
Cameron Gray
[Any views expressed on this forum are my own, and may not represent the views of any employer or organisation that I am connected with.]
__________________
Registered User
|
|
|
14th April 2006
|
#3 (permalink)
|
Trusted User (886) Platinum User
Join Date: Aug 2004
Location: Central London
Age: 22
Posts: 668
|
agreed.
__________________
Pierce Lynch [Moderator]
NetSolutionsUK Ltd - Flexible Business & Enterprise Solution Provider
(url-removed: need 20 posts)
[All views and opinions are that of my own, and not the company that I formally represent or the forum in which I moderate.]
__________________
Registered User
|
|
|
14th April 2006
|
#4 (permalink)
|
Join Date: Apr 2005
Location: Manchester
Posts: 213
|
I've been using Exim for years and wouldn't use anything else for a large-scale mail deployment.
If you've already got your accounts in LDAP (as implied) then you just need Exim, SpamAssassin and ClamAV. Exim includes the Exiscan-acl patch from version 4.5 so there's no need to mess about patching the source.
If you need any help with setting this up then this is a service Bashton Ltd can provide (link in sig).
__________________
Web Host - Certified Member
|
|
|
14th April 2006
|
#5 (permalink)
|
Trusted User (346) Platinum User
Join Date: Dec 2004
Age: 26
Posts: 336
|
Another one to look at is Mailscanner http://www.sng.ecs.soton.ac.uk/mailscanner/
Buying the book is the way to go if you want to implement it - the online docs seem a little messy to me. However iirc Mailscanner has its limitations in terms of the docs when coming to give users individual settings (iirc it's all there but not documented terribly well).
Recently I've been playing with Postfix + Amavisd-new + SpamAssassin + ClamAv which works nicely tied to OpenLDAP or MySQL.
Exim sounds like a good option though from what's been said above.
There is a need for an open source 'm0n0wall' style app that could bundle all this.
__________________
Registered User
|
|
|
14th April 2006
|
#6 (permalink)
|
Registered User (15) Welcome aboard!
Join Date: Mar 2006
Posts: 12
|
Quote:
|
Originally Posted by Cameron Gray
Exim + EximSA + ClamAV + SpamAssassin + MySQL (Config+MessageStore)
|
Do you mean that Exim is actually storing your spool and/or mailboxes in the MySQL database? I've not come across this feature before and don't seem to be able to find mention of it on Google or the Exim manual. Have you got any pointers to documentation? What are the advantages of doing this?
__________________
Registered User
|
|
|
14th April 2006
|
#7 (permalink)
|
Registered User (15) Welcome aboard!
Join Date: Mar 2006
Posts: 12
|
Quote:
|
Originally Posted by samb
I've been using Exim for years and wouldn't use anything else for a large-scale mail deployment.
If you've already got your accounts in LDAP (as implied) then you just need Exim, SpamAssassin and ClamAV. Exim includes the Exiscan-acl patch from version 4.5 so there's no need to mess about patching the source.
|
We don't have the accounts in LDAP yet but getting them there shouldn't be a problem. I would like to use LDAP as we've got experience using it for mail routing elsewhere and it can easily be reliplicated to every mail server, making the system more scalable and resilient.
We currently have scripts to generate static configuration files from a mysql database and copy these onto each mail server. We will be replacing these with a simpler script to synchronise the data in the master LDAP server to the MySQL database, with a view to phasing out the MySQL database when all the lecacy systems using it have been replaced.
__________________
Registered User
|
|
|
14th April 2006
|
#8 (permalink)
|
Join Date: Jul 2003
Location: London
Posts: 1,031
|
Exim has hooks for all this without needing extra patches or intermediary packages..
* Auth SMTP is built in - you'd need to link exim against LDAP libs and configure suitable queries into the authentication section
* DNSBL is built in - you'd need to add the relevant black lists you want to use
* sender verification callbacks and greylisting - can be configured, look at the greylistd Debian package
* spam filtering - with the exiscan-acl functionality direct hooks to spamassassin
* virus scanning - with the exiscan-acl functionality direct hooks to clamav and others
Quote:
|
We currently have scripts to generate static configuration files from a mysql database and copy these onto each mail server.
|
This is actually a very good idea for many reasons.
__________________
Paul Civati
Rack Sense Ltd
Views expressed are my own and not those of the company.
Company reg no. 05115163, VAT reg no. GB 912 5238 45
__________________
Web Host - Certified Member
|
|
|
14th April 2006
|
#9 (permalink)
|
Trusted User (569) Platinum User
Join Date: Mar 2004
Age: 24
Posts: 544
|
Quote:
|
Originally Posted by BenM
Do you mean that Exim is actually storing your spool and/or mailboxes in the MySQL database?
|
In another MySQL database, yes.
Quote:
|
Originally Posted by BenM
I've not come across this feature before and don't seem to be able to find mention of it on Google or the Exim manual.
|
You won't have done, I wrote the transport for it.
Quote:
|
Originally Posted by BenM
Have you got any pointers to documentation? What are the advantages of doing this?
|
Learn how routers and transports work and what they are for. The advantage from my point of view is that I bridgehead the POP/IMAP connections which a MySQL mailstore so apart from multiple people accessing the same row (header, content or attachment) it scales rather well.
Also included some rudimentary attachment minimisation on storage, i.e. if the attachments MD5 matches one already stored, just include a pointer rather than store the whole attachment again.
__________________
Cameron Gray
[Any views expressed on this forum are my own, and may not represent the views of any employer or organisation that I am connected with.]
__________________
Registered User
|
|
|
15th April 2006
|
#10 (permalink)
|
Join Date: Apr 2005
Location: Manchester
Posts: 213
|
Quote:
|
Originally Posted by Cameron Gray
Learn how routers and transports work and what they are for. The advantage from my point of view is that I bridgehead the POP/IMAP connections which a MySQL mailstore so apart from multiple people accessing the same row (header, content or attachment) it scales rather well.
Also included some rudimentary attachment minimisation on storage, i.e. if the attachments MD5 matches one already stored, just include a pointer rather than store the whole attachment again.
|
Interesting idea, although rather too far from KISS for my liking. When you say 'scales rather well' how many messages/second are we talking? I've set up small clusters (5 nodes) that can comfortably handle 50 messages/second (and probably more, that's as much as the spec required) using the more traditional Maildir on an NFS mount. I've not really done much in the way of writing large blobs to MySQL databases, so it'd be interesting to see how it compares.
In the event of data corruption I know I'd much rather be dealing with some missing files from a Maildir spool than missing parts of a MySQL database too..
All sounds very interesting though, if only as a technical exercise.
__________________
Web Host - Certified Member
|
|
|
15th April 2006
|
#11 (permalink)
|
Trusted User (569) Platinum User
Join Date: Mar 2004
Age: 24
Posts: 544
|
Maximum delivery throughput observed to local MySQL instance 150-200 messages/sec, maximum delivery throughput observed with remote MySQL 120 messages/sec.
The biggest throughput damage occurs when you move AV and SA away from the MTA box in my experience. SpamAssassin 3 isn't so bad, but the Clam Socket system is not great unless you heavily tune EximSA to limit the amount and cases in which the actual AV scanner is run.
__________________
Cameron Gray
[Any views expressed on this forum are my own, and may not represent the views of any employer or organisation that I am connected with.]
__________________
Registered User
|
|
|
17th April 2006
|
#12 (permalink)
|
Registered User (15) Welcome aboard!
Join Date: Mar 2006
Posts: 12
|
(copying static configuraiton files)
Quote:
|
Originally Posted by racksense
This is actually a very good idea for many reasons.
|
Apart from the KISS philosophy, do you have any in mind?
Unfortunately the author of our current scripts didn't believe in KISS, so it's either a complete rewrite or use LDAP.
As I see it, the pros of using LDAP are:
- replication is taken care of for you
- the data can be accessed directly by multiple types and instances of systems
What are the cons? Reliability?
__________________
Registered User
|
|
|
17th April 2006
|
#13 (permalink)
|
Join Date: Apr 2005
Location: Manchester
Posts: 213
|
I was referring to the storing of messages in MySQL when I spoke about KISS.
LDAP sounds to be a good fit for your scenario - make sure the database is replicated to each node and you'll have a nice setup.
Full consultancy is of course available 
__________________
Web Host - Certified Member
|
|
|
17th April 2006
|
#14 (permalink)
|
Registered User (15) Welcome aboard!
Join Date: Mar 2006
Posts: 12
|
Quote:
|
Originally Posted by Cameron Gray
Learn how routers and transports work and what they are for. The advantage from my point of view is that I bridgehead the POP/IMAP connections which a MySQL mailstore so apart from multiple people accessing the same row (header, content or attachment) it scales rather well.
Also included some rudimentary attachment minimisation on storage, i.e. if the attachments MD5 matches one already stored, just include a pointer rather than store the whole attachment again.
|
Quite a neat idea. I can see this is potentially quite scalable if you can build a fast enough mysql server, as you can have multiple mail servers feeding into the same database and multiple POP/IMAP servers reading from it. You could do mysql replication as well (although I'm not sure I'd want to, as our experiences with large replicated MySQL databases is they don't always replicate that reliably). Are the POP & IMAP servesr a complete custom job or based on existing ones?
How many man-hours went into building the system?
__________________
Registered User
|
|
|
17th April 2006
|
#15 (permalink)
|
Registered User (15) Welcome aboard!
Join Date: Mar 2006
Posts: 12
|
Quote:
|
Originally Posted by samb
I was referring to the storing of messages in MySQL when I spoke about KISS.
|
I realised that, but its always a worthwhile philosophy to keep in mind. (I just wish I could convince my colleagues of that. :-( )
Quote:
|
Originally Posted by samb
LDAP sounds to be a good fit for your scenario - make sure the database is replicated to each node and you'll have a nice setup.
Full consultancy is of course available 
|
Thanks for the offer Sam. We should be able to (and will need to) figure it all out ourselves given time, but time is a luxury we're short of at the moment so we might yet take you up on the offer if you have plenty of experience setting up this kind of system and tuning the anti-virus and anti-spam software.
__________________
Registered User
|
|
|
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
|
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is On
|
|
|
|
|