billmercer
asked on
Setting up SpamAssassin to run on Netware with GroupWise GWIA 6.0
I have GroupWise 6.0 running on a Netware 6.0 server.
I'd like to try setting up SpamAssassin to work with my GWIA, maybe via the
/GWIA/THIRD directory. I'm not looking for a commercial product such as Guinevere,
and I would rather not set up a separate email gateway unless that's the only way
Ideally I'd like to find a Netware-compatible implementation of SpamAssassin that
I could run right on the Netware server. Or (if that's not possible for some reason)
if I can run SpamAssassin on a separate Linux or Windows box which can access the
GroupWise folders over the network. This is not a high volume mail server, so I'm
not too worried about network performance.
I'm not a coder, but I can make minor tweaks if needed.
I'd like to try setting up SpamAssassin to work with my GWIA, maybe via the
/GWIA/THIRD directory. I'm not looking for a commercial product such as Guinevere,
and I would rather not set up a separate email gateway unless that's the only way
Ideally I'd like to find a Netware-compatible implementation of SpamAssassin that
I could run right on the Netware server. Or (if that's not possible for some reason)
if I can run SpamAssassin on a separate Linux or Windows box which can access the
GroupWise folders over the network. This is not a high volume mail server, so I'm
not too worried about network performance.
I'm not a coder, but I can make minor tweaks if needed.
PsiCop
Wow. This is the most interesting question to come along in awhile! I've been working on implementing SpamAssassin in the Solaris world, so lemme think about this a bit and get back to you.
ASKER
OK, I'm not in a rush for this.
I haven't forgotten, been diverted with a GroupWise issue, but its getting resolved now.
ASKER
Just to keep this question active, here's what I know about GroupWise's ability to interact with third-party apps.
The GWIA supports third party programs with the optional THIRD folder. If this folder is specified in the GWIA configuration, the GWIA will stick incoming internet email in one directory structure, then retrieve it from a different one. GroupWise then depends on the third party program to incoming message, process it in some way, and then push it to the out folder, where GWIA picks it back up and returns it to the normal GroupWise message flow.
The same process functions in reverse for outgoing messages.
My current system uses a third-party program that decodes attachments, virus-scans them, then sends them on if clean. It also does rudimentary filtering by subject and domain name.
It seems like SpamAssassin could easily be inserted into this message flow and flag or redirect high-scoring messages, but I have not been able to find anyone who has actually done this.
I am willing to run SpamAssassin on a linux box if that's what it takes, but ideally I'd like to run it right on the Netware server. I've considered also setting up a separate box to act as a mail relay and run SpamAssassin on that, but I'd much prefer to keep my existing ip and MX records.
The GWIA supports third party programs with the optional THIRD folder. If this folder is specified in the GWIA configuration, the GWIA will stick incoming internet email in one directory structure, then retrieve it from a different one. GroupWise then depends on the third party program to incoming message, process it in some way, and then push it to the out folder, where GWIA picks it back up and returns it to the normal GroupWise message flow.
The same process functions in reverse for outgoing messages.
My current system uses a third-party program that decodes attachments, virus-scans them, then sends them on if clean. It also does rudimentary filtering by subject and domain name.
It seems like SpamAssassin could easily be inserted into this message flow and flag or redirect high-scoring messages, but I have not been able to find anyone who has actually done this.
I am willing to run SpamAssassin on a linux box if that's what it takes, but ideally I'd like to run it right on the Netware server. I've considered also setting up a separate box to act as a mail relay and run SpamAssassin on that, but I'd much prefer to keep my existing ip and MX records.
Yes, your understanding of how the GroupWise GWIA interacts with 3rd Party apps is fundamentally correct. The "THIRD" directory basically sits between the GWIA (the Agent that messages translates between RFC-822/MIME format and the GroupWise internal LIN format) and the GroupWise SMTP daemon (which handles transmission and receipt of SMTP data).
I'm not sure that MIMEdefang/SpamAssassin in like HOW GroupWise runs that interface, tho.
I'm not sure that MIMEdefang/SpamAssassin in like HOW GroupWise runs that interface, tho.
ASKER
I know SpamAssassin can be used as a filter, accepting a message from STDIN, applying appropriate spammishness headers, and sending the result to STDOUT. Maybe this could be done with some sort of wrapper, either in Perl or maybe in NetBasic?
I can do a little programming/scripting, but I'm no guru.
I can do a little programming/scripting, but I'm no guru.
Whilst in the shower this morning, it finally hit me as to what I think the problem is with this idea. It boils down to 2 words:
Interprocess Communication
Now, I'm not a GroupWise developer. What I'm about to write is based on my knowledge of GroupWise, MIMEdefang, SpamAssassin, NetWare, NFAP, *NIX and sendmail. I *think* I've interpreted it all correctly, and this is my best estimate of the situation. Its possible I'm misinterpreted something about one or more of these things. Frankly, I think it get a *definitive* answer, we'd really need to sit down with someone responsible for the actual coding of the GWIA component of GroupWise.
Anyway, on to my thoughts on the issue....
When you use MIMEdefang (along with any other tool that runs under it, like SpamAssassin) in a sendmail environment, you're using the sendmail Milter interface. Its designed to use UNIX sockets to pass the message along between components of the system (sendmail daemon, MIMEdefang, SpamAssassin, sendmail router), and to let each component hand off to the next.
Sendmail is similar to the GroupWise GWIA (or the GWIA is similar to sendmail, seeing as how sendmail is much older than GWIA) in that message handling is split into two separate steps. The daemon, or MTA, accepts/generates TCP Port 25 connections and transfers the E-Mail, applying a limited number of tests (ACLs, RBLs, etc) while doing so. This is different than the "LOCAL" portion of sendmail that determines if the E-Mail is going to be delivered locally or relayed elsewhere, and queues the message appropriately (so that, for locally-destined E-Mail, it can be picked up by ProcMail or mail or whatever the Local Delivery Agent is that handles delivery to the local mailbox; or have its delivery information resolved/looked up and be placed in the proper queue for retransmission my the MTA portion of sendmail to the next relay point, if its not for local delivery). The Milter interface is used to interpose an intermediate step (or steps) between those two functions of sendmail.
GWIA has something similar. The daemon that accepts/generates connections, receives/transmits the E-Mail, and does do some basic filtering (mainly RBLs), and then the actual GWIA, which handles determining the route that the message needs to take within the GroupWise system (i.e. resolving that "Bob.Smith@thisdomain.com"
Anyway, in the GroupWise environment, the THIRD directory is its equivalent to the sendmail Milter interface. The trouble is, there's no communication between the processes handling the E-mails. The daemon portion of GWIA gets an E-Mail in and places it in the THIRD directory. It "remembers" the filename, so when it later scans that directory looking for outbound E-Mail, it knows not to try to transmit that one, but that's about all it does. The other half of the GWIA, the translator and resolver, is also scanning that directory, looking for messages that it knows it didn't put in there. Those must be INbound messages, so it knows to grab them and delete them from the directory. The same thing happens in reverse for outbound messages - the GWIA's translator/resolver portion dumps an outbound message in the THIRD directory and remembers the filename, but doesn't tell the daemon portion anything. The daemon knows a message is outbound because it doesn't remember putting that file in the THIRD directory.
So, the issue of integrating MIMEdefang (MD) w/SpamAssassin (SA) is that sure, you can NFS export that THIRD directory and make it available to a machine running MD/SA, but:
1) For outbound messages, you have no way to alert MD/SA that a message needs to be dealt with, or tell the daemon not to grab the message from the THIRD directory until MD/SA are done
2) For inbound messages, the daemon won't tell MD/SA that a message needs to be scanned, nor will the GWIA translator/resolver know to wait until a scan is done
The way other products take advantage of that directory is they run ON the NetWare server - and can lock the file they're working on (preventing the GroupWise components from getting it) - and they can work on it faster. MD/SA can take several seconds to scan a multi-MB file (for this reason, most users place a hard limit on the size of file it will scan), but even if its smaller, the overhead imposed by NFAP (NFS is a rather chatty protocol) can result in a significant (remember, a second is a fairly long time to a computer) delay. And I don't think NFAP will allow remote locking over NFS.
The result is that one component or the other of GWIA could come along and process the message before MD/SA had time to get it, analyze it, and make a determination about it.
So...t5hose are my thoughts. This would be a fascinating conversation to have during the Thursday night "Meet the Experts" session at BrainShare, where you get to sit down (well, stand around) and shoot the breeze with the Novell product developers.
I'd love for this to work - it'd be a neat hack. And I'm wondering if the queuing enhancements in GroupWise v7 will address the interprocess communication issues that I see as a roadblock to doing this with the current GroupWise (I also wonder if I'm right about those issues, but for the nonce I'll go with the idea that all my assumptions are correct). If I made any mistakes, I hope another Expert will pipe up - I've been working with GroupWise for about 8 years, but there's always more to learn.
Interprocess Communication
Now, I'm not a GroupWise developer. What I'm about to write is based on my knowledge of GroupWise, MIMEdefang, SpamAssassin, NetWare, NFAP, *NIX and sendmail. I *think* I've interpreted it all correctly, and this is my best estimate of the situation. Its possible I'm misinterpreted something about one or more of these things. Frankly, I think it get a *definitive* answer, we'd really need to sit down with someone responsible for the actual coding of the GWIA component of GroupWise.
Anyway, on to my thoughts on the issue....
When you use MIMEdefang (along with any other tool that runs under it, like SpamAssassin) in a sendmail environment, you're using the sendmail Milter interface. Its designed to use UNIX sockets to pass the message along between components of the system (sendmail daemon, MIMEdefang, SpamAssassin, sendmail router), and to let each component hand off to the next.
Sendmail is similar to the GroupWise GWIA (or the GWIA is similar to sendmail, seeing as how sendmail is much older than GWIA) in that message handling is split into two separate steps. The daemon, or MTA, accepts/generates TCP Port 25 connections and transfers the E-Mail, applying a limited number of tests (ACLs, RBLs, etc) while doing so. This is different than the "LOCAL" portion of sendmail that determines if the E-Mail is going to be delivered locally or relayed elsewhere, and queues the message appropriately (so that, for locally-destined E-Mail, it can be picked up by ProcMail or mail or whatever the Local Delivery Agent is that handles delivery to the local mailbox; or have its delivery information resolved/looked up and be placed in the proper queue for retransmission my the MTA portion of sendmail to the next relay point, if its not for local delivery). The Milter interface is used to interpose an intermediate step (or steps) between those two functions of sendmail.
GWIA has something similar. The daemon that accepts/generates connections, receives/transmits the E-Mail, and does do some basic filtering (mainly RBLs), and then the actual GWIA, which handles determining the route that the message needs to take within the GroupWise system (i.e. resolving that "Bob.Smith@thisdomain.com"
Anyway, in the GroupWise environment, the THIRD directory is its equivalent to the sendmail Milter interface. The trouble is, there's no communication between the processes handling the E-mails. The daemon portion of GWIA gets an E-Mail in and places it in the THIRD directory. It "remembers" the filename, so when it later scans that directory looking for outbound E-Mail, it knows not to try to transmit that one, but that's about all it does. The other half of the GWIA, the translator and resolver, is also scanning that directory, looking for messages that it knows it didn't put in there. Those must be INbound messages, so it knows to grab them and delete them from the directory. The same thing happens in reverse for outbound messages - the GWIA's translator/resolver portion dumps an outbound message in the THIRD directory and remembers the filename, but doesn't tell the daemon portion anything. The daemon knows a message is outbound because it doesn't remember putting that file in the THIRD directory.
So, the issue of integrating MIMEdefang (MD) w/SpamAssassin (SA) is that sure, you can NFS export that THIRD directory and make it available to a machine running MD/SA, but:
1) For outbound messages, you have no way to alert MD/SA that a message needs to be dealt with, or tell the daemon not to grab the message from the THIRD directory until MD/SA are done
2) For inbound messages, the daemon won't tell MD/SA that a message needs to be scanned, nor will the GWIA translator/resolver know to wait until a scan is done
The way other products take advantage of that directory is they run ON the NetWare server - and can lock the file they're working on (preventing the GroupWise components from getting it) - and they can work on it faster. MD/SA can take several seconds to scan a multi-MB file (for this reason, most users place a hard limit on the size of file it will scan), but even if its smaller, the overhead imposed by NFAP (NFS is a rather chatty protocol) can result in a significant (remember, a second is a fairly long time to a computer) delay. And I don't think NFAP will allow remote locking over NFS.
The result is that one component or the other of GWIA could come along and process the message before MD/SA had time to get it, analyze it, and make a determination about it.
So...t5hose are my thoughts. This would be a fascinating conversation to have during the Thursday night "Meet the Experts" session at BrainShare, where you get to sit down (well, stand around) and shoot the breeze with the Novell product developers.
I'd love for this to work - it'd be a neat hack. And I'm wondering if the queuing enhancements in GroupWise v7 will address the interprocess communication issues that I see as a roadblock to doing this with the current GroupWise (I also wonder if I'm right about those issues, but for the nonce I'll go with the idea that all my assumptions are correct). If I made any mistakes, I hope another Expert will pipe up - I've been working with GroupWise for about 8 years, but there's always more to learn.
ASKER
My experience with the GWIA suggests that this is not really a problem. When given the alternate location, the GWIA maintains separate SEND and RECEIVE folders in both the GWIA folder and the GWIA\THIRD folder. The third-party program actually moves the files from one SEND folder to the other, and from one RECEIVE folder to the other. The only thing that ever appears in the RECEIVE folder is the incoming email, so there's no need for GWIA to distinguish which messages it needs to process, it just processes everything that shows up in that subdir.
The THIRD folder acts like a side detour on the normal message flow roadway.
To illustrate, If I shut off my filter NLM but leave GWIA running, incoming internet messages still come in, but just start to pile up in the \GWIA\THIRD\RECEIVE folder. Outgoing messages passed to the GWIA accumulate in the \GWIA\SEND folder. To manually "force" incoming messages through, I simply drag them from \GWIA\THIRD\RECEIVE to \GWIA\RECEIVE, and they are then sucked up into the regular GroupWise message flow. The same thing happens in reverse for outgoing messages.
At one point when I was having problems with the filter NLM causing abends after a service pack, I was able to bypass the filter without having to reconfigure the GWIA settings, simply by running a batch file on my workstation that moved the files from one SEND folder to the other and vice versa. This worked perfectly.
That's the point at which I realized that SpamAssassin should be possible. As long as the file name doesn't change, the GWIA doesn't care what you do with the messages. You can even capture outgoing mail, edit it with Notepad, and then release the modified message, a facility which comes in handy every April 1st. :)
In fact, it seems to me it should be possible to daisy-chain a series of processes doing various things to your email by giving each process its own SEND and RECEIVE folders, and have messages hopscotching around getting stamped and filtered in various ways.
Also, the GWIA does provide a simple form of process communication in the form of the PULSE.TMP file, which is regenerated each time GWIA finishes an idle cycle. By comparing the date and time of the current PULSE.TMP with the system date and time, you could tell how much time you have left before the next activity cycle begins. You could even have a script delete the PULSE.TMP file, then wait for it to exist again before continuing.
It seems like all the pieces exist, and it should just be a matter of putting them together in the right order.
The THIRD folder acts like a side detour on the normal message flow roadway.
To illustrate, If I shut off my filter NLM but leave GWIA running, incoming internet messages still come in, but just start to pile up in the \GWIA\THIRD\RECEIVE folder. Outgoing messages passed to the GWIA accumulate in the \GWIA\SEND folder. To manually "force" incoming messages through, I simply drag them from \GWIA\THIRD\RECEIVE to \GWIA\RECEIVE, and they are then sucked up into the regular GroupWise message flow. The same thing happens in reverse for outgoing messages.
At one point when I was having problems with the filter NLM causing abends after a service pack, I was able to bypass the filter without having to reconfigure the GWIA settings, simply by running a batch file on my workstation that moved the files from one SEND folder to the other and vice versa. This worked perfectly.
That's the point at which I realized that SpamAssassin should be possible. As long as the file name doesn't change, the GWIA doesn't care what you do with the messages. You can even capture outgoing mail, edit it with Notepad, and then release the modified message, a facility which comes in handy every April 1st. :)
In fact, it seems to me it should be possible to daisy-chain a series of processes doing various things to your email by giving each process its own SEND and RECEIVE folders, and have messages hopscotching around getting stamped and filtered in various ways.
Also, the GWIA does provide a simple form of process communication in the form of the PULSE.TMP file, which is regenerated each time GWIA finishes an idle cycle. By comparing the date and time of the current PULSE.TMP with the system date and time, you could tell how much time you have left before the next activity cycle begins. You could even have a script delete the PULSE.TMP file, then wait for it to exist again before continuing.
It seems like all the pieces exist, and it should just be a matter of putting them together in the right order.
Hmmm....well, you might be right. You're definitely right about the message flow.
I think you'll have to hack MD to have it get stuff from the right plave and put it back into the proper subdir.
I dunno - I'll have to think about this some more.
I think you'll have to hack MD to have it get stuff from the right plave and put it back into the proper subdir.
I dunno - I'll have to think about this some more.
ASKER
I'm no Perl programmer, but I'm willing to try some simple path hacking. Has anyone run Mimedefang on Netware's Perl before?
ASKER
I've checked, and the version of Perl on my Netware server is 5.8.0, which seems to be higher than the minimum.
And I tried running the Makefile.pl as-is, and it actually did some stuff before croaking with
MakeMaker could not find "C:\perl\5.8.0\lib\NetWare
And I apparently need some modules I don't currently have, that will be interesting to try and track down.
PMRUS: "Thank you for calling PerlMods'R'Us! How may I help you?"
Me: "Er, I've got a Netware server..."
PMRS: <CLICK> BZZZZZZZZZZ....
Me: "Drat."
I'll be back...
And I tried running the Makefile.pl as-is, and it actually did some stuff before croaking with
MakeMaker could not find "C:\perl\5.8.0\lib\NetWare
And I apparently need some modules I don't currently have, that will be interesting to try and track down.
PMRUS: "Thank you for calling PerlMods'R'Us! How may I help you?"
Me: "Er, I've got a Netware server..."
PMRS: <CLICK> BZZZZZZZZZZ....
Me: "Drat."
I'll be back...
I'm thinking that you can't run the persistent thread-pool that MIMEdefang wants. That's coded for *NIX OSes.
ASKER
Couldn't I bypass MimeDefang entirely, and just run SpamAssassin on every file in the THIRD/RECIEVE folder, redirecting output to the GWIA/RECIEVE folder?
I nboted a Novell Forge project to do this exact thing. I do't have a URL handy right now.
ASKER
Yep, that was it.
I'm also thinking that with GroupWise v7, it will be easier to plug something like this in.
I'm also thinking that with GroupWise v7, it will be easier to plug something like this in.
ASKER
I've been thinking about this some more, and I'm wondering if I can run SpamAssassin on a linux box, and configure the GWIA to use exported NFS shares for messages? That would eliminate configuration-related problems due to running the netware perl.
I'm already using UNC paths in the gwia config, so that's not an obstacle. Any thoughts?
.
I'm already using UNC paths in the gwia config, so that's not an obstacle. Any thoughts?
.
Hmmmm.....
Yeah...NFS really isn't a good idea for this. There are significant file locking issues that come into play, and time-stamping is also difficult to keep consistent. And NFS is bandwidth PIG. Oink oink. You'd think it was NetBIOS.
Since GroupWise has been available for YEARS on Linux (long before the advent of OES), why not simply implement this on OES - and run the GWIA on the same server?
Yeah...NFS really isn't a good idea for this. There are significant file locking issues that come into play, and time-stamping is also difficult to keep consistent. And NFS is bandwidth PIG. Oink oink. You'd think it was NetBIOS.
Since GroupWise has been available for YEARS on Linux (long before the advent of OES), why not simply implement this on OES - and run the GWIA on the same server?
ASKER
'Cuz it's SBS 6.0...
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.