jdhoover123
asked on
Symantec Mail Security for SMTP 5.0 - Filter Hub Errors
We've just started using Symantec Mail Security for SMTP 5.0 on a Windows 2003 Server SP1 Standard Edition system. Our hard drive is quickly filling as every few minutes (or less) the svchost.exe program executes a "svchost.exe -k WinErr" and two files appear in "C:\WINDOWS\PCHEALTH\Error
This continues on and on until very little space is left on the hard drive, at which point smtp connections stop being accepted, and no mail comes through. Incidentally, if you telnet to the smtp port you get back a "432 Please try again later" message back at that time.
I've tried rebooting. I've tried turning of the "Error Reporting" in the "System Properties" window (right click My Computer, then Properties, then Advanced, then Error Reporting). That didn't seem to stop it either.
So, it sounds to me like the filter-hub application is crashing, but perhaps auto restarting, and quite regularly? I have seen a few (very few) "filter-hub.exe has crashed, send this to Microsoft" type pop ups, but not in several days.
I should also mention that CPU utilization reports at 100% for long stretches - sometimes 20 seconds, sometimes a minute or two. This is followed by a brief "rest" in the 40% range (which seems to be where it operates normally). The 100% utilization even occurs while connections are not being accepted. The conduit.exe application takes about 30 to 40 percent, but its the svchost.exe app that seems to peg the cpu utilization. Also, filter-hub.exe is often using 90% or more when the machine is reporting full utilization.
Any thoughts? Right now I'm manually running a batch file to delete those hdmp and mdmp files - ugh! I'm also in the process of building another server to see if it has the same problem. Lastly, I'll mention its running with 1.5 GB of memory and the OS is on a VMWare ESX 2.5 server with local hard drive storage.
This continues on and on until very little space is left on the hard drive, at which point smtp connections stop being accepted, and no mail comes through. Incidentally, if you telnet to the smtp port you get back a "432 Please try again later" message back at that time.
I've tried rebooting. I've tried turning of the "Error Reporting" in the "System Properties" window (right click My Computer, then Properties, then Advanced, then Error Reporting). That didn't seem to stop it either.
So, it sounds to me like the filter-hub application is crashing, but perhaps auto restarting, and quite regularly? I have seen a few (very few) "filter-hub.exe has crashed, send this to Microsoft" type pop ups, but not in several days.
I should also mention that CPU utilization reports at 100% for long stretches - sometimes 20 seconds, sometimes a minute or two. This is followed by a brief "rest" in the 40% range (which seems to be where it operates normally). The 100% utilization even occurs while connections are not being accepted. The conduit.exe application takes about 30 to 40 percent, but its the svchost.exe app that seems to peg the cpu utilization. Also, filter-hub.exe is often using 90% or more when the machine is reporting full utilization.
Any thoughts? Right now I'm manually running a batch file to delete those hdmp and mdmp files - ugh! I'm also in the process of building another server to see if it has the same problem. Lastly, I'll mention its running with 1.5 GB of memory and the OS is on a VMWare ESX 2.5 server with local hard drive storage.
ASKER
I went through the Event Viewer's Application log. The error started on 7/9/2006 at 10:49:54 AM. It had been working just fine until then - so for about the first 10 days in use. I'm wondering if a Windows Update would have been installed then, I'll have to dig further.
At that same time, a DrWatson error was logged, with Event ID 4097. It reads with the following description: "The application, C:\Program Files\Symantec\SMSSMTP\sca
I'm going to look in the Security and System logs to see if anything else occurred around that time.
At that same time, a DrWatson error was logged, with Event ID 4097. It reads with the following description: "The application, C:\Program Files\Symantec\SMSSMTP\sca
I'm going to look in the Security and System logs to see if anything else occurred around that time.
ASKER
System log is clogged with stuff too. Primarily and Event ID of 7031 from the Service Control Manager, with a message of "The SMS Filter Hub service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 0 milliseconds: Run the configured recovery program.".
ASKER
I'm getting close. If I disable the anti-virus portion, there is no crashing of the filter-hub.exe application. Mail flows very nicely, the machine's CPU utilization is not pegged (once in the past few minutes, and only for about 6 seconds). As I write this, though, I notice it is pegging for a long period - about 48 seconds. I'll keep an eye on that. It could be that we just have a lot of email that is flowing through. The filter-hub program is not the only one using CPU now, though - even though utilization is nearly 100% now all the time.
Darn... there it goes again... This time, the faulting module of filter-hub.exe is libspamhunter.dll.
Hmmmm... back to building the other server...
Darn... there it goes again... This time, the faulting module of filter-hub.exe is libspamhunter.dll.
Hmmmm... back to building the other server...
ASKER
I built the other server, installed the Symantec software, and right off the bat I started seeing the conduit.exe and filter-hub.exe start using 100% of the CPU. Keep in mind, the server is not even processing any mail yet!!! Its just got a couple of IP addresses assigned to it, but no mail servers even know about this server yet.
Thoughts? I'm going to switch to this server anyhow, because it has more hard drive space - and so far none of the hdmp and mdmp files are showing up.
Thoughts? I'm going to switch to this server anyhow, because it has more hard drive space - and so far none of the hdmp and mdmp files are showing up.
ASKER
All switched over, mail is flowing, nothing in the "C:\windows\PCHEALTH\Error
ASKER
Incidentally, the other server is still running - 100% utilization now and again, but nothing in the UserDumps folder now. Conduit.exe and filter-hub.exe are both the ones using the CPU, with each at 92 or 97 percent now and then. All this server has now is a random IP address on my internal network, so it shouldn't be doing anything at all. I did throw ethereal on the new servers, and I saw during some of the peaks there was some encrypted messaging going back and forth between a brightmail.com server and my server - presumably to make sure the software is registered properly, is my guess.
ASKER
It appears that svchost.exe uses a lot of CPU (presumably when writing the hdmp and mdmp files). Sometimes the Dr Watson executable starts up, and after svchost.exe pegs the CPU, filter-hub.exe does (I assume as it restarts). Symantec is having me run a diagnostic tool called SymBatchDiag2.78.exe. It is currently running on the original machine, and pegging the CPU while it does it. I had started it on the production machine, but just can't do that during the day, so I cancelled it. This diag tool takes between 5 and 30 minutes is what I've been told. Initial response from Symantec is that it is not compatible with VMWare.
ASKER
We've now installed this software on 4 different computers, by different manufacturers. Each has been a clean build. There is no other software on the systems, yet the problems continue. Usually after a couple of days, we'll see filter-hub.exe begin to crash continuously - every couple of minutes. It seems to restart itself, but sometimes it won't - and then mail does not go through and we reboot the computer, and it works for another day.
As of today, we've stopped using the product completely. I do with they would make a package that used a standard MTA, not a Symantec product. It would be ideal if it supported sendmail as Brightmail did.
Lastly, Symantec support had me run a diagnostic tool. It showed them nothing, and I've not heard from them since July 25. I guess I should change my question to "what other anti-spam software should I consider using".
As of today, we've stopped using the product completely. I do with they would make a package that used a standard MTA, not a Symantec product. It would be ideal if it supported sendmail as Brightmail did.
Lastly, Symantec support had me run a diagnostic tool. It showed them nothing, and I've not heard from them since July 25. I guess I should change my question to "what other anti-spam software should I consider using".
I hope this thread gets archived here, despite the lack of contribution from experts. I subscribed to the thread to see what would show up and can only say that I'm not too surprised at Symantec software causing a bogging-down of your server, although I have no remedy to suggest. I have heard very good things about the open-source antispam tools, like spamassassin and such, but that may not be an option in your case. Your way of describing events and steps taken can be very useful for other admins in a similar position. I suggest you post a Q in the Community Support area and ask them to award you the points!
Cheers
/RID
Cheers
/RID
ASKER
Thanks for the note.
We had previously used SpamAssassin, but its accuracy was pretty poor (compared to Brightmail, which has served us very well in the past). This all started when we simply wanted to renew our brigthmail subscription for our flawlessly running (for two years) Redhat Linux box. However, since we bought brightmail over two years ago, they've since been bought by Symantec. Brightmail support was alway great, but the minute it switched to Symantec we didn't even bother calling when we had problems (which the problems were very few). It was suggested that we upgrade to their new Mail Security for SMTP product. Right away I was concerned as I called support and was told to UINSTALL SENDMAIL. I could hardly contain my laughter. Apparently the new software has its own MTA. Well, we were not about to uninstall sendmail on our Linux box (sendmail is to Linux as water is to life in our opinion). So, we tried it on a Windows box, and another Windows box, and another and another.
I do hope the thread gets archived as well, since Google searches for this problem turned up nothing. I can't imagine how others are dealing with this as every single installation we've performed has died. I wonder if its the 100,000 emails per day that kills it - although our last install ran the longest (for about 4 days). It was a beast of a machine though (we thought throwing raw horespower at it would help) - and it only sat between 5 and 20 percent CPU utilization at most times - hardly ever topped out. But sure enough, we started having problems today, and a quick look at the Event Viewer showed filter-hub crashed every couple of minutes since 5-ish PM on Wednesday.
I'll be sure to update the thread as I hear back from my sales reps and support (whom I've also copied as of today).
We had previously used SpamAssassin, but its accuracy was pretty poor (compared to Brightmail, which has served us very well in the past). This all started when we simply wanted to renew our brigthmail subscription for our flawlessly running (for two years) Redhat Linux box. However, since we bought brightmail over two years ago, they've since been bought by Symantec. Brightmail support was alway great, but the minute it switched to Symantec we didn't even bother calling when we had problems (which the problems were very few). It was suggested that we upgrade to their new Mail Security for SMTP product. Right away I was concerned as I called support and was told to UINSTALL SENDMAIL. I could hardly contain my laughter. Apparently the new software has its own MTA. Well, we were not about to uninstall sendmail on our Linux box (sendmail is to Linux as water is to life in our opinion). So, we tried it on a Windows box, and another Windows box, and another and another.
I do hope the thread gets archived as well, since Google searches for this problem turned up nothing. I can't imagine how others are dealing with this as every single installation we've performed has died. I wonder if its the 100,000 emails per day that kills it - although our last install ran the longest (for about 4 days). It was a beast of a machine though (we thought throwing raw horespower at it would help) - and it only sat between 5 and 20 percent CPU utilization at most times - hardly ever topped out. But sure enough, we started having problems today, and a quick look at the Event Viewer showed filter-hub crashed every couple of minutes since 5-ish PM on Wednesday.
I'll be sure to update the thread as I hear back from my sales reps and support (whom I've also copied as of today).
This makes good reading... :)
/RID
/RID
ASKER
Hey, wow, a response from Symantec - and it only took a scathing email and CC's to everyone at my vendor, and a suggestion that I change to an alternate product.
Good Afternoon,
All data related to the filter hub issues submitted by our customers had been forwarded to development. After anlyzing the data, there was a filter hub issue identified by development that they are currently working on a patch for. While I do not have a specific ETA for this patch, it will be designed to fix issues where the filter hub service is crashing. Regardless of the environment, I would recommend applying the patch when it becomes available.
Best Regards,
Good Afternoon,
All data related to the filter hub issues submitted by our customers had been forwarded to development. After anlyzing the data, there was a filter hub issue identified by development that they are currently working on a patch for. While I do not have a specific ETA for this patch, it will be designed to fix issues where the filter hub service is crashing. Regardless of the environment, I would recommend applying the patch when it becomes available.
Best Regards,
ASKER
I do want to keep this open as we wait for Symantec to release a patch. Can we give it 2 more weeks?
Jon
Jon
Almost 3 weeks later :) Any news?
ASKER
I emailed Symantec for an update yesterday afternoon.
Any news here?
ASKER
I wish there was news to report, but my email of October 9 has gone unanswered. I'll send another off to them today. If this is a pain to keep open, feel free to close it.
Hi Jon, What a great post with alot of insight! I would like to be able to read .hdmp adn .mdmp files to locate such issues and also find viruses that happened to find their way onto my smssmtp server. Thanks for the documentatio!
ASKER
I asked our vendor, Dell, to get involved. They received a response last week. It is below... We'll be trying this in our lab in about 4 weeks, and I'll post again if this is still open.
Begin email from Symantec rep:
Technical support is researching several filterhub.exe issues. In fact, we just released a patch. Client is welcome to try this patch. In fact, I uploaded it for their use.
**** omitted FTP ****
In looking at the support case notes, I recommend the following:
Do not run SMSSMTP on a virtual system (Note from jdhoover123: we had tried with the same errors on both the Virtual and non-virtualized server). (In fact, none of the "Brightmail" software is supported in a virtual environment.) If client wants to run SMSMSTP 5.0 on Windows 2003, then it should be run on Windows 2003 SP1 only.
It is possible that the folks out at client ran into a new issue. If they apply the new patch and filterhub.exe keeps crashing, then I recommend they reopen their support case and ask that Level 2 support take a look at the issue. In order for Level 2 to have a look, client will need to provide debug logs/files when the issue occurs (Note from jdhoover123: we provided these with the original case). These logs/files will include the following:
filterhub and mta set to deug level logging (this is in the settings.log file if I remember correctly)
Create a full crash dump when the problem occurs.
A copy of the MTA folder.
Technical support is the best place obtain instructions on how to set debug logging.
Begin email from Symantec rep:
Technical support is researching several filterhub.exe issues. In fact, we just released a patch. Client is welcome to try this patch. In fact, I uploaded it for their use.
**** omitted FTP ****
In looking at the support case notes, I recommend the following:
Do not run SMSSMTP on a virtual system (Note from jdhoover123: we had tried with the same errors on both the Virtual and non-virtualized server). (In fact, none of the "Brightmail" software is supported in a virtual environment.) If client wants to run SMSMSTP 5.0 on Windows 2003, then it should be run on Windows 2003 SP1 only.
It is possible that the folks out at client ran into a new issue. If they apply the new patch and filterhub.exe keeps crashing, then I recommend they reopen their support case and ask that Level 2 support take a look at the issue. In order for Level 2 to have a look, client will need to provide debug logs/files when the issue occurs (Note from jdhoover123: we provided these with the original case). These logs/files will include the following:
filterhub and mta set to deug level logging (this is in the settings.log file if I remember correctly)
Create a full crash dump when the problem occurs.
A copy of the MTA folder.
Technical support is the best place obtain instructions on how to set debug logging.
ASKER
Lyndy333,
Not sure if I can help on the reading of the hdmp and mdmp files, but you might look at:
http://support.microsoft.com/?id=286350
Jon
Not sure if I can help on the reading of the hdmp and mdmp files, but you might look at:
http://support.microsoft.com/?id=286350
Jon
Thanks Jon, I will experiment...I have saved the .hdmp and .mdmp.
I have a mass mailer on the smssmtp server. Have yet to identify it. Does not show up in processes, can not use hijack this, can not use autoruns, Symantec anti-virus and smssmtp haven't located it...geezie...I am now using filemon and I hope to find it soon...Have a nice day and Thanks again...
I have a mass mailer on the smssmtp server. Have yet to identify it. Does not show up in processes, can not use hijack this, can not use autoruns, Symantec anti-virus and smssmtp haven't located it...geezie...I am now using filemon and I hope to find it soon...Have a nice day and Thanks again...
Any news here?
ASKER
Yes, we've been running a trial now for about 60 days, and while we do occasionally get filter-hub errors like before, its rather rare. Also, hardly a thing shows up in the Event Viewere, whereas originally it was being added to ever few minutes. The spikes to 100% CPU Usage have also been reduced significantly, if not altogether. Looks like the current patch certainly helps a lot, and we've just purchased our licensing (it arrived today in my Inbox). The patch level was 172b. Incidentally, it is working even on our VMWare ESX 3 Infrastructure server, though not officially supported.
Jon
Jon
So what we do with the question?
I'm just wondering where you got the patch from, as I have a similar problem with filter-hub.exe hogging CPU time and eventually causing SMTP connections to not respond after a couple of days or so meaning that a reboot of the server is required. I have been unable to find anything on the Symantec website yet and really do not want to start chasing their support as previous dealings with them have been very painful to say the least!! Excellent post by the way very helpful.
od_c....
Have you checked out the following post?
https://www.experts-exchange.com/questions/21928349/Symantec-Mail-Security-for-SMTP-5-0-Filter-Hub-Errors.html
Have you checked out the following post?
https://www.experts-exchange.com/questions/21928349/Symantec-Mail-Security-for-SMTP-5-0-Filter-Hub-Errors.html
ASKER
@od_c,
Send me an email at sms012907@1234.tv. I've got some contact info for you to get the patch pretty quickly. I just don't want to post the guy's phone and email at Symantec here.
Jon
Send me an email at sms012907@1234.tv. I've got some contact info for you to get the patch pretty quickly. I just don't want to post the guy's phone and email at Symantec here.
Jon
ASKER
@venabili,
If we can leave it here, and just show it "solved" with my comment from 1/18 - that will let people know to try the patch 172b from Symantec (and hopefully avoid reading the entire thread).
Jon
If we can leave it here, and just show it "solved" with my comment from 1/18 - that will let people know to try the patch 172b from Symantec (and hopefully avoid reading the entire thread).
Jon
Thanks jd,
Got patch, found it eventually on symantecs website will install it and see if it resolves the issue thanks for your help.
http://www.symantec.com/region/reg_eu/techsupp/enterprise/products/sym_mail_security/sym_mail_security_5.0_smtp/files.html
Got patch, found it eventually on symantecs website will install it and see if it resolves the issue thanks for your help.
http://www.symantec.com/region/reg_eu/techsupp/enterprise/products/sym_mail_security/sym_mail_security_5.0_smtp/files.html
I'm still having problems with this... The server is patched to 176, but the filter-hub.exe is crashing constantly.
This is driving me nuts! Apparently we're not the only ones having this problem.
Why won't Symantec just fix this!?
This is driving me nuts! Apparently we're not the only ones having this problem.
Why won't Symantec just fix this!?
I ran a debug on the mdmp file... I'm not an expert, but it looks like it's the kvolefio.dll that's causing the problems.. Any suggestions?
Hi Ric....
My issues resolved after I installed all patches and am running version 4.1.15.47....
My issues resolved after I installed all patches and am running version 4.1.15.47....
Did you go back to version 4? My problems is on SMS for SMTP 5.0.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
We had the same exact problem with version 5.01. It was resolved by adding memory (2GB total) to the server.
We're already running on 2GB of RAM :(
This software works real good when it's not having any "troubles". I have three servers running it. One fully patched w/2G of RAM and I still see conduit sucking 100% of the CPU. I also am running one on VMWare, and it runs fine, actually the best of all three. Sometimes I also get emails from them that some/all services have stopped, but usualy they are lies.
Dave
Dave
Yeah, the software is great.
We've now moved on from running SMS on Server2003 to the appliance box (8300).
The appliance is great! Highly recommended! :)
We've now moved on from running SMS on Server2003 to the appliance box (8300).
The appliance is great! Highly recommended! :)
ASKER
libdayzero.dll anyone? Perhaps the anti-virus part of the application? I think I'll search around for a way to turn that feature off.