Solved

Symantec Mail Security for SMTP 5.0 - Filter Hub Errors

Posted on 2006-07-22
41
8,726 Views
Last Modified: 2012-05-05
We've just started using Symantec Mail Security for SMTP 5.0 on a Windows 2003 Server SP1 Standard Edition system. Our hard drive is quickly filling as every few minutes (or less) the svchost.exe program executes a "svchost.exe -k WinErr" and two files appear in "C:\WINDOWS\PCHEALTH\ErrorRep\UserDumps" with names like "filter-hub.exe.20060722-154150-00.mdmp" (usually about 17 MB) and "filter-hub.exe.20060722-154150-00.hdmp" (usually about 115 MB - seemingly the size in memory of Symantec's filter-hub.exe program).

This continues on and on until very little space is left on the hard drive, at which point smtp connections stop being accepted, and no mail comes through. Incidentally, if you telnet to the smtp port you get back a "432 Please try again later" message back at that time.

I've tried rebooting. I've tried turning of the "Error Reporting" in the "System Properties" window (right click My Computer, then Properties, then Advanced, then Error Reporting). That didn't seem to stop it either.

So, it sounds to me like the filter-hub application is crashing, but perhaps auto restarting, and quite regularly? I have seen a few (very few) "filter-hub.exe has crashed, send this to Microsoft" type pop ups, but not in several days.

I should also mention that CPU utilization reports at 100% for long stretches - sometimes 20 seconds, sometimes a minute or two. This is followed by a brief "rest" in the 40% range (which seems to be where it operates normally). The 100% utilization even occurs while connections are not being accepted. The conduit.exe application takes about 30 to 40 percent, but its the svchost.exe app that seems to peg the cpu utilization. Also, filter-hub.exe is often using 90% or more when the machine is reporting full utilization.

Any thoughts? Right now I'm manually running a batch file to delete those hdmp and mdmp files - ugh! I'm also in the process of building another server to see if it has the same problem. Lastly, I'll mention its running with 1.5 GB of memory and the OS is on a VMWare ESX 2.5 server with local hard drive storage.
0
Comment
Question by:jdhoover123
  • 19
  • 5
  • 4
  • +6
41 Comments
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160561
Also noticed in the Event Viewer (Application) in Windows, an Error with event ID of 1000 happening all the time, with the text "Faulting application filter-hub.exe, version 5.0.0.1, faulting module libdayzero.dll, version 5.0.0.1, fault address 0x00024830.". Then there is an Information event with Event ID 512, and a description of "INFO: 'Symantec SMS Filter Hub started.'."

libdayzero.dll anyone? Perhaps the anti-virus part of the application? I think I'll search around for a way to turn that feature off.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160605
I went through the Event Viewer's Application log. The error started on 7/9/2006 at 10:49:54 AM. It had been working just fine until then - so for about the first 10 days in use. I'm wondering if a Windows Update would have been installed then, I'll have to dig further.

At that same time, a DrWatson error was logged, with Event ID 4097. It reads with the following description: "The application, C:\Program Files\Symantec\SMSSMTP\scanner\bin\filter-hub.exe, generated an application error The error occurred on 07/09/2006 @ 10:49:54.653 The exception generated was c0000005 at address 00418770 (filter_hub)".

I'm going to look in the Security and System logs to see if anything else occurred around that time.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160630
System log is clogged with stuff too. Primarily and Event ID of 7031 from the Service Control Manager, with a message of "The SMS Filter Hub service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 0 milliseconds: Run the configured recovery program.".
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160707
I'm getting close. If I disable the anti-virus portion, there is no crashing of the filter-hub.exe application. Mail flows very nicely, the machine's CPU utilization is not pegged (once in the past few minutes, and only for about 6 seconds). As I write this, though, I notice it is pegging for a long period - about 48 seconds. I'll keep an eye on that. It could be that we just have a lot of email that is flowing through. The filter-hub program is not the only one using CPU now, though - even though utilization is nearly 100% now all the time.

Darn... there it goes again... This time, the faulting module of filter-hub.exe is libspamhunter.dll.

Hmmmm... back to building the other server...
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160809
I built the other server, installed the Symantec software, and right off the bat I started seeing the conduit.exe and filter-hub.exe start using 100% of the CPU. Keep in mind, the server is not even processing any mail yet!!! Its just got a couple of IP addresses assigned to it, but no mail servers even know about this server yet.

Thoughts? I'm going to switch to this server anyhow, because it has more hard drive space - and so far none of the hdmp and mdmp files are showing up.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160960
All switched over, mail is flowing, nothing in the "C:\windows\PCHEALTH\ErrorRep\UserDumps" directory yet. In fact, that directory doesn't even exist yet. So far, the log files are clean. I do see a few jumps now and then to 100% - usually for 5 to 10 seconds. Right now it just finished one though that was about 30 seconds. Otherewise, utilization is varying as you might expect - from 20 to 60 percent. I think I'll call Symantec on Monday when their tech support is open, and use the notes I've made here for further troubleshooting. I'll post any further information as I get it.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17160985
Incidentally, the other server is still running - 100% utilization now and again, but nothing in the UserDumps folder now. Conduit.exe and filter-hub.exe are both the ones using the CPU, with each at 92 or 97 percent now and then. All this server has now is a random IP address on my internal network, so it shouldn't be doing anything at all. I did throw ethereal on the new servers, and I saw during some of the peaks there was some encrypted messaging going back and forth between a brightmail.com server and my server - presumably to make sure the software is registered properly, is my guess.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17169498
It appears that svchost.exe uses a lot of CPU (presumably when writing the hdmp and mdmp files). Sometimes the Dr Watson executable starts up, and after svchost.exe pegs the CPU, filter-hub.exe does (I assume as it restarts). Symantec is having me run a diagnostic tool called SymBatchDiag2.78.exe. It is currently running on the original machine, and pegging the CPU while it does it. I had started it on the production machine, but just can't do that during the day, so I cancelled it. This diag tool takes between 5 and 30 minutes is what I've been told. Initial response from Symantec is that it is not compatible with VMWare.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17252208
We've now installed this software on 4 different computers, by different manufacturers. Each has been a clean build. There is no other software on the systems, yet the problems continue. Usually after a couple of days, we'll see filter-hub.exe begin to crash continuously - every couple of minutes. It seems to restart itself, but sometimes it won't - and then mail does not go through and we reboot the computer, and it works for another day.

As of today, we've stopped using the product completely. I do with they would make a package that used a standard MTA, not a Symantec product. It would be ideal if it supported sendmail as Brightmail did.

Lastly, Symantec support had me run a diagnostic tool. It showed them nothing, and I've not heard from them since July 25. I guess I should change my question to "what other anti-spam software should I consider using".
0
 
LVL 31

Expert Comment

by:rid
ID: 17252366
I hope this thread gets archived here, despite the lack of contribution from experts. I subscribed to the thread to see what would show up and can only say that I'm not too surprised at Symantec software causing a bogging-down of your server, although I have no remedy to suggest. I have heard very good things about the open-source antispam tools, like spamassassin and such, but that may not be an option in your case. Your way of describing events and steps taken can be very useful for other admins in a similar position. I suggest you post a Q in the Community Support area and ask them to award you the points!

Cheers
/RID
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17252449
Thanks for the note.

We had previously used SpamAssassin, but its accuracy was pretty poor (compared to Brightmail, which has served us very well in the past). This all started when we simply wanted to renew our brigthmail subscription for our flawlessly running (for two years) Redhat Linux box. However, since we bought brightmail over two years ago, they've since been bought by Symantec. Brightmail support was alway great, but the minute it switched to Symantec we didn't even bother calling when we had problems (which the problems were very few). It was suggested that we upgrade to their new Mail Security for SMTP product. Right away I was concerned as I called support and was told to UINSTALL SENDMAIL. I could hardly contain my laughter. Apparently the new software has its own MTA. Well, we were not about to uninstall sendmail on our Linux box (sendmail is to Linux as water is to life in our opinion). So, we tried it on a Windows box, and another Windows box, and another and another.

I do hope the thread gets archived as well, since Google searches for this problem turned up nothing. I can't imagine how others are dealing with this as every single installation we've performed has died. I wonder if its the 100,000 emails per day that kills it - although our last install ran the longest (for about 4 days). It was a beast of a machine though (we thought throwing raw horespower at it would help) - and it only sat between 5 and 20 percent CPU utilization at most times - hardly ever topped out. But sure enough, we started having problems today, and a quick look at the Event Viewer showed filter-hub crashed every couple of minutes since 5-ish PM on Wednesday.

I'll be sure to update the thread as I hear back from my sales reps and support (whom I've also copied as of today).
0
 
LVL 31

Expert Comment

by:rid
ID: 17252521
This makes good reading... :)
/RID
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17252544
Hey, wow, a response from Symantec - and it only took a scathing email and CC's to everyone at my vendor, and a suggestion that I change to an alternate product.

Good Afternoon,
 
All data related to the filter hub issues submitted by our customers had been forwarded to development.  After anlyzing the data, there was a filter hub issue identified by development that they are currently working on a patch for.  While I do not have a specific ETA for this patch, it will be designed to fix issues where the filter hub service is crashing.  Regardless of the environment, I would recommend applying the patch when it becomes available.
 
Best Regards,
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17547519
I do want to keep this open as we wait for Symantec to release a patch. Can we give it 2 more weeks?

Jon
0
 
LVL 20

Expert Comment

by:Venabili
ID: 17692477
Almost 3 weeks later :) Any news?
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17698549
I emailed Symantec for an update yesterday afternoon.
0
 
LVL 20

Expert Comment

by:Venabili
ID: 17915791
Any news here?
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 17958816
I wish there was news to report, but my email of October 9 has gone unanswered. I'll send another off to them today. If this is a pain to keep open, feel free to close it.
0
 
LVL 1

Expert Comment

by:Lyndy333
ID: 18020461
Hi Jon, What a great post with alot of insight! I would like to be able to read .hdmp adn .mdmp files to locate such issues and also find viruses that happened to find their way onto my smssmtp server. Thanks for the documentatio!
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 1

Author Comment

by:jdhoover123
ID: 18020523
I asked our vendor, Dell, to get involved. They received a response last week. It is below... We'll be trying this in our lab in about 4 weeks, and I'll post again if this is still open.
 
Begin email from Symantec rep:

Technical support is researching several filterhub.exe issues.  In fact, we just released a patch.  Client is welcome to try this patch.  In fact, I uploaded it for their use.
 
**** omitted FTP ****
 
In looking at the support case notes, I recommend the following:
 
Do not run SMSSMTP on a virtual system (Note from jdhoover123: we had tried with the same errors on both the Virtual and non-virtualized server).  (In fact, none of the "Brightmail" software is supported in a virtual environment.) If client wants to run SMSMSTP 5.0 on Windows 2003, then it should be run on Windows 2003 SP1 only.
 
It is possible that the folks out at client ran into a new issue. If they apply the new patch and filterhub.exe keeps crashing, then I recommend they reopen their support case and ask that Level 2 support take a look at the issue.  In order for Level 2 to have a look, client will need to provide debug logs/files when the issue occurs (Note from jdhoover123: we provided these with the original case).  These logs/files will include the following:
 
filterhub and mta set to deug level logging (this is in the settings.log file if I remember correctly)
Create a full crash dump when the problem occurs.
A copy of the MTA folder.
 
Technical support is the best place obtain instructions on how to set debug logging.
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 18020570
Lyndy333,

Not sure if I can help on the reading of the hdmp and mdmp files, but you might look at:

http://support.microsoft.com/?id=286350

Jon
0
 
LVL 1

Expert Comment

by:Lyndy333
ID: 18021853
Thanks Jon, I will experiment...I have saved the .hdmp and .mdmp.

I have a mass mailer on the smssmtp server. Have yet to identify it. Does not show up in processes, can not use hijack this, can not use autoruns, Symantec anti-virus and smssmtp haven't located it...geezie...I am now using filemon and I hope to find it soon...Have a nice day and Thanks again...
0
 
LVL 20

Expert Comment

by:Venabili
ID: 18339721
Any news here?
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 18342361
Yes, we've been running a trial now for about 60 days, and while we do occasionally get filter-hub errors like before, its rather rare. Also, hardly a thing shows up in the Event Viewere, whereas originally it was being added to ever few minutes. The spikes to 100% CPU Usage have also been reduced significantly, if not altogether. Looks like the current patch certainly helps a lot, and we've just purchased our licensing (it arrived today in my Inbox). The patch level was 172b. Incidentally, it is working even on our VMWare ESX 3 Infrastructure server, though not officially supported.

Jon
0
 
LVL 20

Expert Comment

by:Venabili
ID: 18380588
So what we do with the question?
0
 

Expert Comment

by:od_c
ID: 18418829
I'm just wondering where you got the patch from, as I have a similar problem with filter-hub.exe hogging CPU time and eventually causing SMTP connections to not respond after a couple of days or so meaning that a reboot of the server is required. I have been unable to find anything on the Symantec website yet and really do not want to start chasing their support as previous dealings with them have been very painful to say the least!! Excellent post by the way very helpful.  
0
 
LVL 1

Expert Comment

by:Lyndy333
ID: 18419064
od_c....

Have you checked out the following post?

http://www.experts-exchange.com/Applications/Email/Q_21928349.html
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 18419842
@od_c,

Send me an email at sms012907@1234.tv. I've got some contact info for you to get the patch pretty quickly. I just don't want to post the guy's phone and email at Symantec here.

Jon
0
 
LVL 1

Author Comment

by:jdhoover123
ID: 18419861
@venabili,

If we can leave it here, and just show it "solved" with my comment from 1/18 - that will let people know to try the patch 172b from Symantec (and hopefully avoid reading the entire thread).

Jon
0
 

Expert Comment

by:od_c
ID: 18420548
Thanks jd,

Got patch, found it eventually on symantecs website will install it and see if it resolves the issue thanks for your help.

http://www.symantec.com/region/reg_eu/techsupp/enterprise/products/sym_mail_security/sym_mail_security_5.0_smtp/files.html
0
 

Expert Comment

by:ricmik
ID: 19057457
I'm still having problems with this... The server is patched to 176, but the filter-hub.exe is crashing constantly.
This is driving me nuts! Apparently we're not the only ones having this problem.
Why won't Symantec just fix this!?
0
 

Expert Comment

by:ricmik
ID: 19057469
I ran a debug on the mdmp file... I'm not an expert, but it looks like it's the kvolefio.dll that's causing the problems.. Any suggestions?
0
 
LVL 1

Expert Comment

by:Lyndy333
ID: 19057861
Hi Ric....
My issues resolved after I installed all patches and am running version 4.1.15.47....
0
 

Expert Comment

by:ricmik
ID: 19063151
Did you go back to version 4? My problems is on SMS for SMTP 5.0.
0
 
LVL 1

Accepted Solution

by:
Computer101 earned 0 total points
ID: 19329644
PAQed with points refunded (500)

Computer101
EE Admin
0
 

Expert Comment

by:adonoso
ID: 20084371
We had the same exact problem with version 5.01. It was resolved by adding memory (2GB total) to the server.
0
 

Expert Comment

by:ricmik
ID: 20085056
We're already running on 2GB of RAM :(
0
 
LVL 1

Expert Comment

by:Daveatwork
ID: 21824605
This software works real good when it's not having any "troubles".  I have three servers running it. One fully patched w/2G of RAM and I still see conduit sucking 100% of the CPU.  I also am running one on VMWare, and it runs fine, actually the best of all three.  Sometimes I also get emails from them that some/all services have stopped, but usualy they are lies.

Dave
0
 

Expert Comment

by:ricmik
ID: 21826565
Yeah, the software is great.
We've now moved on from running SMS on Server2003 to the appliance box (8300).
The appliance is great! Highly recommended! :)
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

I tend toward trying the newest hardware and software.  Thiss sometimes works out to my benefit, and sometimes not.  Because I downloaded and installed Android 5.x (http://www.experts-exchange.com/articles/18084/Upgrading-to-Android-5-0-Lollipop.htm…
Find out what Office 365 Transport Rules are, how they work and their limitations managing Office 365 signatures.
This video discusses moving either the default database or any database to a new volume.
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now