Solved

Server 2003 DNS Service Very High Memory Utilization

Posted on 2007-11-29
21
3,249 Views
Last Modified: 2008-07-22
System is Windows Server 2003 Standard SP1 running DNS services, fully patched and current with all Windows updates.  This machine is a secondary DNS server.  After being in service nearly 2 years, jsut 2 weeks ago this server suddenly began using excessivly high amounts of memory for the dns.exe process.  By high, I mean that after restarting the DNS service, it begins a near-linear increase (no spikes, just fairly constant slope upward)  in memory use, easily exceeding 1.25 GB of memory after 4 or 5 days.  I restart the process, and the even repeats itself.  Until now, memory use was nearly a flat line.

I have completely uninstalled the DNS service, and re-installed.  

This machine serves only a few hundred zones (stored in zone files & registry, NOT Active Directory, is stand-alone server and not a member of any AD domain) and answers ONLY for its own zones (recursion is disabled, root hints file has been removed).

I have repeatedly googled and searched the MS KB for answers on high memory utilization for the DNS service, but have come up with noting relevant.

Anyone seen this problem, or have a reference to a known issue/fix?
0
Comment
Question by:Ecompro
  • 8
  • 5
  • 4
  • +3
21 Comments
 
LVL 21

Expert Comment

by:from_exp
ID: 20377451
If you haven't done any changes(new updates, new zones, new firewall rules etc) recently, nehavior of dns process is a result, not a root of a problem.
try to sniff traffic with wireshark from the dns machine with dns process running.
0
 
LVL 7

Expert Comment

by:northcide
ID: 20382598
Server 2003 SP2 is available.  You should start there.
0
 
LVL 1

Expert Comment

by:evrenuyanik
ID: 20552297
I have the same problem too. It started about a month ago and is seen only on secondary dns servers. Problem started at the same time on 3 different machines which are running as secondary dns.  Other primary dns servers have no problem. My servers are all updated, all service packs, updates etc., but the problem continues and no recent windows update solved it yet. I think that this issue might be related to an update, because the problem started at the same time on different servers. As windows update is done automatically, an update might have affected all servers at the same time.

Because I couldn't find a solution yet, I scheduled 'dns service restarts' with the command "dnscmd.exe . /restart". As the service restarts itself periodically with this command, the memory consumption is kept low, but of course an exact solution must be found. I hope a future update of windows would solve this problem.
0
 

Author Comment

by:Ecompro
ID: 20553053
evrenuyanik, thanks for the info.  I'm glad someone else has the same symptoms I do, starting around the same time as the Windows updates came out.  The same day as my original post, I did the same thing of having a script fire around 3 a.m. to restart the service.  Hopefully enough people will report it, that MS will diagnose and put out a hotfix or Windows update for it.
0
 
LVL 1

Expert Comment

by:evrenuyanik
ID: 20590013
On a server of ours, we have about 2000 zones, that servers memory is increasing very fast, I set up that server so that it restarts the dns service every 1 hour. Because the restart process takes place in just a few seconds, there is no disadvantage of restarting it frequently; so restarting it every 1 hour keeps memory consumption low. Before setting up these restarts, the server was slowing done distinctly; after setting up scheduling for restarts, the speed of the machine has increased.
0
 

Expert Comment

by:maishsk
ID: 20732615
Do you perhaps have KB941672 installed?
0
 

Author Comment

by:Ecompro
ID: 20733269
>Do you perhaps have KB941672 installed?

Yes, we do.
0
 

Accepted Solution

by:
maishsk earned 250 total points
ID: 20733645
I would be happy to elaborate later on about he whole story we have been going through the past 2 weeks but I will do it later tonight.

To cut a long story short there is a memory leak in the patch.

Will get back later
0
 

Author Comment

by:Ecompro
ID: 20733718
>To cut a long story short there is a memory leak in the patch.

Does Microsoft know about it, or was this an independent discovery?  I can't wait to hear the rest of the story, when you have time.
0
 

Expert Comment

by:maishsk
ID: 20740538
Yes they know about it. I have sent a few hundred emails over the past 2 weeks with a support engineer from Microsoft trying to debug the issue.

My experience:
We had our DNS servers patched with this patch on Jan 4th. One week later we started to get zone that expired and the DNS service started to crash on the servers. After support from Microsoft told us that the process had a memory leak that was found after release. We were trying to debug the problem. In addition to the whole issue we had netowrk problem which were causing even bigger dns issues.

So in short the symptoms could be:
- DNS.exe process uses a lot of memory (It can easily go over 100mb)
- DNS service stops responding - it looks like it works, but does not transfer zones or forward queries.

At the moment the only thing that can be done is restart the DNS to clear the memory.

Microsoft will release a patch for the issue in the future, but do not have a release date yet.

And no this is not public knowledge until now....

If anyone would like more detailed info please contact me off line at maishsk--@ gmail.com (remove the --)

0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 

Expert Comment

by:maishsk
ID: 20879015
Seeing that several people have asked me offline what the solution to this one was, I will post what I know here.

At the moment the only thing I fount that brought down the memory usage for the DNS services was to remove KB941672

Machine with patch - version of dns.exe - 5.2.3790.3027

After patched removed
version of dns.exe - 5.2.3790.2915

I guess we will all have to wait for Microsoft to get their act together and fix this patch up properly
0
 

Author Comment

by:Ecompro
ID: 20879593
Thanks for the update.  It's disappointing that MS can't fix a simple memory leak in this many months.  They know it relates to KB941672, and they know what code changes were written into that release, so it doesn't seem like a monumental task to back track their steps and see what demon was compiled in.  

(Disclaimer:  I am not a programmer.)
0
 
LVL 4

Assisted Solution

by:sivanov
sivanov earned 250 total points
ID: 21359599
0
 
LVL 4

Expert Comment

by:sivanov
ID: 21359615
If previous link does not solve the issue, u can try to use an older dns.exe and dll, before 941672
0
 

Author Comment

by:Ecompro
ID: 21359694
Thanks for the update.  I submitted the request for the hotfix, and am keeping my fingers crossed.  Once the hotfix is installed and I run it for a few days, I'll post back with my observations.
0
 

Expert Comment

by:maishsk
ID: 21362095
From what we have seen in our environment, after deploying this HF from MS, the DNS process has returned to normal limits and is functioning properly.

Ii hope this solves your issues as well...

Good luck
0
 

Author Comment

by:Ecompro
ID: 21362537
I'm struggling to get the hotfix from Microsoft.  The DNS server is running Server 2003 Standard at SP1.  The rep responded back saying the hotfix is not available for this edition, even though the hotfix prerequisites clearly state Server 2003 at SP1 or SP2.  Their support seems to be sliding downhill in recent years.
0
 
LVL 4

Expert Comment

by:sivanov
ID: 21367185
Push them again they must sent you the hotfix, and i thinkg that its good to update to sp2
0
 

Author Comment

by:Ecompro
ID: 21367298
In the message was a link to provide feedback to the manager, along with the manager's name and employee's name.  So I used it... and I replied back to the employee making the decline, with a quote from the KB article.  A few hours later, the link for hotfix download arrived.  :)

I installed it late yesterday, and so far memory utilization has remained flat for more than 12 hours (a very good sign).  Without the fix, after 12 hours, memory utilization would be up 200-300 MB based on the trends of the past 30 days.  I'll give it a few days to be sure before closing this request, but at the moment it looks like maishsk hit the nail on the head, and sivanov provided the correct link to the fix.  
0
 
LVL 4

Expert Comment

by:sivanov
ID: 21374842
ok i got a bird that tells me that the fix works  in 99.9% of the cases - reazon for failure usually are - nor restarting the service after updates, sometimes its needed to reboot the server - i've also read 0.01 % issues which for the moment are solved by workaround with replasing the dns.exe and dll with version previous that KB941672, but must be aware that every MS update will try to replace them with KB941672.
OK FINALY  AGAIN http://support.microsoft.com/kb/946565/
0
 

Author Comment

by:Ecompro
ID: 21376094
I'm going to declare this closed.  It's been almost 2 days and memory utilization has remained flat.  Many thanks go to maishsk for making the correct diagnosis, and sivanov for providing the correct link to the hotfix.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

There have been a lot of times when we have seen the need to enter a large number of DNS entries in a forward lookup zone. The standard procedure would be to launch the DNS Manager console, create the Zone and start adding new hosts using the New…
Learn about cloud computing and its benefits for small business owners.
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now