Solved

Exchange 2003 Information store - possible problem

Posted on 2004-09-14
16
3,373 Views
Last Modified: 2012-08-14
Hello,

I am running a windows 2003 standard edition server as a domain controller with Exchange 2003 standard edition installed.  We haven't had issues with this configuration up until about 2 weeks ago.  I've read other forums on this issue including Nino Bilics post at on the MS Exchange team's BLOG.  Here is the issue: One day the above mentioned server gave us a virtual memory error and informed us that it was changing the virtual memory size.  I opened up task manager after about 7 unsuccesfull tries (it kept giving me memory errors while trying to open it, I finally had to close out the eTrust real time monitor to get it to open) I noticed the following in task manager:

Store.exe
  - memory usage: 881,684 K
  - Page Faults: 1,100,983
  - Virtual memory size: 1,444,728 K and climbing

MAD.exe
  - memory usage: 16,876 K
  - page faults: 26,737
  - virtual memory size: 1,123,044 K

svchost.exe
  - memory usage: 16,876 K
  - page faults: 79,985
  - virtual memory size: 1,111,756 K

This seems odd to me.  I know store.exe is supposed to grab as much memory as it needs, and re-allocate it as it sees fit, but this seems rather high to me.  It looks like these 3 processes are just munching away at the virtual memory on the server.  I've also started getting complaints about outlook running slow and acting funny from end users.  On my own email I am unable to pull up HTML messages that contain images, I get a memory error from Outlook.  This server has 2 gb of RAM and a 2 ghz xeon processor. The only apps running on this server are eTrust antivirus 7 and the backup exec remote client.  I've added the /3GB and /Userva=3030 switches to the boot.ini file as well.  I did install a couple of critical updates before the problem started....so maybe this could be linked also.

Any ideas or help on this matter would be greatly appreciated.  

Thanks!

Dustin
0
Comment
Question by:uucutech
  • 8
  • 7
16 Comments
 
LVL 12

Expert Comment

by:BNettles73
Comment Utility

Have you downloaded and looked through the Troubleshooting documentation? It has quite a few scenario's which would explain why resource issues occur. Make sure you aren't running backups during the day and make sure your AV software is not scanning IIS or Exchange, unless it was designed specifically for that purpose.

A lot of times AV software is the root cause of these types of issues. Not to mention, if the software isn't designed for it - then you may end up with Metabase Corruption. Hopefully your exchange databases are on at least RAID 5.  I wouldn't worry so much about the memory, unless you are receiving the event in the application logs.

Can you tell me what the Free System Page Table Entries counter is for your server? You can use performance monitor ... I believe it will be in Memory ...

Have you deleted or moved a lot of data around recently? It could be fragmentation of your databases if so ...

You should definitely download and review the document below, if you haven't already -

Troubleshooting Exchange Server 2003 Performance
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/e2k3perf.mspx
0
 

Author Comment

by:uucutech
Comment Utility
I just downloaded the Troubleshooting docs....I'll read through them today.  our AV solution scans email attachments and strips them if they meet certain criteria so we don't really want to disable that functionality.  I don't think it is the AV because this server had been running fine for about a month and a half before we had this issue.  Yes our exchange DB's are on RAID 5, thank goodness.  I'm getting tons of errors in the application log that look like they are due to memory issues.  such as:

*Message processing failed because there is not enough available memory (800700DE-82000387)

*LDAP bind was unsuccessful on directory "server.domain.com" for distinguished name ". Directory returned error:[0x5a] No Memory.   DC=domain,DC=com

*Windows cannot bind  to "domain.com" domain. (No Memory). Group Policy processing aborted.
 
The  Free System Page Table Entries counter is at 100 when i go into performance monitor for this server.
We have not deleted or moved any data around on this server....so fragmentation shouldn't be an issue.  Thank you so much for your time and patience.  Hopefully i can get this figured out.

Thanks again,

Dustin




0
 
LVL 7

Expert Comment

by:jaguarpriest
Comment Utility
can you post your hardware config. Are you like running only on 1GB of ram or something?

jaguar
0
 

Author Comment

by:uucutech
Comment Utility
The Critical updates that were installed prior to this problem were KB 867460 and KB 867801
If the list of hot fixes in the "add/remove" programs applet is sequential from top to bottom.  I don't know if this will help or not.

0
 

Author Comment

by:uucutech
Comment Utility
If you read the first post You'll see that this server has 2 gb of RAM.  Is this not enough perhaps?  It's only serving about 70 - 80 inboxes and the private information store is only about 5 gb.

Thanks,

Dustin
0
 
LVL 12

Expert Comment

by:BNettles73
Comment Utility
if the counter is only 100 then you definitely have virtual memory fragmentation issues ... the Free System PTE counter should never drop below 3000.

I would go over this article with a fine tooth comb and make sure you have your settings configured properly:

How to optimize memory usage in Exchange Server 2003
http://support.microsoft.com/default.aspx?kbid=815372

Troubleshoot Virtual Memory Fragmentation in Exchange 2003
http://support.microsoft.com/?id=325044

Do you have multiple storage groups on that server? How many users, how many databases?
Did you set the HeapDeCommitFreeBlockThreshold registry value registry key to 0x00040000 after you configured the /3GB and /USERVA=3030 switch?
The HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\SystemPages key should be set to 0.
0
 
LVL 12

Expert Comment

by:BNettles73
Comment Utility
2 GB should be enough ... I have 3.5 GB on each my E2K3 cluster nodes and they hold about 4000 mailboxes each ...
0
 

Author Comment

by:uucutech
Comment Utility
I'm sorry i might have posted that Free System PTE wrong.  It actually looks like it is 36598, if I'm reading it correctly.  I put 100 because while looking at it in the graph view the indicator scrolling across the screen was all the way at 100 the whole time.  

We only have 1 storage group on this exchange server.  There are about 80 users, we have 2 databases 1 for private and 1 for public......private is the only 1 being used.  Is that what you mean?  I also ran accross an article that said the /3GB switch and the /Userva=XXXX switch are only for enterprise versions of exchange.  If this is correct should i take them out of the boot.ini?  There is also a /maxmem=2048 switch in the boot.ini  this shouldn't be causing problems should it?

I'll go through those documents BNettles73  thanks for your help


Dustin
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 
LVL 12

Expert Comment

by:BNettles73
Comment Utility
KB 815372 ... I'm pretty sure with STD you still need to have the 3GB and Userva=3030 switch set ...

Note Exchange 2003 performs an optimal memory configuration check when the store process starts. If the memory settings are not optimal, you will see event 9665 in Event Viewer. This message appears if any of the following conditions exist:
The server is running any edition of Microsoft Windows 2000 Server and the SystemPages value in the registry is set outside the range of 24000 to 31000.

Note The SystemPages subkey is located in the registry in the following path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\SystemPages.

The server is running Microsoft Windows 2000 Advanced Server or Microsoft Windows 2000 Datacenter Server, has 1 GB or more of physical memory, but does not have the /3GB switch set in the boot.ini file.

The server is running Microsoft Windows Server 2003 Standard, Microsoft Windows Server 2003 Enterprise or Microsoft Windows Server 2003 Datacenter edition, and the SystemPages value in the registry is set to something other than 0.

***The server is running Microsoft Windows Server 2003 Standard Edition, Microsoft Windows Server 2003 Enterprise Edition or Microsoft Windows Server 2003 Datacenter Edition, has 1 GB or more of physical memory and the /3GB switch is set, but the /USERVA setting is not present in the boot.ini file or is outside the range of 3030 to 2970.

The server is running any edition of Microsoft Windows 2000 Server or Microsoft Windows Server 2003, and the HeapDeCommitFreeBlockThreshold value in the registry is set to something other than 0x00040000.
The memory configuration check
0
 

Author Comment

by:uucutech
Comment Utility
I just read that and noticed that this registry entry on the server doesn't coincide with what that article says:

HeapDeCommitFreeBlockThreshold=0x41000400

should i go ahead and set this to 0x00040000?  If so there are a few different places in the registry that it needs to be changed.  Should i change it in all of them?

Thanks,

Dustin
0
 
LVL 12

Accepted Solution

by:
BNettles73 earned 500 total points
Comment Utility
Definitely need to correct the issue ...

If you have 1 GB or more physical memory installed, configure the HeapDeCommitFreeBlockThreshold registry value in the following registry key, and then restart the server:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
Value name: HeapDeCommitFreeBlockThreshold
Value type: REG_DWORD
Value data: 0x00040000 (recommended)
Value default: not present

For additional information about how to use and configure this registry value, click the following article number to view the article in the Microsoft Knowledge Base:
315407 The "HeapDecommitFreeBlockThreshold" registry key
0
 

Author Comment

by:uucutech
Comment Utility
I'll do this tonight after hours and let you know how the server is performing in the morning.  Hopefully we're on to something here.

Thanks again for all of your help,

Dustin
0
 

Author Comment

by:uucutech
Comment Utility
I just read a little more on this subject and saw this :

Note: For Exchange Server computers which are at the same time Active Directory Domain Controllers or Global Catalog servers we do also not recommend setting the /3GB switch in boot.ini. We recommend having dedicated Active Directory Domain Controllers or Global Catalog servers.

Should i go ahead and take out the /3GB and /Userva=XXXX switches.  Or is it OK to leave them the way they are?
0
 
LVL 12

Expert Comment

by:BNettles73
Comment Utility
I'd leave it as is .... you'll end up with serious VM fragmentation issues if you don't ... fyi ... you can read through this article about why not to install E2K3 on a DC ... I understand in a smaller environment it is sometimes necessary, so you'll want to understand the implications of it ... you should be ok with a small user base though ...

http://www.petri.co.il/problems_with_exchange_2003_installed_on_domain_controllers.htm
0
 

Author Comment

by:uucutech
Comment Utility
Thanks BNettles73,

That registry entry fixed the problem.  Now the only process that is hogging memory is store.exe, which should be the case.  :)
all the points go to you, and I give you an A.  We are also going to demote the exchange server to a member server, we have 1 other domain controller running Win2k3 AD, and we are a smaller environment so I think we should be OK.

Thanks again,

Dustin
0
 
LVL 12

Expert Comment

by:BNettles73
Comment Utility
congrats
0

Featured Post

Promote certifications in your email signature

Has your company recently won an award or achieved a certification? They'll no doubt want to show it off. Email signature images used to promote certifications & awards can instantly establish credibility with a recipient and provide you with numerous benefits.

Join & Write a Comment

Resolve Outlook connectivity issues after moving mailbox to new Exchange 2016 server
Marketers need statistics and metrics like everybody else needs oxygen. In this article we explain how to enable marketing campaign statistics for Microsoft Exchange mail.
how to add IIS SMTP to handle application/Scanner relays into office 365.
This video discusses moving either the default database or any database to a new volume.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now