uucutech
asked on
Exchange 2003 Information store - possible problem
Hello,
I am running a windows 2003 standard edition server as a domain controller with Exchange 2003 standard edition installed. We haven't had issues with this configuration up until about 2 weeks ago. I've read other forums on this issue including Nino Bilics post at on the MS Exchange team's BLOG. Here is the issue: One day the above mentioned server gave us a virtual memory error and informed us that it was changing the virtual memory size. I opened up task manager after about 7 unsuccesfull tries (it kept giving me memory errors while trying to open it, I finally had to close out the eTrust real time monitor to get it to open) I noticed the following in task manager:
Store.exe
- memory usage: 881,684 K
- Page Faults: 1,100,983
- Virtual memory size: 1,444,728 K and climbing
MAD.exe
- memory usage: 16,876 K
- page faults: 26,737
- virtual memory size: 1,123,044 K
svchost.exe
- memory usage: 16,876 K
- page faults: 79,985
- virtual memory size: 1,111,756 K
This seems odd to me. I know store.exe is supposed to grab as much memory as it needs, and re-allocate it as it sees fit, but this seems rather high to me. It looks like these 3 processes are just munching away at the virtual memory on the server. I've also started getting complaints about outlook running slow and acting funny from end users. On my own email I am unable to pull up HTML messages that contain images, I get a memory error from Outlook. This server has 2 gb of RAM and a 2 ghz xeon processor. The only apps running on this server are eTrust antivirus 7 and the backup exec remote client. I've added the /3GB and /Userva=3030 switches to the boot.ini file as well. I did install a couple of critical updates before the problem started....so maybe this could be linked also.
Any ideas or help on this matter would be greatly appreciated.
Thanks!
Dustin
I am running a windows 2003 standard edition server as a domain controller with Exchange 2003 standard edition installed. We haven't had issues with this configuration up until about 2 weeks ago. I've read other forums on this issue including Nino Bilics post at on the MS Exchange team's BLOG. Here is the issue: One day the above mentioned server gave us a virtual memory error and informed us that it was changing the virtual memory size. I opened up task manager after about 7 unsuccesfull tries (it kept giving me memory errors while trying to open it, I finally had to close out the eTrust real time monitor to get it to open) I noticed the following in task manager:
Store.exe
- memory usage: 881,684 K
- Page Faults: 1,100,983
- Virtual memory size: 1,444,728 K and climbing
MAD.exe
- memory usage: 16,876 K
- page faults: 26,737
- virtual memory size: 1,123,044 K
svchost.exe
- memory usage: 16,876 K
- page faults: 79,985
- virtual memory size: 1,111,756 K
This seems odd to me. I know store.exe is supposed to grab as much memory as it needs, and re-allocate it as it sees fit, but this seems rather high to me. It looks like these 3 processes are just munching away at the virtual memory on the server. I've also started getting complaints about outlook running slow and acting funny from end users. On my own email I am unable to pull up HTML messages that contain images, I get a memory error from Outlook. This server has 2 gb of RAM and a 2 ghz xeon processor. The only apps running on this server are eTrust antivirus 7 and the backup exec remote client. I've added the /3GB and /Userva=3030 switches to the boot.ini file as well. I did install a couple of critical updates before the problem started....so maybe this could be linked also.
Any ideas or help on this matter would be greatly appreciated.
Thanks!
Dustin
ASKER
I just downloaded the Troubleshooting docs....I'll read through them today. our AV solution scans email attachments and strips them if they meet certain criteria so we don't really want to disable that functionality. I don't think it is the AV because this server had been running fine for about a month and a half before we had this issue. Yes our exchange DB's are on RAID 5, thank goodness. I'm getting tons of errors in the application log that look like they are due to memory issues. such as:
*Message processing failed because there is not enough available memory (800700DE-82000387)
*LDAP bind was unsuccessful on directory "server.domain.com" for distinguished name ". Directory returned error:[0x5a] No Memory. DC=domain,DC=com
*Windows cannot bind to "domain.com" domain. (No Memory). Group Policy processing aborted.
The Free System Page Table Entries counter is at 100 when i go into performance monitor for this server.
We have not deleted or moved any data around on this server....so fragmentation shouldn't be an issue. Thank you so much for your time and patience. Hopefully i can get this figured out.
Thanks again,
Dustin
*Message processing failed because there is not enough available memory (800700DE-82000387)
*LDAP bind was unsuccessful on directory "server.domain.com" for distinguished name ". Directory returned error:[0x5a] No Memory. DC=domain,DC=com
*Windows cannot bind to "domain.com" domain. (No Memory). Group Policy processing aborted.
The Free System Page Table Entries counter is at 100 when i go into performance monitor for this server.
We have not deleted or moved any data around on this server....so fragmentation shouldn't be an issue. Thank you so much for your time and patience. Hopefully i can get this figured out.
Thanks again,
Dustin
can you post your hardware config. Are you like running only on 1GB of ram or something?
jaguar
jaguar
ASKER
The Critical updates that were installed prior to this problem were KB 867460 and KB 867801
If the list of hot fixes in the "add/remove" programs applet is sequential from top to bottom. I don't know if this will help or not.
If the list of hot fixes in the "add/remove" programs applet is sequential from top to bottom. I don't know if this will help or not.
ASKER
If you read the first post You'll see that this server has 2 gb of RAM. Is this not enough perhaps? It's only serving about 70 - 80 inboxes and the private information store is only about 5 gb.
Thanks,
Dustin
Thanks,
Dustin
if the counter is only 100 then you definitely have virtual memory fragmentation issues ... the Free System PTE counter should never drop below 3000.
I would go over this article with a fine tooth comb and make sure you have your settings configured properly:
How to optimize memory usage in Exchange Server 2003
http://support.microsoft.com/default.aspx?kbid=815372
Troubleshoot Virtual Memory Fragmentation in Exchange 2003
http://support.microsoft.com/?id=325044
Do you have multiple storage groups on that server? How many users, how many databases?
Did you set the HeapDeCommitFreeBlockThres
The HKEY_LOCAL_MACHINE\SYSTEM\
I would go over this article with a fine tooth comb and make sure you have your settings configured properly:
How to optimize memory usage in Exchange Server 2003
http://support.microsoft.com/default.aspx?kbid=815372
Troubleshoot Virtual Memory Fragmentation in Exchange 2003
http://support.microsoft.com/?id=325044
Do you have multiple storage groups on that server? How many users, how many databases?
Did you set the HeapDeCommitFreeBlockThres
The HKEY_LOCAL_MACHINE\SYSTEM\
2 GB should be enough ... I have 3.5 GB on each my E2K3 cluster nodes and they hold about 4000 mailboxes each ...
ASKER
I'm sorry i might have posted that Free System PTE wrong. It actually looks like it is 36598, if I'm reading it correctly. I put 100 because while looking at it in the graph view the indicator scrolling across the screen was all the way at 100 the whole time.
We only have 1 storage group on this exchange server. There are about 80 users, we have 2 databases 1 for private and 1 for public......private is the only 1 being used. Is that what you mean? I also ran accross an article that said the /3GB switch and the /Userva=XXXX switch are only for enterprise versions of exchange. If this is correct should i take them out of the boot.ini? There is also a /maxmem=2048 switch in the boot.ini this shouldn't be causing problems should it?
I'll go through those documents BNettles73 thanks for your help
Dustin
We only have 1 storage group on this exchange server. There are about 80 users, we have 2 databases 1 for private and 1 for public......private is the only 1 being used. Is that what you mean? I also ran accross an article that said the /3GB switch and the /Userva=XXXX switch are only for enterprise versions of exchange. If this is correct should i take them out of the boot.ini? There is also a /maxmem=2048 switch in the boot.ini this shouldn't be causing problems should it?
I'll go through those documents BNettles73 thanks for your help
Dustin
KB 815372 ... I'm pretty sure with STD you still need to have the 3GB and Userva=3030 switch set ...
Note Exchange 2003 performs an optimal memory configuration check when the store process starts. If the memory settings are not optimal, you will see event 9665 in Event Viewer. This message appears if any of the following conditions exist:
The server is running any edition of Microsoft Windows 2000 Server and the SystemPages value in the registry is set outside the range of 24000 to 31000.
Note The SystemPages subkey is located in the registry in the following path: HKEY_LOCAL_MACHINE\SYSTEM\
The server is running Microsoft Windows 2000 Advanced Server or Microsoft Windows 2000 Datacenter Server, has 1 GB or more of physical memory, but does not have the /3GB switch set in the boot.ini file.
The server is running Microsoft Windows Server 2003 Standard, Microsoft Windows Server 2003 Enterprise or Microsoft Windows Server 2003 Datacenter edition, and the SystemPages value in the registry is set to something other than 0.
***The server is running Microsoft Windows Server 2003 Standard Edition, Microsoft Windows Server 2003 Enterprise Edition or Microsoft Windows Server 2003 Datacenter Edition, has 1 GB or more of physical memory and the /3GB switch is set, but the /USERVA setting is not present in the boot.ini file or is outside the range of 3030 to 2970.
The server is running any edition of Microsoft Windows 2000 Server or Microsoft Windows Server 2003, and the HeapDeCommitFreeBlockThres
The memory configuration check
Note Exchange 2003 performs an optimal memory configuration check when the store process starts. If the memory settings are not optimal, you will see event 9665 in Event Viewer. This message appears if any of the following conditions exist:
The server is running any edition of Microsoft Windows 2000 Server and the SystemPages value in the registry is set outside the range of 24000 to 31000.
Note The SystemPages subkey is located in the registry in the following path: HKEY_LOCAL_MACHINE\SYSTEM\
The server is running Microsoft Windows 2000 Advanced Server or Microsoft Windows 2000 Datacenter Server, has 1 GB or more of physical memory, but does not have the /3GB switch set in the boot.ini file.
The server is running Microsoft Windows Server 2003 Standard, Microsoft Windows Server 2003 Enterprise or Microsoft Windows Server 2003 Datacenter edition, and the SystemPages value in the registry is set to something other than 0.
***The server is running Microsoft Windows Server 2003 Standard Edition, Microsoft Windows Server 2003 Enterprise Edition or Microsoft Windows Server 2003 Datacenter Edition, has 1 GB or more of physical memory and the /3GB switch is set, but the /USERVA setting is not present in the boot.ini file or is outside the range of 3030 to 2970.
The server is running any edition of Microsoft Windows 2000 Server or Microsoft Windows Server 2003, and the HeapDeCommitFreeBlockThres
The memory configuration check
ASKER
I just read that and noticed that this registry entry on the server doesn't coincide with what that article says:
HeapDeCommitFreeBlockThres
should i go ahead and set this to 0x00040000? If so there are a few different places in the registry that it needs to be changed. Should i change it in all of them?
Thanks,
Dustin
HeapDeCommitFreeBlockThres
should i go ahead and set this to 0x00040000? If so there are a few different places in the registry that it needs to be changed. Should i change it in all of them?
Thanks,
Dustin
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I'll do this tonight after hours and let you know how the server is performing in the morning. Hopefully we're on to something here.
Thanks again for all of your help,
Dustin
Thanks again for all of your help,
Dustin
ASKER
I just read a little more on this subject and saw this :
Note: For Exchange Server computers which are at the same time Active Directory Domain Controllers or Global Catalog servers we do also not recommend setting the /3GB switch in boot.ini. We recommend having dedicated Active Directory Domain Controllers or Global Catalog servers.
Should i go ahead and take out the /3GB and /Userva=XXXX switches. Or is it OK to leave them the way they are?
Note: For Exchange Server computers which are at the same time Active Directory Domain Controllers or Global Catalog servers we do also not recommend setting the /3GB switch in boot.ini. We recommend having dedicated Active Directory Domain Controllers or Global Catalog servers.
Should i go ahead and take out the /3GB and /Userva=XXXX switches. Or is it OK to leave them the way they are?
I'd leave it as is .... you'll end up with serious VM fragmentation issues if you don't ... fyi ... you can read through this article about why not to install E2K3 on a DC ... I understand in a smaller environment it is sometimes necessary, so you'll want to understand the implications of it ... you should be ok with a small user base though ...
http://www.petri.co.il/problems_with_exchange_2003_installed_on_domain_controllers.htm
http://www.petri.co.il/problems_with_exchange_2003_installed_on_domain_controllers.htm
ASKER
Thanks BNettles73,
That registry entry fixed the problem. Now the only process that is hogging memory is store.exe, which should be the case. :)
all the points go to you, and I give you an A. We are also going to demote the exchange server to a member server, we have 1 other domain controller running Win2k3 AD, and we are a smaller environment so I think we should be OK.
Thanks again,
Dustin
That registry entry fixed the problem. Now the only process that is hogging memory is store.exe, which should be the case. :)
all the points go to you, and I give you an A. We are also going to demote the exchange server to a member server, we have 1 other domain controller running Win2k3 AD, and we are a smaller environment so I think we should be OK.
Thanks again,
Dustin
congrats
Have you downloaded and looked through the Troubleshooting documentation? It has quite a few scenario's which would explain why resource issues occur. Make sure you aren't running backups during the day and make sure your AV software is not scanning IIS or Exchange, unless it was designed specifically for that purpose.
A lot of times AV software is the root cause of these types of issues. Not to mention, if the software isn't designed for it - then you may end up with Metabase Corruption. Hopefully your exchange databases are on at least RAID 5. I wouldn't worry so much about the memory, unless you are receiving the event in the application logs.
Can you tell me what the Free System Page Table Entries counter is for your server? You can use performance monitor ... I believe it will be in Memory ...
Have you deleted or moved a lot of data around recently? It could be fragmentation of your databases if so ...
You should definitely download and review the document below, if you haven't already -
Troubleshooting Exchange Server 2003 Performance
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/e2k3perf.mspx