• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 871
  • Last Modified:

Production IIS RAM Usage is out of control

This is a Urgent item,
We have 3 servers.
- Production Server
- Qualification Server
- Developmental Server

We have a Web Application that is installed on on three servers with identical Web.config settings on all three systems, only difference would be the connection string to the SQL Servers. All three systems are connected to identical copies of the database so the data loaded is the same.

My Issue is the Production system is running and 5 Times the RAM usage than the other two systems, and this usage shows up right as the application starts up.

- Production Server RAM Usage at start up is 19.4 GB
- Qualification Server RAM Usage at start up is 4 GB
- Developmental Server RAM Usage at start up is 4.2 GB

I have gone through settings to compare the application pools and I do not see any differences.

I am looking for a setting that might have been accidentally enabled on the Production system that is causing this high RAM Usage.

The Production system cannot be taken down over and over, so I have to be careful in any edits I do to fix the problem, so I am more trying to reproduce the high memory issue in the Qualification environment.

This is urgent so any help is appreciated.
Thanks
image004.png
image003.png
0
EazyWorks
Asked:
EazyWorks
  • 11
  • 5
  • 4
  • +1
3 Solutions
 
gaurav05Commented:
Hi,

have you installed any windows update on production system  ?

Try to install debug diagnostic tool and attach it with performance counter on worker
process.

Try to generate dump and debug that dump file.

when you attach debug diag tool on worker process, worker process utilization
becomes normal and when it increased it will generate the dump file and
you will able to find thread which cause more utilization of worker process.
0
 
madginoCommented:
Maybe it's because the application is used by much more users in production and this is the amount of memory it requires.

Try to emulate the same traffic (and the same amount of data) as in production on your Qualification environment and see what happens with memory.
0
 
EazyWorksAuthor Commented:
The ram usage is taken at start up before any users access the system.
The screen shot us not showing that, but that is how it acts.
The usage over time and when users access the system will go up but not that much.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
EazyWorksAuthor Commented:
I just noticed that the production system has a newer version of IIS. I will post all the details shortly.
0
 
EazyWorksAuthor Commented:
IIS Versions
- Qualification System : Version 7.5.7600.16385
- Production System : Version  7.5.7601.17514
0
 
TI2HeavenCommented:
It may be an application component. Some components query the OS to see how much memory is free to decide how much memory they do need.
¿Can you give more details of your WebApplication?.  Sizes of read cache happen to use this type of query to choose its size at runtime.
0
 
madginoCommented:
Anything else running on the IIS? The memory is increasing when you start the site or when you start the IIS? For example if you are stopping only the website but keep the IIS working, how is the memory? Did youy check application pool settings?
0
 
EazyWorksAuthor Commented:
This is the size of the w3wp.exe process we are comparing only.
I have launched another environment on a server and it as well is running at around 4GB after the application starts.
0
 
EazyWorksAuthor Commented:
For example,
If we put the application offline using the "App_Offline.html"  it will be around 500mb for the system and the w3wp.exe process is at 500k. This is the same for all systems, I put the application online, and only the w3wp.exe process RAM usage on the various systems is out of control on the production system.

One problem I have is we are running production so I cannot mess around with the production environment. I can capture the information but I cannot restart, nor do updates at this time.

The compiled source running on all the systems is the same.
0
 
EazyWorksAuthor Commented:
I have a question then,
can he number of cores, or number of processors affect the RAM utilization of a Executable like IIS w3pw.exe. I have started another identical production server with no users logged in or accessing the system, and the RAM footprint is still around 18 GB of RAM.
0
 
madginoCommented:
Probably you'll need to plan for some downtime and do some tests on production.
My suggestions are:
- try to restart the server
- try to play with the application pool (change it , create new one, play with memory settings)
- try to do some memory debugging:
http://support.microsoft.com/kb/919790
- check the IIS forum for other suggestions
http://forums.iis.net/search/SearchResults.aspx?q=memory

In the end it might be a "normal" IIS behavior, maybe it's getting from start a percent of available memory, probably you have a lot more RAM on production that test and that's why this happens.
How are the percentages Used RAM/Total RAM for all 3 machines?
0
 
TI2HeavenCommented:
I still need to know more about de application....
Does you have access to the source code?
¿does the application use global.asax?
In a web application there are three memory scopes: page level memory (stack), session level memory (heap) and application level memory (heap).
On application starts events, application level memory is allocated.
Memory allocation depend only of code logic, some Microsoft components can query the system to take a decision about memory usage.
I assume that memory usage its high but you are not finding errors on the system.
Just because I only know you are using a SQL server, I would suggest you to trace the application connection on SQL server to find more clues.
0
 
EazyWorksAuthor Commented:
How about the ConcurrentDictionary class?
I am going to make a test application, but I am leaning towards the class uses more memory when more cores are present to synchronize the access.
Production has 24 cores and all other systems range from 2 to 4 cores.
We see about a 5x memory increase on the production server.
0
 
EazyWorksAuthor Commented:
I also have a duplicate production server i can alter to test. I have managed to exclude the api access and the user load.
On the cloned system with identical server specs, the server acts the same way. It is a little lower in usage but still a factor of 5 bigger than on the development and qualification systems.
0
 
TI2HeavenCommented:
ConcurrentDictionary class might be the answer.  This object class has a private variable array, that automatically resize itself when is full. This means that if the object is full, a new array is allocated and all the elements of the old array need to be copied from the old array to the new one. So the event of getting full has such penalty that this class worth to be oversized.
I have readied the help but there is no mention of limit the first allocation size.
The following articles explain what is going on behind the scenes:
Article
0
 
TI2HeavenCommented:
The article talks about single thread algorithms but you can there is and parallel processing strategy where you divide memory in chunks an you execute the same procedure en each chunk. Optimal number of chunks depends on number of cores.
0
 
EazyWorksAuthor Commented:
I am putting up some of my findings, we will be running some additional testing as well as a test case not using the ConcurrentDictonary.

I used the Processor Affinity Mask to attempt to simulate the application running with fewer cores. It did show that the number of cores affected the application intial RAM footprint, but not at the level we are seeing with actual physical core differnces.

- 24 Cores = 14.78 GB RAM
- 12(IIS Affinity Mask) Cores = 13.26 GB RAM
- 4(IIS Affinity Mask) Cores = 12.57 GB RAM
- 2(IIS Affinity Mask) Cores = 12.48 GB RAM
- 2(Qual) Cores = 4.4 GB RAM
- 4(DEV) Cores = 3.4 GB RAM

On all of these system we are running the same compiled .NET web application as well as the same database attached.

We are leaning towards the fact that it might be more based on the threads running that access the ConncurrentDictornary, I am extended a test program that I hope will mimic the same behavior.

We are also going to replace the ConcurrentDictionary and give it a test as well.

We will be marking out answers in a few days and any extra information that is posted is appreciated.
0
 
madginoCommented:
I guess after all these tests you will be the expert on this thread :).
0
 
EazyWorksAuthor Commented:
0
 
TI2HeavenCommented:
I fell like you start making a question and now you are sharing your learning experiences.
I have advanced you that as you suspect cores and free memory are taken into account to allocate initial memory in a program. I introduce you to the reason and I give you a very interesting article that talks about classes that are hungry of memory (the collections).
It seems that you want to get further and I would like to warm you that this topic is huge and the pill you send us is not the best way to start with. Before you can really understand JIT designers’ decisions you need to get some background of hardware efficiency guides.
This guides use c++ as programing language but the concepts they describe may be used in any language. I am sending you some references that are available for free from internet.
Agner manuals
Intel old guide
(There are more recent intel’s manuals but they skip some very important concepts given in this guide. Once you read this one you can go to most recent one).
Please don’t leave this question open for ever.
0
 
EazyWorksAuthor Commented:
Closing thread, thanks for the feedback.
0

Featured Post

Get quick recovery of individual SharePoint items

Free tool – Veeam Explorer for Microsoft SharePoint, enables fast, easy restores of SharePoint sites, documents, libraries and lists — all with no agents to manage and no additional licenses to buy.

  • 11
  • 5
  • 4
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now