Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Page File/ memory usage

Posted on 2007-07-25
22
Medium Priority
?
374 Views
Last Modified: 2013-12-16
I have a windows server 2003 that houses an application taht is very thread and socket intensive. the application, when under normal running conditions, has approximately 600 threads and 600 listeners running. MY question relates to the memory/ page file usage on the server. According to Task Manager, My available memory is usually around 780M and the system has 1G installed. The page file usage is around 900m - 1G. I would like to know why this is. I know it directly relates to the application, but I am not sure how and I can find nothing on the WWW to help.
Also, the application in question was develeoped in house by me using C#, if that helps in anyway.
Thanks in advance.
0
Comment
Question by:codejunky
  • 11
  • 11
22 Comments
 
LVL 37

Accepted Solution

by:
gregoryyoung earned 1000 total points
ID: 19568324
Well ... each thread has a stack ... that stack is 1 MB so 600 threads would quickly ramp you up to 600 mb of memory before these threads even do anything.

Can i ask *why* you have 600 threads open? Are you using a thread per socket model? This model is generally not preferred (async sockets are preferred) because they get run in IOCP threads which already exist on your system and are far more scalable.

Cheers,

Greg
0
 

Author Comment

by:codejunky
ID: 19568381
my memory usage is remaining low, but the page file usage is what is ramping up. I create each listener in its own thread, but maybe I shouldn't. What you are saying regarding the 1MB stack seems logical considering what I see when I start the service. is the stack held in the page file exclusively?
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19568409
"my memory usage is remaining low"

How are you measuring this? Task manager by chance? Task manager doesn't show you actual memory usage it shows you your working set size. http://www.itwriting.com/dotnetmem.php explains more on the difference between the two. You want to use perfmon to check this (look at private bytes).

0
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

 

Author Comment

by:codejunky
ID: 19568410
Oh, and I am using async sockets.
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19568423
if you are creating every listener in its own thread how are you using async sockets? Perhaps there is a terminology problem here?
0
 

Author Comment

by:codejunky
ID: 19570148
basically I have a listener class that creates it's own thread for the listner to wait in. It is based on this example I found on the web.

http://www.csharphelp.com/archives3/archive486.html

0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19570261
ok but how many "Listeners" do you have? You only need one per port you are listening on, are you listening to 600 ports?
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19570307
btw: that code is a fairly bad example there are some issues with it :)

Cheers,

Greg
0
 

Author Comment

by:codejunky
ID: 19570491
Yes, I am listening on 600 different ports and growing.

I would welcome any examples that are better... This was my first attempt at a socket applcation and it was a rather indepth project. since I first used the example, I have changed probably 90% of the code, but the thread creation has remained the same. I may not be understanding something , but it seems like the only way to handle multiple listeners.
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19570536
why are you listenning on 600 ports? you only need to listen to 1 port generally then when you accept the socket it gets accepted on another port. I just put up a small example of a purely async server to illustrate some buffer management work I did.

http://codebetter.com/blogs/gregyoung/archive/2007/06/18/async-sockets-and-buffer-management.aspx original discussion

http://codebetter.com/blogs/gregyoung/archive/2007/07/20/async-sockets-and-buffer-management-ctd.aspx (example)

There are no managed threads used in the example at all (only IOCP) and it supports 10k users easily.
0
 

Author Comment

by:codejunky
ID: 19571105
the multiple ports is a design necessity. I will check out the links above. I appreciate your help. Your answers have explained what I was originally asking. I was completely unaware of 1M stack per thread. I am now trying to find a way to change that. Thanks alot for your help.
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19571111
you say you are listenning on 600 ports ... can i ask why? this is very unusual.
0
 

Author Comment

by:codejunky
ID: 19571327
I really like to be different...

Actually, it's an application conceptually similar to what they are running at gotomypc.com.
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19572058
Yes i understand but you onlyneed one listener, it accepts sockets to other ports
0
 

Author Comment

by:codejunky
ID: 19575050
How exactly to do I create listener that will answere calls on both port 11000 and 12000 at the same time?
0
 

Author Comment

by:codejunky
ID: 19576130
Maybe I should have said create a single listener bound to 2 ports simultaneously
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19576481
well for that you create 2  using your current methodology ... but you can do it without threads ... the example I gave above does listen on a port without any threads.

I can understand 2 or even 10 but 600 seems a bit over the top :)
0
 

Author Comment

by:codejunky
ID: 19577035
It probably wouldn't if I could explain the applicaiton in more detail, but I am not at liberty to speak very intensively about the inner workings without some sort of confidentiality in place. I am really not trying to be difficult, I am just limited regarding what I can and can't say.

Thanks,
Chad
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19577060
No worries. Either way check out my code, it doesn't need a thread for a listenner. I am copying/pasting the relevant portion here for you.

    class SimpleServer
    {
        private IPEndPoint m_LocalEndPoint;
        private Socket m_Listener;
        private List<Connection> m_ActiveConnections = new List<Connection>();

        public SimpleServer(IPEndPoint _LocalEndPoint)
        {
            if (_LocalEndPoint == null) { throw new ArgumentNullException("_LocalEndPoint"); }
            m_LocalEndPoint = _LocalEndPoint;
        }

        public void BeginListenning()
        {
            if (m_Listener != null) { throw new Exception("Already listenning"); }

            m_Listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            Console.WriteLine("Begin Listenning on " + m_LocalEndPoint.ToString());
            try
            {
                m_Listener.Bind(m_LocalEndPoint);
                m_Listener.Listen(100);
                m_Listener.BeginAccept(new AsyncCallback(AcceptCallback), m_Listener);
            }
            catch (Exception Ex)
            {
                Console.WriteLine("BeginListenning" + Ex.ToString());
            }
        }

        private void AcceptCallback(IAsyncResult ar)
        {
            Socket Listener = (Socket)ar.AsyncState;
            Socket Handler = Listener.EndAccept(ar);
            Listener.BeginAccept(new AsyncCallback(AcceptCallback), Listener);
            Console.WriteLine("Received Connection From " + Handler.RemoteEndPoint.ToString());
            Connection c = new Connection(Handler);
            c.ConnectionLost += new ConnectionLostHandler(Client_ConnectionLost);
            lock (m_ActiveConnections)
            {
                m_ActiveConnections.Add(c);
            }
        }

        void Client_ConnectionLost(Connection _Connection)
        {
            Console.WriteLine("Lost connection from " + _Connection.RemoteEndPoint.ToString());
            lock (m_ActiveConnections)
            {
                m_ActiveConnections.Remove(_Connection);
            }
        }
    }




Then it is called in the form:

    class Program
    {
        static void Main(string[] args)
        {
            IPEndPoint p = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 12345);
            SimpleServer server = new SimpleServer(p);
            server.BeginListenning();
            Console.WriteLine("Server up and listenning hit enter to end program");
            Console.ReadLine();
        }
    }
0
 

Author Comment

by:codejunky
ID: 19577544
that makes more sense than the infinte loop in my example. thanks for the advice. that should significantly reduce the memory foot print. not that it is overly necessary( I am only putting around 900 listeneres on a single server and memory is cheaper than time.), but it definately seems like a better implementation. since you illustrated the 1M stack size, I have reduced the stack size to 512k for these threads and that has helped alot form a size perspective.
0
 

Author Comment

by:codejunky
ID: 19577914
well, that was quick. I changed the code and the private bytes footprint with 600 active connections is 70M as opposed to 600M before. I appreciate your help.

Chad
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 19577989
:)



glad to help. see the rest of that post on buffer management as well (makes a big difference)
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Great sound, comfort and fit, excellent build quality, versatility, compatibility. These are just some of the many reasons for choosing a headset from Sennheiser.
In the modern office, employees tend to move around the workplace a lot more freely. Conferences, collaborative groups, flexible seating and working from home require a new level of mobility. Technology has not only changed the behavior and the expe…
The viewer will learn how to use and create keystrokes in Netbeans IDE 8.0 for Windows.
The viewer will learn how to use and create new code templates in NetBeans IDE 8.0 for Windows.
Suggested Courses

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question