Link to home
Start Free TrialLog in
Avatar of codejunky
codejunky

asked on

Page File/ memory usage

I have a windows server 2003 that houses an application taht is very thread and socket intensive. the application, when under normal running conditions, has approximately 600 threads and 600 listeners running. MY question relates to the memory/ page file usage on the server. According to Task Manager, My available memory is usually around 780M and the system has 1G installed. The page file usage is around 900m - 1G. I would like to know why this is. I know it directly relates to the application, but I am not sure how and I can find nothing on the WWW to help.
Also, the application in question was develeoped in house by me using C#, if that helps in anyway.
Thanks in advance.
ASKER CERTIFIED SOLUTION
Avatar of gregoryyoung
gregoryyoung
Flag of Canada image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of codejunky
codejunky

ASKER

my memory usage is remaining low, but the page file usage is what is ramping up. I create each listener in its own thread, but maybe I shouldn't. What you are saying regarding the 1MB stack seems logical considering what I see when I start the service. is the stack held in the page file exclusively?
"my memory usage is remaining low"

How are you measuring this? Task manager by chance? Task manager doesn't show you actual memory usage it shows you your working set size. http://www.itwriting.com/dotnetmem.php explains more on the difference between the two. You want to use perfmon to check this (look at private bytes).

Oh, and I am using async sockets.
if you are creating every listener in its own thread how are you using async sockets? Perhaps there is a terminology problem here?
basically I have a listener class that creates it's own thread for the listner to wait in. It is based on this example I found on the web.

http://www.csharphelp.com/archives3/archive486.html

ok but how many "Listeners" do you have? You only need one per port you are listening on, are you listening to 600 ports?
btw: that code is a fairly bad example there are some issues with it :)

Cheers,

Greg
Yes, I am listening on 600 different ports and growing.

I would welcome any examples that are better... This was my first attempt at a socket applcation and it was a rather indepth project. since I first used the example, I have changed probably 90% of the code, but the thread creation has remained the same. I may not be understanding something , but it seems like the only way to handle multiple listeners.
why are you listenning on 600 ports? you only need to listen to 1 port generally then when you accept the socket it gets accepted on another port. I just put up a small example of a purely async server to illustrate some buffer management work I did.

http://codebetter.com/blogs/gregyoung/archive/2007/06/18/async-sockets-and-buffer-management.aspx original discussion

http://codebetter.com/blogs/gregyoung/archive/2007/07/20/async-sockets-and-buffer-management-ctd.aspx (example)

There are no managed threads used in the example at all (only IOCP) and it supports 10k users easily.
the multiple ports is a design necessity. I will check out the links above. I appreciate your help. Your answers have explained what I was originally asking. I was completely unaware of 1M stack per thread. I am now trying to find a way to change that. Thanks alot for your help.
you say you are listenning on 600 ports ... can i ask why? this is very unusual.
I really like to be different...

Actually, it's an application conceptually similar to what they are running at gotomypc.com.
Yes i understand but you onlyneed one listener, it accepts sockets to other ports
How exactly to do I create listener that will answere calls on both port 11000 and 12000 at the same time?
Maybe I should have said create a single listener bound to 2 ports simultaneously
well for that you create 2  using your current methodology ... but you can do it without threads ... the example I gave above does listen on a port without any threads.

I can understand 2 or even 10 but 600 seems a bit over the top :)
It probably wouldn't if I could explain the applicaiton in more detail, but I am not at liberty to speak very intensively about the inner workings without some sort of confidentiality in place. I am really not trying to be difficult, I am just limited regarding what I can and can't say.

Thanks,
Chad
No worries. Either way check out my code, it doesn't need a thread for a listenner. I am copying/pasting the relevant portion here for you.

    class SimpleServer
    {
        private IPEndPoint m_LocalEndPoint;
        private Socket m_Listener;
        private List<Connection> m_ActiveConnections = new List<Connection>();

        public SimpleServer(IPEndPoint _LocalEndPoint)
        {
            if (_LocalEndPoint == null) { throw new ArgumentNullException("_LocalEndPoint"); }
            m_LocalEndPoint = _LocalEndPoint;
        }

        public void BeginListenning()
        {
            if (m_Listener != null) { throw new Exception("Already listenning"); }

            m_Listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            Console.WriteLine("Begin Listenning on " + m_LocalEndPoint.ToString());
            try
            {
                m_Listener.Bind(m_LocalEndPoint);
                m_Listener.Listen(100);
                m_Listener.BeginAccept(new AsyncCallback(AcceptCallback), m_Listener);
            }
            catch (Exception Ex)
            {
                Console.WriteLine("BeginListenning" + Ex.ToString());
            }
        }

        private void AcceptCallback(IAsyncResult ar)
        {
            Socket Listener = (Socket)ar.AsyncState;
            Socket Handler = Listener.EndAccept(ar);
            Listener.BeginAccept(new AsyncCallback(AcceptCallback), Listener);
            Console.WriteLine("Received Connection From " + Handler.RemoteEndPoint.ToString());
            Connection c = new Connection(Handler);
            c.ConnectionLost += new ConnectionLostHandler(Client_ConnectionLost);
            lock (m_ActiveConnections)
            {
                m_ActiveConnections.Add(c);
            }
        }

        void Client_ConnectionLost(Connection _Connection)
        {
            Console.WriteLine("Lost connection from " + _Connection.RemoteEndPoint.ToString());
            lock (m_ActiveConnections)
            {
                m_ActiveConnections.Remove(_Connection);
            }
        }
    }




Then it is called in the form:

    class Program
    {
        static void Main(string[] args)
        {
            IPEndPoint p = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 12345);
            SimpleServer server = new SimpleServer(p);
            server.BeginListenning();
            Console.WriteLine("Server up and listenning hit enter to end program");
            Console.ReadLine();
        }
    }
that makes more sense than the infinte loop in my example. thanks for the advice. that should significantly reduce the memory foot print. not that it is overly necessary( I am only putting around 900 listeneres on a single server and memory is cheaper than time.), but it definately seems like a better implementation. since you illustrated the 1M stack size, I have reduced the stack size to 512k for these threads and that has helped alot form a size perspective.
well, that was quick. I changed the code and the private bytes footprint with 600 active connections is 70M as opposed to 600M before. I appreciate your help.

Chad
:)



glad to help. see the rest of that post on buffer management as well (makes a big difference)