sQL 2008 R2 Windows 2008 R2 Memory issues

I have a QA server running SQL 2008 R2 on Windows 2008 Server, 16gb of physical memory was installed, only 3-4 users are using the server via an application quering sql server database and sql is taking up all the memory.  No other databases or application is installed on this server.  

I have configured the memory down to 5gb in SQL memory settings, was ok for a day then the SQL took up all the physical memory, paging file increased and cache did.  spoke with several DBAs we have reconfigured the memory settings to 13gb we are no using.


For Windows Performance Monitor on this server I have set up the following counters  as first port of call

Memory Object – Cache Byes counter for the system working set
-      Pool Nonpaged Bytes counter for size of unpaged pool
-      Available Bytes Counter
-      Modified Page List Bytes counter

We have not enabled AWE memory settings

I have read the various blogs on sql and memory.  I am not convinced that the DBAs answer of setting 13gb of memory will work.

I need to rule out SQL, I need to rule out the Web tool calling on sQL, I need to confirm if windows 2008 is at fault, it is a 64 bit operating system 4 processors 2.67 GHz.