How can I prevent SQL Server 2008 from exceeding the maximum memory?

Hm?! This option should to it. Have you rebooted the Server!?
http://searchsqlserver.techtarget.com/tip/Configuring-SQL-Server-memory-settings

Is it better only restart the SQL Server
However may be the problem eas not in the server but in the tables indexes
Did you have made a maintenance plan to the databases ?

@merowinger: The max memory setting was already set to 8Gb. The server has been rebooted but somehow SQL seems to ignore the setting.

@Aquinos: Haven't done that yet. I'll give it a try but I can't understand why SQL keeps lurking up all memory although I have set it to a maximum of 8Gb.

Hi,
Not problem with sql server.
Check for your paging settings on the sql box.

We need to think like that!
If the SQL need that memory he gonna pick it, right ?

So...

If you have a bigger database with a lots of users, and any replication, and the log file in the same server....

The users try to open the tables and (re)create the indexes.
If you don't shrink it it keep growing and growing

>> I have set the max memory setting in SQL to 8Gb but the server keeps using 11Gb.

I Seriously doubt it..
Just follow the steps specified below:

1. Right Click your Server in SSMS(after connecting it)
2. Choose Properties -> Click Memory
3. Make sure Maximum Memory field is set to 8000 MB ( Not that the default is MB).

Kindly confirm the value which is set now.

@rrjegan17:

AWE is turned off.
Min memory is 0 MB.
Max memory is set to 8096 MB.  (however sql is currently using 10.4Gb and growing)
Index creation memory is set to 0 KB.
Minimum memory per query is set to 1024 KB.

>> Min memory is 0 MB.

Its recommended to set Min Memory to at least 2048 MB or 4096 MB.

>> Max memory is set to 8096 MB.

Ok, Try restarting the SQL Server Service once the Min Memory is set..
Ideally Restarting is not required but a try to see what happens once.

@rrjegan17: I have changed the settings and restarted the server. SQL still uses 11Gb of memory. (so there's 1Gb left for the system)

Seems like that you have more than one instance of SQL Server installed in your machine..
Kindly post a snapshot of the list of services running in SQL Server Configuration manager to confirm it..

An instance of SQL Server will never use memory allocated / set by Max Server Memory but other instances can also use it..

Try applying the latest SP once for both SQL Server as well as OS and try it out.

@rrjegan17: All service packs and updates have been applied.

Can you post the result of the query below once:

select *
from sys.configurations
where name like '%server memory%'

As you see, the value_in_use for Maximum memory is still on 8096..
Therefore, do the following things to nail down where exactly the issue is:

1. Open task Manager -> Click View -> Select Columns
2. Choose Memory - working set, Memory - Peak working set, Memory - Commit size, etc.,
3. Now, you can see the difference that SQL server process is still using 8096

So, I hope the issue is that Virtual memory is not set in your system.
Kindly configure your Virtual Memory by following the steps below:

1. Right click My Computer -> Properties
2. Choose Advanced System Settings
3. Choose Advanced -> Click Settings -> Click Advanced menu again
4. Under Virtual memory -> Click Change
5. Make sure that you have Virtual memory set more  than the recommended value.
Rule of Thumb is that Virtual Memory = 1.5 * Physical Memory
Eg: If you have 4 GB of RAM, set Virtual memory to 6 GB

Restart your machine and try once.

@rrjegan17: Virtual memory was set to Automaticcly manage. I have changed this to manual and set it to 18432. (12288 * 1.5) I'm currently unable to reset the server but I'll let you know the results. Thanks for all the help. I appreciate it!

Sure, let me know if you face any issues so that I can help you out.

I have done a reset but the memory keeps lurking up all the memory.

I have reported the issue to Microsoft. Although the problem still exists all the points  are rwarded to rrjegan17 for al l the help.