Machine not booting using /PAE /3GB

I am using this machine as a dedicated Oracle Server. This has 8 GB memory installed, but while upgrading it to 32 GB memory and updating the boot.ini with /3GB switch (multi(0)disk(0)rdisk(0)partition(1)\WINNT="Windows Server 2003, Enterprise" /fastdetect /pae /3gb /NoExecute=OptOut), the server didn't boot. Now I have reverted it to 8 GB and it has booted successfully. Can anybody let me know why upgrading memory failed to boot the system and if /3GB switch has any relation to memory upgrade. How do I solve this problem? I need to upgrade the RAM and also use /3GB concurrently, is this possible?
Who is Participating?
schwertnerConnect With a Mentor Commented:

Just flipping /3GB isn't enough for most programs. It has the effect on the kernel, true, but unless your process's executable has the Large Address Space Aware flag set, Windows won't actually give you the full 3GB space. Link your executable with /LARGEADDRESSAWARE or use EDITBIN.

The reason you have to do this is that some programs were written to assume that bit 31 of a user-mode pointer would always be 0 and therefore that this bit could be reused for some other purpose. Some applications do this with bits 1 and 0 because for a 4-byte aligned quantity, bits 1 and 0 are also 0. When it wants to use the pointer, the repurposed bits values are stored somewhere, then they're set to 0.

Of course, as soon as you start getting pointers in the 2-3GB region, bit 31 is set. The application sets the bit to 0 - whoops, you're pointing at the wrong place.

So unless you're working with an application you know is large address aware (Exchange Information Store and SQL Server spring to mind) don't bother enabling /3GB, it'll hurt kernel address space but do nothing for your processes.

As for what gets loaded into kernel VAs: memory sections for all loaded kernel mode modules (NTOSKRNL.EXE, HAL.DLL, WIN32K.SYS and a whole load of drivers), paged pool and non-paged pool (used for all the drivers' and system components' private memory heap allocations - non-paged pool for memory that may be accessed when paging is disabled, paged pool for everything else), system and process page tables, system and process working set lists, and the system caches.

Reducing the system space to 1GB with /3GB causes the maximum paged pool size to drop, according to "Inside Windows 2000", from 482MB to 160MB, and removes a whole area 448MB in size which is used for system page table entries and system cache. The file system cache in Windows is based on mapped files: the system maps blocks of files 256KB at a time into virtual memory; when a process reads from or writes to a file, the virtual addresses are touched which causes the memory manager to swap data in from the file. This includes filesystem metadata - my system is currently showing 15MB 'valid' (currently paged in), 6MB 'standby/dirty' (removed from the working set and will be reused at some point, after being written to disk if dirty) and 542 views for a total of about 135MB of VA space for $Mft, the NTFS Master File Table. This is on a 512MB RAM system.

So reducing the amount of space available to the kernel can seriously affect disk caching and the ability of various components to allocate memory - probably reducing the number of asynchronous I/Os that can be pending. For the two components I mentioned before, Exchange Information Store and SQL Server, it can be worthwhile because they do a lot of their own disk buffering and a lot of unbuffered I/O to their transaction logs (e.g. STORE writes the entire content of every message to the transaction log before storing it in the appropriate database).

You can tune things even more on Windows XP and Server 2003 with the /USERVA switch.
scuthberConnect With a Mentor Commented:
I didn't think you had to use both /3GB and /PAE? I thought it was neccesary only to use /PAE with Enterprise Edition?
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

fcadenaAuthor Commented:
I should have been more clear about the server "not booting". The BIOS recognized the 32GB OK, Windows loaded and was able to see 16GB as expected (since the /3GB on a 32bit 2003 OS) but the behavior was erratic. I was not able to login to the my normal domain, if I attached a keyboard the machine would not recognize it, RDP wouldn' work (I had to use ILO and login locally). Once I removed the new memory and put back my old 8GB, everything work perfectly. If I use the 32GB without the /3GB the server works perfectly also.
I am the Sys Admin and not the DBA and they are telling me their DBs are fine-tuned to used the /3GB. My question to them (no answer yet) was:
1) Would oracle work better with 16GB of RAM (using /3GB, 3GB for Apps, 1GB for the OS) or using 32GB RAM (without /3GB, 2GB for OS, 2 GB for Apps)?

I can only have 1 maintenance window a week, so I will try next using the /3GB without the /PAE as sugegsted.

schwertnerConnect With a Mentor Commented:
Oracle ... has many faces when it goes to implementation.
What do you mean?
1.  Do you mean many Oracle instances on this computer thats executables (background processes) and caches (SGA, user processes) need more then 3 GB RAM?
2. You need for particular Oracle instances space bigger than 3GB?

I think that 2 GB is more then enough for most Oracle installation (I mean background processes, user processes and SGA - System Global Area (the Oracle caches). Big SGA uses complex mechanism for maintanance, so  it will cause you problems.

So if you have many Oracle instances on this machine first see how the run without  any attempts to use large RAM chunks. I do not believe you use big SGA.

Make the following experiment:
1. Do not use the Windows RAM keys.
2. Start all Oracle instances
3. Using Task manager see if they utilize the RAM normally.

To utilize big RAM is not an easy task. Technically there are some details:

Enabling support at the OS level:
    AWE can be enabled at the OS by adding the /PAE switch to the boot.ini  
    as such:
multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /PAE
    It IS possible to have BOTH the /PAE and /3GB switch in place on the same
    machine, as such:
multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /3GB /PAE
    However, be aware that if BOTH switches are in place, the server will only
    be able to recognize up to 16GB of RAM.  If you are working with a server  
    with more than 16GB of RAM, you will need to choose between the two.
    It is important to note that once either or both of these switches are in
    place in the boot.ini, ALL processes running can take advantage of these
    switches.  Thus, in a case where multiple Oracle instances are running on
    the same server, ALL instances can take advantage of the additional memory                
    afforded by these switches, up to the amount of physical memory on the box.
Operating System Privileges Needed at the OS Level:
   In order to take advantage of the additional memory afforded through PAE,
   the operating system user account which is used to start the OracleService<SID>
   must be granted the 'Lock Pages in Memory' system privilege at the operating system
   level.   By default, the OracleService<SID> starts as the LocalSystem account.
   The LocalSystem account has the privilege to Lock Pages in Memory granted to  
   it by default.
   However, if you change the OracleService<SID> to logon as a user OTHER than
   LocalSystem, you may see the following errors when attempting to start the
   database with USE_INDIRECT_DATA_BUFFERS set to TRUE :
   SQL> startup pfile=c:\temp\initscott.ora
   ORA-27102: out of memory
   OSD-00010: Message 10 not found;  product=RDBMS; facility=SOSD
   O/S-Error: (OS 1300) Not all privileges referenced are assigned to the caller.
   To rectify this, you must grant the 'Lock pages in memory' privilege to the user
   that the OracleService<SID> starts as.  To do this, click on:
   Start ->  Programs -> Administrative Tools -> Local Security Policy
   (on a Domain Controller, click on 'Domain Security Policy' instead of 'Local Security Policy')
   Double-click on the 'Lock Pages in memory' policy.
   Add the appropriate user and click 'Ok'.
   Restart the OracleService<SID>
fcadenaAuthor Commented:
It is fine, but my concern is why RDP, Domain Logon and Keyboard attached to the system, does not work after upgrading the memory? Is it something like the address space where the NIC and PS/2 driver loads cannot be pointed / accessed or are these devices not loaded after memory upgradation. How to confirm? Will there be any IRQ conflict?
schwertnerConnect With a Mentor Commented:
You have to see how many NIC your computer has.
Does they are properly configured.

Also we understand the basics of Oracle the stuff, but the real Windows experts are in another thread.
Possibly they will be able to help you.
fcadenaAuthor Commented:
schewertner, thank you so much for all this great information. I'll will take it to the DBAs an see what they think. This is their box and they were the ones telling me to add these switches. However, this may have been from their previous 2000 OS and /PAE may not be needed in their 2003 version. Unfortunately, this is a production box running many oracle instances and they cannot make changes but once a month. I'll post any new information. In the mean time the DBAs think that this hotfix may help our problem. We'll try implement it on some other servers and see if it has any benefits.
schwertnerConnect With a Mentor Commented:
It will be important to measure the SGA of the instances.
Possibly if they are less then 2GB the box will run
good without switces. It will divide the available RAM
in virtual chunks, but they will be small.
For anyone who may see odd and weird problems after implementing the /3gb switch, I have some additional info on this issue.

Especially if you run a system with both /pae and /3b you may run into trouble. As mentioned earlier in the thread the maximum kernel space is reduced, and since memory pool is mapped inside this space, the max. memory that can be allocated running /3gb is about 15.5 GB.

IF you have allocated max memory 15.5 GB and are using /3gb, pay special attention to the amount of Free System Page Table Entries (PTE's). The value can be seen by adding a monitor in perfmon.

You would want at least 10k free entries and you would want as many as possible. If you go below 5-10k free entries, you may see very odd behavior from your system, which can include some of the symptons mentioned in this thread, for example (but not limited to):

- Random network failures on the box.
- Random components failing.
- Random Stuff in general not working.

This is because windows without enough free PTE's is running out of working space for whatever it has to do, and as such everything in the system will be randomly affected by this. Unfortunately there is no builtin monitoring that will alert you of this issue, and you has to check this value yourself.

Also be aware that even tho a system may seem to run just fine with max memory and /3gb, always always always check the PTE's before doing ANY upgrades to drivers or installing patches/new software/servicepacks to the system, because you may (very likely) have a system running on the edge, and even a driver update adding new functionality can push it over the edge.

In general - rule of thumb - if you can in any way avoid to use /3gb and /pae at the same time, do it!

This info is gathered from my own bad experience with the /3gb switch, and I now monitor the PTE's on all systems I have, since several of them need to use the /3gb switch.

I hope this is of use to other people using this boot parameter!
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.