What is the registry key for IE8 setting "Enable memory protection to help mitigate online attacks"?

While I'm not sure about the registry settings, Microsoft says you can do it with the boot.ini as a system-level policy:

http://technet.microsoft.com/en-us/library/cc700810.aspx

You can also use RegMon (http://live.sysinternals.com/regmon.exe) to monitor which registry entries are changed when you modify the DEP setting in the CP applet (System -> Performance -> DEP).  Chances are, though, since it appears as a boot-time setting, you'll probably need a reboot for it to engage properly.  That's just a guess.

Some further research indicates this key has something relevance here:

HKLM\Software\Microsoft\WindowsNT\CurrentVersion\AppCompatFlags\Layers

I have not seen an explicit explanation of the keys, but it sounds like it holds the in/exclude list for DEP.

Thanks for the comment but I was hoping for the reg key the reflects the setting in Internet Explorer-->Tools-->Internet Options-->Advanced-->Enable memory protection to help mitigate online attacks

The information I read implied that is just an IE-specific setting for the same DEP UI found in the system applet.  It sounds like changing that setting in IE simply creates an opt-in entry for DEP.

Still, you can use RegMon to verify the registry keys used.  That should expose exactly what you are looking for.