Disable Data Execution Protection in installshield

Posted on 2006-04-12
Last Modified: 2013-12-03

In my software I am using a third party activex component which is using some kind of encryption to protect itself. In windows server 2003 this component does not properly register, and my application will not properly execute. When Windows Data Execution Protection (DEP) is disabled, it is working fine.

So I want to disable DEP for the 2 executables in question. Is it possible to do this automatically from the installshield script? (I am using Installshield Developer 8). An alternative would be opening the dialog (Control Panel-System-Advanced-(Performance) Settings-Data Execution Protection, and instruct the user to add two exceptions there.
Question by:cwwkie
    LVL 86

    Accepted Solution

    See ("How to Configure Memory Protection in Windows XP SP2"), where you should be able to specify a list of exceptions for DEP. This should be the same for W2k3. ("You receive a "Data Execution Prevention" error message in Windows XP Service Pack 2 or in Windows XP Tablet PC Edition 2005") also describes this method. I am afraid that there probably won't be any programmatic way that does not require a reboot, since that kinda would defeat the purpose of DEP.
    LVL 14

    Author Comment

    > I am afraid that there probably won't be any programmatic way that does not require a reboot, since that kinda would defeat the purpose of DEP.

    I was afraid of that. But then it would be better to pause the installation process and instruct the user to add the exceptions, and continue after the user confirms the exceptions are added. I can get the advanced tab with (something equivalent to) "rundll32 shell32.dll,Control_RunDLL sysdm.cpl,,3", but the other steps must be done manually.

    But I leave this question open for some days in case someone else knows a better way.
    LVL 86

    Expert Comment

    You could use RegMon ( to see where these changes are stored in the registry (if so) in order to do these steps programmatically.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Why You Should Analyze Threat Actor TTPs

    After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

    zlib is a free compression library (a DLL) on which the popular gzip utility is built.  In this article, we'll see how to use the zlib functions to compress and decompress data in memory; that is, without needing to use a temporary file.  We'll be c…
    For most people, the WrapPanel seems like a magic when they switch from WinForms to WPF. Most of us will think that the code that is used to write a control like that would be difficult. However, most of the work is done by the WPF engine, and the W…
    This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA.…
    Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…

    794 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now