?
Solved

Detecting whether a debugger is running

Posted on 2007-07-20
14
Medium Priority
?
1,188 Views
Last Modified: 2012-06-27
I need to detect whether a debugger is running under Windows, Linux or Mac in C++.  How can I do this?
0
Comment
Question by:hach-que
  • 5
  • 4
  • 2
  • +2
14 Comments
 
LVL 86

Expert Comment

by:jkr
ID: 19536828
Under Windows, that's quite straightforward, just use 'IsDebuggerPresent()' (http://msdn2.microsoft.com/en-us/library/ms680345.aspx):

BOOL bDebuggerPresent = IsDebuggerPresent();
0
 
LVL 13

Expert Comment

by:josgood
ID: 19536832
For Windows, I believe you'll have to write a virtual device driver to hook Int 3.

I don't know enough to tell you how to do it.  VXD == virtual device driver.
0
 
LVL 13

Expert Comment

by:josgood
ID: 19536833
jkr's answer is SO much better!
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 2

Author Comment

by:hach-que
ID: 19537620
That's for windows, but does anyone know whether there is a similar library for Linux/Mac?
0
 
LVL 22

Expert Comment

by:grg99
ID: 19538056
There is no foolproof or portable way to do this, especially if the debugger is trying to hide, like any good debugger should.

On most OS'es, the system API's can be trapped, and in fact are by debuggers, so it's trivial for the debugger to notice you're calling IsDebuggerPresent() and return FALSE.



0
 
LVL 2

Author Comment

by:hach-que
ID: 19541514
But that does not explain how such software like Softwrap is able to detect debuggers then.
0
 
LVL 22

Expert Comment

by:grg99
ID: 19542399
It's a constant competition, every app that wants to detect debuggers or virtual machines or rootkits keeps adding tests, then (not so much the debuggers), but the virtual machines and rootkit writers patch those holes.  Almost every hole can be patched somehow.

Some techniques and their countermeasures:

(1)  Time how long it takes for the CPU to handle a divide by zero or other similar hardware exceptions.  If there's no debugger or VM or rootkit, the time should be relatively short,  otherwise the takes quite a bit longer as the VM or debugger gets involved.   Countermeasure:   simple, have RDTSC and all other timers return a fudged virtual time.

(2)  Follow a few API call paths and ensure they end up in a system DLL.   Countermeasure:  have ReadProcessMemory and other tracing API's return resonable looking call paths into the system.

(3)  Check for tricky hardware hat is rarely emulated correctly.  For instance the serial port 16550x chip has lots of features that are never used, and somewhat hard to emulate.  So checking whether bit 7 of status word 4 goes high when CTS is asserted but not after the 8th byte of the FIFO is empty but not in  trans mode on the second interrupt following a buffer empty, that might tell you something.    

But all those checks can be fooled by a clever debugger or VM or rootkit.



0
 
LVL 86

Expert Comment

by:jkr
ID: 19542446
No regular debugger would hide it's presence. No regular debuger patches API calls for that purpose, so that's a moot point. We're talking debuggers, not "malware".
0
 
LVL 22

Expert Comment

by:grg99
ID: 19544040
We  can't read the OP's mind, maybe they can elaborate on exactly what and why they're trying to detect a debugger.   Usually  the reasons are not so good,  or they might just be trying to keep people from  poking through their code.   That's really hard to do-- someone can either disassemble the code, or run it through a CPU emulator or VM, just far enough to see where they're testng for a debugger, then NOP that test.

0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
ID: 19550142
>>>> BOOL bDebuggerPresent = IsDebuggerPresent();
That checks whether the calling process was currently debugged, which is or is not the same as the asker meant with
>>>> I need to detect whether a debugger is running under Windows

>>>> especially if the debugger is trying to hide, like any good debugger should.
>>>> No regular debugger would hide it's presence.
>>>> We're talking debuggers, not "malware".
>>>> maybe they can elaborate on exactly what and why they're trying to detect
>>>> a debugger.   Usually  the reasons are not so good

Funny. A simple question and views so different !

jkr, if I am right, grg99 tells not from debuggers being malware but from programs trying to detect debuggers.

I wonder, whether we will hear the good reasons, grg99 was asking for.

Regards, Alex



0
 
LVL 22

Expert Comment

by:grg99
ID: 19550775
hmmm, it's a complex situation.

There are several kinds of "debuggers":

(1) A simple debugger,  it aims to let you run and debug your average plain-vanilla programs.   It does not have to take any extra measures to hide itself.  It can do simple API trapping and not hide its workings very much.  It doesn't have to simulate actual machine timing.

(2) A "system" debugger, it aims to let you run and debug more advanced and system level programs.  It has to take extra measures so you can step through the user to system API calls without seeing the debugger's hooks.

(3)  A "driver" debugger.  This is the real McCoy.  It has extra code to handle traps, interrupts, and exceptions.  It has to hide its own trap, interrupt, and exception hooks.

(4)  A "VM" debugger.  This is McCoy Sr.  It puts the machine in virtual mode,  the better to trap each and every thing that has to be emulated.  Since you often want to use it to debug other debuggers, it has to be extra careful to run and not impede the running of other debuggers under it.   Real tricky as it has to present the CPU as being in non-virtual mode when it actually is.

So it's a bit convoluted but the question remains, why does the OP want to do this?



0
 
LVL 2

Author Comment

by:hach-que
ID: 19629923
The reason for this question, is that I am creating a game engine with built-in encryption.  I want to make sure a debugger is not running so that they cannot simply see how a file is encrypted.
0
 
LVL 2

Accepted Solution

by:
hach-que earned 0 total points
ID: 19629926
I also want to stop disassemblers and if possible detect if it's running in a VM (not likely that I can do this, but oh well...)
0
 
LVL 22

Expert Comment

by:grg99
ID: 19630855
Several ways you can stymie disassemblers:

(1)  Encrypt the code.
(2)  Every other statement, insert some random unrelated statement.
(3)  Compile your code to some mythical machine's code.
(4)  Do one or more of the above, one or more times.


0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
We live in a world of interfaces like the one in the title picture. VBA also allows to use interfaces which offers a lot of possibilities. This article describes how to use interfaces in VBA and how to work around their bugs.
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question