Detecting whether a debugger is running

Under Windows, that's quite straightforward, just use 'IsDebuggerPresent()' (http://msdn2.microsoft.com/en-us/library/ms680345.aspx):

BOOL bDebuggerPresent = IsDebuggerPresent();

For Windows, I believe you'll have to write a virtual device driver to hook Int 3.

I don't know enough to tell you how to do it.  VXD == virtual device driver.

jkr's answer is SO much better!

That's for windows, but does anyone know whether there is a similar library for Linux/Mac?

There is no foolproof or portable way to do this, especially if the debugger is trying to hide, like any good debugger should.

On most OS'es, the system API's can be trapped, and in fact are by debuggers, so it's trivial for the debugger to notice you're calling IsDebuggerPresent() and return FALSE.

But that does not explain how such software like Softwrap is able to detect debuggers then.

It's a constant competition, every app that wants to detect debuggers or virtual machines or rootkits keeps adding tests, then (not so much the debuggers), but the virtual machines and rootkit writers patch those holes.  Almost every hole can be patched somehow.

Some techniques and their countermeasures:

(1)  Time how long it takes for the CPU to handle a divide by zero or other similar hardware exceptions.  If there's no debugger or VM or rootkit, the time should be relatively short,  otherwise the takes quite a bit longer as the VM or debugger gets involved.   Countermeasure:   simple, have RDTSC and all other timers return a fudged virtual time.

(2)  Follow a few API call paths and ensure they end up in a system DLL.   Countermeasure:  have ReadProcessMemory and other tracing API's return resonable looking call paths into the system.

(3)  Check for tricky hardware hat is rarely emulated correctly.  For instance the serial port 16550x chip has lots of features that are never used, and somewhat hard to emulate.  So checking whether bit 7 of status word 4 goes high when CTS is asserted but not after the 8th byte of the FIFO is empty but not in  trans mode on the second interrupt following a buffer empty, that might tell you something.    

But all those checks can be fooled by a clever debugger or VM or rootkit.

No regular debugger would hide it's presence. No regular debuger patches API calls for that purpose, so that's a moot point. We're talking debuggers, not "malware".

We  can't read the OP's mind, maybe they can elaborate on exactly what and why they're trying to detect a debugger.   Usually  the reasons are not so good,  or they might just be trying to keep people from  poking through their code.   That's really hard to do-- someone can either disassemble the code, or run it through a CPU emulator or VM, just far enough to see where they're testng for a debugger, then NOP that test.

>>>> BOOL bDebuggerPresent = IsDebuggerPresent();
That checks whether the calling process was currently debugged, which is or is not the same as the asker meant with
>>>> I need to detect whether a debugger is running under Windows

>>>> especially if the debugger is trying to hide, like any good debugger should.
>>>> No regular debugger would hide it's presence.
>>>> We're talking debuggers, not "malware".
>>>> maybe they can elaborate on exactly what and why they're trying to detect
>>>> a debugger.   Usually  the reasons are not so good

Funny. A simple question and views so different !

jkr, if I am right, grg99 tells not from debuggers being malware but from programs trying to detect debuggers.

I wonder, whether we will hear the good reasons, grg99 was asking for.

Regards, Alex

hmmm, it's a complex situation.

There are several kinds of "debuggers":

(1) A simple debugger,  it aims to let you run and debug your average plain-vanilla programs.   It does not have to take any extra measures to hide itself.  It can do simple API trapping and not hide its workings very much.  It doesn't have to simulate actual machine timing.

(2) A "system" debugger, it aims to let you run and debug more advanced and system level programs.  It has to take extra measures so you can step through the user to system API calls without seeing the debugger's hooks.

(3)  A "driver" debugger.  This is the real McCoy.  It has extra code to handle traps, interrupts, and exceptions.  It has to hide its own trap, interrupt, and exception hooks.

(4)  A "VM" debugger.  This is McCoy Sr.  It puts the machine in virtual mode,  the better to trap each and every thing that has to be emulated.  Since you often want to use it to debug other debuggers, it has to be extra careful to run and not impede the running of other debuggers under it.   Real tricky as it has to present the CPU as being in non-virtual mode when it actually is.

So it's a bit convoluted but the question remains, why does the OP want to do this?

The reason for this question, is that I am creating a game engine with built-in encryption.  I want to make sure a debugger is not running so that they cannot simply see how a file is encrypted.

Several ways you can stymie disassemblers:

(1)  Encrypt the code.
(2)  Every other statement, insert some random unrelated statement.
(3)  Compile your code to some mythical machine's code.
(4)  Do one or more of the above, one or more times.