SetWindowsHookEx() fails inside a thread of Windows service created using AfxBeginThread()

I’ve written a piece of code that works perfectly fine inside an executable on Win 2k platform, but does not work when invoked from a multi threaded Windows Service.

The task is to track change in value of a status bar and items of a list control on a 3rd party executable. The window for this target application is first searched using EnumWindows() and than the following code is executed

g_hinstDll = LoadLibrary("hookDLL.dll");

if (g_hinstDll == NULL) {
// Set WH_CALLWNDPROCRET hook on the 3rd party application thread            
g_hhook = ::SetWindowsHookEx(WH_CALLWNDPROCRET, (HOOKPROC)CallWndProcRet, g_hinstDll, m_target_threadid);

if (g_hhook == NULL) {
TRACE("Can't create hook! Error Code = %d\n",GetLastError());

This code works perfectly when compiled as an executable. But if we run this code in a thread invoked using AfxBeginThread(), inside a Windows Service, it fails with the error code ERROR_ACCESS_DENIED after the SetWindowsHookEx call.

Q - Can somebody give some pointers on what could be the reason for the error message?

Additional Information –
•      The service has the option ‘Allow service to interact with the desktop’ option selected, to facilitate search of the target application window thread.

•      Currently logged on user is part of the Administrators group.

•      Inside this particular thread in the service, i’m able to open the target process using these calls
OpenProcess(STANDARD_RIGHTS_REQUIRED, FALSE, m_target_processid)

•      Call for starting the thread is
Who is Participating?
jkrConnect With a Mentor Commented:
>>but does not work when invoked from a multi threaded Windows Service.

What kind of messages of would you like to hook on an invisible desktop that has no windows? See ("Why Do Certain Win32 Technologies Misbehave in Windows NT Services?"):

On Windows NT 4.0, only one window station can be visible, WinSta0. The visible window station is also defined as interactive. A nonvisible window station is noninteractive. A nonvisible window station does not receive any user input and no display device is associated with it.
As stated earlier, desktops are contained within window station objects, just as a directory contains files. A desktop object contains a logical display surface and includes windows, menus, and hooks. Only desktops belonging to the visible window station can be seen and receive user input and only one desktop can be seen and receive input at a time. This desktop is known as the active desktop.
__SargeAuthor Commented:
Thanks for your comment. The article is very informative.

The application that has to be hooked is on the interactive desktop. The service thread that is trying to hook this interactive desktop application, is running under LocalSystem account, with 'Allow service to interact desktop' option selected. This allows the service thread access to the interactive desktop.

The service thread has following piece of code

g_hinstDll = LoadLibrary("hookDLL.dll");

g_hhook = ::SetWindowsHookEx(WH_CALLWNDPROCRET, (HOOKPROC)CallWndProcRet, g_hinstDll, m_target_threadid);

SetWindowsHookEx() fails with  ERROR_ACCESS_DENIED.

Could it be related to g_hinstDll handle? This module of this handle is under LocalSystem context. Is this creating a problem?

Any other idea on what could be reason for this error?

>>Could it be related to g_hinstDll handle?

No. It is related to the fact that you cannot hook anything from a service running on an invisible desktop.
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

>No. It is related to the fact that you cannot hook anything from a service running on an invisible desktop.

Isn't it more accurate to say: You cannot hook anything in a thread that belongs to a desktop other than the current one?  

I imagine the ERROR_ACCESS_DENIED relates to the attempt to cross this desktop boundary.  However, threads running within the scope of the service will be receiving messages (just not WM_PAINTs), and I think these could be hooked if desired.  (Though that's not what Sarge is trying to do.)

'Allow service to interact desktop' does not get around the above restriction. If you need the main part of your program to run as a service, I think you will have to factor out the part that establishes the hook into a separate process. The service could create this on the interactive desktop and perhaps communicate with it via a named pipe, if necessary.

Hi Sarge,

Well, I may have been wrong.  My apologies.  

I thought that "interactive services" still ran in a separate Window Station and desktop, and yet somehow were allowed to interact with the user.  Having looked for more information via Google, it sounds like they actually run in the interactive Window Station/desktop.  I can't find an explicit statement of this, but it's implied by various things I've seen.

I found this MS KB article relating to NT4 -;EN-US;163892

So, unless Win2k has been deliberately changed to prevent this, it sounds like what you are trying ought to work.  Although, MS actually recommend the approach I mentioned above, for security reasons  -

__SargeAuthor Commented:
Thanks for all your inputs.

I have already tried launching a new process(Win32 exe) through CreateProcess() in the interactive desktop. I could see the GUI of the launched exe, but even this exe could not hook the target application. The error returned was the same  ERROR_ACCESS_DENIED.

I cannot use CreateProcessWithLogonW() as the service also needs to run on WinNT4.0. I would see if CreateProcessAsUser(), works in this case.

colmccConnect With a Mentor Commented:
Hi Sarge,

Yes.  CreateProcess would still give ERROR_ACCCESS_DENIED because the user would still be LocalSystem. (Assuming that the problem is the one described in the MS KB article.)

CreateProcessAsUser sounds like a good idea.  You would need to get hold of an access token for the logged-on user, which can be a bit tricky I think.  Possibly OpenProcessToken on the hook-target process, followed by ImpersonateLoggedOnUser might do the trick.

But it is still a bit worrying that the MS KB article suggests that the problem you are seeing with the service was fixed in NT4 SP2.  Maybe there is something else going on here?

A few things you could try :

Run the service on NT4 (SP >= 2).  Do you get the same error?

Replace AfxBeginThread with the Win32 function CreateThread. (It seems unlikely, but maybe MFC is screwing things up somehow.)

If the problem really is that LocalSystem has no access to the user's desktop, perhaps it would be possible to allow this access (DF_ALLOWOTHERACCOUNTHOOK), using SetUserObjectInformation.

None of the above is particularly well thought through, but maybe it will give you a few ideas.

Good luck,
All Courses

From novice to tech pro — start learning today.