SetWindowsHookEx crashes 16-bit apps

I have a program which uses SetWindowsHookEx() to install a WH_GETMESSAGE hook. I set the thread id to 0 so that the hook is loaded in every process. My problem is that my hook procedure is being called for 16-bit processes, which crash everything.
What should I do to make my SetWindowsHookEx() work ignoring the running 16-bit applications?
(Other than rewriting my hook DLL in 16-bit...)

Thanks!
olegolegAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

MadshiCommented:
Normally that should be no problem. AFAIK it's Windows' problem to make your hook function work for 16-bit. So I guess, there's something wrong with your hook function code or calling convention. Can you please post the hook function code here?
0
eppsmanCommented:
I've had the exact same problem, and what I ended up doing was enumerating the processes using ToolHelp, then calling SetWindowsHookEx() for each one. For each process, i'd grab the path to the main .exe module from ToolHelp, and feed that to SHGetFileInfo() to tell me what platform the EXE was targeted for. For more info on SHGetFileInfo(), see http://msdn.microsoft.com/library/psdk/shellcc/shell/Functions/SHGetFileInfo.htm

Another way to approach it is to try using GetProcessVersion() on each process as you enumerate them, but it doesn't give as much detailed info.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
MadshiCommented:
You could ask "GetProcessVersion(processID)" instead of calling SHGetFileInfo. Should be a bit faster, I guess...
0
Become a Certified Penetration Testing Engineer

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

MadshiCommented:
And the toolhelp functions are not available under NT4.
0
olegolegAuthor Commented:
Thanks for your comments eppsman.
Unfortunately, as Madshi pointed out, the Toolhelp API is not available under NT 4 (I'd have to run through the registry there), and more importantly - your suggested solution is only good for "currently running" processes.  
I need my hook to be applicable for processes created after my Hook install proc.

After further investigation on this issue, I did manage to solve the problem.
It turns out that the 16-bit application crash only if I use MSVCRT.DLL (The runtime libraries) or do some heap allocations, in my hook code.  Once I removed the runtime libraries, and took care of checking the process version on the DLL_ATTACH (Using GetProcessVersion()), returning False if its a 16bit process, everything worked.
I now have a 32bit hook DLL, which once installed using the standard SetWindowsHookEx(0...) API, doesnt crash the 16-bit applications.

Thank you all for your insights.
0
MadshiCommented:
Olegoleg, please review this question and the comments and then tell me WHY did you accept eppsman's answer??? Don't you think that the points should have gone to me???
0
olegolegAuthor Commented:
Hi Madshi,

Well to be honest, neither one of you actually "desrved" the points, as no one pointed out what was the problem.
I guess you do desrve them more than eppsman with your "AFAIK it's Windows' problem to make your hook function work for 16-bit" (although this too is not %100 correct because who said using the MSVCRT runtime libraries is a no-no?).

Anyway, eppsman posted an answer first.
Please don't take this personally.
Thank you for your valuable insights.
(Wish I could split the points among you too)
0
MadshiCommented:
Well, I couldn't tell you what the problem was, because you didn't give me the sources (I asked about them in my comment). And that's why I only posted a comment instead of an answer. Well, it's not the first time I get punished for being so polite to write a comment instead of an answer first...
But forget about it. It's not so important.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Development

From novice to tech pro — start learning today.