I'm writing an anti-cheat client that monitors a user's system while they play games to make sure they aren't cheating. I realize this may seem unethical, but the idea is that the player will voluntarily install and run the client in order to participate in online leagues and tournaments (where cheating is becoming more and more common).
One of the measures I use to make sure the player isn't cheating is to periodically capture their screen. Many cheats allow users to see through walls or gain other similar unfair advantages. If I can spot these artificial enhancements in the screenshot, I've caught a cheater.
This method works wonderfully against most cheats, but fails against the superior ones. Expensive, subscription based cheats have started "spoofing" screenshots by hooking bitblt/stretchblt, disabling the cheat, and returning a "clean" screenshot, after which the cheat is reactivated. I've tried getting around this by using directx and the front buffer to capture the screen (since all the cheats I've studied only worry about bitblt/stretchblt, and now glreadpixels), but when I test this method in fullscreen games I get black screenshots.
To make matters worse, most "good" cheats run in kernel mode, making it a lot tougher to spot/control them. This is why I'm hoping I can find a way to insure real screenshots so I don't have to make a driver to scan for cheats. So, is it possible to keep these cheats from spoofing my screenshots (or get clean directx screenshots in full screen mode)?