I'm writing an MMO game in VB.NET and C#, and am researching anti-cheat methods. Here's what I've got so far in terms of actual code and ideas:
1) Active detection of cheat applications.
I can detect Cheat Engine, ArtMoney, etc. through process monitoring (process name, process directory and file hash) and FindWindow'ing application-unique classes (e.g. Cheat Engine's window is easily detected by looking for the TMemoryBrowser class). The definitions of such applications can be updated within the source code for each version.
2) Self-hashing of executables.
The SHA1 hash of the executable is calculated post-build and is signed using RSA with a key pair stored on my development machine. The signed hash is appended to the executable file and the application checks it upon run. The public key is stored inside the executable as a CSP blob in bytes. Without the correct hash, the application just crashes.
3) Server side code handles all operations.
The game application itself simply serves as a frontend to the server. Each operation performed in the GUI by the user causes a message to be sent to the server. This means that no operations are actually performed on the client side - instead the client requests that the operation is performed by the server on its behalf. The server runs Apache and the backend is simply coded in PHP. Communication is over SSL and the game application checks that the URL of the certificate and the issuer information is correct. Each message sent has a handshake protocol in which the client's request sends a random number, which the server joins with another random number and an unchanging string and hashes it. The response is sent back with the server's random number and the resulting hash. The application checks the hash and if it is correct the message is authenticated.
4) Encryption on stored sensitive data.
Certain bits of sensitive information that must be stored within the game executable are encrypted using a simple algorithm. This is intended more as a deterrant than a security measure.
5) Security violation logs.
If unexpected requests are made by a client, the server records them in a table. If an account or IP address makes too many violations within a set period of time, they are temporaraly banned. Repeat offenders get permenantly banned (accounts that are permabanned get deleted after 7 days if an admin doesn't intervene). Certain violations result in an instant permaban.
6) Purposeful crashes.
If a user attempts to circumvent security measures, the application can crash itself. Unconventional crash methods are used (calling APIs with bad signatures, invalid casting, division by 0, etc) along with the usual Throw New Exception to make preventing the crash more difficult. These methods are not all called from a single method, but rather are hard coded separately to different cases, In most cases, more than one method is used and may be called from a separate thread or sequentially in the method code.
7) Dual session IDs
Not only is the usual PHPSESSID used, but a second session ID is used to track a client to a specific user account.
8) Secure variables
To make memory searching/editing harder, some values are stored in a secure variable class that encrypts them and decrypts them for use.
As far as all of this go, most of it can be circumvented by a talented cheater. I'd like to know if there are any specific extra security measures I can take to help secure my game from cheaters, without resorting to shelling out a million pounds to get PunkBuster or VAC on my game. I had a look at DMW and it's interesting but not really what I had in mind. What to games like Dawn of War and Command and Conquer do to stop people from cheating?