Mark
asked on
Need help decoding VNC challenge/response string
I'm trying to run x11vnc with the --passwdfile option. This option sends a challenge string to the VNC client. The client opens a dialog on the user's screen asking for a password. The client then sends the response string back to the VNC server. The VNC server calls the program referenced by the --passwdfile parameter with this string in the following format:
A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30 .next.16.#6....0
10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29 ....L...\,.\..!)
20: 3A DF C2 A2 7C E9 1F 1A D7 :...|....Challenge:
23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C
Response:
2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7ASKER
A Q Choudary: That is truly a great link! I've been blundering around experimenting on my own without such a good resource. I'll study that and perhaps change my VNC server since x11vnc seem to no longer be actively supported. But, that will take some time and further experimentation.
Meanwhile, the X11vnc server's -passwdfile switch has an option as described below, which does not appear to be described in your link.
At this point, what I'm looking for is not "HOWTO" on VNC generally, but rather how to use the challenge string to decode the response string. I'm not familiar enough with DES encryption to figure that out and have found nothing so far to help.
Meanwhile, the X11vnc server's -passwdfile switch has an option as described below, which does not appear to be described in your link.
If filename is prefixed with "custom:" then a custom password checker isI created such a custom command and my vnc client was presented with a login screen. The string sent to my custom command was as I presented in my OP. I believe this string contains the DES encoded password.
supplied as an external command following the ":". The command will be run
when a client authenticates. If the command exits with 0 the client is
accepted, otherwise it is rejected. The environment variables are set as in
-accept.
The standard input to the custom command will be a decimal digit "len" fol-
lowed by a newline. "len" specifies the challenge size and is usually 16
(the VNC spec). Then follows len bytes which is the random challenge string
that was sent to the client. This is then followed by len more bytes holding
the client's response (i.e. the challenge string encrypted via DES with the
user password in the standard situation).
At this point, what I'm looking for is not "HOWTO" on VNC generally, but rather how to use the challenge string to decode the response string. I'm not familiar enough with DES encryption to figure that out and have found nothing so far to help.
ASKER
Could I get some help finding an answer to this question? If nobody has any insights at all, then we might as well delete it. I'm looking for how to decode a DES encrypted string.
ASKER CERTIFIED SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
http://www.stuartellis.name/articles/vnc-on-linux/