Need help decoding VNC challenge/response string
I'm trying to run x11vnc with the --passwdfile option. This option sends a challenge string to the VNC client. The client opens a dialog on the user's screen asking for a password. The client then sends the response string back to the VNC server. The VNC server calls the program referenced by the --passwdfile parameter with this string in the following format:
A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30 .next.16.#6....0
10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29 ....L...\,.\..!)
20: 3A DF C2 A2 7C E9 1F 1A D7 :...|....
Challenge:
23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C
Response:
2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7
Asked:
Who is Participating?
Need to check document properly below is link of document:
http://www.stuartellis.name/articles/vnc-on-linux/
http://www.stuartellis.name/articles/vnc-on-linux/
A Q Choudary: That is truly a great link! I've been blundering around experimenting on my own without such a good resource. I'll study that and perhaps change my VNC server since x11vnc seem to no longer be actively supported. But, that will take some time and further experimentation.
Meanwhile, the X11vnc server's -passwdfile switch has an option as described below, which does not appear to be described in your link.
At this point, what I'm looking for is not "HOWTO" on VNC generally, but rather how to use the challenge string to decode the response string. I'm not familiar enough with DES encryption to figure that out and have found nothing so far to help.
Meanwhile, the X11vnc server's -passwdfile switch has an option as described below, which does not appear to be described in your link.
If filename is prefixed with "custom:" then a custom password checker isI created such a custom command and my vnc client was presented with a login screen. The string sent to my custom command was as I presented in my OP. I believe this string contains the DES encoded password.
supplied as an external command following the ":". The command will be run
when a client authenticates. If the command exits with 0 the client is
accepted, otherwise it is rejected. The environment variables are set as in
-accept.
The standard input to the custom command will be a decimal digit "len" fol-
lowed by a newline. "len" specifies the challenge size and is usually 16
(the VNC spec). Then follows len bytes which is the random challenge string
that was sent to the client. This is then followed by len more bytes holding
the client's response (i.e. the challenge string encrypted via DES with the
user password in the standard situation).
At this point, what I'm looking for is not "HOWTO" on VNC generally, but rather how to use the challenge string to decode the response string. I'm not familiar enough with DES encryption to figure that out and have found nothing so far to help.
Question has a verified solution.
Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.
Have a better answer? Share it in a comment.
All Courses
From novice to tech pro — start learning today.
It seems to me you've been quite patient -- your question having been open for 2 full months now.
I'm afraid I can't help you with the decode mechanism (although you might want to look into the openssl command to decrypt the DES encrypted password string. If that doesn't work for you, here's a page with 6 other encrypt/decrypt tools:
https://www.tecmint.com/linux-password-protect-files-with-encryption/
All of them can use stream input, so your shell can cat or echo the encrypted data into them.
But I must say, as a programmer myself, I often find I hit a roadblock in developing my own solution only to find a perfectly viable and easy-to-use solution is available elsewhere, if I'll just back away from the problem enough to notice the other options. (I personally refer to these projects as rabbit-hole projects, and I find I'm particularly susceptible to them myself).
In this case, I might suggest you look at the tigervnc project. They have a set of servers and clients already coded for all of the major Linux distros. Its Open Source, so if you really WANT to keep digging, you can always delve into their code to see how they did it!
Good luck!
Dan