• Status: Solved
  • Priority: High
  • Security: Public
  • Views: 150
  • Last Modified:

Need help decoding VNC challenge/response string

I'm trying to run x11vnc with the --passwdfile option. This option sends a challenge string to the VNC client. The client opens a dialog on the user's screen asking for a password. The client then sends the response string back to the VNC server. The VNC server calls the program referenced by the --passwdfile parameter with this string in the following format:

A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
 0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30    .next.16.#6....0
10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29    ....L...\,.\..!)
20: 3A DF C2 A2 7C E9 1F 1A D7                         :...|....

Open in new window

The <newline>next<newline> can be ignored as this is debug stuff from my script. The strings are:
23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C

2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7

Open in new window

The manpage defines the response string as, "client's response (i.e. the challenge string encrypted via DES with the user password in the standard situation)." I have no further information or documentation. I assume one must use the challenge string to decode the response string and get the user-entered password, but I've no idea how to do that.
  • 2
1 Solution
A Q ChoudaryJunior Linux EngineerCommented:
Need to check document properly below is link of document:
jmarkfoleyAuthor Commented:
A Q Choudary: That is truly a great link! I've been blundering around experimenting on my own without such a good resource. I'll study that and perhaps change my VNC server since x11vnc seem to no longer be actively supported. But, that will take some time and further experimentation.

Meanwhile, the X11vnc server's -passwdfile switch has an option as described below, which does not appear to be described in your link.
If  filename  is  prefixed  with "custom:" then a custom password checker is
supplied as an external command following the ":". The command will  be  run
when  a  client  authenticates.   If  the command exits with 0 the client is
accepted, otherwise it is rejected.  The environment variables are set as in

The  standard input to the custom command will be a decimal digit "len" fol-
lowed by a newline. "len" specifies the challenge size  and  is  usually  16
(the VNC spec).  Then follows len bytes which is the random challenge string
that was sent to the client. This is then followed by len more bytes holding
the  client's response (i.e. the challenge string encrypted via DES with the
user password in the standard situation).
I created such a custom command and my vnc client was presented with a login screen. The string sent to my custom command was as I presented in my OP. I believe this string contains the DES encoded password.

At this point, what I'm looking for is not "HOWTO" on VNC generally, but rather how to use the challenge string to decode the response string. I'm not familiar enough with DES encryption to figure that out and have found nothing so far to help.
jmarkfoleyAuthor Commented:
Could I get some help finding an answer to this question? If nobody has any insights at all, then we might as well delete it. I'm looking for how to decode a DES encrypted string.
Daniel McAllisterPresident, IT4SOHO, LLCCommented:
Greetings jMarkFoley...

It seems to me you've been quite patient -- your question having been open for 2 full months now.
I'm afraid I can't help you with the decode mechanism (although you might want to look into the openssl command to decrypt the DES encrypted password string. If that doesn't work for you, here's a page with 6 other encrypt/decrypt tools:
All of them can use stream input, so your shell can cat or echo the encrypted data into them.

But I must say, as a programmer myself, I often find I hit a roadblock in developing my own solution only to find a perfectly viable and easy-to-use solution is available elsewhere, if I'll just back away from the problem enough to notice the other options. (I personally refer to these projects as rabbit-hole projects, and I find I'm particularly susceptible to them myself).

In this case, I might suggest you look at the tigervnc project. They have a set of servers and clients already coded for all of the major Linux distros. Its Open Source, so if you really WANT to keep digging, you can always delve into their code to see how they did it!

Good luck!

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Easily manage email signatures in Office 365

Managing email signatures in Office 365 can be a challenging task if you don't have the right tool. CodeTwo Email Signatures for Office 365 will help you implement a unified email signature look, no matter what email client is used by users. Test it for free!

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now