We help IT Professionals succeed at work.
Get Started

Need help decoding VNC challenge/response string

435 Views
Last Modified: 2018-01-26
I'm trying to run x11vnc with the --passwdfile option. This option sends a challenge string to the VNC client. The client opens a dialog on the user's screen asking for a password. The client then sends the response string back to the VNC server. The VNC server calls the program referenced by the --passwdfile parameter with this string in the following format:

A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
 0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30    .next.16.#6....0
10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29    ....L...\,.\..!)
20: 3A DF C2 A2 7C E9 1F 1A D7                         :...|....

Open in new window

The <newline>next<newline> can be ignored as this is debug stuff from my script. The strings are:
Challenge:
23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C

Response:
2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7

Open in new window

The manpage defines the response string as, "client's response (i.e. the challenge string encrypted via DES with the user password in the standard situation)." I have no further information or documentation. I assume one must use the challenge string to decode the response string and get the user-entered password, but I've no idea how to do that.
Comment
Watch Question
President, IT4SOHO, LLC
CERTIFIED EXPERT
Commented:
This problem has been solved!
Unlock 1 Answer and 4 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE