troubleshooting Question

Need help decoding VNC challenge/response string

Avatar of Mark
Mark asked on
VirtualizationServer HardwareActive DirectoryStorage SoftwareServer Software
4 Comments1 Solution439 ViewsLast Modified:
I'm trying to run x11vnc with the --passwdfile option. This option sends a challenge string to the VNC client. The client opens a dialog on the user's screen asking for a password. The client then sends the response string back to the VNC server. The VNC server calls the program referenced by the --passwdfile parameter with this string in the following format:

A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
 0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30    .next.16.#6....0
10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29    ....L...\,.\..!)
20: 3A DF C2 A2 7C E9 1F 1A D7                         :...|....
The <newline>next<newline> can be ignored as this is debug stuff from my script. The strings are:
Challenge:
23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C

Response:
2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7
The manpage defines the response string as, "client's response (i.e. the challenge string encrypted via DES with the user password in the standard situation)." I have no further information or documentation. I assume one must use the challenge string to decode the response string and get the user-entered password, but I've no idea how to do that.
ASKER CERTIFIED SOLUTION
Daniel McAllister
President, IT4SOHO, LLC

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 4 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 4 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros