Need help decoding VNC challenge/response string

I'm trying to run x11vnc with the --passwdfile option. This option sends a challenge string to the VNC client. The client opens a dialog on the user's screen asking for a password. The client then sends the response string back to the VNC server. The VNC server calls the program referenced by the --passwdfile parameter with this string in the following format:

A two-byte length (in this case 16), followed by the challenge string (16 bytes), followed by the response string (16 bytes). What I get back, in hex is:
 0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30    .next.16.#6....0
10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29    ....L...\,.\..!)
20: 3A DF C2 A2 7C E9 1F 1A D7                         :...|....

Open in new window

The <newline>next<newline> can be ignored as this is debug stuff from my script. The strings are:
Challenge:
23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C

Response:
2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7

Open in new window

The manpage defines the response string as, "client's response (i.e. the challenge string encrypted via DES with the user password in the standard situation)." I have no further information or documentation. I assume one must use the challenge string to decode the response string and get the user-entered password, but I've no idea how to do that.
LVL 1
MarkAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

A Q ChoudaryJunior Linux EngineerCommented:
Need to check document properly below is link of document:
http://www.stuartellis.name/articles/vnc-on-linux/
0
MarkAuthor Commented:
A Q Choudary: That is truly a great link! I've been blundering around experimenting on my own without such a good resource. I'll study that and perhaps change my VNC server since x11vnc seem to no longer be actively supported. But, that will take some time and further experimentation.

Meanwhile, the X11vnc server's -passwdfile switch has an option as described below, which does not appear to be described in your link.
If  filename  is  prefixed  with "custom:" then a custom password checker is
supplied as an external command following the ":". The command will  be  run
when  a  client  authenticates.   If  the command exits with 0 the client is
accepted, otherwise it is rejected.  The environment variables are set as in
-accept.

The  standard input to the custom command will be a decimal digit "len" fol-
lowed by a newline. "len" specifies the challenge size  and  is  usually  16
(the VNC spec).  Then follows len bytes which is the random challenge string
that was sent to the client. This is then followed by len more bytes holding
the  client's response (i.e. the challenge string encrypted via DES with the
user password in the standard situation).
I created such a custom command and my vnc client was presented with a login screen. The string sent to my custom command was as I presented in my OP. I believe this string contains the DES encoded password.

At this point, what I'm looking for is not "HOWTO" on VNC generally, but rather how to use the challenge string to decode the response string. I'm not familiar enough with DES encryption to figure that out and have found nothing so far to help.
0
MarkAuthor Commented:
Could I get some help finding an answer to this question? If nobody has any insights at all, then we might as well delete it. I'm looking for how to decode a DES encrypted string.
0
Daniel McAllisterPresident, IT4SOHO, LLCCommented:
Greetings jMarkFoley...

It seems to me you've been quite patient -- your question having been open for 2 full months now.
I'm afraid I can't help you with the decode mechanism (although you might want to look into the openssl command to decrypt the DES encrypted password string. If that doesn't work for you, here's a page with 6 other encrypt/decrypt tools:
   https://www.tecmint.com/linux-password-protect-files-with-encryption/
All of them can use stream input, so your shell can cat or echo the encrypted data into them.

But I must say, as a programmer myself, I often find I hit a roadblock in developing my own solution only to find a perfectly viable and easy-to-use solution is available elsewhere, if I'll just back away from the problem enough to notice the other options. (I personally refer to these projects as rabbit-hole projects, and I find I'm particularly susceptible to them myself).

In this case, I might suggest you look at the tigervnc project. They have a set of servers and clients already coded for all of the major Linux distros. Its Open Source, so if you really WANT to keep digging, you can always delve into their code to see how they did it!

Good luck!

Dan
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Virtualization

From novice to tech pro — start learning today.