8 bit or 16 bit algorhythm determination (security)

The good encryption secret should be based on algorithm but not any magical things. Remember you are trying to implement the same algorithm but not going to break it so the chance to find out and implement the same algorithm may not be impossible.

You have the plantext and ciphertext, for example if it is encrypted by MD5 hash then you simply implement the MD5 algorithm in your new application then you should have the same hash result. Even the MD5 added some salt still can find out by brute force method (just take some time but not forever).  It may use DES or other well known algorithm but you still can easy to implement it because all the algorithm you can obtain source from the Internet.

If the algorithm just their own algorithm then the chance to break it still possible because it may not strong enough as well known algorithm such as simply applied the XOR with a pre-defined string e.g.: "#s7Dxz*-^...". You may try to browse the binary see any speical chars there in binary data area. Of course you still need some debugging skill to do such task.

I think the algorithm is a simple one for they will not put many effort on powerbuilder algorithm because they're writing a encrypt function for application but not the encryption application.  Search the powerbuilder 4 see any crypt function build in then you save a lot of time sure they will use it. If not then it should be the well known or their own one.

All you have to do is find out what algorithm it is using and this is a critical issue. Try post some out see any luck because I'm not the Cryptanalysis but my 0.02 cents.

You may try in this way:
Create 3 new user account as following:

Login ID          Name            Password
AAAAA            AAAAA           AAAAA
AAAAB            AAAAA           AAAAA
AAAAC            AAAAA           AAAAAA

For the simply self develop crypto algorithm may uses XOR with pre-defined string, or XOR Login ID (if not auto gen.), Name or bit shift, or time a small value prime number.

See the encrypted result from database then you may have an idea.

Exactly what is this "security functionality" that this application provides?  Why can't everyone just pick new passwords?

If you actually have the plaintext password and the encrypted password of two users, you can test for a simple XOR by XORing the plaintext and encrypted to come up with the "key" and then testing that key on the other password.
If A XOR B = C, then A XOR C = B.

If you go the route of reverse-engineering the algorithm, then SoftICE is your best bet.  You might also try Win32DAsm, though.  The difficulty of this route is going to be dependent on how "well" PowerBuilder compiles code.  If it does a "good" job there will hopefully be little clutter within the algorithm.

Bradley

Before a PowerBuilder 4.0 developer provides an application to users, the application code goes through a partial translation to pseudo-code or P-Code like the other players such as VB 1-4, dBase 3/4, Clipper 87/5.x, FoxPro2-5 at that time. At run time, interpretation occurs before the code is executed. So it is possible to decompile the P-Code to Source code as VB, dBase, FoxPro, Clipper because they already has decompiler.

Actually I found some very simple PB 6,7, or 8 decompiler in the Net but still in the alpha state. However PB 4 is much simpler design then nowaday. If you really necessary need to study the SoftICE and Assembler to trace the code from scratch, I suggested you consider write your own PB 4 decompiler instead .

My suggestion still try to find out the password algorithm may more practical.