RAS Connection over ISDN under Windows XP

Posted on 2005-03-16
Medium Priority
Last Modified: 2013-12-29

I have two machines, each running Windows XP Pro, each with their own ISDN Modem.
I have set one machine up to listen for incomming connections and the other to dial in to it.
With the 'Server' configured to allow unsecured connections and the 'Client' set up as 'Optional Encryption' I can dial up and connect ok.
As soon as I set the Server to 'Must have encryption' and the Client to use encryption, it connects ok, but hangs on the 'Verifying Username & Password'???

Any ideas?


Question by:James Atkin
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 6
LVL 12

Expert Comment

ID: 13553449
Of these two ISDN's, are they remotely located, or in the same premises?

Before getting into this, I would suggest you get a packet capture and monitor program, Ethereal, http://www.Ethereal.com/ because it works better than anything else in troubleshooting bad handshakes/username-password-challenge-response

It is most likely that you're trying to use old encryption like MD5 or something, and the server doesn't support it, i.e., available protocols.

You can really only see these in a packet capture.

Ethereal must be installed on both machines.

I'm just curious too as to why you chose ISDN, old establish account?  Or no DSL or fibre in your area?

Author Comment

by:James Atkin
ID: 13553495
At the moment, this is a test with both ISDN modems in the same premises.

As for the choice of ISDN - not mine I'm afraid; one of those Customer requests I'm afraid :-(

Regarding the encryption type, I have tried several conbinations including the Windows XP defaults - i.e. Must use an encrypted password & data - but no luck :-(

Before I go ahead with Ethereal, is this going to be able to help in this case - both ISDN modems are serial devices connected directly to each of the PCs....
LVL 12

Expert Comment

ID: 13554325
Yes, it should help.  I've seen this problem before.  It's the handshake and protocol used.  They must pick a mutually agreeable encryption/decryption scheme.  If they don't, it fails because the server can't decrypt it.  It's common.  The only way to really see which protocol was selected, if indeed one was agreed upon, is using the packet capture because you can't see what's in the encrypted packets.

XP specifically has this problem.  Even between only two devices, PC's, connected by any means, ISDN, modem, DSL, whatever.  That's because all of these use TCP/IP transport.

The modems don't really care,they just send and receive packets, whether by ATM, ISDN, or whatever.

If you set "Server Must Have Encryption," then the server and client have to first agree on a protocol, sometimes called a hash, like MD5, ssl, whatever.

So, the client has to be set to the same hash as the server.


The server sends a message showing available protocols, the client picks one, and they agree.  If it's ssl, they have to share a public key and all that.  It gets pretty involved.

If they choose something like Lanman 2.0, and the server doesn't support it, no go.

Similar things can go wrong.

Offhand, I'd say they are not agreeing on the protocol, unless you know exactly which ones they are using, it gets hard to figure out why they aren't talking over a secure line.

Ethereal just bypasses the day or two it can take in trying to figure out why they're not talking to each other.  That is, you'd have to do a lot of troubleshooting to get to the point where you realise that the protocol is the issue.

Maybe you can avoid installing and testing by clicking on properties or something, and seeing exactly what secure method both the server and client are using.  Can you do that?
Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.


Author Comment

by:James Atkin
ID: 13554953
This is what I suspect is happening - makes sense as it 'hangs' at the 'Verifying Username & Password' point...
I'll install Ethereal and check out the information flow....

Author Comment

by:James Atkin
ID: 13555086
Ok, we have a little problem with Etherreal...
As the connections are made directly through the modem Ethereal had nothing to bind itself to (there is no LAN adapter)
Am I missing something in the config of Ethereal that allows be to see what is happening in Serial Devices???
Sorry if I am asking a daft question....

Author Comment

by:James Atkin
ID: 13555202
Whilst I think about it...

I have tried the exact same configuration using the same machines using standard 56k Modems (same manufacturer; Multitech) and it works fine!

Just thought I'd throw this into the equation  ;-)
LVL 12

Expert Comment

ID: 13561883
Yes, I forgot about the serial lines on ISDN, haven't used them in a while.

I'm sure there is something in Ethereal, the guy who wrote it and maintains is a big professor or something, who is from way back when on the Internet; I'm sure he's worked on ISDN, and others before it.

You are going to send me back to the books!

Okay, it's got to be that serial is really nothing more than a direct ATM switch connection.  Which also has to use a form of Ethernet; ethernet is very basic to the entire concept of the Internet and was written to accomplish packet switching back when the Internet was started.

So, first I have to know what's at the other end of your connection; obviously not a DSLAM, a switch of some kind, but what kind?  An ATM?  See if you can find out from the telco.

Meanwhile, I'll open some books that I haven't opened in a long time!
LVL 12

Expert Comment

ID: 13561982
Well, I have to go shortly, to get some sleep.  I did check a couple of things, 1.) some ISDN use mostly ATM [the basic telephone switch of all telephone systems, i.e., the packet switching actual switch throughout most phone companies],2.) there are routers for ISDN that use Ethernet, just as there are ISDN modems that do.

So, having reread the question, you have two modems, what make and model?

I'll check the docs when I get up in the morning, unless you read through them and find the answer.  But dimes to dollars says that it's somewhere in the protocol that's being used simply because ISDN is what a lot of ISP's actually use for things like a DS3, DSO connections, etc..

It's also very expensive!  Check this out:


Be back in the A.M. [my A.M., not yours, GMT-4]


Author Comment

by:James Atkin
ID: 13562966

We have two Multitech MTA128ST modems and they are being used to create a peer-peer network.  There is no ISP involved, just a direct phonecall between the two PCs.

I did a little monitoring of the Serial Traffic using a COM port monitor yesterday and discovered something very strange:

During the attempt to make the connection:
   1. Modem 1 will dial Modem 2 and establishes an initial connection.
   2. During the negotiation of passwords/encryption Modem 1 sends a chunk of info to Modem 2
   3. Modem 2 then 'appears' to be continuously communicating with the PC, but sends nothing out to Modem 1.
   4. This continues with Modem 1 waiting for communication and eventually timing out.

Ever seen anything like this?????
LVL 12

Expert Comment

ID: 13564471

Boy, this really brings back memories.  What you're doing is what people used to do before about 1985 or so, for BBS connections.  It's actually how the first Internet connection, "ever," was made.  Over common phone line, between two computers [all manual dialed, and so on, of course].

First, let me try to explain something about serial communication.  Nearly all nodes below the backbones are "serial."  Whether through ATM, Frame Relay, PPPoE, and all the others, their mostly either synchronous or asynchronous serial transmissions.  The only parallel transmissions are those of the core mainframes, currently mostly over fibre optics bundles [fibre trunk lines].  PC's currently have no such thing as a 64-bit [indeed, not even a 2-bit], parallel bus.  The parallel port may or may not be the one exception, but it is at most 8-bit, and was never developed enough for peer to peer other than for printers and the like.

ISDN used to be a synchronous system, that is, one dependent upon a mutual clock.  Today, they are asynchrouns mostly using ATM.

Aside from all of that, which is just to refresh mostly my memory, if modem 1 sends first an RTS [Request To Send], modem 2 answers with a CTS [Clear To Send]. This "handshake" goes on until a point is decided "when" to send and receive.  After the data burst is sent, modem 2 returns an ACK and the next sequence starts.  This is what makes the transmission "asynchronous."  Both cannot send at the same time or receive at the same time.  So the system is interleaved with these signals, RTS, CTS, ACK, and others to regulate who is sending and who is receiving over any given time frame [hence the term "frame relay"].  This interleaving is known as Full Duplex, the ability to send and receive, seemingly achieving both, at the same time.  Half Duplex means that only one modem can control the transmission per transmissions series [as opposed to "sequence"].

Now the problem arises that one modem may be in Full Duplex and the other in Half Duplex, thus confusing the transmission, and because this is at a very basic level of communication, an error may go completely undetected causing an open connection, or what seems to be a freeze or lockup of the transceiving process by one or the other modems.

Modem 2 will not, and probably has not, signalled a CTS to modem 1, or, it did not get back a ready to send, and make itself ready to receive, or any combination thereof.

Data segment ends in this scheme are signalled by a very specific pattern of ones and zeroes, something like alternate ones and zeroes for 256 bits followed by all ones, and EOF [End of File].  Other series and sequences include and EOT [End of Transmission].  Being a specific pattern, "they cannot be encrypted and decrypted" by the communicating modems.

An EOF is an EOF, and no other pattern, and an EOT is an EOT and no other pattern.

Some of the other signals are RST [Reset - resets the communication protocol to initilize the RTS and CTS sequence].

To find out why the two Multitech MTA128ST modems are having a hard time, we have to know what protocols they are actually using for encryption, as opposed to plain transmission.  Some modern modems are "smart" modems or "intelligent" modems in that they can order the protocol, such as Ethernet instead of ATM, etc..  Some also have their own encryption/decryption protocols.

Before getting too much further, I have to look at how these modems actually handle data and encryption.

Yes, I have seen this before, many times.  Because normal Internet traffic can send and receive packets that either never get there, in which case it tries to resend, or can reassemble packets out of order, over the telcos phone network [which is packet switched ATM for the most part], a data packet can arrive before a control signal, such as ICMP, and throw the whole synchronization scheme off, freezing transmission.  But so can a finicky modem, handshake, protocol confusion, etc..

Let me find the MA128ST specs [specifications] this morning and try to see what could cause this failure from a logician's point of view.

Be back when I have something.
LVL 12

Accepted Solution

GinEric earned 2000 total points
ID: 13564933
Some pointers:

Your modems support PAP and CHAP, start here:


Note that PAP is cleartext and CHAP uses the "hash" algorithim I talked about earlier.  May require PPP utilization.  Fine for Peer To Peer networking, should work.

You may need other software, see the "Capture Buffer Display" menu image for ISDN here:


I don't know your Operating System, however, Linux documentation should be good for any OS on ISDN implementation.  Also, Linux should have ISDN monitor software.  I still think there is a way to use Ethereal for this because it should capture any protocol packets, including ISDN and related.

Things like MTA and MUX are early mainframe stuff, which is what ISDN is, I haven't seen these terms in years!  A MUX is just a multiplexor, also called a Data Communications Unit, or Data Comm.  These things are of monstrous size!  But in the end, they're still just a fancy modem, handling 65,536 communications lines [phone lines] per "mod."  Many have 1,204 mods per cluster.  The get up in the tens of millions of phone connections, i.e., a backbone, all supervised by a very large mainframe.

They're the basic switch of the actual telephone network.  An integral part of Automated Telephone Switching Network.  The mainframe being the "automator" which makes all the switches work without manual intervention.

Some of the sites I listed have suggestions on what causes problems, such as the one you're having.  One of the points from Cisco was that the two hosts must have the same username/password. The others being the choice of protocol, configuration of the modems, etc..

Check your encryption.  Is it CHAP?  Or is it something else?

Author Comment

by:James Atkin
ID: 13566239
Found it!!!!!!!

It would appear that the Multitech modem has a default configuration of "Client Side Authentication Protocol Negotioation" set to PAP ONLY!  (S58=1)

I have changed this on the initialisation string for the modems to ATS58=3, which should auto-negotiate and it works great!!

Thanks for all your input on this - points well deserved!

Best regards,


LVL 12

Expert Comment

ID: 13569325
Wow!  Thanks.  This is one for the archives.

Featured Post


Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article is a collection of issues that people face from time to time and possible solutions to those issues. I hope you enjoy reading it.
This article explains the fundamentals of industrial networking which ultimately is the backbone network which is providing communications for process devices like robots and other not so interesting stuff.
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Monitoring a network: how to monitor network services and why? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the philosophy behind service monitoring and why a handshake validation is critical in network monitoring. Software utilized …
Suggested Courses

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question