RPC-over-HTTPS: Can't get past Outlook authentication popup.

I have a single-server environment running Small Business Server 2003 Enterprise.

I'm trying to enable RPC-over-HTTPS for Outlook over the Internet.

I enabled the RPC proxy through the Server Management snap-in and then went through this (http://www.petri.co.il/configure_rpc_over_https_on_a_single_server.htm) guide to get everything properly configured.

I have a valid third-party SSL certificate with the correct FQDN, I am passing ports 80 and 443 through the (external) firewall, Outlook Web Access works just fine.

The problem that I am having is that when I try to connect with the Outlook client, I can't get past authentication.  I get a regular IE-style popup saying "Connecting to <servername.internaldomain.local>" and asking for the username and password.  I've tried every variation (username and pass, domain\username and pass, username@domain.local and pass) with no luck.  It just continues to ask for authentication.

When I run Outlook with the /rpcdiag switch, the only entry in the popup window is "Server name: servername; Type: Directory; Status: Connecting" with nothing else.

I can't get anywhere past this point.

I've tried it over the LAN and WAN with the same results, so I don't believe this is a firewall issue.

Who is Participating?
SembeeConnect With a Mentor Commented:
The common problem I see is where there is an authentication mismatch.

In IIS Manager, on the /rpc virtual directory, ensure that both integrated and basic authentication is enabled and anonymous is disabled.

In Outlook, if the machine is a member of the same domain as the Exchange server, use NTLM Authentication. If it is not, then use Basic Authentication.

Is this a single Exchange server enviroment?

Have a look at this.

Check the permissions on the RPC virtual directory in IIS
Is the certificate installed on the PC or trusted?  My understanding is you have to get this 100% or it will fail, with no real indication of why it's failing.  Try browsing to https://SERVER_FQDN and see if a certificate warning/accept prompt pops up, if it does you have a problem because Outlook can't do the "Click OK" that you do to get past this.

Sembee has some great articles on his site, specifically http://amset.info/exchange/rpc-http.asp, I'd recommend a look and see if you've maybe missed something.
Easily manage email signatures in Office 365

Managing email signatures in Office 365 can be a challenging task if you don't have the right tool. CodeTwo Email Signatures for Office 365 will help you implement a unified email signature look, no matter what email client is used by users. Test it for free!

titan6400Author Commented:
Yes, this is a single-server environment.

The cert is good and I get no cert warnings when going through OWA or going to https://fqdn/rpc.

I'm looking through the links you guys gave, thanks.
titan6400Author Commented:
I looked through all of those links and I've already tried all of those things (but I still double-checked.)

Thanks anyway.
RPC over HTTPS is one of those features that either works or it doesn't.

Where I get servers where it doesn't work, I tend to remove the RPC Proxy service and then reinstall it. I then use my own registry settings, applying both the Exchange and domain controller settings to the same box. Using my own registry settings (amset.info) I have a 100% success rate.

titan6400Author Commented:
OK well I did as you suggested and started from scratch and followed the amset.info instructions.

It seems to be working from machines on the LAN now (rpcdiag indicates connections via HTTPS only), but the same problem exists off the LAN.  I just get the popup to enter credentials, which it never accepts, and rpcdiag doesn't show anything being connected at all.

Do you need any ports beyond 443 to be open on the firewall?  Any other troubleshooting ideas?
443 should be the only port needed.
titan6400Author Commented:
I'm playing with the rpcping util now.  From on the LAN, I tried running this command:

rpcping -t ncacn_http -s exchange.server.local -o RpcProxy=fqdn.tld -P "<user>,<domain>,*" -I "<user>,<domain>,*" -H 2 -u 10 -a connect -F 3 -v 3 -E -R none

and got back:

RPCPinging proxy server fqdn.tld with Echo Request Packet
Sending ping to server
Response from server received: 401
Client is not authorized to ping RPC proxy
Ping failed.

I'm honestly not completely sure what to make of that, but if that helps anyone see what the problem is...
MATTHEW_LConnect With a Mentor Commented:
Have you checked the authentication on the RPC virtual directory.  Make sure that basic authentication is checked.
titan6400Author Commented:
Well, it was permissions, but not like you'd think.

The IP restrictions were set on that directory to deny anything outside the local subnet.  Why, I don't know, but that's what the problem was.

All is well now.

I accepted sembee with the majority of the points because before he suggested to reinstall, it wasn't working from inside or outside the network and MATTHEW L with an assist since he was also pointing towards a permissions problem.

Thanks, guys.
Thanks very much.  Glad it is working.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.