asked on

Exchange 2010 EWS Error

We have a multiple site Exchange 2010 SP2 organisation. 4 sites have a single multiple role server, 2 sites have 2 multi role servers with a HLB. I have a problem on one of the HLB sites and one of the other sites. We have an archiving service installed on a standalone server on each site which uses EWS to access Exchange mailboxes. It works perfectly on 4 sites but not the other 2. As part of the troubleshooting I ran test-OutlookWebServices -identity: user1@domain.com which returned the errors below. I did an iisreset and ran it again, everything was successful. I run it a third time and I get the same error output. Iisreset resolves it but it always reverts to the error after a single successful run. This is true for both problematic sites.

All the Exchange servers are in the proxy bypass list for IE. I have checked the EWS virtual directory under IIS and this is set to use Windows authentication and basic authentication. From some Windows clients using IE to browse to the Autodiscover URL I get prompted for Windows credentials.

Any ideas on what is wrong or how I can troubleshoot this?

RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1019
Type : Information
Message : A valid Autodiscover service connection point was found. The Autodiscover URL on this object is https://server1.internaldomain.local/Autodiscover/Autodiscover.xml.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1013
Type : Error
Message : When contacting https://server1.internaldomain.local/Autodiscover/Autodiscover.xml received the error The remote server returned an error: (407) Proxy Authentication Required.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1023
Type : Error
Message : The Autodiscover service couldn't be contacted.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1013
Type : Error
Message : When contacting https://server1.internaldomain.local/EWS/Exchange.asmx received the error The remote server returned an error: (407) Proxy Authentication Required.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1025
Type : Error
Message : [EXCH] Error contacting the AS service at https://server1.internaldomain.local/EWS/Exchange.asmx. Elapsed time was 0 milliseconds.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1013
Type : Error
Message : When contacting https://server1.internaldomain.local/EWS/Exchange.asmx received the error The remote server returned an error: (407) Proxy Authentication Required.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1027
Type : Error
Message : [EXCH] Error contacting the UM service at https://server1.internaldomain.local/EWS/Exchange.asmx. Elapsed time was 0 milliseconds.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1113
Type : Error
Message : When contacting https://server1.internaldomain.local/ews/exchange.asmx received the error The remote server returned an error: (407) Proxy Authentication Required.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1125
Type : Error
Message : [Server] Error contacting the AS service at https://server1.internaldomain.local/ews/exchange.asmx. Elapsed time was 0 milliseconds.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1113
Type : Error
Message : When contacting https://server1.internaldomain.local/ews/exchange.asmx received the error The remote server returned an error: (407) Proxy Authentication Required.
RunspaceId : 8d6bfa3a-706c-4f30-8f75-08ad287dcaf2
Id : 1127
Type : Error
Message : [Server] Error contacting the UM service at https://server1.internaldomain.local/ews/exchange.asmx. Elapsed time was 15 milliseconds.

Simon Butler (Sembee)

The error is pretty clear.
There is a proxy involved somewhere. You need to remove the proxy configuration from the server, or add in exclusions for the internal addresses so they don't go through the proxy.

Simon.

Mayo1

ASKER

Simon

Thanks for the response. I think the proxy bit is a red herring, I think it is a generic IIS authentication error message. I have added all the Exchange servers to the proxy bypass list as the proxy was previously causing an issue for Outlook clients. This command is running from powershell on the server hosting the autodiscover service, would this be redirected to a proxy? I have changed IE not to use a proxy. Still the same issue. How can I tell if this is being redirected to a proxy?

Exchange_Geek

Why are you using Windows Authentication and Basic - remove Basic from the authentication, EWS works good with Windows Authentication.

Also check if your connections are going through ISA / TMG / Router internally?

Run a simple tracert and understand the hops involved to reach CAS box server1.internaldomain.local

What is the response when you access from a client machine and that on server.
https://server1.internaldomain.local/ews/exchange.asmx

If the response is good, you should see XML file information holding loads of entries such as

<wsdl:definitions xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:s="http://www.w3.org/2001/XMLSchema" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" targetNamespace="http://schemas.microsoft.com/exchange/services/2006/messages">

Regards,
Exchange_Geek

Mayo1

ASKER

Exchange geek

Thanks for your response. Basic authentication was enabled as I was doing some testing with the third party archiving software. I will be removing it, but it does not have any link to this particular issue. The error occurs with or without it being enabled.

Tracert shows a single hop to server1.internaldomain.local with no other device being accessed. I also did a netsh show proxy on server1.internaldomain.local and this returns “direct access (no proxy server)”. I am now certain that I am not being redirected to a proxy, also iisreset on server1.internaldomain.local fixes the problem for one run of test-OutlookWebServices. If this problem was due to another device iisreset would have no effect.

Browsing to that URL gets me redirected to the ews/services.wsdl page and returns connection information as you have suggested. Strangely today when I browse https:// server1.internaldomain.local/Autodiscover/Autodiscover.xml I am not prompted for credentials, it returns the content of the xml page. However from powershell I still get errors when running test-OutlookWebServices.

Exchange_Geek

Can you run the following cmdlet and provide us the output, please change the names of the FQDN involved - however, please provide us the details for us to understand which FQDN means what (internal / external)

test-outlookwebservices |FL

Regards,
Exchange_Geek

Mayo1

ASKER

Please see below for output. Same results as when I run against a defined mailbox. First time it succeeded, thereafter it fails with the 407 error. The server does not have external URLs defined for any services as I want external users to be proxied via the CAS servers on the internet facing site.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1019
Type : Information
Message : A valid Autodiscover service connection point was found. The Autodiscover URL on this object is https://server1.internaldomain.local/Autodiscover/Autodiscover.xml.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1013
Type : Error
Message : When contacting https://server1.internaldomain.local/Autodiscover/Autodiscover.xml received the error The remote server returned an error: (407) Proxy Authentication Required.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1023
Type : Error
Message : The Autodiscover service couldn't be contacted.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1013
Type : Error
Message : When contacting https://server1.internaldomain.local/EWS/Exchange.asmx received the error The request failed with HTTP status 407: Proxy Authentication Required (Access is denied).

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1025
Type : Error
Message : [EXCH] Error contacting the AS service at https://server1.internaldomain.local/EWS/Exchange.asmx. Elapsed time was 199 milliseconds.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1013
Type : Error
Message : When contacting https://server1.internaldomain.local/EWS/Exchange.asmx received the error The remote server returned an error: (407) Proxy Authentication Required.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1027
Type : Error
Message : [EXCH] Error contacting the UM service at https://server1.internaldomain.local/EWS/Exchange.asmx. Elapsed time was 107 milliseconds.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1113
Type : Error
Message : When contacting https://server1.internaldomain.local/ews/exchange.asmx received the error The request failed with HTTP status 407: Proxy Authentication Required (Access is denied).

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1125
Type : Error
Message : [Server] Error contacting the AS service at https://server1.internaldomain.local/ews/exchange.asmx. Elapsed time was 199 milliseconds.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1113
Type : Error
Message : When contacting https://server1.internaldomain.local/ews/exchange.asmx received the error The remote server returned an error: (407) Proxy Authentication Required.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
Id : 1127
Type : Error
Message : [Server] Error contacting the UM service at https://server1.internaldomain.local/ews/exchange.asmx. Elapsed time was 92 milliseconds.

Thanks.

Mayo1

Exchange_Geek

How do you plan to achieve this?

The server does not have external URLs defined for any services as I want external users to be proxied via the CAS servers on the internet facing site

Regards,
Exchange_Geek

Mayo1

ASKER

Both problematic servers are on non internet facing AD sites. For a user on these sites connecting to OWA they connect to a URL https://owa.externaldomain.com/owa. To reach the mailbox it traverses the following path. TMG 2010 server acting as a front end server, hardware load balancer on the internet facing site, CAS server on the internet facing site, mailbox server on non internet facing site. As I have not defined external URLs for any services (OWA, etc) the CAS server on the internet facing site will proxy the connection to the mailbox server.

Thanks.

Mayo1

Exchange_Geek

You're issue of proxy is TMG, please note down the rules of TMG and remove them for testing purpose.

You're EWS issue is being caused an issue because of TMG. get TMG out of the picture just for once.

Regards,
Exchange_Geek

Mayo1

ASKER

Internally EWS traffic is not going via the TMG server. The host name for the TMG server is not referenced anywhere on the problematic servers. Output from get-webserviceveirtualdirectory is included below. I have enabled logging on the TMG and I cannot see any connections from the Exchange server when I run the test-outlookwebservices cmdlet.

RunspaceId : 284ac102-d078-449f-b110-b2a6decc499d
CertificateAuthentication :
InternalNLBBypassUrl : https://server1.internaldomain.local/ews/exchange.asmx
GzipLevel : High
MRSProxyEnabled : False
MRSProxyMaxConnections : 100
Name : EWS (Default Web Site)
InternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity}
ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity}
LiveIdSpNegoAuthentication : False
WSSecurityAuthentication : True
LiveIdBasicAuthentication : False
BasicAuthentication : True
DigestAuthentication : False
WindowsAuthentication : True
MetabasePath : IIS://server1.internaldomain.local/W3SVC/1/ROOT/EWS
Path : C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\exchweb\EWS
ExtendedProtectionTokenChecking : None
ExtendedProtectionFlags : {}
ExtendedProtectionSPNList : {}
Server : server1
InternalUrl : https://server1.internaldomain.local/EWS/Exchange.asmx
ExternalUrl :
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=EWS (Default Web Site),CN=HTTP,CN=Protocols,CN=server1,CN=Servers,CN=Exchange Ad
ministrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=catfish,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=internaldomain,DC=global
Identity : server1\EWS (Default Web Site)
Guid : fa8a4a79-6112-4c7a-824c-e5430dedab95
ObjectCategory : internaldomain.local/Configuration/Schema/ms-Exch-Web-Services-Virtual-Directory
ObjectClass : {top, msExchVirtualDirectory, msExchWebServicesVirtualDirectory}
WhenChanged : 23/08/2012 19:21:43
WhenCreated : 09/06/2012 10:53:59
WhenChangedUTC : 23/08/2012 17:21:43
WhenCreatedUTC : 09/06/2012 08:53:59
OrganizationId :
OriginatingServer : DC01.internaldomain.local
IsValid : True

I have compared the output to a site that is working and it is exactly the same.

Thanks.

Mayo1

Exchange_Geek

Couple of points

Though this might be linked OR may-not be, you said you've got two sites that are facing issues and two which aren't

To reach the mailbox it traverses the following path. TMG 2010 server acting as a front end server, hardware load balancer on the internet facing site, CAS server on the internet facing site, mailbox server on non internet facing site. As I have not defined external URLs for any services (OWA, etc) the CAS server on the internet facing site will proxy the connection to the mailbox server.

I think you mean CAS-to-CAS proxy OR you meant CAS-to-mailbox proxy, cause if it is the latter - that isn't possible, considering CAS needs to be local to the mailbox server in AD Site. So, what is the difference in the EWS V.Directory on these two sites and the ones that work.

Regards,
Exchange_Geek

Mayo1

ASKER

CAS to CAS proxy. I cannot see any difference between the sites or the virtual directories. I have checked via IIS admin console and the get-webserviceveirtualdirectory cmdlet. Not sure what else to look at. Should I just recreate the EWS and autodiscover vdirectories? Not sure if this would cause problems moving forward.

Exchange_Geek

That's the last step in troubleshooting this. Can you add the external URL to 1 of those sites and check if EWS is working for you.

Recreating EWS and AD Virt.Dir of course is the last bet.

Regards,
Exchange_Geek

ASKER CERTIFIED SOLUTION

Mayo1

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Mayo1

ASKER

Explanation in the final entry.