Email router is now getting 503 errors (Server Unavailable) has been working previously

CRM 4.0 Email router has previously been working fine.  I checked it recently when some emails didn't come through and testing reults in;
Incoming Status: Failure - The remote server returned an error: (503) Server Unavailable - for all incoming accounts

Outgoing Status: Failure - An error occured while checking the connection to the Microsoft Dynamics CRM server (http://crmurl).  The request failed with HTTP status 503: Service Unavailable.

I can click the url in the outgoing failure and it takes me to the crm default page.  I've tried restarting services but no luck.

Event log is filled with application errors:
#23379 - Could not verify the version of Microsoft Dynamics CRM at http://s-vclcrm03/QuantumMarketResearch. The request failed with HTTP status 503: Service Unavailable.

Any help would be greatly appreciated.


nick34567Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

WilyGuyCommented:
What changed?

Host Headers?  Port Change? Any other URL change?
If you go from the router server via IE to the CRM server URL, can you get in? (using the URL that is in the Deployment Section.

What happens if you try to Load Data in the 3rd tab?
0
WilyGuyCommented:
Also, did you put Anti-Virus on the CRM server? (recently)
0
nick34567Author Commented:
There are no changes that I'm aware of - which is what has me so baffled.

I can load data and it populates the accounts.  It's when I 'test' that it coems up with the errors.

No change to the anti-virus that I'm aware of either  (will confirm)
0
Cloud Class® Course: MCSA MCSE Windows Server 2012

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

WilyGuyCommented:
What are you using to authenticate in the Deployment section? (Local System? A User account?, etc.)
0
nick34567Author Commented:
Using Windows Authentication - Other Specified.  It's  an account we made specifically for crm admin stuff, and it's worked in the past.  I have checked and I can reach the accounts inbox via webamil form the server (with same credentials).

If it's an authertication problem isn't the error different?
0
WilyGuyCommented:
Can you IIRESET the CRM box?  the 503 errors from what your errors are showing says they are coming from the CRM server.
Is the Async Service running?
0
nick34567Author Commented:
Have done IISRESET and a full reboot, neither helped.

Both CRM services are running...
0
WilyGuyCommented:
If you restart the Router Service, does the error go away?
0
WilyGuyCommented:
Also, is the Other Specified Account in the PrivUserGroup? (what about the router Machine$ account?)
0
nick34567Author Commented:
The account wasn't in the group - just added.  The machine account was.  I can't get onto that DC directly at the moment so am waiting for the changes to replicate from another one.

It's unliekly anyone had removed the account from the group though, so is it possible that when it was working previously is was doing so with just the machine account in PrivUserGroup?
0
WilyGuyCommented:
Well, the machine account in the PrivUserGroup usually indicates what is running the service.

503 errors happen under load usually, so that is why I was wondering on the iisreset and the Async reset.

There is a known issue with the router where it will operate for X minutes and then fail for the same X minutes afterwards. See http://support.microsoft.com/kb/952019
0
nick34567Author Commented:
Thanks, I've had a look at this one but it doesnt seem to be relevant to this issue (permanently broken).  Is there any reason I shouldn't try uninstalling and reinstalling the router?

When I click 'test', do you know if the outgoing check is contacting the Async service or the email router service?  Maybe there is something wrong with one of the services?
0
WilyGuyCommented:
I think uninstalling and reinstalling isn't going to help.

Reconfiguration could.

How many users do you have in the system?  Are they set for Email Router for Outgoing?
What about Incoming? (Email Router, Forward Mailbox, Outlook client?)

I would concentrate on one aspect only if possible.  Perhaps disconnect the Incoming Method from the deployment (just change it to blank) and test the outgoing.  You may want to reset the router service after making the change and publishing.  Your event log should show the stop and start, then see what comes up.

There is a nice tool at blogs.msdn.com/benlec for CRM Tracing as well as for Email Router testing. (I have lots of experience with the CRMDiagTurbo, but have not played around too much with the email router test tool)
0
nick34567Author Commented:
It's a bit of a mix at the moment - so this problem wasn't noticed straight away.  The admin and queues are set to use router.  Users are set to use outlook.

I've turned logging to 2 globally, but still just get;
#23379 - Could not verify the version of Microsoft Dynamics CRM at http://s-vclcrm03/QuantumMarketResearch. The request failed with HTTP status 503: Service Unavailable.

I'm now just using the outgoing profile to test with.

Starting to think the problem might lie with the Asynch service.  Do you know what will happen if I run a repair off the installation disc?


0
gregowensCommented:
Are you using proxy servers between CRM server and machine running the email router? How about firewalls? (Yeah I know this is unlikely as it's all internal)...
Review the Internet Explorer proxy settings on the machine running the email connector (remove any proxies if they are set) and try again.
Also, definitely recommend Wily's solution to use the email connector logger.
Are there any stack traces in your event viewer logs?
0
nick34567Author Commented:
I tried the email connector logger and it just returns the error above.  

#23379 - Could not verify the version of Microsoft Dynamics CRM at http://s-vclcrm03/QuantumMarketResearch. The request failed with HTTP status 503: Service Unavailable.

Which seems like it might be not finding the Async service.   Funny thing is, I click the link, and it loads up the CRM default page like normal.

When I restarted the service I noticed an error;
Host S-VCLCRM03: error while processing organization 63c76c37-fe38-dd11-951b-000c2986423b. Exception: Microsoft.Crm.Metadata.EntityMetadataNotFoundException: The entity with ObjectTypeCode = 0 was not found in the MetadataCache.
   at Microsoft.Crm.Metadata.DynamicMetadataCache.GetEntity(Int32 entityCode)
   at Microsoft.Crm.Asynchronous.QueueDataAccess.GetEntityLogicalNameFromObjectTypeCode(Int32 objectTypeCode)
   at Microsoft.Crm.Asynchronous.QueueDataAccess.GetLookup(Object objectId, String objectIdName, Int32 objectDsc, Int32 objectTypeCode)
   at Microsoft.Crm.Asynchronous.QueueDataAccess.<>c__DisplayClass2.<SelectEvents>b__0(Object[] values)
   at Microsoft.Crm.Asynchronous.DataAccessBase.ExecuteSqlCommandAndProcessRecords(IDbCommand command, RecordProcessor recordProcessor)
   at Microsoft.Crm.Asynchronous.QueueDataAccess.SelectEvents(Int32 numberOfEvents)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The GUID at the start of the error does not match the guid in the PRIVUSERGROUP security group on the DC.

I'm starting to think it might be a problem with the Async service itself.  Does anyone know if I do a repair or reinstallation of the application server, what customisations (which aren't stored in the db) I will lose?

Am I right in thinking that a reinstall of the app server will retain all the form, entity and workflow customisations that have been done?

Thanks again for all the help - this is now officially driving me crazy....


0
WilyGuyCommented:
Google this : "Exception: Microsoft.Crm.Metadata.EntityMetadataNotFoundException: The entity with ObjectTypeCode = 0 was not found in the MetadataCache." and make sure you show the omitted results.  LOTS of hits.

Was this an Imported Org?  An Upgrade from 3.0?

Almost seems like the CRM is the problem and you are just noticing problems on the Router server.  Anything in the event log on the CRM server?
0
nick34567Author Commented:
I've now cleaned up the async table and am no longer getting errors with that service.

There is still some peculiar behaviour though and MSCRMEmail errors persist when testing access...
Changing Authentication to Local System, I get 503 errors Incoming and Outgoing.
If I change server name in he deployments section from
http://<servername>/<Organisation>
to
http://localhost/<Organisation>
Outgoing mail tests ok.

If i change it to http://<ip of crm server>/<organisation>
I get a 401 error for outgoing.

Also, if I change the server name in the Incoming profiles to the mailserver's IP, I get 401s for incoming too.

Any ideas?  
0
WilyGuyCommented:
Alright, lets look at some random things:
Run this against MSCRM_CONFIG database:
SELECT NvarCharColumn FROM DeploymentProperties WHERE ColumnName = 'AsyncSdkRootDomain'  SELECT NvarCharColumn FROM DeploymentProperties WHERE ColumnName = 'ADSdkRootDomain'   SELECT NvarCharColumn FROM DeploymentProperties WHERE ColumnName = 'ADWebApplicationRootDomain'  SELECT NvarCharColumn FROM DeploymentProperties WHERE ColumnName = 'ADRootDomainScheme'

Also, do you have IE Enhanced Security turned on on CRM or Router server?

As Greg asked, do you have proxy (at all) in the environment?  Try setting the CRM server in the proxy settings as a local machine.

You can ping CRM from the Router and it resolves. (different machines right?)

Any anti-virus / firewall on CRM server?

What generically is in the PrivUserGroup? (everything is in the same domain)

What is running the CRM AppPool?  How about the Router service?

What 401 error are you getting? (401.1, .2, etc.)

I realize that all of this seems random, but we need that spark.

0
nick34567Author Commented:
Thanks for all your help with this WilyGuy, it's much  appreciated.
The script returned;
1. (blank)
2. (CRM server name):80
3. (CRM server name):80
4. http

There are proxies in teh environment but they should not be a factor here.  I have tested and can get to mail server from CRM and vice-versa.

CRM and router are on the same machine.

Have tested with anti virus disbaled - no difference

The privusergroup contains the CRM Machine, and the 'other' account im trying to use to access the mail server.

CRMapp pool is running as netowrk service.  Router is running as localsystem.

0
nick34567Author Commented:
Forgot to mention.  The error is just striaght 401: Unauthorised.
0
WilyGuyCommented:
What I am finding out with a current project is that what happens while you are configuring the router isn't necessarily the behavior for the "service"

think of it like if I have configured a setting in the IE settings of my profile, it doesn't apply to your IE settings on the same machine necessarily.

I am not sure what Proxy server you use, but if there is a way to push out an exception we should do that. (at least to try it).  (this is another case where making the change in the IE settings may not be indicative of the use of the router)

Try adding the url to the Local Intranet zone in IE (not guaranteeing this will work for the service, but maybe for the test access).

The email router needs to use the name you found in the script (localhost is sexy, but won't work).

When you go to CRM based on the script name from the Router/CRM server, does it prompt for authentication at all?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
WilyGuyCommented:
Any luck?
0
nick34567Author Commented:
I got it working again on a different server.  I have no idea what changed on the original server (suspect GPO) but quite frankly after spending this long on it I'm just glad it's working again.  It's strange how <crm server name> no longer works when referencing itself, but from another server works fine.

Thanks for all the help, although it didnt end up with an answer to what had gone wrong, your comments were a tremendous help - and were what stopped me from tearing my hair out! - when debugging the issue.
0
nick34567Author Commented:
Thanks for the help.  Much appreciated.  

These accelerators look good....
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Dynamics

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.