Project Server 2010 task sync problem with Exchange 2010

We have a Project Server 2010. When tasks are assigned to resources and project is published, the tasks appears in the resource user's Outlook 2010 tasks folder properly. When when the users updates the completion of the task (in Outlook), the status is not sync'd back to Project server (update is not seen at PWA).

On the Exchange 2010 (server name filbert), the app log has repeating warnings and errors related to the task sync. The sharepoint/project 2010 is hosted on server name chestnut.

The events begin with a warning of event ID 5:

A notification for subscription [IABmaWxiZXJ0Lm1zYy5hbHR1cmFzYW5hbHl0aWNzLmNvbRAAAAA9CJYtGpFdTrK+jHra/wqyq78G1cee0gg=] against endpoint [http://chestnut/PWA/_vti_bin/PSI/ExchangeSync.asmx] couldn't be sent. The operation will be retried. Details: WebException: The remote server returned an error: (500) Internal Server Error. StatusCode: InternalServerError Internal Server Error    at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.Services.Core.NotificationServiceClient.HandleResponse(IAsyncResult responseAsyncResult)

then after 30 seconds, a warning of event ID 6:

A notification for subscription [IABmaWxiZXJ0Lm1zYy5hbHR1cmFzYW5hbHl0aWNzLmNvbRAAAAA9CJYtGpFdTrK+jHra/wqyq78G1cee0gg=] against endpoint [http://chestnut/PWA/_vti_bin/PSI/ExchangeSync.asmx] couldn't be sent. (Send attempts: 2) Details: WebException: The remote server returned an error: (500) Internal Server Error. StatusCode: InternalServerError Internal Server Error    at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.Services.Core.NotificationServiceClient.HandleResponse(IAsyncResult responseAsyncResult)

After 8 repeats, finally an error of event ID 7:

After 8 unsuccessful attempts to send a notification for subscription [IABmaWxiZXJ0Lm1zYy5hbHR1cmFzYW5hbHl0aWNzLmNvbRAAAAA9CJYtGpFdTrK+jHra/wqyq78G1cee0gg=] against endpoint [http://chestnut/PWA/_vti_bin/PSI/ExchangeSync.asmx], the subscription has been removed. Details: WebException: The remote server returned an error: (500) Internal Server Error. StatusCode: InternalServerError Internal Server Error    at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.Services.Core.NotificationServiceClient.HandleResponse(IAsyncResult responseAsyncResult)

It appears Exchange (on filbert) attempted to send task completion updates to Project (on chestnut) but was not successful - for reason I don't know.

Any suggestion on where I should look?

The Exchange works fine by itself with respect to email/calendar etc function. Project server itself works fine except it does not get task completion update from Outlook/Exchange.

Thanks.
shutterhackerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Amit KumarCommented:
Project server is using EWS url as HTTP but it should be HTTPS, may be there is conflagration issue.

Also always use FQDN to configure such products. Please follow this guide and check all settings are intact.
shutterhackerAuthor Commented:
@Amit
Thanks for the reply. The Guide you pointed to was exact how the setup was done. Watched the video multiple times and read the text guide multiple times and followed it exactly.

Do you think the HTTP vs HTTPS or the FQDN has something to do with the task sync error?
Amit KumarCommented:
Yes, it is as Exchange always works on SSL only.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

shutterhackerAuthor Commented:
Interesting.
The video in the guide the project server shows URL http://app-server-1/pwa all the time.
shutterhackerAuthor Commented:
Here is the jobs stuck in the queue.
Screenshot-2015-08-13-13.37.26.png
Amit KumarCommented:
Yeah you are right here... check once if "chestnut" is pingable from either side (Exchange <<-->> Project Server).

Open http://chestnut/PWA/_vti_bin/PSI/ExchangeSync.asmx on any machine as well see if it works and ask for authentication. what happens when authentication is done.
Amit KumarCommented:
The worst part is I have worked with MS on same issue in my previous Org, and that case has been troubleshooted for more than a month even but when I left there was no solution.. Now not sure if this is the same situation here.
shutterhackerAuthor Commented:
Here is the details in the queue error when I click the "click to view the error details"

Failure in updating the Exchange subscription for user 3641794f-2cf3-4013-b0ce-62ddcd99ad08:
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='56668f99-9227-48dc-b58e-eb115c420901' teamMemberUid='3641794f-2cf3-4013-b0ce-62ddcd99ad08' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='1bc69870-e289-48b8-83dc-048ade92de27' teamMemberUid='3641794f-2cf3-4013-b0ce-62ddcd99ad08' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='fc5134b1-d172-4fe3-821a-dd5e8bce4b72' teamMemberUid='3641794f-2cf3-4013-b0ce-62ddcd99ad08' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='eb64e95f-5841-4bf2-9827-f0f2f8609472' teamMemberUid='3641794f-2cf3-4013-b0ce-62ddcd99ad08' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='59da70aa-a559-4a15-88f7-462f6babc9ad' teamMemberUid='3641794f-2cf3-4013-b0ce-62ddcd99ad08' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='0c6749cb-5e92-4979-9431-76c290be57f2' teamMemberUid='3641794f-2cf3-4013-b0ce-62ddcd99ad08' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
Failure in updating the Exchange subscription for user b8f2e0cd-9882-4677-be61-b2780b32bae5:
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='4667ea90-8121-48b9-8299-9570aaa80989' teamMemberUid='b8f2e0cd-9882-4677-be61-b2780b32bae5' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='d7c79e69-881c-47e5-9ea2-c803118f6938' teamMemberUid='b8f2e0cd-9882-4677-be61-b2780b32bae5' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='9987d760-8d17-4c38-b44e-13895db9a3d6' teamMemberUid='b8f2e0cd-9882-4677-be61-b2780b32bae5' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='fffb614b-83e9-4c52-b863-db72d3893231' teamMemberUid='b8f2e0cd-9882-4677-be61-b2780b32bae5' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='068ade4c-3c64-4140-9a83-be28149100bb' teamMemberUid='b8f2e0cd-9882-4677-be61-b2780b32bae5' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
ExchangeSyncExchangeSubscriptionUpdateForUserFailed (40511). Details: id='40511' name='ExchangeSyncExchangeSubscriptionUpdateForUserFailed' uid='1fcdeb64-8026-41c3-9c3b-c2833f8e2739' teamMemberUid='b8f2e0cd-9882-4677-be61-b2780b32bae5' exception='System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid, SubscriptionContext context) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSyncUtil.UpdateExchangeSubscription(Guid resourceUid) at Microsoft.Office.Project.Server.ExchangeSync.ExchangeSubscriptionRefreshProcessor.Refresh(PlatformContext ctx)'.
Queue:
GeneralQueueJobFailed (26000) - ExchangeSyncTasks.ExchangeSubscriptionRefresh. Details: id='26000' name='GeneralQueueJobFailed' uid='76e20959-86c1-4cd6-b92b-cda357f62333' JobUID='dc1f0b70-9181-4e64-a258-ccd76e8339a4' ComputerName='CHESTNUT' GroupType='ExchangeSyncTasks' MessageType='ExchangeSubscriptionRefresh' MessageId='1' Stage=''. For more details, check the ULS logs on machine CHESTNUT for entries with JobUID dc1f0b70-9181-4e64-a258-ccd76e8339a4.
Amit KumarCommented:
for SSL point of view please read below quote on same article:

It needs to be done on Project server.

SSL Certificates

Project Server uses Secure Sockets Layer (SSL) to access Exchange Server and must trust the SSL certificate that was used by the Exchange farm. If you have a certificate issued by a trusted authority such as VeriSign, Project Server will trust the certificate. If your SSL certificate has not been issued by a trusted authority, you should export the certificate from the Exchange farm and import it as a trusted certificate on the computer that is running Project Server.
Previous versions of Microsoft Exchange Server 2010 let you use Internet Information Services (IIS) to save a copy of the certificate. Although IIS will still let you save a copy of the certificate in Exchange Server 2010, we don't recommend that you do this. Use the Export-ExchangeCertificate cmdlet to generate a copy of the certificate for importing to Project Server. After you've exported the SSL certificate in the form of a PKCS #12 file, the file can then be imported by using either the Import-ExchangeCertificate cmdlet or by using the Certificate Import wizard in the Exchange Management Console.
shutterhackerAuthor Commented:
The errors refer to a GUID 3641794f-2cf3-4013-b0ce-62ddcd99ad08
I checked the GUID of related users. None has such GUID.
Amit KumarCommented:
Please try search if subject GUID's user available in AD.

get-aduser -id {3641794f-2cf3-4013-b0ce-62ddcd99ad08}
Amit KumarCommented:
If above command does not work please try below one

get-adobject -id {3641794f-2cf3-4013-b0ce-62ddcd99ad08}

as per second error it seems users is not available.
shutterhackerAuthor Commented:
Now it gets more interesting.

GUID ...........ad08 was not found in AD either by get-aduser or by get adobject


PS C:\Users\Administrator> Get-ADobject -identity 3641794f-2cf3-4013-b0ce-62ddcd
99ad08
Get-ADObject : Directory object not found
At line:1 char:13
+ Get-ADobject <<<<  -identity 3641794f-2cf3-4013-b0ce-62ddcd99ad08
    + CategoryInfo          : ObjectNotFound: (3641794f-2cf3-4013-b0ce-62ddcd9
   9ad08:ADObject) [Get-ADObject], ADIdentityNotFoundException
    + FullyQualifiedErrorId : Directory object not found,Microsoft.ActiveDirec
   tory.Management.Commands.GetADObject

PS C:\Users\Administrator> Get-ADuser -identity 3641794f-2cf3-4013-b0ce-62ddcd99
ad08
Get-ADUser : Directory object not found
At line:1 char:11
+ Get-ADuser <<<<  -identity 3641794f-2cf3-4013-b0ce-62ddcd99ad08
    + CategoryInfo          : ObjectNotFound: (3641794f-2cf3-4013-b0ce-62ddcd9
   9ad08:ADUser) [Get-ADUser], ADIdentityNotFoundException
    + FullyQualifiedErrorId : Directory object not found,Microsoft.ActiveDirec
   tory.Management.Commands.GetADUser
shutterhackerAuthor Commented:
Even more, I got to the user management page and grab the GUID of existing Project users, and tried get-adobject and get-aduser. They also returned ObjectNotFound.
shutterhackerAuthor Commented:
For example I grab both GUID and AD GUID for this user and get-aduser/get-adobject did not find the user. Same for many other users.

The weird thing is that the users were not created in Project. They were replicated from AD when the users were created in AD.
Screenshot-2015-08-13-14.18.28.png
Amit KumarCommented:
I checked in my environment I can find user by GUID. please get an user in AD and see what is his/her guid and try to search that user by guid.  If that works means that Project server is taking any stale entry.

I hope you are using Domain controller to check user by GUID with AD module imported powershell.
shutterhackerAuthor Commented:
Amit, to answer your previous questions, the Project server and Exchange server are on the same subnet. They can ping each other no problem.

The URL http://chestnut/PWA/_vti_bin/PSI/ExchangeSync.asmx on workstations returned:

XML Parsing Error: no element found
Location: http://chestnut/PWA/_vti_bin/PSI/ExchangeSync.asmx
Line Number 1, Column 1:

I searched the physical location of the VD _vti_bin/PSI. There isn't a file that remotely resembles ExchangeSync.asmx. I don't know if the file should exist there.

Lastly, the Exchange server does have a commercial SSL certificate from Comodo that is trusted.
shutterhackerAuthor Commented:
Yes I was running AD module for power shell and I was running it on a DC.

get-aduser did work.

I did a get-aduser -filter * and grabbed one user's GUID. and did a test with get-aduser -identity with that GUID. It did return the correct user.

Just those GUIDs on the users in project server have no match in AD.
shutterhackerAuthor Commented:
Another weird thing I noticed in the video is:

The guy showed us how to do the configuration then finally he showed how the tasks were sent to his Outlook. He never showed or said how the completion update in Outlook is sync'd back to Project - there wasn't a word on completion sync back. In other words, he showed half of work and half of the results. We just assume the sync' back to project "should" work with the configuration he showed on the video.
shutterhackerAuthor Commented:
Amit, in your setup, when you get an AD GUID form a project user (like the blue shaded in the screen cap in my post ID 40928970), and run get-aduser with it, it returns the correct user from AD?
Amit KumarCommented:
I dont have project server in production, but it was in lab. But unfortunately lab got messed up and now things are not working properly. I will have reimage all of them.

I was meant to confirm that I am able to find user by GUID it was only from AD not from project server.
Amit KumarCommented:
If this is your production box I suggest log a case in MS and get help from there, because project server is the product where even MS engineers get confused that where is the issue. It is kind of unfortunate software even and Project server 2010 are full of bugs I think.
shutterhackerAuthor Commented:
Amit, I would bring this question back to its life.

We are totally new to this Project server and may have misunderstanding in how exchange sync works. We expected the task update made in Outlook would go directly to Project Center in Project Manager's view on the web app. This might be wrong. As we did more reading it appears the sync isn't supposed to work that way. It appears the sync just puts the completion update into "my tasks" in the resources' view on the web app. The resource must hit "Send Status" button to get the updates to the approve center. From there the PM approves the progress.

If that's the case, the exchange sync is working properly in my setup. The update reliably gets into "My tasks" section in the resources' view.

But problem remains. When the resource hits the send status button, a message "Your changes have not been submitted. Check displayed errors." Upon clicking OK. A red status bar shows "Status:  An unknown error has occurred."

If I understand correctly, the problem now is within the scope of Project web app/sharepoint.

Can you advise? including how is the task status update supposed to work?

Thanks again.
shutterhackerAuthor Commented:
The Exchange sync is working properly. The task completion is sync'd back to the resource's "My tasks" pane. The problem is with submitting the tasks for PM's approval, which causes an unknown error.

Set up a support ticket with MS for this unknown error. They checked it out and suggested us apply a hotfix. It did fixed the problem (now it's fully working) and MS refunded the ticket ($500) on the basis that their product had a problem.
shutterhackerAuthor Commented:
The sync error posted in earlier posts are likely due to a resource that has been deleted but still has pending tasks.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
shutterhackerAuthor Commented:
Thanks Amit for the help. We got solution from MS support and posted here for potentially helping others.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Applications

From novice to tech pro — start learning today.