Cloud PBX Inbound Calls failure in Hybrid Configurations

First, a quick description of my environment.

• Hybrid configuration between on-premises skype for business servers and Skype for business online servers.
• Skype for business on-premise Enterprise Voice
• Have some Skype for business users homed on-premises and others homed in Skype for Business Online.
• Must Continue to use On-premise PSTN Connectivity (my case, CISCO).

Some Background Information

Due to cost restrictions, the company planned to keep both environments. Some users will be moved to Skype online while others will remain on-premise. We needed to leverage existing Cisco Voice Systems (on-premise PBX). So, the best option for us was to use a Cloud PBX Connectivity option. In the cloud, we bought an E 3 + Phone System license.

Hybrid has been configured and Cloud PBX Connectivity been set up as described like in this article

You have to ensure phone license is assigned to the user otherwise you will not able to flip enterprise voice setting from False to True in Skype Online Power shell for the user.

If you use exchange Online UM as mailbox server, HostedVoiceMailPolicy is needed too for the user to allow them to receive Voicemails. So, these are settings needed.  

• HostedVoiceMail – set True
• EnterpriseVoiceEnabled set True
• Assign HostedVoiceMailPolicy
• Assign Voice Policy

Back to the fun part.

Symptom:

External callers seeing server error when they call skype for business online enterprise voice users.

Problem:

Skype for business online Enterprise Voice users did not receive inbound calls. But they could make outbound calls.

Prior to moving the pilot users, I like to test out my work and verify that everything is working fine before I say it’s ready. I enabled a test user. My test user ( skype online) could make outbound calls using Cloud PBX connectivity ( on-premise PSTN) but inbound calls to the test user were getting rejected in between the skype on-premises server and Skype online servers. The external caller will see a "server error" message or call "cannot be completed" error message on their cell phone when they try to call my Skype for Business online test user.

I started my Centralized logging (cls logging) on the on-premise Skype for business servers (both “AlwaysOn” and “IncomingAndOutgoingCall”). Searched the log and opened in snooper. Then I saw two INVITES ( IN and OUT), Trying, and then “504 server time Error“ originating from my Edge Server which I believed was also obtained from Skype for business online servers.

Interestingly, invites successfully came into on-premise Skype servers and skype on premises sent another invite out to Skype Online servers via the Edge server, but the online servers rejected the call due to (what I believed) it could not recognize the domain/does not like the format of the domain being passed across within the diversion header. The header forwarded was from Cisco gateway /PBX and inconsistent with user sip Domain.

Invite (IN) – invites from PSTN/PBX

---------------------------------------------------------

ms-call-source: non-ms-rtc
Diversion: "Test User" <sip:+1XXX-XXX-XXXX@172.16.100.100>;reason=unconditional;privacy=off;screen=yes
ms-trunking-peer: 172.16.100.100;trunk=Sfb15_172.16.100.100;User-Agent="Cisco-CUCM10.5"
Session-Expires: 1800
Min-SE: 90
Message-Body:

INVITE (OUT): Invite Sent to Skype Online.

----------------------------------------------------------------

ALLOW: CANCEL,BYE,INVITE,REFER,NOTIFY,PRACK,UPDATE
ms-gateway-callid: 285b1800-a1311759-583388-90ae10ac@172.16.100.100
ms-endpoint-location-data: NetworkScope;ms-media-location-type=intranet
ms-call-source: non-ms-rtc
Diversion: "Test  User" <sip:XXX-XXX-XXXX@172.16.100.100>;reason=unconditional;privacy=off;screen=yes
ms-trunking-peer: 172.16.100.100 ;trunk=Sfb15_172.16.100.100;User-Agent="Cisco-CUCM10.5"
Session-Expires: 1800
Min-SE: 90

Server Time out Error

ms-diagnostics: 1032;reason="Message diversion user domain traffic type does not match To user domain";to-user-type="domain-type-remote";diversion-type="domain-type-none";source="sipfed1A.online.lync.com";source-server="DM21A01EDG14.infra.lync.com"
ms-split-domain-info: ms-traffic-type=SplitIntra;ms-remote-fqdn=sipfed.online.lync.com
ms-telemetry-id: D45BF8C0-AAD6-5857-9278-01E788F1115E
Server: RTC/7.0

Solution/workaround: 

Was to turn off diversion header in CCM from the Sip Trunk between on-premises Skype for business mediation and Cisco gateway (PBX). Then skype for business online users started getting inbound calls.