asked on

Looonng Replication delays- Exchange 2013

I have an organization with Exchange 2013 servers at three sites running a DAG. Latency between sites 1 and 2 is about 1-2ms, and latency between either site and site 3 is 25-35 ms. get-exchangeserver shows all three servers using the DC at site 2. we have been noticing serious delays in replication when settings change- for example, 2 weeks ago we removed a delegation setting from a mailbox, and one of the users still has the access while the other does not. Sometimes settings take a few days to go through. Any idea where I should look for the cause of this? I haven't noticed anything abnormal in the event logs. The only change I made recently was this morning where I removed Hyper-V time sync from the guest services for the DC, which was causing a 3-minute time difference between sites- well within the 5 minute SSL/Kerberos 'skew' limit. Let me know- I appreciate any suggestions!

Systech Admin

what is the IOPs of the storage and please check if time is correct on all exchange servers

it_medcomp

ASKER

IOPS vary since the servers' environments vary- all are Hyper-V running on 2019, but one is DAS, and the other two are on SAN devices- Both of them are IBM Storwize 5000 arrays, connected by FO, but one of the two servers is on a 3-server cluster, and the other is a standalone server. Is there a tool that I can adequately compare the IOPS with?

Thanks!

Systech Admin

there are lot of tools to measure the performance.

refer this one

https://gallery.technet.microsoft.com/DiskSpd-A-Robust-Storage-6ef84e62

also hows ur email queue ?

Amit

How is your AD replication. Can you run below command:

repadmin /replsum

it_medcomp

ASKER

@systechadmin:
Thanks! Email queue is empty. Here are the IOPS on the three servers- I included all drives, as I'm not sure if this is related to databases and logs, or the OS drive:
Site 1:
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
1 | 299761664 | 4574 | 28.59 | 457.40 | c: (299GiB)
2 | 523632640 | 7990 | 49.94 | 799.00 | e: (572GiB)
3 | 2742419456 | 41846 | 261.54 | 4184.60 | f: (749GiB)
4 | 534249472 | 8152 | 50.95 | 815.20 | g: (877GiB)
5 | 629866496 | 9611 | 60.07 | 961.10 | h: (1099GiB)
6 | 629669888 | 9608 | 60.05 | 960.80 | I: (1023GiB)
7 | 689831936 | 10526 | 65.79 | 1052.60 | j: (1199GiB)
8 | 675807232 | 10312 | 64.45 | 1031.20 | k: (1423GiB)
9 | 683671552 | 10432 | 65.20 | 1043.20 | L: (2420GiB)
------------------------------------------------------------------------------
total: 7545618432 | 115137 | 719.61 | 11513.70

Site 2:
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
1 | 1012465664 | 15449 | 96.41 | 1542.51 | c: (299GiB)
2 | 1030160384 | 15719 | 98.09 | 1569.47 | e: (399GiB)
3 | 4352966656 | 66421 | 414.49 | 6631.83 | f: (352GiB)
4 | 1047134208 | 15978 | 99.71 | 1595.33 | g: (899GiB)
5 | 963969024 | 14709 | 91.79 | 1468.63 | h: (1090GiB)
6 | 1189478400 | 18150 | 113.26 | 1812.19 | I: (1023GiB)
7 | 958201856 | 14621 | 91.24 | 1459.84 | j: (1199GiB)
8 | 1024458752 | 15632 | 97.55 | 1560.78 | k: (808GiB)
9 | 880476160 | 13435 | 83.84 | 1341.42 | L: (2559GiB)
------------------------------------------------------------------------------
total: 12726370304 | 194189 | 1211.80 | 19388.88

Site 3:
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
1 | 285409280 | 4355 | 27.18 | 434.82 | c: (299GiB)
2 | 203292672 | 3102 | 19.36 | 309.72 | e: (499GiB)
3 | 210436096 | 3211 | 20.04 | 320.60 | f: (749GiB)
4 | 267452416 | 4081 | 25.47 | 407.47 | g: (499GiB)
5 | 187039744 | 2854 | 17.81 | 284.96 | h: (649GiB)
6 | 264962048 | 4043 | 25.23 | 403.67 | I: (749GiB)
7 | 248119296 | 3786 | 23.63 | 378.01 | j: (1199GiB)
8 | 251396096 | 3836 | 23.94 | 383.01 | k: (499GiB)
9 | 245301248 | 3743 | 23.36 | 373.72 | L: (1023GiB)
------------------------------------------------------------------------------
total: 4313186304 | 65814 | 410.70 | 6571.19

it_medcomp

ASKER

@Amit:
PS C:\Users\Administrator.CORP> repadmin /replsum
Replication Summary Start Time: 2019-12-05 10:17:41

Beginning data collection for replication summary, this may take awhile:
...................

Source DSA largest delta fails/total %% error
MXMTSVRDC4 30m:43s 0 / 15 0
MXMTSVRDC7-2 30m:43s 0 / 10 0
MXMTSVRDC8 27m:23s 0 / 15 0
MXMTSVRDC8-2 21m:58s 0 / 10 0
USMDSVRADVDC 02m:50s 0 / 5 0
USMDSVRDC 30m:04s 0 / 20 0
USMDSVRDC1-2 30m:04s 0 / 10 0
USMDSVRDC2 27m:35s 0 / 30 0
USMDSVRDC3 30m:37s 0 / 15 0
USMDSVRDC3-1 29m:42s 0 / 15 0
USMDSVRDC4 27m:35s 0 / 10 0
USMDSVRDC5 30m:37s 0 / 10 0
USMTSVRDC2 18m:29s 0 / 15 0
USPMSVRDC 02m:51s 0 / 5 0
USSKSVRDC 02m:52s 0 / 10 0
USYMSVRDC2 03h:28m:49s 0 / 20 0

Destination DSA largest delta fails/total %% error
MXMTSVRDC4 21m:59s 0 / 10 0
MXMTSVRDC7-2 20m:15s 0 / 10 0
MXMTSVRDC8 18m:31s 0 / 15 0
MXMTSVRDC8-2 30m:48s 0 / 10 0
USMDSVRADVDC 12m:35s 0 / 10 0
USMDSVRDC 30m:38s 0 / 20 0
USMDSVRDC1-2 26m:06s 0 / 10 0
USMDSVRDC2 29m:43s 0 / 15 0
USMDSVRDC3 26m:43s 0 / 10 0
USMDSVRDC3-1 28m:24s 0 / 10 0
USMDSVRDC4 30m:29s 0 / 25 0
USMDSVRDC5 30m:53s 0 / 10 0
USMTSVRDC2 27m:23s 0 / 10 0
USPMSVRDC 03h:28m:49s 0 / 15 0
USSKSVRDC 03h:20m:17s 0 / 15 0
USYMSVRDC2 02m:59s 0 / 20 0

Amit

Replication looks fine with no errors, however, i can see some dc with 3h+ replication time. Like USYMSVRDC2. Any reason, why all Exchange pointing to Site 2. If that is the case, then there are configuration issue in your AD sites, which need to be corrected. Because, if your Exchange is going to remote site, that will impact the replication. I assume, here replication issue related to AD and Exchange attributes, not DAG logs.

it_medcomp

ASKER

The ones with the longer times are remote sites. I forgot to include some detail- All of the servers with names like USMDSVRDC are site 1; USMTSVRDC2 is site 2, and site 3 has USPMSVRDC2.

Amit

What subnet configured for your all Exchange server.

it_medcomp

ASKER

Site 1: 192.168.1.0/24
Site 2: 192.168.2.0/24
Site 3: 192.168.10.0/24

This question needs an answer!

Become an EE member today

7 DAY FREE TRIAL

Members can start a 7-Day Free trial then enjoy unlimited access to the platform.

View membership options

Learn why we charge membership fees

We get it - no one likes a content blocker. Take one extra minute and find out why we block content.