We help IT Professionals succeed at work.

Looonng Replication delays- Exchange 2013

it_medcomp
it_medcomp asked
on
Medium Priority
24 Views
Last Modified: 2019-12-05
I have an organization with Exchange 2013 servers at three sites running a DAG. Latency between sites 1 and 2 is about 1-2ms, and latency between either site and site 3 is 25-35 ms. get-exchangeserver shows all three servers using the DC at site 2. we have been noticing serious delays in replication when settings change- for example, 2 weeks ago we removed a delegation setting from a mailbox, and one of the users still has the access while the other does not. Sometimes settings take a few days to go through. Any idea where I should look for the cause of this? I haven't noticed anything abnormal in the event logs. The only change I made recently was this morning where I removed Hyper-V time sync from the guest services for the DC, which was causing a 3-minute time difference between sites- well within the 5 minute SSL/Kerberos 'skew' limit. Let me know- I appreciate any suggestions!
Comment
Watch Question

Systech AdminChief Technology Officer
CERTIFIED EXPERT

Commented:
what is the IOPs of the storage and please check if time is correct on all exchange servers

Author

Commented:
IOPS vary since the servers' environments vary- all are Hyper-V running on 2019, but one is DAS, and the other two are on SAN devices- Both of them are IBM Storwize 5000 arrays, connected by FO, but one of the two servers is on a 3-server cluster, and the other is a standalone server. Is there a tool that I can adequately compare the IOPS with?

Thanks!
Systech AdminChief Technology Officer
CERTIFIED EXPERT

Commented:
there are lot of tools to measure the performance.

refer this one

https://gallery.technet.microsoft.com/DiskSpd-A-Robust-Storage-6ef84e62

also hows ur email queue ?
AmitIT Architect
CERTIFIED EXPERT
Distinguished Expert 2017

Commented:
How is your AD replication. Can you run below command:

repadmin /replsum

Author

Commented:
@systechadmin:
Thanks! Email queue is empty. Here are the IOPS on the three servers- I included all drives, as I'm not sure if this is related to databases and logs, or the OS drive:
Site 1:
Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     1 |       299761664 |         4574 |      28.59 |     457.40 | c: (299GiB)
     2 |       523632640 |         7990 |      49.94 |     799.00 | e: (572GiB)
     3 |      2742419456 |        41846 |     261.54 |    4184.60 | f: (749GiB)
     4 |       534249472 |         8152 |      50.95 |     815.20 | g: (877GiB)
     5 |       629866496 |         9611 |      60.07 |     961.10 | h: (1099GiB)
     6 |       629669888 |         9608 |      60.05 |     960.80 | I: (1023GiB)
     7 |       689831936 |        10526 |      65.79 |    1052.60 | j: (1199GiB)
     8 |       675807232 |        10312 |      64.45 |    1031.20 | k: (1423GiB)
     9 |       683671552 |        10432 |      65.20 |    1043.20 | L: (2420GiB)
------------------------------------------------------------------------------
total:        7545618432 |       115137 |     719.61 |   11513.70


Site 2:
Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     1 |      1012465664 |        15449 |      96.41 |    1542.51 | c: (299GiB)
     2 |      1030160384 |        15719 |      98.09 |    1569.47 | e: (399GiB)
     3 |      4352966656 |        66421 |     414.49 |    6631.83 | f: (352GiB)
     4 |      1047134208 |        15978 |      99.71 |    1595.33 | g: (899GiB)
     5 |       963969024 |        14709 |      91.79 |    1468.63 | h: (1090GiB)
     6 |      1189478400 |        18150 |     113.26 |    1812.19 | I: (1023GiB)
     7 |       958201856 |        14621 |      91.24 |    1459.84 | j: (1199GiB)
     8 |      1024458752 |        15632 |      97.55 |    1560.78 | k: (808GiB)
     9 |       880476160 |        13435 |      83.84 |    1341.42 | L: (2559GiB)
------------------------------------------------------------------------------
total:       12726370304 |       194189 |    1211.80 |   19388.88

Site 3:
Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
    1 |       285409280 |         4355 |      27.18 |     434.82 | c: (299GiB)
    2 |       203292672 |         3102 |      19.36 |     309.72 | e: (499GiB)
    3 |       210436096 |         3211 |      20.04 |     320.60 | f: (749GiB)
    4 |       267452416 |         4081 |      25.47 |     407.47 | g: (499GiB)
    5 |       187039744 |         2854 |      17.81 |     284.96 | h: (649GiB)
    6 |       264962048 |         4043 |      25.23 |     403.67 | I: (749GiB)
    7 |       248119296 |         3786 |      23.63 |     378.01 | j: (1199GiB)
    8 |       251396096 |         3836 |      23.94 |     383.01 | k: (499GiB)
    9 |       245301248 |         3743 |      23.36 |     373.72 | L: (1023GiB)
------------------------------------------------------------------------------
total:        4313186304 |        65814 |     410.70 |    6571.19

Author

Commented:
@Amit:
PS C:\Users\Administrator.CORP> repadmin /replsum
Replication Summary Start Time: 2019-12-05 10:17:41

Beginning data collection for replication summary, this may take awhile:
  ...................


Source DSA          largest delta    fails/total %%   error
 MXMTSVRDC4                30m:43s    0 /  15    0
 MXMTSVRDC7-2              30m:43s    0 /  10    0
 MXMTSVRDC8                27m:23s    0 /  15    0
 MXMTSVRDC8-2              21m:58s    0 /  10    0
 USMDSVRADVDC              02m:50s    0 /   5    0
 USMDSVRDC                 30m:04s    0 /  20    0
 USMDSVRDC1-2              30m:04s    0 /  10    0
 USMDSVRDC2                27m:35s    0 /  30    0
 USMDSVRDC3                30m:37s    0 /  15    0
 USMDSVRDC3-1              29m:42s    0 /  15    0
 USMDSVRDC4                27m:35s    0 /  10    0
 USMDSVRDC5                30m:37s    0 /  10    0
 USMTSVRDC2                18m:29s    0 /  15    0
 USPMSVRDC                 02m:51s    0 /   5    0
 USSKSVRDC                 02m:52s    0 /  10    0
 USYMSVRDC2            03h:28m:49s    0 /  20    0


Destination DSA     largest delta    fails/total %%   error
 MXMTSVRDC4                21m:59s    0 /  10    0
 MXMTSVRDC7-2              20m:15s    0 /  10    0
 MXMTSVRDC8                18m:31s    0 /  15    0
 MXMTSVRDC8-2              30m:48s    0 /  10    0
 USMDSVRADVDC              12m:35s    0 /  10    0
 USMDSVRDC                 30m:38s    0 /  20    0
 USMDSVRDC1-2              26m:06s    0 /  10    0
 USMDSVRDC2                29m:43s    0 /  15    0
 USMDSVRDC3                26m:43s    0 /  10    0
 USMDSVRDC3-1              28m:24s    0 /  10    0
 USMDSVRDC4                30m:29s    0 /  25    0
 USMDSVRDC5                30m:53s    0 /  10    0
 USMTSVRDC2                27m:23s    0 /  10    0
 USPMSVRDC             03h:28m:49s    0 /  15    0
 USSKSVRDC             03h:20m:17s    0 /  15    0
 USYMSVRDC2                02m:59s    0 /  20    0
AmitIT Architect
CERTIFIED EXPERT
Distinguished Expert 2017

Commented:
Replication looks fine with no errors, however, i can see some dc with 3h+ replication time. Like USYMSVRDC2. Any reason, why all Exchange pointing to Site 2. If that is the case, then there are configuration issue in your AD sites, which need to be corrected. Because, if your Exchange is going to remote site, that will impact the replication. I assume, here replication issue related to AD and Exchange attributes, not DAG logs.

Author

Commented:
The ones with the longer times are remote sites. I forgot to include some detail- All of the servers with names like USMDSVRDC are site 1; USMTSVRDC2 is site 2, and site 3 has USPMSVRDC2.
AmitIT Architect
CERTIFIED EXPERT
Distinguished Expert 2017

Commented:
What subnet configured for your all Exchange server.

Author

Commented:
Site 1: 192.168.1.0/24
Site 2: 192.168.2.0/24
Site 3: 192.168.10.0/24

Explore More ContentExplore courses, solutions, and other research materials related to this topic.