CONTENT INDEX STATE FOR ALL DATABASES APPEAR AS "UNKNOWN".

1. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller service.
 
2. Remove all the files under “Fsis” file:

C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis
We moved these files to another folder for backup purposes.

3. Open EMS and “Run as Administrator"

4. Navigate to directory C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer

5. Launch the script and run the command below:

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.

6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):

.\installconfig.ps1 -action U -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

Then again installed Search component with below command :

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2016 server with repadmin /syncall /APeD.

8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatus and review the ContentIndexState.

Hi there,

I'm sure running dag but in case you don't then you still can run it.

1- Stop the mailbox store service.
2- Stop the Microsoft exchange search
3- Stop the Microsoft exchange host controller.
4- Dive into your mailbox folder and look for a folder like this "B889F121-8D8A-4617-9C06-751B0B3F656312.29.Single" - then delete it-
5- Start all the services back and will re-index everything back, give it some time.

Cheers,

Hi Emil,

I did that a couple of times since 2-3 weeks ago, on the smallest DB that only has 3 users and the state didn't change, remains UNKNOWN.

Thanks,

Are u using DAG?

No.

Just 1 server as a domain controller with Windows 2016 Standard and another server with Windows 2016 Standard and Exchange 2016.

And I tried stopping the 2 services, deleting the GUID for one of the databases (that only has 3 users), restarting the services and waiting 2-3 days.
Nothing accomplished, that DB remains as UNKNOWN.

Can you run this command and show me the output?

Get-MailboxDatabaseCopyStatus

Then


[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Unkown"}

Both commands show the same result...
results.jpg

Okay, here is what you have to do. You have 9 MaiboxDatabase up and running. You need to follow the instruction I have given to you

1- Stop the mailbox store service.
2- Stop the Microsoft exchange search
3- Stop the Microsoft exchange host controller.
4- Dive into your mailbox folder and look for a folder like this "B889F121-8D8A-4617-9C06-751B0B3F656312.29.Single" - then delete it on each database-
5- Start all the services back and will re-index everything back, give it some time. It can take an hour or so depending on how big your DBs are.
6- Wait some time. first, you will see it as "Unknown" then it will transition to Crawling, then healthy.

There's nothing wrong with your database, the Indexing means, if a user wants to look for more emails in outlook it won't be able to.
Cheers,

Why don;t you follow my action plan which I provided above since you already hv  stopped the 2 search services, renamed and deleted the GUID folder in the database directory and restarting the services.

OR create contentsubmitters group in AD.

1. Add a ContentSubmitters group to AD.
   I did this manually, creating it as a Security enabled Universal group.
2. Force or wait for Active Directory replication.
3. Assign Network Service and Administrators groups the “Full Access” permission (I think they mean “Full Control” permission).
   Ended up using the PowerShell commands
   Add-ADPermission -Id ContentSubmitters -User “Network Service” -Access Rights GenericAll
   Add-ADPermission -Id ContentSubmitters -User “Administrators” -Access Rights GenericAll
4. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services
5. Delete the Content Index data for the affected database(s)
   This is the folder with a GUID name inside the folder where the mailbox database lives.
6. Start the  Microsoft Exchange Search and Microsoft Exchange Search Host Controller services

Thanks Emil & Saif,

I tried this procedure deleting the subfolder but for only ONE of the databases.

QUESTION:
Do I need to delete the directories like "B889F121-8D8A-4617-9C06-751B0B3F656312.29.Single" for ALL and EACH of the databases and then start the services again?

I thought that I could try the procedure with just one of the databases and wait for the result.

Thanks again for your time!

Do I need to delete the directories like "B889F121-8D8A-4617-9C06-751B0B3F656312.29.Single" for ALL and EACH of the databases and then start the services again?

Yes, you need to do it, otherwise, It won't work properly. The system store handles all the database connectivity and it's mapped to the host controller which control the indexing for each one of the database. "Theoretically" you can do one but you'd get UNKNOWN you won't see the crawling transition.

I had this issue many times, that's how I got it solved. I don't know about @Saif Shaikh solution, but you can try doing his solution if mine doesn't work for you. But again, that's how I always get it done.

ok.

- I stopped both services.
- I deleted all sub-directories identified like "B889F121-8D8A-4617-9C06-751B0B3F656312.29.Single" for all databases.
- I restarted both services.

There is only one of the databases with 20-30 users.
All others have only a couple of test users.

I will leave them running and check on them first thing tomorrow morning.
Will keep you posted.

Again, thanks!



NOTE:
Oooooppssss!
I just noticed on the instructions that I have to also stop the Microsoft Exchange Information Store service.
Am I right?

I wanted to confirm the steps:
- Stop the Microsoft Exchange Information Store
- Stop the Microsoft Exchange Search
- Stop the Microsoft Exchange Search Host Controller
- Delete all directories like "B889F121-8D8A-4617-9C06-751B0B3F656312.29.Single" for ALL databases.
- Restart the Microsoft Exchange Information Store
- Restart the Microsoft Exchange Search
- Restart the Microsoft Exchange Search Host Controller

I always like to stop the database but that is optional, you don't need to.
From unknown should be crawling after some time, cause it has to rebuild the index of the database.

8 databases have only 3 users and almost no activity at all (200MB for the xxx.edb files of each one).
1 database have 30 users with normal activity (90GB for the xxx.edb file)

This morning, after 15 hours of running the procedure suggested, all database still appear as UNKNOWN.   :(

Something I am missing?

Have you tried rebooting the server?

Hi Saif,

I was trying your first recommendation. When I attempted to proceed with step two (deleting the files in the "FSIS" directory) I was unsuccessful in backing up the files. I attempted to move, copy, and rename the subfolders but no command seems to work. It seems that there is some service (or program) that is still using these files and do not let me do anything with them.I noticed when i renamed the folder "InteractionEngineNode1" to "OLD InteractionEngineNode1" the original directory was created again almost immediately. I am assuming there is a service running that created the directory again.

- How should I proceed?
- Is there an additional service that I should stop that would allow me to follow through with all your steps?
- Should I not back-up the files and proceed with the steps?
- Or should I follow your last recommendation about the "contentsubmitters group"?

Thanks!

HI Emil,

Yes, I did.
Same result.

Yes it will create the directory again. Follow all steps mentioned above.

Hi Saif,

Should I follow all steps without moving/renaming/deleting the directories inside the FSIS?

Thanks,

Yes since it is not allowing you to. When you reinstall the search component then, it will automatically recreate those folders and files inside it.

Taking backup is recommend, not mandatory.

I am receiving the following error:

Old nodes belonging to the system 'Fsis', already exist in 'd:\Program Files\Microsoft\Exchange
Server\V15\Bin\Search\Ceres\HostController\Data'. Rerun the configuration in attach mode, if you need to reuse them.

Hi,

Stop and disable the search services and rename the Data Folder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

I did what you recommended.

-Stopped the Search and Search Host Controller services.
-Renamed the Data directory as OLD-Data.
-The install step finished fine.
-The uninstall step (as originally suggested) finished fine.
-The install step for the 2nd. time finished fine.
-Confirmed that the Data directory was recreated.
-Restarted the Search and Search Host Controller services.

Should I reboot or just wait until tomorrow to inspect the Content Index State of each database.

Thanks again Saif!

Run command: get-mailboxdatabasecopystatus * 

From EMS and check if the database content index is in crawling state.

If yes then you can monitor the CI state by running command: 

get-mailboxdatabasecopystatus * | name, *content*, *crawl*

The above command show how many mailboxes are now left to crawl.

I am receiving the following error for all databases:
COULD NOT FIND REGISTRY KEY OF INDEX STATUS.
I guess the rebuilding process has aborted/stopped or is it still running in the background?

Something I should do to correct the error and run everything again?

Thanks!
error.jpg

1. Browse to the registry key. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus.
2. Check to see if any of the mailbox database keys exist.  They will be labeled by GUID.
3. Is the time stamp being updated for any of the keys.  Keep in mind, the default is GMT time.

If the registry keys do not exist and/or some of them are not updated, you have a HUGE c:\windows\temp folder.

1. Open cmd as administrator
2. Run [copy]del c:\windows\temp\* /Q /S[/copy]
3. Once the delete completes, restart both of the exchange search services.
4. Wait 10-15 minutes and run a [copy]get-mailboxdatabasecopystatus[/copy] and the Content Index State should be something other than unknown.  If will likely Healthy or Crawling.

I found the same suggestion yesterday on some google pages.
"If the registry keys do not exist and/or some of them are not updated, you have a HUGE c:\windows\temp folder"

Seeing that there wouldn't be any harm in deleting temporary files, I did it yesterday and after that, did your whole process again around 6pm.
Came this morning at 9am and the error message still is present and all DB are still UNKNOWN.

Any suggestion on correcting the:
COULD NOT FIND REGISTRY KEY OF INDEX STATUS

- Could be some kind of protection of the registry that is blocking the writing of these needed values?

Do this, try creating a new mailbox database, restart the mailbox store and tell me if you see the same issue on the newly created database.

Done.

Attaching screenshots of the registry (no new keys created at all under SEARCH) and the index state of the new TEST-DATABASE.

Same result...   :(
registry.jpg
test.jpg

Dude,

Just for curiosity, what CU you have installed at this moment? Your issue sounds like a bug to me. The reason why is because if you created a new mailbox database and the database still "Unknown" then, that means there something wrong with the updates.

Tell me what CU you have installed.

Check the permission on the "c:\programData\microsoft\crypto\rsa\machinekeys" folder.

AND

Give full access permission for the below accounts on MachineKeys folder
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys
-Administrator
-Exchange Servers

- System
-Exchange Trusted Subsystems
-The account which you have logged into the server  your admin ID.
- Restart the Microsoft Exchange Search Host Controller and Microsoft Exchange Search.

OR

Reboot the exchange server

Find screenshot attached.

The version displayed with the command, corresponds to:
Exchange Server 2016 CU13
June 18, 2019
15.1.1779.2
15.01.1779.002

Already started the download for CU14 & CU15.
edition.jpg

Whenever you can please install this CU: https://www.microsoft.com/en-us/download/details.aspx?id=100302

Thanks Hemil,

I am planning to install CU14 & CU15, but it will have to be on Monday.
I am having surgery tomorrow morning.

Hi Saif,

I checked the permissions on that directory and indeed there were no users or groups added.
Make the changes you mentioned, and ran the whole process again.

It was done about 20 minutes and so far the state is still UNKNOWN.

No worries, good luck with the surgery. Having your index unknown won't do anything to the user but what it would is, if a user has tons of emails and wants to look for more emails via Outlook/web app, it won't be able to due to index issue.

Thanks Hemil,

Will contact you again first thing Monday.

I think upgrade might not resolve your issue since you are already on a good one. This has not caused due to an update I'm sure. 

First Troubleshooting which you need to perform before upgrade:

=================================================

You said that you found there was no users or members added on the Machinekeys folder. If you have added them correctly which I said above, then you need to reboot the server and see.

Also if this does not fix after reboot, you can try to  rename the folder to \Machinekeys.old and reboot the server. on the affected node and see the behaviour.

Second Troubleshooting which you need to perform before upgrade:

=================================================

More information on fixing the error:  COULD NOT FIND REGISTRY KEY OF INDEX STATUS

You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:

Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:

{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single

1.)    The initial {GUID} is the GUID of the database.

There will be 1 entry per database on the DAG node.
2.)    ‘SPECIAL’ means that the CI is reseeding

The source server, port, source database, and cell instance trail this value
3.)    The double numeric values (1,1; 4,7; 8,8; etc.) signify the state[s] of the CI

4.)    'Single' references which type of cell we are dealing with.

5.)    The two digit number, after the second GUID, signifies the schema version of the index (in this case, the schema version is 12). You will not be able to find the database in question, if you use the second GUID and leave the schema version appended to it.

 

The first number in the series is the Index Status. The second number in the series is the error code. The codes break-down as follows:

Index Status:

Unknown = 0
Healthy = 1
Crawling =2
Failed = 3
Seeding = 4
Failed And Suspended = 5
Suspended = 6
Disabled = 7
Auto-Suspended = 8
Healthy And Upgrading = 9

Error Code:

1 Success
2 Internal Error        
3 Crawling Database
4 DatabaseOffline
5 MapiNetworkError
6 Catalog Corruption
7 Seeding Catalog        
8 Catalog Suspended        
9 Catalog Reseed
10 Index Not Enabled        
11 Catalog Excluded
12 Activation Preference skipped        
13  Lag Copy skipped        
14 Recovery Database Skipped        
15 Fast Error        
16 Service Not Running
17 Index Timestamp too Old    

When a CI is reseeding, the status will – obviously – be ‘Seeding’ when you run ‘Get-MailboxDatabaseCopyStatus’. In order to see where the seeding source is, find the ‘ContentIndexSeedingSource’ value for that specific copy via 'Get-MailboxDatabaseCopyStatus' or you can query the registry key. Keep in mind, though, that the source port and cell are not exposed via ‘Get-MailboxDatabaseCopyStatus’ and are only found in the registry.

- You may recreate the registry keys and set it to 3 as an example below:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus]
"{b7772c8c-2f3a-48d7-bbcb-75df0befdaf5}"="3,3,51539607564,2014-06-12 21:54:48Z,0,,0,0"
"{9b169239-79e0-4325-9f03-dee6d1085738}"="3,3,51539607564,2014-09-15 16:02:09Z,0,,0,0"
"{cb1c7d99-2f45-4cd2-b5ed-db2b22077692}"="3,3,4294967297,2014-09-15 16:05:51Z,0,,18,0"
"{bcd21097-dd52-4533-9ddf-d8b47e4d0134}"="3,3,4294967297,2014-09-15 16:04:34Z,0,,18,0"
"{2bf3d91c-cd12-4789-ac9a-a1ec063c6038}"="3,3,51539607564,2014-09-15 16:05:01Z,0,,0,0"
"{2d08b685-44f4-455b-92c3-0211f331e15f}"="3,3,4294967297,2014-09-15 16:05:47Z,0,,0,0"

This is very interesting but I'm sure upgrade and all is not a part of solution since your issue is more of Microsoft support case now. But I would suggest before you upgrade your server do this step and see the result. But a reboot is required since this is a registry change.

Make sure you recreate the keys, Create a string value, in the name, insert the GUID of the database, in the Data according to the example above and if you have 5 databases in DAG you will have to create 5 registry keys as done in the example above. 

@Saif - keep in mind he does not have DAG. He has a standalone server with one exchange server. I believe the solution you're giving him is more like for DAG, from where you are going to re-seed a database when it comes from the source? I'm not saying it may be an update issue I'm just saying updating your system might fix bogus internal issues we do not know about.

But hopefully what you're saying may be helpful. That issue he's experiencing comes from DAG but he's not replicating its database.

@Hemil did you verified what fixes the CU is going to fix.

Under the section 

Issues that this cumulative update fixes

It does not mention content index unknown is a bug in earlier version and that is gonna fix in cu14.

https://support.microsoft.com/en-in/help/4514140/cumulative-update-14-for-exchange-server-2016

OK so let's say that even if he has astandalone server then I change my step with stand alone one.

So my troubleshooting step remains the same but this time I give u output of stand alone server as well as server in Dag, go for it and try.

You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:

Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

Assuming your mailbox server is not a member of a DAG, when you run the command you'll get outputs that look something like the following:

{79853a57-6c9e-46a2-a5e3-70db036754f1} : 1,1,4294967297,2013-07-16 00:33:17Z,0,
{115d21d9-8b88-4574-b71e-5a7142daaf78} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
{aab61984-7df3-412f-9c0b-f44bf0eaadbc} : 1,1,4294967297,2013-07-16 00:33:16Z,0,

Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:

{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single

Thanks guys,

I just came back today and going to start doing the suggestions made about doing the queries and will let you know.

This is the result of the query of the keys:

[PS] C:\Windows\system32>Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus
Get-ItemProperty : Cannot find path 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus' because it does not exist.
At line:1 char:1
+ Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\I ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (HKLM:\SOFTWARE\...rch\IndexStatus:String) [Get-ItemProperty], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetItemPropertyCommand

Try to rename machinekeys to machinekeysold and restarted search services or server.

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\

Make sure you have following permissions on the newly created machinekeys folder after the search services or server is restarted.

-Administrator
-Exchange Servers

- System
-Exchange Trusted Subsystems
-The account which you have logged into the server   your admin ID.

Hi Saif,
Tried this last suggestion with no results.


Hi Emil,
After no results, tried to install CU14, but it stuck on step 1 of 17: Stopping Services.
After several minutes, I got the following error:

Error:
The following error was generated when "$error.Clear();
          & $RoleBinPath\ServiceControl.ps1 -Operation:DisableServices -Roles:($RoleRoles.Replace('Role','').Split(',')) -SetupScriptsDirectory:$RoleBinPath;
          & $RoleBinPath\ServiceControl.ps1 -Operation:Stop -Roles:($RoleRoles.Replace('Role','').Split(',')) -IsDatacenter:([bool]$RoleIsDatacenter)
        " was run: "Microsoft.Exchange.Configuration.Tasks.ServiceDidNotReachStatusException: Service 'MSExchangeTransport' failed to reach status 'Stopped' on this server.
   at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
   at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
   at Microsoft.Exchange.Management.Tasks.ManageSetupService.WaitForServiceStatus(ServiceController serviceController, ServiceControllerStatus status, Unlimited`1 maximumWaitTime, Boolean ignoreFailures, Boolean sendWatsonReportForHungService)
   at Microsoft.Exchange.Management.Tasks.ManageSetupService.StopService(ServiceController serviceController, Boolean ignoreServiceStopTimeout, Boolean failIfServiceNotInstalled, Unlimited`1 maximumWaitTime)
   at Microsoft.Exchange.Management.Tasks.ManageSetupService.StopService(String serviceName, Boolean ignoreServiceStopTimeout, Boolean failIfServiceNotInstalled, Unlimited`1 maximumWaitTime)
   at Microsoft.Exchange.Management.Tasks.StopSetupService.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()
   at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".

There was no ROLLBACK button or CANCEL, just the EXIT button.
After exiting the CU14 update, clients were not able to connect, so I rebooted the server.

After rebooting, a lot of services in the server (from Exchange and also the IIS, w3svc and others) were DISABLED.
So nothing was working.

I set the services as AUTOMATIC and rebooted the server again.
Now the OWA and ECP are able to ask for credentials, but after typing user and password, I get this error:

The website cannot display the page
  HTTP 500
 
Most likely causes:
•The website is under maintenance.
•The website has a programming error.
 
What you can try:
  Refresh the page.  
  Go back to the previous page.  
 
More information  More information  


I tried recreating the virtual directories for the OWA and ECP, but it keeps failing after the credentials.



PLEASE HELP ASAP!!

UPDATE:

Without alternatives, I tried CU15, and at least it did find something:

Error:
This computer requires .NET Framework 4.8 (https://support.microsoft.com/kb/4503548).
For more information, visit: https://docs.microsoft.com/Exchange/plan-and-deploy/system-requirements?view=exchserver-2016

So it seems that CU14 had time to DISABLE some services and UNINSTALL some programs.

Downloaded .NET 4.8 and now it is installing, to then RETRY the CU15 again.

UPDATE:

CU15 is now on step 3 of 17: Remove Exchange Files at 90%.

At least step 1 (Stopping Services) and step 2 finished fine.

UPDATE:

CU15 in Step 8 of 17: Languages

So far, no errors.

UPDATE:

Step 9 of 17: Management Tools 11%

Hey Roberto,

Sorry for my long delay, I had too much going on the past weeks. I see you still unable to resolve this issue.
Honestly, you shouldn't have any trouble while doing so. If you did then, perhaps cause of renaming the above suggestion? "I can't tell for sure" - I installed the cumulative updates few weeks ago and never had any issue related to it.

Here is a helpful link for you: https://eclat.tech/software/microsoft/exchangeserver/exchange-server-2013/microsoft-exchange-server-2016-cumulative-update-14-nightmare/

Beware sometimes exchange likes to shut down services for some reason and never get it back online. You can manually turn it on.
How's everything doing?

Thanks Hemil,

Now is at step 12 of 17:Mailbox Role -  Unified Messaging Service
Now errors so far.

I just wish that when everything finishes, the server will be operative.

Great! Let's hope everything gets resolved.

Bad news,

At step 13 of 17: Mailbox Role: Mailbox Services got the following error:

Error:
The following error was generated when "$error.Clear();
      buildToBuildUpgrade-ExsetdataAtom -AtomName SystemAttendant -DomainController $RoleDomainController

" was run: "Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'.
   at Microsoft.Exchange.Configuration.Tasks.Task.ThrowTerminatingError(Exception exception, ErrorCategory category, Object target)
   at Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.HandleExsetdataReturnCode(UInt32 scErr)
   at Microsoft.Exchange.Management.Deployment.ManageExsetdataAtom.BuildToBuildUpgradeAtom(AtomID atomID)
   at Microsoft.Exchange.Management.Deployment.BuildToBuildUpgradeExsetdataAtom.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__91_1()
   at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".



No idea what to do next.

Remove the update from the control panel and reboot the server.

@Hemil my troubleshooting steps are all accurate and may not cause the upgrade to fail. Renaming the machine keys folder will not impact the upgrade to fail. You can see where and on which role it is failing. 

It does not fail on search component or machine keys. It is failing on 13 of 17: Mailbox Role: Mailbox Services

"Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'.

 

Before upgrade you should also have suggested him to take a snapshot/backup of exchange and system state backup. I know snapshots are not supported by MS, but during upgrade failures they save life. 

But nevertheless since your plan was not feasible and suggestions made that the fixes that are applicable to new CU. It does not mention anywhere in the fixes that CI state of unknown is a bug and will be fixed in CU 14 for exchange 2016. 

Now that the damaged has been done, you say to revert back as per your update "Remove the update from the control panel and reboot the server." Even this is not possible now.

"Given the error message “An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'” It’s recommended to check if any disc related issue (for example: Event 55) on the drive where your Exchange is installed in the Application log.

If exists, it’s recommended to take a backup of the DB data , server back up and then run chkdisk on the drive where your exchange was located."

@Saif Shaikh

Let me clarify something that may be misunderstood. I have never questioned your steps, you are a professional IT like any of us. But it seems to me you're taking this thread a little bit personal.

Please read a little more carefully a thread before you start shooting words that I have never said.

Here is what you stated

Before upgrade, you should also have suggested to him take a snapshot/backup of exchange and system state backup. I know snapshots are not supported by MS, but during upgrade failures they save life.

First and foremost, It is true, I give you that. I should've suggested he take a snapshot/backup - BUT with all due respect to "Roberto", I'm expecting any sysadmin to know better to have backups before starting any type of system modifications - we are IT, not users. AND, if that was the case, why didn't you tell him to make a backup of its system before he starts modifying default parameters on the registry? he is trusting you applying your steps as wells as me right? and perhaps he has no clue what he's doing on the registry, what if something goes wrong? It sounds to me like Irony doesn't it? but anyway, I just wanted to point it out.

Also, keep in mind any IT knows better than anyone  "SYSTEMS ARE NOT STABLE, WE AS ADMIN DO OUR BEST TO MAINTAIN IT"

but nevertheless, since your plan was not feasible and suggestions made that the fixes that are applicable to new CU/

I have never made a comment explicitly stating "an update will fix it". I clearly said "It sounds like a bug to me, perhaps you should check the update, what CU you have? - and on a second thread, I said, Whenever you can please install this CU: https://www.microsoft.com/en-us/download/details.aspx?id=100302 "This is the part you mentioned I did not tell him to back up its system."

I never gave him the effectiveness of that's the REAL issue might be, instead, my thoughts. "Your issue sounds like a bug to ME" I'm giving my opinion.

"The reason why is because if you created a new mailbox database and the database still "Unknown" then, that means there something wrong with the updates." that means don't take my word from granted let's investigate.

Last but not least

Now that the damage has been done, you say to revert back as per your update " Remove the update from the control panel and reboot the server." Even this is not possible now.

Okay, based upon that comment you sound like "I did the damage and therefore everything is messed up because of it."

Look, pal, I will give you this advice if you work within an IT environment do not look for a culprit but instead a solution. With that remark, you are creating a poisonous environment wherever you go. Keep in mind one thing.

1- I never doubted your solution
2- I even told Roberto to follow-up with you ever since you sound certain modifying registry. Myself personally, I do not like modifying the registry unless I have a trusted source from Microsoft. But, since you too certain then why not? should've given it a try.

We are here to share our thoughts and help one another and not to belittle on another.


To get around into this issue, we need to restore an exchange backup, if possible an image and find what's wrong with the system. His issue shouldn't be complicated as I mentioned before, Uknown Indexing is only shown when you use DAG. I have never seen indexing issues without DAG.

Hi guys,

Sorry I caused a bit of trouble.
I just had a training course with Microsoft that only covered installing the Exchange and configuring accounts, groups, rules and some other basic things. Let's forget about this and just keep going.

Finally I had to get support from our provider, which put us in touch with Microsoft Support and they enabled a remote login session.
The most critical problem (the server was not operative) was resolved by them.

It seems that the disk where Exchange 2016 was installed and the other disk where all the databases are, BOTH were somehow corrupted and this was determined by them. All these disks were on a virtual environment with VMWare (and I don't even have access to that console). They "requested" to create new disks for both and they recovered the installation from the databases to the new disks.

Almost everything is now working fine, but they will enable another remote session to correct the infamous SEARCH FUNCTION not working.

I appreciate all the help you provided.

Good to know, you have resolved the issue. Here are few additional suggestions that I need to add. I had faced same issue because I have not upgraded the server with latest updates. So first Confirm that your Exchange server has the latest updates installed. Sometimes, issues are resolved with updates. Also Ensure there is sufficient disk space available on the drive hosting your Exchange databases.