Why don;t you follow my action plan which I provided above since you already hv stopped the 2 search services, renamed and deleted the GUID folder in the database directory and restarting the services.
OR create contentsubmitters group in AD.
Do I need to delete the directories like "B889F121-8D8A-4617-9C06-751B0B3F656 312.29.Sin gle" for ALL and EACH of the databases and then start the services again?
Yes it will create the directory again. Follow all steps mentioned above.
Yes since it is not allowing you to. When you reinstall the search component then, it will automatically recreate those folders and files inside it.
Taking backup is recommend, not mandatory.
Hi,
Stop and disable the search services and rename the Data Folder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
Run command: get-mailboxdatabasecopystatus *
From EMS and check if the database content index is in crawling state.
If yes then you can monitor the CI state by running command:
get-mailboxdatabasecopystatus * | name, *content*, *crawl*
The above command show how many mailboxes are now left to crawl.
If the registry keys do not exist and/or some of them are not updated, you have a HUGE c:\windows\temp folder.
Check the permission on the "c:\programData\microsoft\crypto\rsa\machinekeys" folder.
AND
Give full access permission for the below accounts on MachineKeys folder
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys
-Administrator
-Exchange Servers
- System
-Exchange Trusted Subsystems
-The account which you have logged into the server your admin ID.
- Restart the Microsoft Exchange Search Host Controller and Microsoft Exchange Search.
OR
Reboot the exchange server
I think upgrade might not resolve your issue since you are already on a good one. This has not caused due to an update I'm sure.
First Troubleshooting which you need to perform before upgrade:
=================================================
You said that you found there was no users or members added on the Machinekeys folder. If you have added them correctly which I said above, then you need to reboot the server and see.
Also if this does not fix after reboot, you can try to rename the folder to \Machinekeys.old and reboot the server. on the affected node and see the behaviour.
Second Troubleshooting which you need to perform before upgrade:
=================================================
More information on fixing the error: COULD NOT FIND REGISTRY KEY OF INDEX STATUS
You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:
Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus
Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:
{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single
1.) The initial {GUID} is the GUID of the database.
There will be 1 entry per database on the DAG node.
2.) ‘SPECIAL’ means that the CI is reseeding
The source server, port, source database, and cell instance trail this value
3.) The double numeric values (1,1; 4,7; 8,8; etc.) signify the state[s] of the CI
4.) 'Single' references which type of cell we are dealing with.
5.) The two digit number, after the second GUID, signifies the schema version of the index (in this case, the schema version is 12). You will not be able to find the database in question, if you use the second GUID and leave the schema version appended to it.
The first number in the series is the Index Status. The second number in the series is the error code. The codes break-down as follows:
Index Status:
Unknown = 0
Healthy = 1
Crawling =2
Failed = 3
Seeding = 4
Failed And Suspended = 5
Suspended = 6
Disabled = 7
Auto-Suspended = 8
Healthy And Upgrading = 9
Error Code:
1 Success
2 Internal Error
3 Crawling Database
4 DatabaseOffline
5 MapiNetworkError
6 Catalog Corruption
7 Seeding Catalog
8 Catalog Suspended
9 Catalog Reseed
10 Index Not Enabled
11 Catalog Excluded
12 Activation Preference skipped
13 Lag Copy skipped
14 Recovery Database Skipped
15 Fast Error
16 Service Not Running
17 Index Timestamp too Old
When a CI is reseeding, the status will – obviously – be ‘Seeding’ when you run ‘Get-MailboxDatabaseCopyStatus’. In order to see where the seeding source is, find the ‘ContentIndexSeedingSource’ value for that specific copy via 'Get-MailboxDatabaseCopyStatus' or you can query the registry key. Keep in mind, though, that the source port and cell are not exposed via ‘Get-MailboxDatabaseCopyStatus’ and are only found in the registry.
- You may recreate the registry keys and set it to 3 as an example below:
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus]
"{b7772c8c-2f3a-48d7-bbcb-75df0befdaf5}"="3,3,51539607564,2014-06-12 21:54:48Z,0,,0,0"
"{9b169239-79e0-4325-9f03-dee6d1085738}"="3,3,51539607564,2014-09-15 16:02:09Z,0,,0,0"
"{cb1c7d99-2f45-4cd2-b5ed-db2b22077692}"="3,3,4294967297,2014-09-15 16:05:51Z,0,,18,0"
"{bcd21097-dd52-4533-9ddf-d8b47e4d0134}"="3,3,4294967297,2014-09-15 16:04:34Z,0,,18,0"
"{2bf3d91c-cd12-4789-ac9a-a1ec063c6038}"="3,3,51539607564,2014-09-15 16:05:01Z,0,,0,0"
"{2d08b685-44f4-455b-92c3-0211f331e15f}"="3,3,4294967297,2014-09-15 16:05:47Z,0,,0,0"
This is very interesting but I'm sure upgrade and all is not a part of solution since your issue is more of Microsoft support case now. But I would suggest before you upgrade your server do this step and see the result. But a reboot is required since this is a registry change.
Make sure you recreate the keys, Create a string value, in the name, insert the GUID of the database, in the Data according to the example above and if you have 5 databases in DAG you will have to create 5 registry keys as done in the example above.
@Hemil did you verified what fixes the CU is going to fix.
Under the section
It does not mention content index unknown is a bug in earlier version and that is gonna fix in cu14.
https://support.microsoft.com/en-in/help/4514140/cumulative-update-14-for-exchange-server-2016
OK so let's say that even if he has astandalone server then I change my step with stand alone one.
So my troubleshooting step remains the same but this time I give u output of stand alone server as well as server in Dag, go for it and try.
You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:
Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus
Assuming your mailbox server is not a member of a DAG, when you run the command you'll get outputs that look something like the following:
{79853a57-6c9e-46a2-a5e3-70db036754f1} : 1,1,4294967297,2013-07-16 00:33:17Z,0,
{115d21d9-8b88-4574-b71e-5a7142daaf78} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
{aab61984-7df3-412f-9c0b-f44bf0eaadbc} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:
{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single
Try to rename machinekeys to machinekeysold and restarted search services or server.
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\
Make sure you have following permissions on the newly created machinekeys folder after the search services or server is restarted.
-Administrator
-Exchange Servers
- System
-Exchange Trusted Subsystems
-The account which you have logged into the server your admin ID.
@Hemil my troubleshooting steps are all accurate and may not cause the upgrade to fail. Renaming the machine keys folder will not impact the upgrade to fail. You can see where and on which role it is failing.
It does not fail on search component or machine keys. It is failing on 13 of 17: Mailbox Role: Mailbox Services
"Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'.
Before upgrade you should also have suggested him to take a snapshot/backup of exchange and system state backup. I know snapshots are not supported by MS, but during upgrade failures they save life.
But nevertheless since your plan was not feasible and suggestions made that the fixes that are applicable to new CU. It does not mention anywhere in the fixes that CI state of unknown is a bug and will be fixed in CU 14 for exchange 2016.
Now that the damaged has been done, you say to revert back as per your update "Remove the update from the control panel and reboot the server." Even this is not possible now.
"Given the error message “An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'” It’s recommended to check if any disc related issue (for example: Event 55) on the drive where your Exchange is installed in the Application log.
If exists, it’s recommended to take a backup of the DB data , server back up and then run chkdisk on the drive where your exchange was located."
Before upgrade, you should also have suggested to him take a snapshot/backup of exchange and system state backup. I know snapshots are not supported by MS, but during upgrade failures they save life.First and foremost, It is true, I give you that. I should've suggested he take a snapshot/backup - BUT with all due respect to "Roberto", I'm expecting any sysadmin to know better to have backups before starting any type of system modifications - we are IT, not users. AND, if that was the case, why didn't you tell him to make a backup of its system before he starts modifying default parameters on the registry? he is trusting you applying your steps as wells as me right? and perhaps he has no clue what he's doing on the registry, what if something goes wrong? It sounds to me like Irony doesn't it? but anyway, I just wanted to point it out.
but nevertheless, since your plan was not feasible and suggestions made that the fixes that are applicable to new CU/
Now that the damage has been done, you say to revert back as per your update " Remove the update from the control panel and reboot the server." Even this is not possible now.
1. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller service.
2. Remove all the files under “Fsis” file:
C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis
We moved these files to another folder for backup purposes.
3. Open EMS and “Run as Administrator"
4. Navigate to directory C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer
5. Launch the script and run the command below:
.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.
6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):
.\installconfig.ps1 -action U -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
Then again installed Search component with below command :
.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2016 server with repadmin /syncall /APeD.
8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatus and review the ContentIndexState.