- Virtualization
- Exchange
- Installation
- Dell
- Windows OS
CONTENT INDEX STATE FOR ALL DATABASES APPEAR AS "UNKNOWN".
I am running a brand new installation of Exchange 2016.I have a Windows 2016 Standard working as the domain, and another Windows 2016 Standard with all the roles of Exchange 2016.
So far everything is working fine with the users, groups, rules, etc. with the exception of the SEARCH function (either on OWA or on local OUTLOOK 365).
The state of every database appear as unknown.
After reading and trying some suggestions, the problem still persists.
- I've tried stopping the 2 search services, renaming and deleting the GUID folder in the database directory and restarting the services.
- I've tried moving one the database (the one that only has 2 users) to a new database and mounting the new one.
Nothing has worked and top management is starting to put pressure on solving this issue ASAP.
Here is a screenshot of the databases states,
Thanks for your help,
Roberto.
databases.jpg
So far everything is working fine with the users, groups, rules, etc. with the exception of the SEARCH function (either on OWA or on local OUTLOOK 365).
The state of every database appear as unknown.
After reading and trying some suggestions, the problem still persists.
- I've tried stopping the 2 search services, renaming and deleting the GUID folder in the database directory and restarting the services.
- I've tried moving one the database (the one that only has 2 users) to a new database and mounting the new one.
Nothing has worked and top management is starting to put pressure on solving this issue ASAP.
Here is a screenshot of the databases states,
Thanks for your help,
Roberto.
databases.jpg
1. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller service.
2. Remove all the files under “Fsis” file:
C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis
We moved these files to another folder for backup purposes.
3. Open EMS and “Run as Administrator"
4. Navigate to directory C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer
5. Launch the script and run the command below:
.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.
6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):
.\installconfig.ps1 -action U -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
Then again installed Search component with below command :
.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2016 server with repadmin /syncall /APeD.
8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatus and review the ContentIndexState.
Hi there,
I'm sure running dag but in case you don't then you still can run it.
1- Stop the mailbox store service.
2- Stop the Microsoft exchange search
3- Stop the Microsoft exchange host controller.
4- Dive into your mailbox folder and look for a folder like this "B889F121-8D8A-4617-9C06-7
5- Start all the services back and will re-index everything back, give it some time.
Cheers,
I'm sure running dag but in case you don't then you still can run it.
1- Stop the mailbox store service.
2- Stop the Microsoft exchange search
3- Stop the Microsoft exchange host controller.
4- Dive into your mailbox folder and look for a folder like this "B889F121-8D8A-4617-9C06-7
5- Start all the services back and will re-index everything back, give it some time.
Cheers,
Hi Emil,
I did that a couple of times since 2-3 weeks ago, on the smallest DB that only has 3 users and the state didn't change, remains UNKNOWN.
Thanks,
I did that a couple of times since 2-3 weeks ago, on the smallest DB that only has 3 users and the state didn't change, remains UNKNOWN.
Thanks,
No.
Just 1 server as a domain controller with Windows 2016 Standard and another server with Windows 2016 Standard and Exchange 2016.
And I tried stopping the 2 services, deleting the GUID for one of the databases (that only has 3 users), restarting the services and waiting 2-3 days.
Nothing accomplished, that DB remains as UNKNOWN.
Just 1 server as a domain controller with Windows 2016 Standard and another server with Windows 2016 Standard and Exchange 2016.
And I tried stopping the 2 services, deleting the GUID for one of the databases (that only has 3 users), restarting the services and waiting 2-3 days.
Nothing accomplished, that DB remains as UNKNOWN.
Can you run this command and show me the output?
Get-MailboxDatabaseCopySta
Then
[PS] C:\>Get-MailboxDatabaseCop
Get-MailboxDatabaseCopySta
Then
[PS] C:\>Get-MailboxDatabaseCop
Okay, here is what you have to do. You have 9 MaiboxDatabase up and running. You need to follow the instruction I have given to you
1- Stop the mailbox store service.
2- Stop the Microsoft exchange search
3- Stop the Microsoft exchange host controller.
4- Dive into your mailbox folder and look for a folder like this "B889F121-8D8A-4617-9C06-7
5- Start all the services back and will re-index everything back, give it some time. It can take an hour or so depending on how big your DBs are.
6- Wait some time. first, you will see it as "Unknown" then it will transition to Crawling, then healthy.
There's nothing wrong with your database, the Indexing means, if a user wants to look for more emails in outlook it won't be able to.
Cheers,
1- Stop the mailbox store service.
2- Stop the Microsoft exchange search
3- Stop the Microsoft exchange host controller.
4- Dive into your mailbox folder and look for a folder like this "B889F121-8D8A-4617-9C06-7
5- Start all the services back and will re-index everything back, give it some time. It can take an hour or so depending on how big your DBs are.
6- Wait some time. first, you will see it as "Unknown" then it will transition to Crawling, then healthy.
There's nothing wrong with your database, the Indexing means, if a user wants to look for more emails in outlook it won't be able to.
Cheers,
Why don;t you follow my action plan which I provided above since you already hv stopped the 2 search services, renamed and deleted the GUID folder in the database directory and restarting the services.
OR create contentsubmitters group in AD.
- Add a ContentSubmitters group to AD.
I did this manually, creating it as a Security enabled Universal group. - Force or wait for Active Directory replication.
- Assign Network Service and Administrators groups the “Full Access” permission (I think they mean “Full Control” permission).
Ended up using the PowerShell commands
Add-ADPermission -Id ContentSubmitters -User “Network Service” -Access Rights GenericAll
Add-ADPermission -Id ContentSubmitters -User “Administrators” -Access Rights GenericAll - Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services
- Delete the Content Index data for the affected database(s)
This is the folder with a GUID name inside the folder where the mailbox database lives. - Start the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services
Thanks Emil & Saif,
I tried this procedure deleting the subfolder but for only ONE of the databases.
QUESTION:
Do I need to delete the directories like "B889F121-8D8A-4617-9C06-7
I thought that I could try the procedure with just one of the databases and wait for the result.
Thanks again for your time!
I tried this procedure deleting the subfolder but for only ONE of the databases.
QUESTION:
Do I need to delete the directories like "B889F121-8D8A-4617-9C06-7
I thought that I could try the procedure with just one of the databases and wait for the result.
Thanks again for your time!
Do I need to delete the directories like "B889F121-8D8A-4617-9C06-751B0B3F656 312.29.Sin gle" for ALL and EACH of the databases and then start the services again?
Yes, you need to do it, otherwise, It won't work properly. The system store handles all the database connectivity and it's mapped to the host controller which control the indexing for each one of the database. "Theoretically" you can do one but you'd get UNKNOWN you won't see the crawling transition.
I had this issue many times, that's how I got it solved. I don't know about @Saif Shaikh solution, but you can try doing his solution if mine doesn't work for you. But again, that's how I always get it done.
ok.
- I stopped both services.
- I deleted all sub-directories identified like "B889F121-8D8A-4617-9C06-7
- I restarted both services.
There is only one of the databases with 20-30 users.
All others have only a couple of test users.
I will leave them running and check on them first thing tomorrow morning.
Will keep you posted.
Again, thanks!
NOTE:
Oooooppssss!
I just noticed on the instructions that I have to also stop the Microsoft Exchange Information Store service.
Am I right?
I wanted to confirm the steps:
- Stop the Microsoft Exchange Information Store
- Stop the Microsoft Exchange Search
- Stop the Microsoft Exchange Search Host Controller
- Delete all directories like "B889F121-8D8A-4617-9C06-7
- Restart the Microsoft Exchange Information Store
- Restart the Microsoft Exchange Search
- Restart the Microsoft Exchange Search Host Controller
- I stopped both services.
- I deleted all sub-directories identified like "B889F121-8D8A-4617-9C06-7
- I restarted both services.
There is only one of the databases with 20-30 users.
All others have only a couple of test users.
I will leave them running and check on them first thing tomorrow morning.
Will keep you posted.
Again, thanks!
NOTE:
Oooooppssss!
I just noticed on the instructions that I have to also stop the Microsoft Exchange Information Store service.
Am I right?
I wanted to confirm the steps:
- Stop the Microsoft Exchange Information Store
- Stop the Microsoft Exchange Search
- Stop the Microsoft Exchange Search Host Controller
- Delete all directories like "B889F121-8D8A-4617-9C06-7
- Restart the Microsoft Exchange Information Store
- Restart the Microsoft Exchange Search
- Restart the Microsoft Exchange Search Host Controller
I always like to stop the database but that is optional, you don't need to.
From unknown should be crawling after some time, cause it has to rebuild the index of the database.
From unknown should be crawling after some time, cause it has to rebuild the index of the database.
8 databases have only 3 users and almost no activity at all (200MB for the xxx.edb files of each one).
1 database have 30 users with normal activity (90GB for the xxx.edb file)
This morning, after 15 hours of running the procedure suggested, all database still appear as UNKNOWN. :(
Something I am missing?
1 database have 30 users with normal activity (90GB for the xxx.edb file)
This morning, after 15 hours of running the procedure suggested, all database still appear as UNKNOWN. :(
Something I am missing?
Hi Saif,
I was trying your first recommendation. When I attempted to proceed with step two (deleting the files in the "FSIS" directory) I was unsuccessful in backing up the files. I attempted to move, copy, and rename the subfolders but no command seems to work. It seems that there is some service (or program) that is still using these files and do not let me do anything with them.I noticed when i renamed the folder "InteractionEngineNode1" to "OLD InteractionEngineNode1" the original directory was created again almost immediately. I am assuming there is a service running that created the directory again.
- How should I proceed?
- Is there an additional service that I should stop that would allow me to follow through with all your steps?
- Should I not back-up the files and proceed with the steps?
- Or should I follow your last recommendation about the "contentsubmitters group"?
Thanks!
I was trying your first recommendation. When I attempted to proceed with step two (deleting the files in the "FSIS" directory) I was unsuccessful in backing up the files. I attempted to move, copy, and rename the subfolders but no command seems to work. It seems that there is some service (or program) that is still using these files and do not let me do anything with them.I noticed when i renamed the folder "InteractionEngineNode1" to "OLD InteractionEngineNode1" the original directory was created again almost immediately. I am assuming there is a service running that created the directory again.
- How should I proceed?
- Is there an additional service that I should stop that would allow me to follow through with all your steps?
- Should I not back-up the files and proceed with the steps?
- Or should I follow your last recommendation about the "contentsubmitters group"?
Thanks!
Hi Saif,
Should I follow all steps without moving/renaming/deleting the directories inside the FSIS?
Thanks,
Should I follow all steps without moving/renaming/deleting the directories inside the FSIS?
Thanks,
Yes since it is not allowing you to. When you reinstall the search component then, it will automatically recreate those folders and files inside it.
Taking backup is recommend, not mandatory.
I am receiving the following error:
Old nodes belonging to the system 'Fsis', already exist in 'd:\Program Files\Microsoft\Exchange
Server\V15\Bin\Search\Cere
Old nodes belonging to the system 'Fsis', already exist in 'd:\Program Files\Microsoft\Exchange
Server\V15\Bin\Search\Cere
Hi,
Stop and disable the search services and rename the Data Folder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
I did what you recommended.
-Stopped the Search and Search Host Controller services.
-Renamed the Data directory as OLD-Data.
-The install step finished fine.
-The uninstall step (as originally suggested) finished fine.
-The install step for the 2nd. time finished fine.
-Confirmed that the Data directory was recreated.
-Restarted the Search and Search Host Controller services.
Should I reboot or just wait until tomorrow to inspect the Content Index State of each database.
Thanks again Saif!
-Stopped the Search and Search Host Controller services.
-Renamed the Data directory as OLD-Data.
-The install step finished fine.
-The uninstall step (as originally suggested) finished fine.
-The install step for the 2nd. time finished fine.
-Confirmed that the Data directory was recreated.
-Restarted the Search and Search Host Controller services.
Should I reboot or just wait until tomorrow to inspect the Content Index State of each database.
Thanks again Saif!
Run command: get-mailboxdatabasecopystatus *
From EMS and check if the database content index is in crawling state.
If yes then you can monitor the CI state by running command:
get-mailboxdatabasecopystatus * | name, *content*, *crawl*
The above command show how many mailboxes are now left to crawl.
I am receiving the following error for all databases:
COULD NOT FIND REGISTRY KEY OF INDEX STATUS.
I guess the rebuilding process has aborted/stopped or is it still running in the background?
Something I should do to correct the error and run everything again?
Thanks!
error.jpg
COULD NOT FIND REGISTRY KEY OF INDEX STATUS.
I guess the rebuilding process has aborted/stopped or is it still running in the background?
Something I should do to correct the error and run everything again?
Thanks!
error.jpg
- Browse to the registry key. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus.
- Check to see if any of the mailbox database keys exist. They will be labeled by GUID.
- Is the time stamp being updated for any of the keys. Keep in mind, the default is GMT time.
If the registry keys do not exist and/or some of them are not updated, you have a HUGE c:\windows\temp folder.
- Open cmd as administrator
- Run [copy]del c:\windows\temp\* /Q /S[/copy]
- Once the delete completes, restart both of the exchange search services.
- Wait 10-15 minutes and run a [copy]get-mailboxdatabasecopystatus[/copy] and the Content Index State should be something other than unknown. If will likely Healthy or Crawling.
I found the same suggestion yesterday on some google pages.
"If the registry keys do not exist and/or some of them are not updated, you have a HUGE c:\windows\temp folder"
Seeing that there wouldn't be any harm in deleting temporary files, I did it yesterday and after that, did your whole process again around 6pm.
Came this morning at 9am and the error message still is present and all DB are still UNKNOWN.
Any suggestion on correcting the:
COULD NOT FIND REGISTRY KEY OF INDEX STATUS
- Could be some kind of protection of the registry that is blocking the writing of these needed values?
"If the registry keys do not exist and/or some of them are not updated, you have a HUGE c:\windows\temp folder"
Seeing that there wouldn't be any harm in deleting temporary files, I did it yesterday and after that, did your whole process again around 6pm.
Came this morning at 9am and the error message still is present and all DB are still UNKNOWN.
Any suggestion on correcting the:
COULD NOT FIND REGISTRY KEY OF INDEX STATUS
- Could be some kind of protection of the registry that is blocking the writing of these needed values?
Do this, try creating a new mailbox database, restart the mailbox store and tell me if you see the same issue on the newly created database.
Done.
Attaching screenshots of the registry (no new keys created at all under SEARCH) and the index state of the new TEST-DATABASE.
Same result... :(
registry.jpg
test.jpg
Attaching screenshots of the registry (no new keys created at all under SEARCH) and the index state of the new TEST-DATABASE.
Same result... :(
registry.jpg
test.jpg
Dude,
Just for curiosity, what CU you have installed at this moment? Your issue sounds like a bug to me. The reason why is because if you created a new mailbox database and the database still "Unknown" then, that means there something wrong with the updates.
Tell me what CU you have installed.
Just for curiosity, what CU you have installed at this moment? Your issue sounds like a bug to me. The reason why is because if you created a new mailbox database and the database still "Unknown" then, that means there something wrong with the updates.
Tell me what CU you have installed.
Check the permission on the "c:\programData\microsoft\crypto\rsa\machinekeys" folder.
AND
Give full access permission for the below accounts on MachineKeys folder
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys
-Administrator
-Exchange Servers
-
System
-Exchange Trusted Subsystems
-The account which you have logged into the server
your admin ID.
- Restart the Microsoft Exchange Search Host Controller and Microsoft Exchange Search.
OR
Reboot the exchange server
Find screenshot attached.
The version displayed with the command, corresponds to:
Exchange Server 2016 CU13
June 18, 2019
15.1.1779.2
15.01.1779.002
Already started the download for CU14 & CU15.
edition.jpg
The version displayed with the command, corresponds to:
Exchange Server 2016 CU13
June 18, 2019
15.1.1779.2
15.01.1779.002
Already started the download for CU14 & CU15.
edition.jpg
Whenever you can please install this CU: https://www.microsoft.com/en-us/download/details.aspx?id=100302
Thanks Hemil,
I am planning to install CU14 & CU15, but it will have to be on Monday.
I am having surgery tomorrow morning.
I am planning to install CU14 & CU15, but it will have to be on Monday.
I am having surgery tomorrow morning.
Hi Saif,
I checked the permissions on that directory and indeed there were no users or groups added.
Make the changes you mentioned, and ran the whole process again.
It was done about 20 minutes and so far the state is still UNKNOWN.
I checked the permissions on that directory and indeed there were no users or groups added.
Make the changes you mentioned, and ran the whole process again.
It was done about 20 minutes and so far the state is still UNKNOWN.
No worries, good luck with the surgery. Having your index unknown won't do anything to the user but what it would is, if a user has tons of emails and wants to look for more emails via Outlook/web app, it won't be able to due to index issue.
I think upgrade might not resolve your issue since you are already on a good one. This has not caused due to an update I'm sure.
First Troubleshooting which you need to perform before upgrade:
=================================================
You said that you found there was no users or members added on the Machinekeys folder. If you have added them correctly which I said above, then you need to reboot the server and see.
Also if this does not fix after reboot, you can try to rename the folder to \Machinekeys.old and reboot the server. on the affected node and see the behaviour.
Second Troubleshooting which you need to perform before upgrade:
=================================================
More information on fixing the error: COULD NOT FIND REGISTRY KEY OF INDEX STATUS
You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:
Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus
Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:
{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single
1.) The initial {GUID} is the GUID of the database.
There will be 1 entry per database on the DAG node.
2.) ‘SPECIAL’ means that the CI is reseeding
The source server, port, source database, and cell instance trail this value
3.) The double numeric values (1,1; 4,7; 8,8; etc.) signify the state[s] of the CI
4.) 'Single' references which type of cell we are dealing with.
5.) The two digit number, after the second GUID, signifies the schema version of the index (in this case, the schema version is 12). You will not be able to find the database in question, if you use the second GUID and leave the schema version appended to it.
The first number in the series is the Index Status. The second number in the series is the error code. The codes break-down as follows:
Index Status:
Unknown = 0
Healthy = 1
Crawling =2
Failed = 3
Seeding = 4
Failed And Suspended = 5
Suspended = 6
Disabled = 7
Auto-Suspended = 8
Healthy And Upgrading = 9
Error Code:
1 Success
2 Internal Error
3 Crawling Database
4 DatabaseOffline
5 MapiNetworkError
6 Catalog Corruption
7 Seeding Catalog
8 Catalog Suspended
9 Catalog Reseed
10 Index Not Enabled
11 Catalog Excluded
12 Activation Preference skipped
13 Lag Copy skipped
14 Recovery Database Skipped
15 Fast Error
16 Service Not Running
17 Index Timestamp too Old
When a CI is reseeding, the status will – obviously – be ‘Seeding’ when you run ‘Get-MailboxDatabaseCopyStatus’. In order to see where the seeding source is, find the ‘ContentIndexSeedingSource’ value for that specific copy via 'Get-MailboxDatabaseCopyStatus' or you can query the registry key. Keep in mind, though, that the source port and cell are not exposed via ‘Get-MailboxDatabaseCopyStatus’ and are only found in the registry.
- You may recreate the registry keys and set it to 3 as an example below:
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus]
"{b7772c8c-2f3a-48d7-bbcb-75df0befdaf5}"="3,3,51539607564,2014-06-12 21:54:48Z,0,,0,0"
"{9b169239-79e0-4325-9f03-dee6d1085738}"="3,3,51539607564,2014-09-15 16:02:09Z,0,,0,0"
"{cb1c7d99-2f45-4cd2-b5ed-db2b22077692}"="3,3,4294967297,2014-09-15 16:05:51Z,0,,18,0"
"{bcd21097-dd52-4533-9ddf-d8b47e4d0134}"="3,3,4294967297,2014-09-15 16:04:34Z,0,,18,0"
"{2bf3d91c-cd12-4789-ac9a-a1ec063c6038}"="3,3,51539607564,2014-09-15 16:05:01Z,0,,0,0"
"{2d08b685-44f4-455b-92c3-0211f331e15f}"="3,3,4294967297,2014-09-15 16:05:47Z,0,,0,0"
This is very interesting but I'm sure upgrade and all is not a part of solution since your issue is more of Microsoft support case now. But I would suggest before you upgrade your server do this step and see the result. But a reboot is required since this is a registry change.
Make sure you recreate the keys, Create a string value, in the name, insert the GUID of the database, in the Data according to the example above and if you have 5 databases in DAG you will have to create 5 registry keys as done in the example above.
@Saif - keep in mind he does not have DAG. He has a standalone server with one exchange server. I believe the solution you're giving him is more like for DAG, from where you are going to re-seed a database when it comes from the source? I'm not saying it may be an update issue I'm just saying updating your system might fix bogus internal issues we do not know about.
But hopefully what you're saying may be helpful. That issue he's experiencing comes from DAG but he's not replicating its database.
But hopefully what you're saying may be helpful. That issue he's experiencing comes from DAG but he's not replicating its database.
@Hemil did you verified what fixes the CU is going to fix.
Under the section
Issues that this cumulative update fixes
It does not mention content index unknown is a bug in earlier version and that is gonna fix in cu14.
https://support.microsoft.com/en-in/help/4514140/cumulative-update-14-for-exchange-server-2016
OK so let's say that even if he has astandalone server then I change my step with stand alone one.
So my troubleshooting step remains the same but this time I give u output of stand alone server as well as server in Dag, go for it and try.
You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:
Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus
Assuming your mailbox server is not a member of a DAG, when you run the command you'll get outputs that look something like the following:
{79853a57-6c9e-46a2-a5e3-70db036754f1} : 1,1,4294967297,2013-07-16 00:33:17Z,0,
{115d21d9-8b88-4574-b71e-5a7142daaf78} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
{aab61984-7df3-412f-9c0b-f44bf0eaadbc} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:
{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single
Thanks guys,
I just came back today and going to start doing the suggestions made about doing the queries and will let you know.
I just came back today and going to start doing the suggestions made about doing the queries and will let you know.
This is the result of the query of the keys:
[PS] C:\Windows\system32>Get-It
Get-ItemProperty : Cannot find path 'HKLM:\SOFTWARE\Microsoft\
At line:1 char:1
+ Get-ItemProperty HKLM:\SOFTWARE\Microsoft\E
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (HKLM:\SOFTWARE\...rch\Ind
+ FullyQualifiedErrorId : PathNotFound,Microsoft.Pow
[PS] C:\Windows\system32>Get-It
Get-ItemProperty : Cannot find path 'HKLM:\SOFTWARE\Microsoft\
At line:1 char:1
+ Get-ItemProperty HKLM:\SOFTWARE\Microsoft\E
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (HKLM:\SOFTWARE\...rch\Ind
+ FullyQualifiedErrorId : PathNotFound,Microsoft.Pow
Try to rename machinekeys to machinekeysold and restarted search services or server.
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\
Make sure you have following permissions on the newly created machinekeys folder after the search services or server is restarted.
-Administrator
-Exchange Servers
- System
-Exchange Trusted Subsystems
-The account which you have logged into the server your admin ID.
Hi Saif,
Tried this last suggestion with no results.
Hi Emil,
After no results, tried to install CU14, but it stuck on step 1 of 17: Stopping Services.
After several minutes, I got the following error:
Error:
The following error was generated when "$error.Clear();
& $RoleBinPath\ServiceContro
& $RoleBinPath\ServiceContro
" was run: "Microsoft.Exchange.Config
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Configu
There was no ROLLBACK button or CANCEL, just the EXIT button.
After exiting the CU14 update, clients were not able to connect, so I rebooted the server.
After rebooting, a lot of services in the server (from Exchange and also the IIS, w3svc and others) were DISABLED.
So nothing was working.
I set the services as AUTOMATIC and rebooted the server again.
Now the OWA and ECP are able to ask for credentials, but after typing user and password, I get this error:
The website cannot display the page
HTTP 500
Most likely causes:
•The website is under maintenance.
•The website has a programming error.
What you can try:
Refresh the page.
Go back to the previous page.
More information More information
I tried recreating the virtual directories for the OWA and ECP, but it keeps failing after the credentials.
PLEASE HELP ASAP!!
Tried this last suggestion with no results.
Hi Emil,
After no results, tried to install CU14, but it stuck on step 1 of 17: Stopping Services.
After several minutes, I got the following error:
Error:
The following error was generated when "$error.Clear();
& $RoleBinPath\ServiceContro
& $RoleBinPath\ServiceContro
" was run: "Microsoft.Exchange.Config
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Configu
There was no ROLLBACK button or CANCEL, just the EXIT button.
After exiting the CU14 update, clients were not able to connect, so I rebooted the server.
After rebooting, a lot of services in the server (from Exchange and also the IIS, w3svc and others) were DISABLED.
So nothing was working.
I set the services as AUTOMATIC and rebooted the server again.
Now the OWA and ECP are able to ask for credentials, but after typing user and password, I get this error:
The website cannot display the page
HTTP 500
Most likely causes:
•The website is under maintenance.
•The website has a programming error.
What you can try:
Refresh the page.
Go back to the previous page.
More information More information
I tried recreating the virtual directories for the OWA and ECP, but it keeps failing after the credentials.
PLEASE HELP ASAP!!
UPDATE:
Without alternatives, I tried CU15, and at least it did find something:
Error:
This computer requires .NET Framework 4.8 (https://support.microsoft.com/kb/4503548).
For more information, visit: https://docs.microsoft.com/Exchange/plan-and-deploy/system-requirements?view=exchserver-2016
So it seems that CU14 had time to DISABLE some services and UNINSTALL some programs.
Downloaded .NET 4.8 and now it is installing, to then RETRY the CU15 again.
Without alternatives, I tried CU15, and at least it did find something:
Error:
This computer requires .NET Framework 4.8 (https://support.microsoft.com/kb/4503548).
For more information, visit: https://docs.microsoft.com/Exchange/plan-and-deploy/system-requirements?view=exchserver-2016
So it seems that CU14 had time to DISABLE some services and UNINSTALL some programs.
Downloaded .NET 4.8 and now it is installing, to then RETRY the CU15 again.
UPDATE:
CU15 is now on step 3 of 17: Remove Exchange Files at 90%.
At least step 1 (Stopping Services) and step 2 finished fine.
CU15 is now on step 3 of 17: Remove Exchange Files at 90%.
At least step 1 (Stopping Services) and step 2 finished fine.
Hey Roberto,
Sorry for my long delay, I had too much going on the past weeks. I see you still unable to resolve this issue.
Honestly, you shouldn't have any trouble while doing so. If you did then, perhaps cause of renaming the above suggestion? "I can't tell for sure" - I installed the cumulative updates few weeks ago and never had any issue related to it.
Here is a helpful link for you: https://eclat.tech/software/microsoft/exchangeserver/exchange-server-2013/microsoft-exchange-server-2016-cumulative-update-14-nightmare/
Beware sometimes exchange likes to shut down services for some reason and never get it back online. You can manually turn it on.
How's everything doing?
Sorry for my long delay, I had too much going on the past weeks. I see you still unable to resolve this issue.
Honestly, you shouldn't have any trouble while doing so. If you did then, perhaps cause of renaming the above suggestion? "I can't tell for sure" - I installed the cumulative updates few weeks ago and never had any issue related to it.
Here is a helpful link for you: https://eclat.tech/software/microsoft/exchangeserver/exchange-server-2013/microsoft-exchange-server-2016-cumulative-update-14-nightmare/
Beware sometimes exchange likes to shut down services for some reason and never get it back online. You can manually turn it on.
How's everything doing?
Thanks Hemil,
Now is at step 12 of 17:Mailbox Role - Unified Messaging Service
Now errors so far.
I just wish that when everything finishes, the server will be operative.
Now is at step 12 of 17:Mailbox Role - Unified Messaging Service
Now errors so far.
I just wish that when everything finishes, the server will be operative.
Bad news,
At step 13 of 17: Mailbox Role: Mailbox Services got the following error:
Error:
The following error was generated when "$error.Clear();
buildToBuildUpgrade-Exsetd
" was run: "Microsoft.Exchange.Manage
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Configu
No idea what to do next.
At step 13 of 17: Mailbox Role: Mailbox Services got the following error:
Error:
The following error was generated when "$error.Clear();
buildToBuildUpgrade-Exsetd
" was run: "Microsoft.Exchange.Manage
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Managem
at Microsoft.Exchange.Configu
at Microsoft.Exchange.Configu
No idea what to do next.
@Hemil my troubleshooting steps are all accurate and may not cause the upgrade to fail. Renaming the machine keys folder will not impact the upgrade to fail. You can see where and on which role it is failing.
It does not fail on search component or machine keys. It is failing on 13 of 17: Mailbox Role: Mailbox Services
"Microsoft.Exchange.Management.Deployment.ExsetdataException: An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'.
Before upgrade you should also have suggested him to take a snapshot/backup of exchange and system state backup. I know snapshots are not supported by MS, but during upgrade failures they save life.
But nevertheless since your plan was not feasible and suggestions made that the fixes that are applicable to new CU. It does not mention anywhere in the fixes that CI state of unknown is a bug and will be fixed in CU 14 for exchange 2016.
Now that the damaged has been done, you say to revert back as per your update " Remove the update from the control panel and reboot the server." Even this is not possible now.
"Given the error message “An error occurred with error code '3221685616' and message 'The file or directory is corrupted and unreadable.'” It’s recommended to check if any disc related issue (for example: Event 55) on the drive where your Exchange is installed in the Application log.
If exists, it’s recommended to take a backup of the DB data , server back up and then run chkdisk on the drive where your exchange was located."
@Saif Shaikh
Let me clarify something that may be misunderstood. I have never questioned your steps, you are a professional IT like any of us. But it seems to me you're taking this thread a little bit personal.
Please read a little more carefully a thread before you start shooting words that I have never said.
Here is what you stated
Also, keep in mind any IT knows better than anyone "SYSTEMS ARE NOT STABLE, WE AS ADMIN DO OUR BEST TO MAINTAIN IT"
I have never made a comment explicitly stating "an update will fix it". I clearly said "It sounds like a bug to me, perhaps you should check the update, what CU you have? - and on a second thread, I said, Whenever you can please install this CU: https://www.microsoft.com/en-us/download/details.aspx?id=100302 "This is the part you mentioned I did not tell him to back up its system."
I never gave him the effectiveness of that's the REAL issue might be, instead, my thoughts. "Your issue sounds like a bug to ME" I'm giving my opinion.
"The reason why is because if you created a new mailbox database and the database still "Unknown" then, that means there something wrong with the updates." that means don't take my word from granted let's investigate.
Last but not least
Okay, based upon that comment you sound like "I did the damage and therefore everything is messed up because of it."
Look, pal, I will give you this advice if you work within an IT environment do not look for a culprit but instead a solution. With that remark, you are creating a poisonous environment wherever you go. Keep in mind one thing.
1- I never doubted your solution
2- I even told Roberto to follow-up with you ever since you sound certain modifying registry. Myself personally, I do not like modifying the registry unless I have a trusted source from Microsoft. But, since you too certain then why not? should've given it a try.
We are here to share our thoughts and help one another and not to belittle on another.
To get around into this issue, we need to restore an exchange backup, if possible an image and find what's wrong with the system. His issue shouldn't be complicated as I mentioned before, Uknown Indexing is only shown when you use DAG. I have never seen indexing issues without DAG.
Let me clarify something that may be misunderstood. I have never questioned your steps, you are a professional IT like any of us. But it seems to me you're taking this thread a little bit personal.
Please read a little more carefully a thread before you start shooting words that I have never said.
Here is what you stated
Before upgrade, you should also have suggested to him take a snapshot/backup of exchange and system state backup. I know snapshots are not supported by MS, but during upgrade failures they save life.First and foremost, It is true, I give you that. I should've suggested he take a snapshot/backup - BUT with all due respect to "Roberto", I'm expecting any sysadmin to know better to have backups before starting any type of system modifications - we are IT, not users. AND, if that was the case, why didn't you tell him to make a backup of its system before he starts modifying default parameters on the registry? he is trusting you applying your steps as wells as me right? and perhaps he has no clue what he's doing on the registry, what if something goes wrong? It sounds to me like Irony doesn't it? but anyway, I just wanted to point it out.
Also, keep in mind any IT knows better than anyone "SYSTEMS ARE NOT STABLE, WE AS ADMIN DO OUR BEST TO MAINTAIN IT"
but nevertheless, since your plan was not feasible and suggestions made that the fixes that are applicable to new CU/
I have never made a comment explicitly stating "an update will fix it". I clearly said "It sounds like a bug to me, perhaps you should check the update, what CU you have? - and on a second thread, I said, Whenever you can please install this CU: https://www.microsoft.com/en-us/download/details.aspx?id=100302 "This is the part you mentioned I did not tell him to back up its system."
I never gave him the effectiveness of that's the REAL issue might be, instead, my thoughts. "Your issue sounds like a bug to ME" I'm giving my opinion.
"The reason why is because if you created a new mailbox database and the database still "Unknown" then, that means there something wrong with the updates." that means don't take my word from granted let's investigate.
Last but not least
Now that the damage has been done, you say to revert back as per your update " Remove the update from the control panel and reboot the server." Even this is not possible now.
Okay, based upon that comment you sound like "I did the damage and therefore everything is messed up because of it."
Look, pal, I will give you this advice if you work within an IT environment do not look for a culprit but instead a solution. With that remark, you are creating a poisonous environment wherever you go. Keep in mind one thing.
1- I never doubted your solution
2- I even told Roberto to follow-up with you ever since you sound certain modifying registry. Myself personally, I do not like modifying the registry unless I have a trusted source from Microsoft. But, since you too certain then why not? should've given it a try.
We are here to share our thoughts and help one another and not to belittle on another.
To get around into this issue, we need to restore an exchange backup, if possible an image and find what's wrong with the system. His issue shouldn't be complicated as I mentioned before, Uknown Indexing is only shown when you use DAG. I have never seen indexing issues without DAG.
Hi guys,
Sorry I caused a bit of trouble.
I just had a training course with Microsoft that only covered installing the Exchange and configuring accounts, groups, rules and some other basic things. Let's forget about this and just keep going.
Finally I had to get support from our provider, which put us in touch with Microsoft Support and they enabled a remote login session.
The most critical problem (the server was not operative) was resolved by them.
It seems that the disk where Exchange 2016 was installed and the other disk where all the databases are, BOTH were somehow corrupted and this was determined by them. All these disks were on a virtual environment with VMWare (and I don't even have access to that console). They "requested" to create new disks for both and they recovered the installation from the databases to the new disks.
Almost everything is now working fine, but they will enable another remote session to correct the infamous SEARCH FUNCTION not working.
I appreciate all the help you provided.
Sorry I caused a bit of trouble.
I just had a training course with Microsoft that only covered installing the Exchange and configuring accounts, groups, rules and some other basic things. Let's forget about this and just keep going.
Finally I had to get support from our provider, which put us in touch with Microsoft Support and they enabled a remote login session.
The most critical problem (the server was not operative) was resolved by them.
It seems that the disk where Exchange 2016 was installed and the other disk where all the databases are, BOTH were somehow corrupted and this was determined by them. All these disks were on a virtual environment with VMWare (and I don't even have access to that console). They "requested" to create new disks for both and they recovered the installation from the databases to the new disks.
Almost everything is now working fine, but they will enable another remote session to correct the infamous SEARCH FUNCTION not working.
I appreciate all the help you provided.
Explore More ContentExplore courses, solutions, and other research materials related to this topic.
-
![]()
received a Solution
Exchange 2016 and Hyper-V Install -
![]()
wrote an Article
![]()
Steps To Solve Content Index Failed Error
-
![]()
created a Video
![]()
Exchange 2016 Installation - Part 15 - Creating your Databases with the GUI and PowerShell
-
Explore Cloud Class® ![]()
Introduction to Windows Vista for Intermediate Users
Premium Content
You need an Expert Office subscription to comment.Start Free Trial