Exchange 2010 DB failure

We are on Exchagne 2010 SP2 with two servers. We have 24 DB's in the DAG with 12 active on each server and replicas on the other.  Last evening we had one of the DBs fail during the backup. attached is a event message, there are many.  Now all of the DBs have migrated to be active on one of the servers instead of being split. Only one shows as failed.

Not sure how to proceed.
DB-Failed.docx
WildgenAsked:
Who is Participating?
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
but two of them show "Resynchronizing" on the 01 server and the other shows as Healthy - which 2 DB's are in that state ?

How long will these show at Resynching?  They are very small databased with only a very few mailboxes - Depends what the issue is ?

Try Resume first if it still fails only Option is Reseed ? - Resume means that it will sync changes since last failover or stuff.

Update means that it will reseed the entire DB and Logs from Active server to Passive :)

Let me know if you have any doubts still

- Rancy
0
 
TheGeezer2010Commented:
Can you not just migrate the 11 back over - they all show Healthy and have queues which look fine ? If you are asking why they migrated over then would need the output of a command get-databaseavailabilitygroup <DAGname> -fl

If you check the event logs for the time of failure for a Database which failed over, is there anything in there which might explain it ?
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
Buddy till the time Copy status is in Failed state the backups will never be able to complete the backups of the server.

You would need to Suspend and Update (reseed) the database to the other node and wait for it to come into healthy state and then work with Moving 12DB's per server and backups would also work fine.

- Rancy
0
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

 
TheGeezer2010Commented:
Further to Rancy's comment, you may also need to update the Index as well

http://busbar.blogspot.ie/2010/05/exchange-2010-mailbox-database-in-dag.html
0
 
WildgenAuthor Commented:
I have attached more info.  DAG info and a list of the Evnets.  This all happened during the nightly backup.  Are you saying that this was just some sort of anomoly on the db15 and thus all failed over to the 01 server.  Now I can just move them back with a reseed. All the copies on the 02 report as healthy but the one.  I will read up on the process but still not clear. Is tehre a good document explaining this process.
Thanks for your assist while I muddle through.
DB-Fail-2.docx
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
Did the other server Reboot ?
Or did the Cluster service fail on that node ?
Now I can just move them back with a reseed. All the copies on the 02 report as healthy but the one - I guess DB15 :( yes i would say Reseed and then failover this DB while you can failover the rest DB's ..... the reason i say Update cause if even you failover it might be in failed state as this ideally happens due to Drive space empty issues on DB or Logs drive) as currently the Database has more information of new data on the New Node compared to the Database on the Old node at the moment).

What kind of documentation are you looking for Failover Reseed or why it happened ?

- Rancy
0
 
WildgenAuthor Commented:
No servers rebooted.  On the 02 server that reported the once active db's now reporting as Healthy, I have used the EMC to "Activate Database Copy"  They now show as Mounted on the 02 server. but two of them show "Resynchronizing" on the 01 server and the other shows as Healthy.
How long will these show at Resynching?  They are very small databased with only a very few mailboxes.

On the Failed DB on the 02 server I have the options to Resume or Update.  Can I use one of those options?  WIll update reseed?
0
 
WildgenAuthor Commented:
The  Resync has cleared up and now shows as Mounted on the 02 and Healthy on the 01.  That is what I wanted.   Most of these DS's are in the 25gb range in size.
I will try the Resume on the Failed db.  

thanks Rancy.
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
Your most welcome !!

Let me know if any issues :)
0
 
WildgenAuthor Commented:
The Resume  on the failed db was successful. It now reports healthy.
I am proceeding to Activate the Healthy databases on the 02 servers where they should be.
They all report as healthy now.
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
Good :)

Relaxed ... lets have some Fun !!

- Rancy
0
 
WildgenAuthor Commented:
Yes. more relaxed now.  Will be even more when all have been Activated on the correct server.
Is there a command to check the DB index?
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus

- Rancy
0
 
TheGeezer2010Commented:
Get-MailboxDatabaseCopyStatus | Format-Table Identity,ActiveDatabaseCopy,ContentIndexState -Auto
0
 
WildgenAuthor Commented:
Appreciate the good input and timelyness.
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
Shouldnt i have been chosen as well for helping ..... just trying to understand if i lacked somewhere ?

- Rancy
0
 
WildgenAuthor Commented:
OOps I thought I did select you.  Sorry can I change that.
0
 
Manpreet SIngh KhatraSolutions Architect, Project LeadCommented:
You need to hit "Request Attention" at the top and proceed.

- Rancy
0
 
TheGeezer2010Commented:
You are most welcome and glad we could be of assistance
0
 
WildgenAuthor Commented:
very helpful discussion of the issue.  provided solutions worked .
Thanks
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.