Link to home
Start Free TrialLog in
Avatar of Steven Richford
Steven Richford

asked on

Microsoft Azure Site Recovery (ASR) - Hyper-V Replication failing due to large data churn

Hello
EDITED 12th Oct 18
  • Summary
  • We have attempted to run Microsoft Azure Site Recovery (ASR) - Hyper-V Replication without success.
  • We are running Windows Server 2016 with 3 VMs
  • The ASR service has been installed by an external service company
  • We have been able to successfully complete the initial backup of the servers to the cloud
  • However the system  cannot keep up to date with the amount of changes on the server
  • All the upload bandwidth is being saturated (80 Mbps) and not keeping up with changes
  • (See attached screen shot of message from ASR)

These are the answers I received from the service company to my questions:
1. Could you tell me what the replication method we are using is called.
Azure Site Recovery - Hyper-V Replication
2. What the problem is. Is it the MySQL databases?
We don't know if it is the MySQL Databases for sure. As explained before the problem is too many changes are being made and the Azure servers cannot keep up with the constant changes.
3. Is the problem correctly described as being too much “data churn”?
I have forwarded the screenshot I had of the exact error message. (see uploaded image)

The service company cannot resolve the issue and say it is up to us to solve the data churn issue
They have asked Microsoft for help but they say they cannot because the issue is likely with our MYSQL databases
- It surprises me that they are not certain about this.

Our in-house developers have made some changes and we stopped creating large temporary tables on the server - but this did not help.

My thoughts - I "think" we need help to understand
EITHER how we find out what is triggering the ASR service to upload data so we can explore if we we can avoid this
OR we find out how to monitor MYSQL to identify why the databases are changing and triggering ASR to see a change
OR both of these
AS WELL - we would like to know if there is any way that MYSQL could be replicated apart from ASR and yet be available for Disaster Recovery to Azure if we need it.

Best regards

Steven
Azure-message.png
Avatar of Aaron Tomosky
Aaron Tomosky
Flag of United States of America image

Backup the server once. Then run dB exports (gzip) and then use azure replication to sync those. In a DR scenario, spin up the server and restore the latest backup.
Avatar of Steven Richford
Steven Richford

ASKER

Hello Aaron - thank you for your message, much appreciated.
My apologies. My problem was not very well described and I have completely re-edited it. I have also added a screen shot of the problem.
I think your suggestion to use gzip relates to Azure Content Delivery Network?  I have now properly described our service as Azure Site Recovery - Hyper-V Replication which I think already has compression or could gzip be relevant?
Best regards
Steven
ASKER CERTIFIED SOLUTION
Avatar of Aaron Tomosky
Aaron Tomosky
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Thanks Aaron
This is great - a fresh idea! I'll work on it with the team on Tuesday and let you know how it goes.
Thanks a lot for your contribution!
Steven
Glad to help. It's important to consider how you would actually fail over and back, write down the steps and test periodically.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Arne - more useful info - thanks very much!
This will help us consider out options when I talk to the service company on Monday/Tuesday.
I am not sure what expectations for RTO and RPO we asked them for. I think it was
  • RTO of 1 hour
  • RPO 15 minutes
Once we get it working we can explore what the limitations are - we could increase the bandwidth if that then became the bottleneck.
Best regards
Steven
Sounds like a plan. Backups every 15 min and you ha e an hour to fail over? I'm sure you can document the steps to make that happen. Don't forget about fail back, that can be a PITA if not planned well.
Good morning

I now have some more information. My colleague confirms that he recently ran a utility that revealed that MYSQL was very busy every 10 or 15 minutes with a lot of disk activity. He assumes that it is doing some housekeeping and presumably ASR is responding to changes arising from this. So we have a high level of confidence that MYSQL is causing ASR a problem.

Some time ago we moved temporary MYSQL data files (created as part of our processing) to a new  drive (T:) and that drive is excluded from the ASR  service.

Our new plan is to run MYSQL server on a VM which is included in ASR with ALL our data elsewhere, on the T: drive . We know we can configure MYSQL to store its temp files on the T: drive as well. Is there any activity that MYSQL carries out that we can't control that creates data churn locally - which could be picked up by ASR as changes.

Will the plan to replicate MYSQL to the cloud outside of ASR work with all the data and temp files on a drive excluded from ASR and yet MYSQL server itself included in hte ASR replication?

I hope my questions make sense.

Best regards

Steven
Short answer: yes.
The thing is, you will need to create the same configuration at the DR location for it to turn up, so whatever steps are necessary to create those drives and restore the data that you backed up and synced outside of ASR will need to be documented and tested.
We are on the way! At the weekend we moved all the MYSQL data files to a drive (T:) which ASR will ignore. Then we will monitor the changes that Azure will still have to contend with using MS site recovery deployment planner which we have set up today.

At the same time my colleague is starting to set up the MYSQL replication to the cloud.

I'll update you in a few days on the outcome.

Many thanks for your help.

Steven
Good afternoon Aaron and Arne
Everything is working out well. The ASR is now fully complete and fine without the MYSQL data (which is stored on a drive that is excluded - and backed up to another local system at this time.)
The next step is that we are going to move the MYSQL data away from our local server to Amazon Web Services. So the VMs will be able to look in the same place for the data - whether running from our office or if we fail-over to Azure.
Thanks for your advice and guidance!
Steven