NTFS Event ID: 55 - File Corruption on Hyper-V
Hi all,
We had ransomware infect the network from client computer, which spread to all network shares it had access to. The infection encrypted media, documents and database files on all 3 VM servers we have in our Hyper-V setup. We than paid for Decryptor to decrypt those files but not all files decrypted so we started a data recovery from backup.
We installed a new Synology NAS drive and attached this via iSCSI to both our Hyper-V hosts. A new VHDX was created from Hyper-V on the Synology which would house the restored data files. Straight away we started seeing NTFS corruptions shown in Event Viewer on the SBS 2011 (SP1) with Event ID: 55. We than removed the iSCSI connection to the second Hyper-V host as we read that this could be the cause and run chkdsk on the SBS 2011 which stopped the NTFS corruptions on the iSCSI drive.
We then restored files from before the corruption started back onto the VHDX created on the Synology NAS and overwrote any new files the client worked on since.
However, the NTFS corruptions came back and after some research, I found that VSS could be the cause so I've disabled the shadow copies (which deleted all previous versions) and I could see that the frequency of NTFS corruptions decreased by a noticeable amount (Event viewer logs was showing ID 55), however, they continued until I ran a new chkdsk with shadow copies disabled first and then re-ran chkdsk after re-enabling shadow copies. There don't seem to be any NTFS corruptions on the SBS 2011 at the moment.
On the Hyper-V host, there are Events for drive corruption with Event ID: 98 at around 8 o'clock everyday. And although VM level backups start at 7 o'clock, I believe the problem to be snapshots causing NTFS corruptions on the iSCSI drive. Most likely related to the same NTFS corruptions on the SBS 2011 VM.
We are looking for help/advice to see why backups (snapshots/shadowcopies) are causing corruption on the newly created drive when backups are running. Assuming this is the cause of the NTFS corruption. Backups have been disabled at present to check if it repeats today and there haven't been any signs of corruption so far but I am afraid that once I turn the backup on, same thing will happen.
We also have LACP configured for the Synology but we've gone through the logs and we see no errors. However, we only had basic logs enabled but iSCSI looks healthy from Synology.
Has anyone experienced this? And if so, how do I tackle this to find the cause of this corruption? We think it could be a conflict with the backups, maybe when the snapshots are being created or when deep scan is running, or the shadow copies and if it is shadow copies, what else can be done? We fear this NTFS corruption could come back at any time if not already done so. As such, we are reluctant to make any changes whilst everything is working but also means we aren't backing up which is a major concern.
Any help and advice is appreciated.
Thank You
We had ransomware infect the network from client computer, which spread to all network shares it had access to. The infection encrypted media, documents and database files on all 3 VM servers we have in our Hyper-V setup. We than paid for Decryptor to decrypt those files but not all files decrypted so we started a data recovery from backup.
We installed a new Synology NAS drive and attached this via iSCSI to both our Hyper-V hosts. A new VHDX was created from Hyper-V on the Synology which would house the restored data files. Straight away we started seeing NTFS corruptions shown in Event Viewer on the SBS 2011 (SP1) with Event ID: 55. We than removed the iSCSI connection to the second Hyper-V host as we read that this could be the cause and run chkdsk on the SBS 2011 which stopped the NTFS corruptions on the iSCSI drive.
We then restored files from before the corruption started back onto the VHDX created on the Synology NAS and overwrote any new files the client worked on since.
However, the NTFS corruptions came back and after some research, I found that VSS could be the cause so I've disabled the shadow copies (which deleted all previous versions) and I could see that the frequency of NTFS corruptions decreased by a noticeable amount (Event viewer logs was showing ID 55), however, they continued until I ran a new chkdsk with shadow copies disabled first and then re-ran chkdsk after re-enabling shadow copies. There don't seem to be any NTFS corruptions on the SBS 2011 at the moment.
On the Hyper-V host, there are Events for drive corruption with Event ID: 98 at around 8 o'clock everyday. And although VM level backups start at 7 o'clock, I believe the problem to be snapshots causing NTFS corruptions on the iSCSI drive. Most likely related to the same NTFS corruptions on the SBS 2011 VM.
We are looking for help/advice to see why backups (snapshots/shadowcopies) are causing corruption on the newly created drive when backups are running. Assuming this is the cause of the NTFS corruption. Backups have been disabled at present to check if it repeats today and there haven't been any signs of corruption so far but I am afraid that once I turn the backup on, same thing will happen.
We also have LACP configured for the Synology but we've gone through the logs and we see no errors. However, we only had basic logs enabled but iSCSI looks healthy from Synology.
Has anyone experienced this? And if so, how do I tackle this to find the cause of this corruption? We think it could be a conflict with the backups, maybe when the snapshots are being created or when deep scan is running, or the shadow copies and if it is shadow copies, what else can be done? We fear this NTFS corruption could come back at any time if not already done so. As such, we are reluctant to make any changes whilst everything is working but also means we aren't backing up which is a major concern.
Any help and advice is appreciated.
Thank You
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks for the help.
We're going with the advice from kevinhsieh... We're moving all data off the NAS and onto the servers local drives. This will then allow us to totally remove the NAS iSCSI connections, delete/format them totally from the NAS and re-create the LUN's onto the single server.
Without going into the detail, we don't have the Hyper-V setup in a cluster and as per kevinhsieh's suggestion this is most likely the cause of the NTFS corruptions.
Because of the size of the data, this is taking some time to do and also has to be done out of hours. I will update you guys once we have got it all up and running to let you know how we got on.
Thanks.
We're going with the advice from kevinhsieh... We're moving all data off the NAS and onto the servers local drives. This will then allow us to totally remove the NAS iSCSI connections, delete/format them totally from the NAS and re-create the LUN's onto the single server.
Without going into the detail, we don't have the Hyper-V setup in a cluster and as per kevinhsieh's suggestion this is most likely the cause of the NTFS corruptions.
Because of the size of the data, this is taking some time to do and also has to be done out of hours. I will update you guys once we have got it all up and running to let you know how we got on.
Thanks.
ASKER
The suggestion by kevinhsieh seems to have solved our issue. We took the store of both VM's and this no longer causes corruption.
The issue has been resolved.
The issue has been resolved.
Are the two hosts fairly identical? If yes, you can install the Failover Cluster Role then stand up the cluster.
Once done, add the LUN that is attached to both hosts to Cluster Storage, then to Cluster Shared Volumes.
From there, you can attach the VHDX file that would now be resident in C:\ClusterStorage\VolumeX where X is your LUN.
NOTE: A 1.5GB LUN needs to be set up and added to both hosts, then added to Cluster Storage, then the Cluster Quorum Wizard run to choose that LUN for a Witness Disk. Do _<I>NOT</I>_ add this LUN to Cluster Shared Volumes.