Ryan Snapp
asked on
Server 2019 latest patches and updates, BSOD caused by dedup.sys during large file copy
I have a Windows Server 2019 ESXi 7 VM that has been getting BSOD's with the Dedup.sys. After reviewing the logs, I decided to do an unoptimize of the volume in question and hopefully resolve the issue. It blues screens when trying that as well.
At this point, I decided to mount a backup of the virtual disk on my Veeam Server (a separate Physical Windows Server 2019 Box) and started a copy from it as well, and sure enough, it blue screens too with Dedup.sys being flagged as the reason again.
I've run mem test and all that on both these servers. They otherwise perform fine and the other VM's on my ESXi servers are fine. Chdsk and all that has been run as well.
Any suggestions?
At this point, I decided to mount a backup of the virtual disk on my Veeam Server (a separate Physical Windows Server 2019 Box) and started a copy from it as well, and sure enough, it blue screens too with Dedup.sys being flagged as the reason again.
I've run mem test and all that on both these servers. They otherwise perform fine and the other VM's on my ESXi servers are fine. Chdsk and all that has been run as well.
Any suggestions?
ASKER
Here is the Minidump. I'm trying to find the stop code, I may have to make one of these bluescreens again to get it.
I do have a bugcheck in the event logs:
The computer has rebooted from a bugcheck. The bugcheck was: 0x0000003b (0x00000000c0000094, 0xfffff80118462f29, 0xffff9e861d4446d0, 0x0000000000000000). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 93084700-0bde-4192-a7dd-35ac2b18b555.
070721-11640-01.dmp
I do have a bugcheck in the event logs:
The computer has rebooted from a bugcheck. The bugcheck was: 0x0000003b (0x00000000c0000094, 0xfffff80118462f29, 0xffff9e861d4446d0, 0x0000000000000000). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 93084700-0bde-4192-a7dd-35ac2b18b555.
070721-11640-01.dmp
the first parameter is the exception code - in this case 0xC0000094 which is a divide by zero
is the system is fully patched?
have you tried sfc /scannow and dism /online /cleanup-image /restorehealth ?
is the system is fully patched?
have you tried sfc /scannow and dism /online /cleanup-image /restorehealth ?
ASKER
I had not because I just rebuilt the server running Veeam last week and everything is fully patched due to a failed RAID card, so when the physical server running Veeam got the BSOD as well when I mounted the restored VHD in Veeam, I was really shocked.
And just to be clear, the Server 2019 ESXi VM that I originally had this issue with is fully patched. On the volumes that do not have Deduping enable, I have no issues with large file copying. So this issue has been replicated on two completely different servers using the copies of the virtual disk.
I just finished running sfc (Windows Resource Protection did not find any integrity violations.)
and the dism image restore on the physical server running Veeam and I'll see if it gets another BSOD with a large file copy.
And just to be clear, the Server 2019 ESXi VM that I originally had this issue with is fully patched. On the volumes that do not have Deduping enable, I have no issues with large file copying. So this issue has been replicated on two completely different servers using the copies of the virtual disk.
I just finished running sfc (Windows Resource Protection did not find any integrity violations.)
and the dism image restore on the physical server running Veeam and I'll see if it gets another BSOD with a large file copy.
ASKER
The physical server just blue screened copying a 1.5GB MP4 file. I also cause the issue by copying ISO files and other large files.
trying to find a commonality here...
so it crashed on the vm and also the physical veeam server?
what is the storage where these volumes reside?
so it crashed on the vm and also the physical veeam server?
what is the storage where these volumes reside?
ASKER
The VM server is on our DellEMC SAS SSD SAN and the host servers are two ESXi 7 U2.
The physical server is an older Dell R710 with spinning disks, a completely different setup.
The only common denominator is that this volume I'm trying to fix is the same, it's just restored from one server to the other. I was so frustrated when I found restoring the virtual disk and mounting on my physical Veeam server did the same exact bluescreen. I've tried chdsk with all the commands I could think of too.
The physical server is an older Dell R710 with spinning disks, a completely different setup.
The only common denominator is that this volume I'm trying to fix is the same, it's just restored from one server to the other. I was so frustrated when I found restoring the virtual disk and mounting on my physical Veeam server did the same exact bluescreen. I've tried chdsk with all the commands I could think of too.
is there deduplication on the san?
if so, wondering if having it there and on the windows server is causing an issue
if so, wondering if having it there and on the windows server is causing an issue
ASKER
No, there isn't. I even double-checked to be sure.
do you have storage from another location that can be attached to this vm as a deduplicating volume and do a file copy test? will help to narrow down if this is a storage issue or with the OS
Just run the dism /online /cleanup-image /checkhealth and sfc /scannow commands anyway, as suggested by Seth.
The Veeam backups may have copied over an image with corrupt system files. Veeam doesn't care if it's corrupt or not, it just copies the disk images as is.
Even if your system is "fully patched," the patches may not have updated the corrupted files. The /checkhealth option is quick. Run it with /restorehealth if it does detect something.
The Veeam backups may have copied over an image with corrupt system files. Veeam doesn't care if it's corrupt or not, it just copies the disk images as is.
Even if your system is "fully patched," the patches may not have updated the corrupted files. The /checkhealth option is quick. Run it with /restorehealth if it does detect something.
ASKER
Last night I did the following with no resolve:
On the original production 2019 VM:
1) SFC /scannow
"Windows Resource Protection did not find any integrity violations." from SFC
2) Ran dism /online /cleanup-image /checkhealth
3) Ran SCF /scannow again.
4) Rebooted.
5) Tried to copy a yet different large file and it bluescreened as soon as the copy started.
6) After it came back up, I tried to unoptimize the volume causing issues:
"Start-DedupJob -Type Unoptimization g:"
It blue screened. I attached the mini-dump. 070821-17031-01.dmp
Bug check in event logs:
"The computer has rebooted from a bugcheck. The bugcheck was: 0x0000003b (0x00000000c0000094, 0xfffff80893d62f29, 0xffff838604d65ac0, 0x0000000000000000). A dump was saved in: C:\WINDOWS\MEMORY.DMP. Report Id: 3489669a-f191-423a-9375-f2d3d627fde5."
I ran another backup on the VM. Did a Server 2019 reinstall over the top just for kicks. Same exact issues.
I restored the backup. I do believe I have ruled out system file/wim corruption.
This one volume seems to cause any Server 2019 VM or physical to crash when I mount it to them. Dedup.sys is always the culprit. I need to be able to dedupe this volume and get all the data copied to a new clean volume that has never used Windows Deduplication and I won't be using it in the future.
Does anyone have experience fixing issues on volumes that have been deduped? I can't find a whole lot googling it.
I should add just to be clear when I mount this volume (restored) to my physical Dell R710 with spinning disks (completely different physical server and storage), it blue screens.. It is a brand new OS install from this last week too.
Thanks,
Ryan
On the original production 2019 VM:
1) SFC /scannow
"Windows Resource Protection did not find any integrity violations." from SFC
2) Ran dism /online /cleanup-image /checkhealth
3) Ran SCF /scannow again.
4) Rebooted.
5) Tried to copy a yet different large file and it bluescreened as soon as the copy started.
6) After it came back up, I tried to unoptimize the volume causing issues:
"Start-DedupJob -Type Unoptimization g:"
It blue screened. I attached the mini-dump. 070821-17031-01.dmp
Bug check in event logs:
"The computer has rebooted from a bugcheck. The bugcheck was: 0x0000003b (0x00000000c0000094, 0xfffff80893d62f29, 0xffff838604d65ac0, 0x0000000000000000). A dump was saved in: C:\WINDOWS\MEMORY.DMP. Report Id: 3489669a-f191-423a-9375-f2d3d627fde5."
I ran another backup on the VM. Did a Server 2019 reinstall over the top just for kicks. Same exact issues.
I restored the backup. I do believe I have ruled out system file/wim corruption.
This one volume seems to cause any Server 2019 VM or physical to crash when I mount it to them. Dedup.sys is always the culprit. I need to be able to dedupe this volume and get all the data copied to a new clean volume that has never used Windows Deduplication and I won't be using it in the future.
Does anyone have experience fixing issues on volumes that have been deduped? I can't find a whole lot googling it.
I should add just to be clear when I mount this volume (restored) to my physical Dell R710 with spinning disks (completely different physical server and storage), it blue screens.. It is a brand new OS install from this last week too.
Thanks,
Ryan
If you're restoring from a VM image to a physical device, you need to have the correct drivers. The wrong drivers will cause it to blue screen.
ASKER
I must not be explaining this correctly. I apologize for that.
I'm mounting a restored copy of the volume in question as a mounted virtual disk on the physical server. Not the entire VM. I have Deduplication enabled on this server for this purpose so it can read the volume.
Please note, I have several other virtual disks with volumes on the 2019 VM that is having the issues, but Deduplication has never been enabled on those volumes.. They all work fine, I can't get them to cause the issue. The OS virtual disk has never had deduplication running on it either. The blue screens follow the one volume anywhere I mount it, whether it be a physical or another VM.
I'm mounting a restored copy of the volume in question as a mounted virtual disk on the physical server. Not the entire VM. I have Deduplication enabled on this server for this purpose so it can read the volume.
Please note, I have several other virtual disks with volumes on the 2019 VM that is having the issues, but Deduplication has never been enabled on those volumes.. They all work fine, I can't get them to cause the issue. The OS virtual disk has never had deduplication running on it either. The blue screens follow the one volume anywhere I mount it, whether it be a physical or another VM.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
can you post a mini dump?