Solved

To locate and remove Ghost & corrupted log files files in VMFS Datstore

Posted on 2016-08-03
23
211 Views
Last Modified: 2016-08-09
Related to vSphere 5.5[upgraded from 4.1] ..recently we did in place vmfs upgrade & there was firmware upgrade done @san level as well
now from last few days...we are getting  few errors

"ON-DISK ERROR: FB inconsistency found: (8010,0) allocated in bitmap, but never used"

and due to this we are unable to delete few orphaned vmdk from data store, we can see same folder there on multiple lun.

Can we remove such file using any valid method wich can be action with no interruption to running services ?
also would like to confirm for below given quires...

>What is ghost file and corrupted log  files in vmfs data store? and what could be reason behind it ?
>Can we avoid such file creation ?
>Will that ghost or corrupted file cause datastore corruption or any other major impact ?
>is Orphaned vmdk related  to such ghost or corrupted files ?
>How can we remove such files easily with no service interruption or how to fix this issue on permanent basis ?
>what could be reason behind such ghost or corrupted files ? is in place upgrade reason behind it ?
In Place upgrade is not recommended by VMware ?
0
Comment
Question by:patron
  • 13
  • 10
23 Comments
 
LVL 1

Author Comment

by:patron
ID: 41740455
we are unable to remove such log file or vmdk ,because file allocation table or root dir seems to be corrupted ? and VMFS unable to locate all blocks and entries for the files to be deleted.
What could be reason behind it ?
0
 
LVL 119
ID: 41740473
You can use RV Tools, to check for orphaned files and delete them, if not in use.

http://www.robware.net/
0
 
LVL 1

Author Comment

by:patron
ID: 41740480
I am able to locate orphaned files ,but when trying to delete no luck ?

is it related to ghsot file and what could be done to get this fixed ?
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 
LVL 119

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 41740569
if you cannot delete them check they are not in use.

what ghost files do you have ?
0
 
LVL 1

Author Comment

by:patron
ID: 41741776
yes, they r not in use,but yet not able to delete n files are like log files n vmdk etc not in use??
0
 
LVL 119
ID: 41741904
log files from what ?

screenshots please
0
 
LVL 1

Author Comment

by:patron
ID: 41741950
normally we should have 10 folder on data store  as there r 10 vms created on datastore ,but here we found 12 folders n under those 2 extra folder we found few vmdk, vmk or logs which r created for any virtual machine , so when we try to delete them it seem files r locked ..unable to delete ??
0
 
LVL 119
ID: 41742261
you need to check if they are in use, does RV Tools suggest they are orphaned.
0
 
LVL 1

Author Comment

by:patron
ID: 41743687
Strange that file/vmdk identified by rvtool n powergui as orphaned are like...

>few of them are of cource not in use n dirty/duplicate file to be removed
>but few of them are attached to running vms as second hard drives ..but why those are consider as orphaned ?

can in place or scratch upgrade cause a datastore to be corrupted ?
0
 
LVL 119

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 41743750
well if they are connected and in use by a VM, they are not orphaned, and in use.

can in place or scratch upgrade cause a datastore to be corrupted ?

anything is possible, but we've not seen it.
0
 
LVL 1

Author Comment

by:patron
ID: 41743801
okay, further i need to use voma to look for more detail ..

Can we use voma on running infra..i mean will there be any impact or prerequisites if required ?

i need to check vmfs consistency if my datsore is okay   using voma tool.

please confirm if we have other  better way to check if lun is okay with no issues n impcat ?
0
 
LVL 119
ID: 41744171
Yes, you can use VOMA to check for VMFS corruption.
0
 
LVL 1

Author Comment

by:patron
ID: 41746129
got your point ,Thanks

Can we use voma for running infra if vms are running on datasore with prod infra ?

or if any better tool to locate n fix the problem @datasore level in case there is any corruption or crash due to orphaned ghost files ??

i mean m looking for healthcheck for my data store, further orphaned i can remove accordingly

can we get orphaned due to vaai feature enabled in vmfs 5 ..causes svmotion delayed due to drivers or etc not compatible ?
0
 
LVL 119

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 41746138
if you have a concern about the corruption of a datastore - what do you think ? would be wise about the VMs currently hosted on A CORRUPT DATASTORE ?
0
 
LVL 1

Author Comment

by:patron
ID: 41746141
yes ,its big question here, so need your expert advise on this, as yet there are 30+ vms running on same dataore ,but we have have orphaned disk/files for 3-4 vms on same datastore [which was recently prepared from scratch with vmfs5]

Now if we look for below given logs...
FS3_ADDR_INVALID  

Below errors were seen after running VMFS check using VOMA tool by someone else.
Phase 5: Checking resource reference counts.
ON-DISK ERROR: FB inconsistency found: (8010,0) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (8021,27) allocated in bitmap, but never used


and logs like
Mapping for file abcd.vmdk (34376533 bytes in size):
[           0:    12582912] --> [VMFS -- LVID:56bacca5-61a9bfe9-c083-5cf3fc5b9b14/56bacca5-51e8abc6-4675-5cf3fc5b9b14/1:( 1184909754368 -->  1184922337280)]
[    12582912:     8388608] --> [VMFS Z- LVID:56bacca5-61a9bfe9-c083-5cf3fc5b9b14/56bacca5-51e8abc6-4675-5cf3fc5b9b14/1:( 1184922337280 -->  1184930725888)]
[    20971520:    15728640] --> [VMFS -- LVID:56bacca5-61a9bfe9-c083-5cf3fc5b9b14/56bacca5-51e8abc6-4675-5cf3fc5b9b14/1:( 1184930725888 -->  1184946454528)]


so above logs indicate its all due to corruption @datasore level ?
If Yes -How can we resolve/fix it with minimal efforts and avoid such issue in future ?
If No -How to locate exact issue behind it....
>any tool we can use in running infra  ?
> due to VAAI feature enabled on newly formated lun
>due to in place upgrade
>due to SAN firmware upgrade
>due to patching  of vc/esxi and vms pending since long time including vmtool alos
0
 
LVL 119
ID: 41746142
if you cannot answer that question yourselves.

MOVE THE VMs OFF IMMEDIATELY.
0
 
LVL 1

Author Comment

by:patron
ID: 41746143
we did vmfs upgrade almost  6 month before and it was  90% we did in place upgrade but 10% we did from scratch adding new luns

earlier we had few orphaned disk..but now days [in june-july]getting more and more

how can we control on creation of such orphaned n ghost files
0
 
LVL 1

Author Comment

by:patron
ID: 41746144
i believe its not lun/datasore corruption , can we use any better tool to look n fix more on this within running infra
0
 
LVL 119

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 41746149
NONE.

orphaned disks are not as a result of a corrupt datastore.

what makes you think they are orphaned ?  are they really not in use.

if you have doubts about a datastore , you should move all the VM off it, and see what remains.

and then delete/remove/re-create.
0
 
LVL 1

Author Comment

by:patron
ID: 41746151
Thanks Andrew

yes will check for that, i am not directly involved team in this issue,else i could have done more,but will try for sure

Many Thanks for all your support on this.

let me try to look more into this
0
 
LVL 1

Author Comment

by:patron
ID: 41748253
can we remove this question ?
0
 
LVL 119
ID: 41748528
Why, select an Answer and Award Points.
0
 
LVL 1

Author Closing Comment

by:patron
ID: 41749807
Thanks
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

If your vDisk VHD file gets deleted from the image store accidentally or on purpose, you won't be able to remove the vDisk from the PVS console. There is a known workaround that is solid.
Concerto Cloud Services, a provider of fully managed private, public and hybrid cloud solutions, announced today it was named to the 20 Coolest Cloud Infrastructure Vendors Of The 2017 Cloud  (http://www.concertocloud.com/about/in-the-news/2017/02/0…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…

790 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question