Solved

solaris file system mount error

Posted on 2012-03-11
15
2,123 Views
Last Modified: 2012-04-02
we have a production database server and now we are receiving an error while mounting one of the core file system where main data is stored.

the details are as


bash-2.03#  mount -F vxfs /dev/vx/dsk/proddata/oradata /oradata

UX:vxfs mount: ERROR: file system on /dev/vx/dsk/proddata/oradata has structural

damage

UX:vxfs mount: ERROR: /dev/vx/dsk/proddata/oradata is corrupted. needs checking

bash-2.03#



the vxprint output is as

bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        1035615958.1025.db-srv

dm disk06       c3t1d0s2     sliced   2888     71124291 -
dm rootdisk     c3t0d0s2     sliced   2888     71124291 -

v  expvol       -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-04    expvol       ENABLED  ACTIVE   31457280 CONCAT    -        RW
sv expvol-S01   expvol-04    expvol-L01 1      31457280 0         3/3      ENA

v  expvol-L01   -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-P01   expvol-L01   ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd rootdisk-08  expvol-P01   rootdisk 39114170 264      LOG       c3t0d0   ENA
pl expvol-P02   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd rootdisk-09  expvol-P02   rootdisk 39117059 31457280 0         c3t0d0   ENA
pl expvol-P03   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd disk06-07    expvol-P03   disk06   39114171 31457280 0         c3t1d0   ENA

v  opt          -            ENABLED  ACTIVE   10241505 fsgen     -        ROUND
pl opt-01       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd rootdisk-03  opt-01       rootdisk 28872665 10241505 0         c3t0d0   ENA
pl opt-02       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd disk06-01    opt-02       disk06   0        10241505 0         c3t1d0   ENA

v  rootvol      -            ENABLED  ACTIVE   6144903  root      -        ROUND
pl rootvol-01   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-02  rootvol-01   rootdisk 8389655  6144903  0         c3t0d0   ENA
pl rootvol-02   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-02    rootvol-02   disk06   10241505 6144903  0         c3t1d0   ENA

v  swapvol      -            ENABLED  ACTIVE   8389656  swap      -        ROUND
pl swapvol-01   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd rootdisk-B0  swapvol-01   rootdisk 71124290 1        0         c3t0d0   ENA
sd rootdisk-01  swapvol-01   rootdisk 0        8389655  1         c3t0d0   ENA
pl swapvol-02   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd disk06-03    swapvol-02   disk06   16386408 8389656  0         c3t1d0   ENA

v  usr          -            ENABLED  ACTIVE   6144903  fsgen     -        ROUND
pl usr-01       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-04  usr-01       rootdisk 22727762 6144903  0         c3t0d0   ENA
pl usr-02       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-04    usr-02       disk06   24776064 6144903  0         c3t1d0   ENA

v  var          -            ENABLED  ACTIVE   8193204  fsgen     -        ROUND
pl var-01       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd rootdisk-05  var-01       rootdisk 14534558 8193204  0         c3t0d0   ENA
pl var-02       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd disk06-05    var-02       disk06   30920967 8193204  0         c3t1d0   ENA

Disk group: proddata

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg proddata     default      default  58000    1035206141.1647.db-srv

dm proddat01    c2t1d0s2     sliced   2888     71124291 -
dm proddat02    c1t9d0s2     sliced   2888     71124291 -
dm proddat03    c1t10d0s2    sliced   2888     71124291 -
dm proddat04    c2t0d0s2     sliced   2888     71124291 -
dm proddat05    c1t8d0s2     sliced   2888     71124291 -

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
bash-2.03# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/rootvol  3008649  389335 2559142    14%    /
/dev/vx/dsk/usr      3008649 1047343 1901134    36%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
/dev/vx/dsk/var      4032504 1275981 2716198    32%    /var
swap                 7549304      16 7549288     1%    /var/run
swap                 7549344      56 7549288     1%    /tmp
/dev/vx/dsk/opt      5040814  538850 4451556    11%    /opt
/dev/vx/dsk/rootdg/expvol
                     14745524 11488160 1782812    87%    /exp
bash-2.03# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c1t8d0s2     sliced    proddat05    proddata     online
c1t9d0s2     sliced    proddat02    proddata     online
c1t10d0s2    sliced    proddat03    proddata     online
c2t0d0s2     sliced    proddat04    proddata     online
c2t1d0s2     sliced    proddat01    proddata     online
c3t0d0s2     sliced    rootdisk     rootdg       online
c3t1d0s2     sliced    disk06       rootdg       online
bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        1035615958.1025.db-srv

dm disk06       c3t1d0s2     sliced   2888     71124291 -
dm rootdisk     c3t0d0s2     sliced   2888     71124291 -

v  expvol       -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-04    expvol       ENABLED  ACTIVE   31457280 CONCAT    -        RW
sv expvol-S01   expvol-04    expvol-L01 1      31457280 0         3/3      ENA

v  expvol-L01   -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-P01   expvol-L01   ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd rootdisk-08  expvol-P01   rootdisk 39114170 264      LOG       c3t0d0   ENA
pl expvol-P02   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd rootdisk-09  expvol-P02   rootdisk 39117059 31457280 0         c3t0d0   ENA
pl expvol-P03   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd disk06-07    expvol-P03   disk06   39114171 31457280 0         c3t1d0   ENA

v  opt          -            ENABLED  ACTIVE   10241505 fsgen     -        ROUND
pl opt-01       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd rootdisk-03  opt-01       rootdisk 28872665 10241505 0         c3t0d0   ENA
pl opt-02       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd disk06-01    opt-02       disk06   0        10241505 0         c3t1d0   ENA

v  rootvol      -            ENABLED  ACTIVE   6144903  root      -        ROUND
pl rootvol-01   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-02  rootvol-01   rootdisk 8389655  6144903  0         c3t0d0   ENA
pl rootvol-02   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-02    rootvol-02   disk06   10241505 6144903  0         c3t1d0   ENA

v  swapvol      -            ENABLED  ACTIVE   8389656  swap      -        ROUND
pl swapvol-01   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd rootdisk-B0  swapvol-01   rootdisk 71124290 1        0         c3t0d0   ENA
sd rootdisk-01  swapvol-01   rootdisk 0        8389655  1         c3t0d0   ENA
pl swapvol-02   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd disk06-03    swapvol-02   disk06   16386408 8389656  0         c3t1d0   ENA

v  usr          -            ENABLED  ACTIVE   6144903  fsgen     -        ROUND
pl usr-01       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-04  usr-01       rootdisk 22727762 6144903  0         c3t0d0   ENA
pl usr-02       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-04    usr-02       disk06   24776064 6144903  0         c3t1d0   ENA

v  var          -            ENABLED  ACTIVE   8193204  fsgen     -        ROUND
pl var-01       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd rootdisk-05  var-01       rootdisk 14534558 8193204  0         c3t0d0   ENA
pl var-02       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd disk06-05    var-02       disk06   30920967 8193204  0         c3t1d0   ENA

Disk group: proddata

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg proddata     default      default  58000    1035206141.1647.db-srv

dm proddat01    c2t1d0s2     sliced   2888     71124291 -
dm proddat02    c1t9d0s2     sliced   2888     71124291 -
dm proddat03    c1t10d0s2    sliced   2888     71124291 -
dm proddat04    c2t0d0s2     sliced   2888     71124291 -
dm proddat05    c1t8d0s2     sliced   2888     71124291 -

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
bash-2.03#


please help me
0
Comment
Question by:kurajesh
  • 9
  • 4
  • 2
15 Comments
 
LVL 1

Author Comment

by:kurajesh
ID: 37708570
the fstab output is

bash-2.03# cat vfstab
#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr          ufs     1       yes     -
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/vx/dsk/swapvol     -       -       swap    -       no      -
/dev/vx/dsk/rootvol     /dev/vx/rdsk/rootvol    /       ufs     1       no
-
/dev/vx/dsk/usr /dev/vx/rdsk/usr        /usr    ufs     1       no      -
/dev/vx/dsk/var /dev/vx/rdsk/var        /var    ufs     1       no      -
/dev/vx/dsk/opt /dev/vx/rdsk/opt        /opt    ufs     2       yes     -
swap    -       /tmp    tmpfs   -       yes     -
#NOTE: volume rootvol (/) encapsulated partition c3t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c3t0d0s1
#NOTE: volume var (/var) encapsulated partition c3t0d0s5
#NOTE: volume usr (/usr) encapsulated partition c3t0d0s6
#NOTE: volume opt (/opt) encapsulated partition c3t0d0s7
#/dev/vx/dsk/proddata/oradata   /dev/vx/rdsk/proddata/oradata   /oradata
vxfs    3       yes     -
#/dev/vx/dsk/proddata/visvol    /dev/vx/rdsk/proddata/visvol    /vis    vxfs
3       yes     -
/dev/vx/dsk/rootdg/expvol       /dev/vx/rdsk/rootdg/expvol      /exp    ufs
3       yes     -
0
 
LVL 76

Expert Comment

by:arnold
ID: 37708639
What was done to this file system? Was there an attempt to resize it? Did someone run fsck on the file system without specifying the filesystem -F vxfs?
fsck -F vxfs /dev/vx/dsk/prodata/oradata
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37708734
hi arnold , can i try fsck -F vxfs /dev/vx/dsk/prodata/oradata
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37708738
never tried to resize the filesystem , but not able to mount it when tried to do so
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37709546
when i tried

 fsck -F vxfs /dev/vx/dsk/prodata/oradata
 the output came as



bash-2.03# fsck -F/dev/vx/dsk/proddata/oradata /oradata

fsck: Fstype /dev/vx/dsk/proddata/oradata exceeds 8 characters

bash-2.03# fsck -F vxfs /dev/vx/dsk/proddata/oradata

no valid ILISTs for fileset 999

no valid ILISTs for fileset 999

UX:vxfs fsck: ERROR: cannot initialize aggregate

file system check failure, aborting ...
0
 
LVL 5

Expert Comment

by:hossamshaaban
ID: 37709888
can you instead use
fsck -F vxfs /dev/vx/rdsk/proddata/oradat
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37710903
is there any otherway as well to repair this file system in case if it is corrupted. i need to try the option fsck -F vxfs /dev/vx/rdsk/proddata/oradata as per hossamshaaban.

could anyone suggest an alternate option to repair the file system as we have very critical data in that
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 76

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 37711189
Where is the underlying resources for this partition.
Is

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
check the status of the referenced disks.
Backup is the ultimate solution.

/dev/vx/rdsk/proddaata/oradata is the correct reference as hossamshaaban pointed out.
What happens when you run this command??

Do you know how the partition was formatted?
fstyp -v /dev/vx/rdsk/proddata/oradata
you could using newfs -N  /dev/vx/dsk/proddata/oradata (note -N tells it not to run but to appear as though it does, the issue might be a corrupt super block. Make sure to check newfs for the -N flag means display but perform no action on the filesystem)
It should output the list of possible super blocks.
you could then use
fsck -F vxfs -b <alternate_block> /dev/vx/rdsk/proddata/oradata
sequentially to locate a valid super_block.
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37713303
i have run fsck -F vxfs /dev/vx/rdsk/proddata/oradata also but same error came.




bash-2.03# fsck -F/dev/vx/dsk/proddata/oradata /oradata

fsck: Fstype /dev/vx/dsk/proddata/oradata exceeds 8 characters

bash-2.03# fsck -F vxfs /dev/vx/dsk/proddata/oradata

no valid ILISTs for fileset 999

no valid ILISTs for fileset 999

UX:vxfs fsck: ERROR: cannot initialize aggregate

file system check failure, aborting ...
0
 
LVL 5

Expert Comment

by:hossamshaaban
ID: 37713584
as i see you also use block device /dev/vx/dsk/proddata/oradata not raw device /dev/vx/rdsk/proddata/oradata as i recommend.
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37714430
please let me knwo about any other workarounds
0
 
LVL 76

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 37715681
fsck -F vxfs /dev/vx/rdsk/prodata/oradata
It is unclear how it is damaged.
running fsck with hosseamshaaban's correction would tell you what the error might be based on which we could try to see whether it is correctable.
What about the status of the disks that make up the veritas partition?
0
 
LVL 1

Accepted Solution

by:
kurajesh earned 0 total points
ID: 37718685
even  /dev/vx/rdsk/proddata/oradata  returns the same error. any other suggestions
0
 
LVL 76

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 37719861
What is the status of the underlying disks that make up the oradata veritas volume?

http://www.symantec.com/business/support/index?page=content&id=TECH74878

run fstyp /dev/vx/rdsk/prodata/oradata
There are references above to try different command to see what is going on with the filesystem.
vxassist
ncheck
0
 
LVL 1

Author Closing Comment

by:kurajesh
ID: 37795070
we have almost recovered the partition , thanks
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Daisy Chaining Scheduled Tasks in Task Scheduler 4 96
Administrator Tools 2 65
LINUX ZIP - UNCOMPRESS WINDOWS PATH 3 58
UNIX SCP 5 47
FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Windows 10 is here and for most admins this means frustration and challenges getting that first working Windows 10 image. As in my previous sysprep articles, I've put together a simple help guide to get you through this process. The aim is to achiev…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now