Solved

solaris file system mount error

Posted on 2012-03-11
15
2,175 Views
Last Modified: 2012-04-02
we have a production database server and now we are receiving an error while mounting one of the core file system where main data is stored.

the details are as


bash-2.03#  mount -F vxfs /dev/vx/dsk/proddata/oradata /oradata

UX:vxfs mount: ERROR: file system on /dev/vx/dsk/proddata/oradata has structural

damage

UX:vxfs mount: ERROR: /dev/vx/dsk/proddata/oradata is corrupted. needs checking

bash-2.03#



the vxprint output is as

bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        1035615958.1025.db-srv

dm disk06       c3t1d0s2     sliced   2888     71124291 -
dm rootdisk     c3t0d0s2     sliced   2888     71124291 -

v  expvol       -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-04    expvol       ENABLED  ACTIVE   31457280 CONCAT    -        RW
sv expvol-S01   expvol-04    expvol-L01 1      31457280 0         3/3      ENA

v  expvol-L01   -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-P01   expvol-L01   ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd rootdisk-08  expvol-P01   rootdisk 39114170 264      LOG       c3t0d0   ENA
pl expvol-P02   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd rootdisk-09  expvol-P02   rootdisk 39117059 31457280 0         c3t0d0   ENA
pl expvol-P03   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd disk06-07    expvol-P03   disk06   39114171 31457280 0         c3t1d0   ENA

v  opt          -            ENABLED  ACTIVE   10241505 fsgen     -        ROUND
pl opt-01       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd rootdisk-03  opt-01       rootdisk 28872665 10241505 0         c3t0d0   ENA
pl opt-02       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd disk06-01    opt-02       disk06   0        10241505 0         c3t1d0   ENA

v  rootvol      -            ENABLED  ACTIVE   6144903  root      -        ROUND
pl rootvol-01   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-02  rootvol-01   rootdisk 8389655  6144903  0         c3t0d0   ENA
pl rootvol-02   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-02    rootvol-02   disk06   10241505 6144903  0         c3t1d0   ENA

v  swapvol      -            ENABLED  ACTIVE   8389656  swap      -        ROUND
pl swapvol-01   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd rootdisk-B0  swapvol-01   rootdisk 71124290 1        0         c3t0d0   ENA
sd rootdisk-01  swapvol-01   rootdisk 0        8389655  1         c3t0d0   ENA
pl swapvol-02   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd disk06-03    swapvol-02   disk06   16386408 8389656  0         c3t1d0   ENA

v  usr          -            ENABLED  ACTIVE   6144903  fsgen     -        ROUND
pl usr-01       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-04  usr-01       rootdisk 22727762 6144903  0         c3t0d0   ENA
pl usr-02       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-04    usr-02       disk06   24776064 6144903  0         c3t1d0   ENA

v  var          -            ENABLED  ACTIVE   8193204  fsgen     -        ROUND
pl var-01       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd rootdisk-05  var-01       rootdisk 14534558 8193204  0         c3t0d0   ENA
pl var-02       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd disk06-05    var-02       disk06   30920967 8193204  0         c3t1d0   ENA

Disk group: proddata

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg proddata     default      default  58000    1035206141.1647.db-srv

dm proddat01    c2t1d0s2     sliced   2888     71124291 -
dm proddat02    c1t9d0s2     sliced   2888     71124291 -
dm proddat03    c1t10d0s2    sliced   2888     71124291 -
dm proddat04    c2t0d0s2     sliced   2888     71124291 -
dm proddat05    c1t8d0s2     sliced   2888     71124291 -

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
bash-2.03# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/rootvol  3008649  389335 2559142    14%    /
/dev/vx/dsk/usr      3008649 1047343 1901134    36%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
/dev/vx/dsk/var      4032504 1275981 2716198    32%    /var
swap                 7549304      16 7549288     1%    /var/run
swap                 7549344      56 7549288     1%    /tmp
/dev/vx/dsk/opt      5040814  538850 4451556    11%    /opt
/dev/vx/dsk/rootdg/expvol
                     14745524 11488160 1782812    87%    /exp
bash-2.03# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c1t8d0s2     sliced    proddat05    proddata     online
c1t9d0s2     sliced    proddat02    proddata     online
c1t10d0s2    sliced    proddat03    proddata     online
c2t0d0s2     sliced    proddat04    proddata     online
c2t1d0s2     sliced    proddat01    proddata     online
c3t0d0s2     sliced    rootdisk     rootdg       online
c3t1d0s2     sliced    disk06       rootdg       online
bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        1035615958.1025.db-srv

dm disk06       c3t1d0s2     sliced   2888     71124291 -
dm rootdisk     c3t0d0s2     sliced   2888     71124291 -

v  expvol       -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-04    expvol       ENABLED  ACTIVE   31457280 CONCAT    -        RW
sv expvol-S01   expvol-04    expvol-L01 1      31457280 0         3/3      ENA

v  expvol-L01   -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-P01   expvol-L01   ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd rootdisk-08  expvol-P01   rootdisk 39114170 264      LOG       c3t0d0   ENA
pl expvol-P02   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd rootdisk-09  expvol-P02   rootdisk 39117059 31457280 0         c3t0d0   ENA
pl expvol-P03   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd disk06-07    expvol-P03   disk06   39114171 31457280 0         c3t1d0   ENA

v  opt          -            ENABLED  ACTIVE   10241505 fsgen     -        ROUND
pl opt-01       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd rootdisk-03  opt-01       rootdisk 28872665 10241505 0         c3t0d0   ENA
pl opt-02       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd disk06-01    opt-02       disk06   0        10241505 0         c3t1d0   ENA

v  rootvol      -            ENABLED  ACTIVE   6144903  root      -        ROUND
pl rootvol-01   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-02  rootvol-01   rootdisk 8389655  6144903  0         c3t0d0   ENA
pl rootvol-02   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-02    rootvol-02   disk06   10241505 6144903  0         c3t1d0   ENA

v  swapvol      -            ENABLED  ACTIVE   8389656  swap      -        ROUND
pl swapvol-01   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd rootdisk-B0  swapvol-01   rootdisk 71124290 1        0         c3t0d0   ENA
sd rootdisk-01  swapvol-01   rootdisk 0        8389655  1         c3t0d0   ENA
pl swapvol-02   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd disk06-03    swapvol-02   disk06   16386408 8389656  0         c3t1d0   ENA

v  usr          -            ENABLED  ACTIVE   6144903  fsgen     -        ROUND
pl usr-01       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-04  usr-01       rootdisk 22727762 6144903  0         c3t0d0   ENA
pl usr-02       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-04    usr-02       disk06   24776064 6144903  0         c3t1d0   ENA

v  var          -            ENABLED  ACTIVE   8193204  fsgen     -        ROUND
pl var-01       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd rootdisk-05  var-01       rootdisk 14534558 8193204  0         c3t0d0   ENA
pl var-02       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd disk06-05    var-02       disk06   30920967 8193204  0         c3t1d0   ENA

Disk group: proddata

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg proddata     default      default  58000    1035206141.1647.db-srv

dm proddat01    c2t1d0s2     sliced   2888     71124291 -
dm proddat02    c1t9d0s2     sliced   2888     71124291 -
dm proddat03    c1t10d0s2    sliced   2888     71124291 -
dm proddat04    c2t0d0s2     sliced   2888     71124291 -
dm proddat05    c1t8d0s2     sliced   2888     71124291 -

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
bash-2.03#


please help me
0
Comment
Question by:kurajesh
  • 9
  • 4
  • 2
15 Comments
 
LVL 1

Author Comment

by:kurajesh
ID: 37708570
the fstab output is

bash-2.03# cat vfstab
#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr          ufs     1       yes     -
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/vx/dsk/swapvol     -       -       swap    -       no      -
/dev/vx/dsk/rootvol     /dev/vx/rdsk/rootvol    /       ufs     1       no
-
/dev/vx/dsk/usr /dev/vx/rdsk/usr        /usr    ufs     1       no      -
/dev/vx/dsk/var /dev/vx/rdsk/var        /var    ufs     1       no      -
/dev/vx/dsk/opt /dev/vx/rdsk/opt        /opt    ufs     2       yes     -
swap    -       /tmp    tmpfs   -       yes     -
#NOTE: volume rootvol (/) encapsulated partition c3t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c3t0d0s1
#NOTE: volume var (/var) encapsulated partition c3t0d0s5
#NOTE: volume usr (/usr) encapsulated partition c3t0d0s6
#NOTE: volume opt (/opt) encapsulated partition c3t0d0s7
#/dev/vx/dsk/proddata/oradata   /dev/vx/rdsk/proddata/oradata   /oradata
vxfs    3       yes     -
#/dev/vx/dsk/proddata/visvol    /dev/vx/rdsk/proddata/visvol    /vis    vxfs
3       yes     -
/dev/vx/dsk/rootdg/expvol       /dev/vx/rdsk/rootdg/expvol      /exp    ufs
3       yes     -
0
 
LVL 77

Expert Comment

by:arnold
ID: 37708639
What was done to this file system? Was there an attempt to resize it? Did someone run fsck on the file system without specifying the filesystem -F vxfs?
fsck -F vxfs /dev/vx/dsk/prodata/oradata
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37708734
hi arnold , can i try fsck -F vxfs /dev/vx/dsk/prodata/oradata
0
Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

 
LVL 1

Author Comment

by:kurajesh
ID: 37708738
never tried to resize the filesystem , but not able to mount it when tried to do so
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37709546
when i tried

 fsck -F vxfs /dev/vx/dsk/prodata/oradata
 the output came as



bash-2.03# fsck -F/dev/vx/dsk/proddata/oradata /oradata

fsck: Fstype /dev/vx/dsk/proddata/oradata exceeds 8 characters

bash-2.03# fsck -F vxfs /dev/vx/dsk/proddata/oradata

no valid ILISTs for fileset 999

no valid ILISTs for fileset 999

UX:vxfs fsck: ERROR: cannot initialize aggregate

file system check failure, aborting ...
0
 
LVL 5

Expert Comment

by:hossamshaaban
ID: 37709888
can you instead use
fsck -F vxfs /dev/vx/rdsk/proddata/oradat
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37710903
is there any otherway as well to repair this file system in case if it is corrupted. i need to try the option fsck -F vxfs /dev/vx/rdsk/proddata/oradata as per hossamshaaban.

could anyone suggest an alternate option to repair the file system as we have very critical data in that
0
 
LVL 77

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 37711189
Where is the underlying resources for this partition.
Is

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
check the status of the referenced disks.
Backup is the ultimate solution.

/dev/vx/rdsk/proddaata/oradata is the correct reference as hossamshaaban pointed out.
What happens when you run this command??

Do you know how the partition was formatted?
fstyp -v /dev/vx/rdsk/proddata/oradata
you could using newfs -N  /dev/vx/dsk/proddata/oradata (note -N tells it not to run but to appear as though it does, the issue might be a corrupt super block. Make sure to check newfs for the -N flag means display but perform no action on the filesystem)
It should output the list of possible super blocks.
you could then use
fsck -F vxfs -b <alternate_block> /dev/vx/rdsk/proddata/oradata
sequentially to locate a valid super_block.
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37713303
i have run fsck -F vxfs /dev/vx/rdsk/proddata/oradata also but same error came.




bash-2.03# fsck -F/dev/vx/dsk/proddata/oradata /oradata

fsck: Fstype /dev/vx/dsk/proddata/oradata exceeds 8 characters

bash-2.03# fsck -F vxfs /dev/vx/dsk/proddata/oradata

no valid ILISTs for fileset 999

no valid ILISTs for fileset 999

UX:vxfs fsck: ERROR: cannot initialize aggregate

file system check failure, aborting ...
0
 
LVL 5

Expert Comment

by:hossamshaaban
ID: 37713584
as i see you also use block device /dev/vx/dsk/proddata/oradata not raw device /dev/vx/rdsk/proddata/oradata as i recommend.
0
 
LVL 1

Author Comment

by:kurajesh
ID: 37714430
please let me knwo about any other workarounds
0
 
LVL 77

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 37715681
fsck -F vxfs /dev/vx/rdsk/prodata/oradata
It is unclear how it is damaged.
running fsck with hosseamshaaban's correction would tell you what the error might be based on which we could try to see whether it is correctable.
What about the status of the disks that make up the veritas partition?
0
 
LVL 1

Accepted Solution

by:
kurajesh earned 0 total points
ID: 37718685
even  /dev/vx/rdsk/proddata/oradata  returns the same error. any other suggestions
0
 
LVL 77

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 37719861
What is the status of the underlying disks that make up the oradata veritas volume?

http://www.symantec.com/business/support/index?page=content&id=TECH74878

run fstyp /dev/vx/rdsk/prodata/oradata
There are references above to try different command to see what is going on with the filesystem.
vxassist
ncheck
0
 
LVL 1

Author Closing Comment

by:kurajesh
ID: 37795070
we have almost recovered the partition , thanks
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Windows 7 does not have the best desktop search built in. This is something Windows 7 users have struggled with. You type something in, and your search results don’t always match what you are looking for, or it doesn’t actually work at all. There ar…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

816 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now