Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2402
  • Last Modified:

solaris file system mount error

we have a production database server and now we are receiving an error while mounting one of the core file system where main data is stored.

the details are as


bash-2.03#  mount -F vxfs /dev/vx/dsk/proddata/oradata /oradata

UX:vxfs mount: ERROR: file system on /dev/vx/dsk/proddata/oradata has structural

damage

UX:vxfs mount: ERROR: /dev/vx/dsk/proddata/oradata is corrupted. needs checking

bash-2.03#



the vxprint output is as

bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        1035615958.1025.db-srv

dm disk06       c3t1d0s2     sliced   2888     71124291 -
dm rootdisk     c3t0d0s2     sliced   2888     71124291 -

v  expvol       -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-04    expvol       ENABLED  ACTIVE   31457280 CONCAT    -        RW
sv expvol-S01   expvol-04    expvol-L01 1      31457280 0         3/3      ENA

v  expvol-L01   -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-P01   expvol-L01   ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd rootdisk-08  expvol-P01   rootdisk 39114170 264      LOG       c3t0d0   ENA
pl expvol-P02   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd rootdisk-09  expvol-P02   rootdisk 39117059 31457280 0         c3t0d0   ENA
pl expvol-P03   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd disk06-07    expvol-P03   disk06   39114171 31457280 0         c3t1d0   ENA

v  opt          -            ENABLED  ACTIVE   10241505 fsgen     -        ROUND
pl opt-01       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd rootdisk-03  opt-01       rootdisk 28872665 10241505 0         c3t0d0   ENA
pl opt-02       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd disk06-01    opt-02       disk06   0        10241505 0         c3t1d0   ENA

v  rootvol      -            ENABLED  ACTIVE   6144903  root      -        ROUND
pl rootvol-01   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-02  rootvol-01   rootdisk 8389655  6144903  0         c3t0d0   ENA
pl rootvol-02   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-02    rootvol-02   disk06   10241505 6144903  0         c3t1d0   ENA

v  swapvol      -            ENABLED  ACTIVE   8389656  swap      -        ROUND
pl swapvol-01   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd rootdisk-B0  swapvol-01   rootdisk 71124290 1        0         c3t0d0   ENA
sd rootdisk-01  swapvol-01   rootdisk 0        8389655  1         c3t0d0   ENA
pl swapvol-02   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd disk06-03    swapvol-02   disk06   16386408 8389656  0         c3t1d0   ENA

v  usr          -            ENABLED  ACTIVE   6144903  fsgen     -        ROUND
pl usr-01       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-04  usr-01       rootdisk 22727762 6144903  0         c3t0d0   ENA
pl usr-02       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-04    usr-02       disk06   24776064 6144903  0         c3t1d0   ENA

v  var          -            ENABLED  ACTIVE   8193204  fsgen     -        ROUND
pl var-01       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd rootdisk-05  var-01       rootdisk 14534558 8193204  0         c3t0d0   ENA
pl var-02       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd disk06-05    var-02       disk06   30920967 8193204  0         c3t1d0   ENA

Disk group: proddata

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg proddata     default      default  58000    1035206141.1647.db-srv

dm proddat01    c2t1d0s2     sliced   2888     71124291 -
dm proddat02    c1t9d0s2     sliced   2888     71124291 -
dm proddat03    c1t10d0s2    sliced   2888     71124291 -
dm proddat04    c2t0d0s2     sliced   2888     71124291 -
dm proddat05    c1t8d0s2     sliced   2888     71124291 -

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
bash-2.03# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/rootvol  3008649  389335 2559142    14%    /
/dev/vx/dsk/usr      3008649 1047343 1901134    36%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
/dev/vx/dsk/var      4032504 1275981 2716198    32%    /var
swap                 7549304      16 7549288     1%    /var/run
swap                 7549344      56 7549288     1%    /tmp
/dev/vx/dsk/opt      5040814  538850 4451556    11%    /opt
/dev/vx/dsk/rootdg/expvol
                     14745524 11488160 1782812    87%    /exp
bash-2.03# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c1t8d0s2     sliced    proddat05    proddata     online
c1t9d0s2     sliced    proddat02    proddata     online
c1t10d0s2    sliced    proddat03    proddata     online
c2t0d0s2     sliced    proddat04    proddata     online
c2t1d0s2     sliced    proddat01    proddata     online
c3t0d0s2     sliced    rootdisk     rootdg       online
c3t1d0s2     sliced    disk06       rootdg       online
bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        1035615958.1025.db-srv

dm disk06       c3t1d0s2     sliced   2888     71124291 -
dm rootdisk     c3t0d0s2     sliced   2888     71124291 -

v  expvol       -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-04    expvol       ENABLED  ACTIVE   31457280 CONCAT    -        RW
sv expvol-S01   expvol-04    expvol-L01 1      31457280 0         3/3      ENA

v  expvol-L01   -            ENABLED  ACTIVE   31457280 fsgen     -        SELEC
T
pl expvol-P01   expvol-L01   ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd rootdisk-08  expvol-P01   rootdisk 39114170 264      LOG       c3t0d0   ENA
pl expvol-P02   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd rootdisk-09  expvol-P02   rootdisk 39117059 31457280 0         c3t0d0   ENA
pl expvol-P03   expvol-L01   ENABLED  ACTIVE   31457280 CONCAT    -        RW
sd disk06-07    expvol-P03   disk06   39114171 31457280 0         c3t1d0   ENA

v  opt          -            ENABLED  ACTIVE   10241505 fsgen     -        ROUND
pl opt-01       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd rootdisk-03  opt-01       rootdisk 28872665 10241505 0         c3t0d0   ENA
pl opt-02       opt          ENABLED  ACTIVE   10241505 CONCAT    -        RW
sd disk06-01    opt-02       disk06   0        10241505 0         c3t1d0   ENA

v  rootvol      -            ENABLED  ACTIVE   6144903  root      -        ROUND
pl rootvol-01   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-02  rootvol-01   rootdisk 8389655  6144903  0         c3t0d0   ENA
pl rootvol-02   rootvol      ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-02    rootvol-02   disk06   10241505 6144903  0         c3t1d0   ENA

v  swapvol      -            ENABLED  ACTIVE   8389656  swap      -        ROUND
pl swapvol-01   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd rootdisk-B0  swapvol-01   rootdisk 71124290 1        0         c3t0d0   ENA
sd rootdisk-01  swapvol-01   rootdisk 0        8389655  1         c3t0d0   ENA
pl swapvol-02   swapvol      ENABLED  ACTIVE   8389656  CONCAT    -        RW
sd disk06-03    swapvol-02   disk06   16386408 8389656  0         c3t1d0   ENA

v  usr          -            ENABLED  ACTIVE   6144903  fsgen     -        ROUND
pl usr-01       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd rootdisk-04  usr-01       rootdisk 22727762 6144903  0         c3t0d0   ENA
pl usr-02       usr          ENABLED  ACTIVE   6144903  CONCAT    -        RW
sd disk06-04    usr-02       disk06   24776064 6144903  0         c3t1d0   ENA

v  var          -            ENABLED  ACTIVE   8193204  fsgen     -        ROUND
pl var-01       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd rootdisk-05  var-01       rootdisk 14534558 8193204  0         c3t0d0   ENA
pl var-02       var          ENABLED  ACTIVE   8193204  CONCAT    -        RW
sd disk06-05    var-02       disk06   30920967 8193204  0         c3t1d0   ENA

Disk group: proddata

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
V  NAME         RVG          KSTATE   STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg proddata     default      default  58000    1035206141.1647.db-srv

dm proddat01    c2t1d0s2     sliced   2888     71124291 -
dm proddat02    c1t9d0s2     sliced   2888     71124291 -
dm proddat03    c1t10d0s2    sliced   2888     71124291 -
dm proddat04    c2t0d0s2     sliced   2888     71124291 -
dm proddat05    c1t8d0s2     sliced   2888     71124291 -

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
bash-2.03#


please help me
0
kurajesh
Asked:
kurajesh
  • 9
  • 4
  • 2
4 Solutions
 
kurajeshSenior Systems AnalystAuthor Commented:
the fstab output is

bash-2.03# cat vfstab
#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr          ufs     1       yes     -
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/vx/dsk/swapvol     -       -       swap    -       no      -
/dev/vx/dsk/rootvol     /dev/vx/rdsk/rootvol    /       ufs     1       no
-
/dev/vx/dsk/usr /dev/vx/rdsk/usr        /usr    ufs     1       no      -
/dev/vx/dsk/var /dev/vx/rdsk/var        /var    ufs     1       no      -
/dev/vx/dsk/opt /dev/vx/rdsk/opt        /opt    ufs     2       yes     -
swap    -       /tmp    tmpfs   -       yes     -
#NOTE: volume rootvol (/) encapsulated partition c3t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c3t0d0s1
#NOTE: volume var (/var) encapsulated partition c3t0d0s5
#NOTE: volume usr (/usr) encapsulated partition c3t0d0s6
#NOTE: volume opt (/opt) encapsulated partition c3t0d0s7
#/dev/vx/dsk/proddata/oradata   /dev/vx/rdsk/proddata/oradata   /oradata
vxfs    3       yes     -
#/dev/vx/dsk/proddata/visvol    /dev/vx/rdsk/proddata/visvol    /vis    vxfs
3       yes     -
/dev/vx/dsk/rootdg/expvol       /dev/vx/rdsk/rootdg/expvol      /exp    ufs
3       yes     -
0
 
arnoldCommented:
What was done to this file system? Was there an attempt to resize it? Did someone run fsck on the file system without specifying the filesystem -F vxfs?
fsck -F vxfs /dev/vx/dsk/prodata/oradata
0
 
kurajeshSenior Systems AnalystAuthor Commented:
hi arnold , can i try fsck -F vxfs /dev/vx/dsk/prodata/oradata
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
kurajeshSenior Systems AnalystAuthor Commented:
never tried to resize the filesystem , but not able to mount it when tried to do so
0
 
kurajeshSenior Systems AnalystAuthor Commented:
when i tried

 fsck -F vxfs /dev/vx/dsk/prodata/oradata
 the output came as



bash-2.03# fsck -F/dev/vx/dsk/proddata/oradata /oradata

fsck: Fstype /dev/vx/dsk/proddata/oradata exceeds 8 characters

bash-2.03# fsck -F vxfs /dev/vx/dsk/proddata/oradata

no valid ILISTs for fileset 999

no valid ILISTs for fileset 999

UX:vxfs fsck: ERROR: cannot initialize aggregate

file system check failure, aborting ...
0
 
hossamshaabanCommented:
can you instead use
fsck -F vxfs /dev/vx/rdsk/proddata/oradat
0
 
kurajeshSenior Systems AnalystAuthor Commented:
is there any otherway as well to repair this file system in case if it is corrupted. i need to try the option fsck -F vxfs /dev/vx/rdsk/proddata/oradata as per hossamshaaban.

could anyone suggest an alternate option to repair the file system as we have very critical data in that
0
 
arnoldCommented:
Where is the underlying resources for this partition.
Is

v  oradata      -            ENABLED  ACTIVE   188743680 raid5    -        RAID
pl oradata-01   oradata      ENABLED  ACTIVE   188749920 RAID     4/32     RW
sd proddat05-01 oradata-01   proddat05 0       62916642 0/0       c1t8d0   ENA
sd proddat02-01 oradata-01   proddat02 0       62916642 1/0       c1t9d0   ENA
sd proddat03-01 oradata-01   proddat03 0       62916642 2/0       c1t10d0  ENA
sd proddat04-01 oradata-01   proddat04 0       62916642 3/0       c2t0d0   ENA
pl oradata-02   oradata      ENABLED  LOG      5778     CONCAT    -        RW
sd proddat01-01 oradata-02   proddat01 0       5778     0         c2t1d0   ENA
check the status of the referenced disks.
Backup is the ultimate solution.

/dev/vx/rdsk/proddaata/oradata is the correct reference as hossamshaaban pointed out.
What happens when you run this command??

Do you know how the partition was formatted?
fstyp -v /dev/vx/rdsk/proddata/oradata
you could using newfs -N  /dev/vx/dsk/proddata/oradata (note -N tells it not to run but to appear as though it does, the issue might be a corrupt super block. Make sure to check newfs for the -N flag means display but perform no action on the filesystem)
It should output the list of possible super blocks.
you could then use
fsck -F vxfs -b <alternate_block> /dev/vx/rdsk/proddata/oradata
sequentially to locate a valid super_block.
0
 
kurajeshSenior Systems AnalystAuthor Commented:
i have run fsck -F vxfs /dev/vx/rdsk/proddata/oradata also but same error came.




bash-2.03# fsck -F/dev/vx/dsk/proddata/oradata /oradata

fsck: Fstype /dev/vx/dsk/proddata/oradata exceeds 8 characters

bash-2.03# fsck -F vxfs /dev/vx/dsk/proddata/oradata

no valid ILISTs for fileset 999

no valid ILISTs for fileset 999

UX:vxfs fsck: ERROR: cannot initialize aggregate

file system check failure, aborting ...
0
 
hossamshaabanCommented:
as i see you also use block device /dev/vx/dsk/proddata/oradata not raw device /dev/vx/rdsk/proddata/oradata as i recommend.
0
 
kurajeshSenior Systems AnalystAuthor Commented:
please let me knwo about any other workarounds
0
 
arnoldCommented:
fsck -F vxfs /dev/vx/rdsk/prodata/oradata
It is unclear how it is damaged.
running fsck with hosseamshaaban's correction would tell you what the error might be based on which we could try to see whether it is correctable.
What about the status of the disks that make up the veritas partition?
0
 
kurajeshSenior Systems AnalystAuthor Commented:
even  /dev/vx/rdsk/proddata/oradata  returns the same error. any other suggestions
0
 
arnoldCommented:
What is the status of the underlying disks that make up the oradata veritas volume?

http://www.symantec.com/business/support/index?page=content&id=TECH74878

run fstyp /dev/vx/rdsk/prodata/oradata
There are references above to try different command to see what is going on with the filesystem.
vxassist
ncheck
0
 
kurajeshSenior Systems AnalystAuthor Commented:
we have almost recovered the partition , thanks
0

Featured Post

[Webinar On Demand] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

  • 9
  • 4
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now