Solved

Multipath

Posted on 2014-02-12
6
1,410 Views
Last Modified: 2014-03-15
why I am getting I/O error here.

[root@node1 ~]# pvcreate /dev/sdd
  Physical volume "/dev/sdd" successfully created
[root@node1 ~]#
[root@node1 ~]# pvs
  /dev/mapper/SAN: read failed after 0 of 4096 at 4244570112: Input/output error
  /dev/mapper/SAN: read failed after 0 of 4096 at 4244627456: Input/output error
  /dev/mapper/SAN: read failed after 0 of 4096 at 0: Input/output error
  /dev/mapper/SAN: read failed after 0 of 4096 at 4096: Input/output error
  /dev/mapper/SANp1: read failed after 0 of 4096 at 2147418112: Input/output error
  /dev/mapper/SANp1: read failed after 0 of 4096 at 2147475456: Input/output error
  /dev/mapper/SANp1: read failed after 0 of 4096 at 0: Input/output error
  /dev/mapper/SANp1: read failed after 0 of 4096 at 4096: Input/output error
  Found duplicate PV XAYwgqMQwzcdNWEdKBRlSOwRzByhgZ8f: using /dev/sde not /dev/sdd
  PV         VG       Fmt  Attr PSize  PFree
  /dev/sda2  VolGroup lvm2 a-   11.51g    0
  /dev/sde            lvm2 a-    3.95g 3.95g
0
Comment
Question by:ittechlab
6 Comments
 
LVL 13

Expert Comment

by:Sandy
ID: 39855383
Check the SAN paths,, It seems your system has lost connectivity with SAN or might be some other SAN issue.

TY/SA
0
 
LVL 1

Expert Comment

by:sysrams
ID: 39856007
These errors are common in Linux. these errors are ignorable and it does not affect your disk/san activity. these bugs were fixed at Redhat 6.
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39856031
can be ignored if these are from the devices which are no more exist in the system.

TY/SA
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:ittechlab
ID: 39856174
I am using RHEL 6
0
 

Author Comment

by:ittechlab
ID: 39857960
[root@node1 ~]# multipath -ll
SAN (1IET     00010001) dm-2 ,
size=4.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- #:#:#:#  -   #:#  failed faulty running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- #:#:#:#  -   #:#  failed faulty running
 

[root@node1 ~]# cat /etc/multipath.conf
defaults {
        user_friendly_names no
}

multipaths {
        multipath {
                wwid                    "1IET     00010001"
                alias                   SAN
        }
}


blacklist {
}


[root@node1 ~]# service iscsi status
iSCSI Transport Class version 2.0-870
version 2.0-872
Target: iqn.2014-01.com.example.cluster1:first
        Current Portal: 172.19.3.4:3260,1
        Persistent Portal: 172.19.3.4:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:cc38f04b6e34
                Iface IPaddress: 172.19.3.1
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 17
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 8192
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 19 State: running
                scsi19 Channel 00 Id 0 Lun: 0
                scsi19 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sde          State: running
        Current Portal: 172.18.3.4:3260,1
        Persistent Portal: 172.18.3.4:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:cc38f04b6e34
                Iface IPaddress: 172.18.3.1
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 18
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 8192
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 20 State: running
                scsi20 Channel 00 Id 0 Lun: 0
                scsi20 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdd          State: running


As you can see my above output. The disk is multipath disk.

/dev/sdd
/dev/sde


also according to multipath configuration this is access by /dev/SAN


When i create partion which path i shouid be using? I am really confused. Can some one explain how this works. Also i noticed there is another path /dev/mapper/SAN. Do we use this path?


I am not clear what is the relations between this path

/dev/sdd
/dev/sde
/dev/SAN
/dev/mapper/SAN


every time i boot the system /dev/sdd and /dev/sde changes. does it not consistent?


Please clarify me how this works.
0
 
LVL 13

Accepted Solution

by:
Daniel Helgenberger earned 500 total points
ID: 39859739
Hello ittechlab,

first of all I assume from your question and the listing above you have dm-multipath enabled?

[root@node1 ~]# multipath -ll
SAN (1IET     00010001) dm-2 ,
size=4.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- #:#:#:#  -   #:#  failed faulty running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- #:#:#:#  -   #:#  failed faulty running

Open in new window

Apparently the config is not working, otherwise the would look like this instance:
mpath_ift16r01_prometa (3600d023100066590000000037314d5d4) dm-4 IFT,S16F-R1840
[size=89G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:3:5  sdac 65:192 [active][ready]
 \_ 2:0:7:5  sdbp 68:48  [active][ready]
\_ round-robin 0 [prio=40][enabled]
 \_ 1:0:4:5  sdal 66:80  [active][ready]
 \_ 2:0:4:5  sdas 66:192 [active][ready]
 \_ 1:0:0:5  sdk  8:160  [active][ready]
 \_ 2:0:0:5  sdl  8:176  [active][ready]
mpath_ift16g03a (3600d0230006f89330f653e15b0a92100) dm-14 IFT,A16F-G2422
[size=24T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=200][active]
 \_ 1:0:7:0  sdav 66:240 [active][ready]
 \_ 1:0:9:0  sdbg 67:160 [active][ready]
 \_ 2:0:9:0  sdbs 68:96  [active][ready]
 \_ 2:0:3:0  sdu  65:64  [active][ready]
......

Open in new window


If so, two things apply:
1. DO NOT use /dev/sdXX devices any more. If dm-multipath is really working, you cannot use pvceate on them (you will get a permission denied error) and need to use your /dev/mapper/<mpath-XX> devices.

2. The device enumeration happens module load (on boot time) and will therefore not always yield the same device ID (race condition). But this is no problem since dm-multipath takes care of this end uses the WWID / WWN to distinguish them.

These errors are common in Linux. these errors are ignorable and it does not affect your disk/san activity. these bugs were fixed at Redhat 6.
I cannot verify this; I operate quite a number of (FC) dm-multipath systems (RHEL5/6) and do not have them. This clearly relates to the failed paths you see.

First step:
Check the iSCSI luns. I assume they are working?

ATTENTION! THIS IS AN EXAMPLE!
Second step:
Fix your multipath conf. First, add a blacklist, to exclude all devices but your whitelisted:
## safety first...
blacklist {
	devnode "^(ramrawloopfdmddm-srscdst)[0-9]*"
	devnode "^hd[a-z]"
	devnode "^cciss!c[0-9]d[0-0]*"
	devnode "cvfsc*"
	devnode "c0d0*"
	device {
		vendor "*"
		product "*"
	}
}

Open in new window


Then, whitelist them again:
blacklist_exceptions {
	device {
		vendor	"IFT*"
		product "*"
	}

Open in new window


As mentioned, you can use WWNs instead of vendors if you like. Get the vendor/products from the lsscsi command. Please see man multipath.conf for details about blacklist_exceptions:
blacklist_exceptions section
       The blacklist_exceptions section is used to revert the actions of the blacklist section, ie to include specific device in the multipath topology. This allows to  selectively  include
       devices which would normally be excluded via the blacklist section.

       The following keywords are recognized:

       wwid             The World Wide Identification of a device.

       devnode          Regular expression of the device nodes to be excluded.

       device           Subsection  for  the  device description. This subsection recognizes the vendor and product keywords. For a full description of these keywords please see the devices
                        section description.

Afterwards, (re)start multipathd:
chkconfig multipathd on
service multipathd restart

Open in new window


Then, do a dry run, and if this is ok create the device mappings:
multipath -vd
multipath -v
multipath -ll

Open in new window


If you done everything right, you should see the mappings as [active][ready] like in my example above.

PS:
If everything is running, continue tweaking the multipath.conf for performance. You should contact your device vendors and read their topics on multipath for appropriate values. Here is an example for FC / IFT storage:
devices {
       device {
               vendor                  "IFT"
               path_grouping_policy    group_by_prio
               getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
               path_checker            tur
               path_selector           "round-robin 0"
	       prio_callout	       "/sbin/mpath_prio_alua /dev/%n"
               hardware_handler        "0"
               failback                immediate
               rr_weight               priorities
               no_path_retry           12
	       rr_min_io	       3
       }

Open in new window


Cheers,
Helge
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now