2010 - The basics of the IBM/LSI MPIO driver RDAC (Redundant Disk Array Controller) for Linux
The Slovak original of this document: 2010 - Základy IBM/LSI MPIO ovládača RDAC (Redundant Disk Array Controller) pre Linux (slovensky).
|=-------------------------------------------------=[ FC SAN ]=-----------------------------------------------=| +--------------------------------------------------------------------------------------------------------------+ | | | IBM / LSI | | | | Redundant Disk Array Controller | | MPP (Multi-Path Proxy Driver) | | | | | +--------------------------------------------------------------------------------------------------------------+ |=-------------------------------------------------=[ LH 2010 ]=----------------------------------------------=|
POZNAMKY k RDAC (Redundant Disk Array Controller) a.k.a. Multi-Path Proxy Driver alebo MPP
- Zakladne INFO
- Instalacia
- Pouzitie (priklady)
- Pouzitie (priklady s vystupmi)
- Problemy
- LINKzZakladne INFO
Ovladac RDAC (Redundant Disk Array Controller a.k.a. Multi-Path Proxy alebo MPP) pridava novu vrstvu do
of the Linux driver stack. This driver is what provides multi-path
smerom k diskovemu polu. Je to kombinacia dvoch ovladacov:
[1] vysoko urovnovy ovladac
[2] virtualny HBA ovladac
--------------------------------------------------------------------------------------------------------
[1] The high-level driver has the same role as "sd" (SCSI disk driver) or "sg" (SCSI generic driver)
driver. It serves the disk arrays and gives user space an ioctl interface. It is
zodpovedny za manazment viacerych ciest k diskovemu polu.
[2] Virtualny HBA ovladac je Linuxovy nizkourovnovy ovladac. Je umiestneny na rovnakej urovni ovladacoveho
stack, like any other driver for a QLogic or LSI fibre channel HBA.
When the host has more than one HBA and all those HBAs have access to one or more IBM TotalStorage DS4xxx,
a multipath driver is required. For the DS4xxx family this is the redundant disk array controller/multipath
proxy (RDAC/MPP) driver. On AIX, the base OS already contains the RDAC driver, also known as MPIO. For Linux,
you must download and install the RDAC/MPP package manually.
The RDAC multipath driver consists of two modules:
* The mppUpper: module is loaded before any HBA and prevents the HBA to publicize LUNs to the system.
* The mppVhba: module is a virtual HBA that sits on top of the real HBA. The virtual HBA is responsible
for bundling the different paths to the same LUN, organizing the traffic from and to the LUN, and
presenting it to the OS as single entity.
Note that if RDAC/MPP is used, a non-failover HBA device driver is required, and auto volume transfer (AVT)
must be disabled at the DS4xxx level. You achieve this by changing the host type from Linux to LNXCL. Check
the latest version of the Readme.txt delivered with the RDAC/MPP driver and follow the instructions.
Dolezite prikazy
--------------------------------------------------------------------------------------------------------
# mppUtil - prikaz na vykonavanie roznych funkcii RDAC ovladaca
# mppUpdate - refreshes the entries in the configuration file /var/mpp/devicemapping
# mppBusRescan - finds a new disk array, removes or detaches devices and refreshes device sizes
# hot_add - linka na mppBusRescan (/usr/sbin/hot_add -> /usr/sbin/mppBusRescan)
# /opt/mpp/lsvdev - shows the basic LUN mapping
# /etc/init.d/mpp - discovers devices at boot (with more than one array it is worth turning off, so a restart causes no unwanted trouble)
# /opt/mpp/mppSupport - writes a log file with the RDAC information -> "mppSupportdata_hostname_RDACversion_datetime.tar.gz" in /tmp
Dolezite subory
--------------------------------------------------------------------------------------------------------
[1] /etc/mpp.conf - konfiguracny subor MPP ovladaca
[2] /var/mpp/* - konfiguracne subory MPP ovladaca (stale zmeny disk. poli, LUNov)
[3] /proc/mpp/* - behove (runtime) informacie MPP ovladaca
[1] The MPP driver's configuration file (man RDAC -> a description of each directive)
--------------------------------------------------------------------------------------------------------
# cat /etc/mpp.conf
--------------------------------------------------------------------------------------------------------
virtualDiskProductId=VirtualDisk
DebugLevel=0x0
NotReadyWaitTime=270
BusyWaitTime=270
QuiescenceWaitTime=270
InquiryWaitTime=60
MaxLunsPerArray=256
MaxPathsPerController=4
ScanInterval=60
InquiryInterval=1
MaxArrayModules=30
ErrorLevel=3
SelectionTimeoutRetryCount=0
UaRetryCount=10
RetryCount=10
SynchTimeout=170
FailOverQuiescenceTime=20
FailoverTimeout=120
FailBackToCurrentAllowed=1
ControllerIoWaitTime=300
ArrayIoWaitTime=600
DisableLUNRebalance=0
SelectiveTransferMaxTransferAttempts=5
SelectiveTransferMinIOWaitTime=3
IdlePathCheckingInterval=60
RecheckFailedPathWaitTime=30
FailedPathCheckingInterval=60
ArrayFailoverWaitTime=300
PrintSenseBuffer=0
ClassicModeFailover=0
AVTModeFailover=0
LunFailoverDelay=3
LoadBalancePolicy=1
ImmediateVirtLunCreate=0
BusResetTimeout=150
AllowHBAsgDevs=0
S2ToS3Key=1b1fa2921abbbc44
--------------------------------------------------------------------------------------------------------
[2] Konfiguracne subory MPP ovladaca (stala konfiguracia disk. poli, LUNov)
--------------------------------------------------------------------------------------------------------
# ls -ltr /var/mpp
--------------------------------------------------------------------------------------------------------
-rw------- 1 root root 361 Sep 7 11:57 600a0b800047dfb8000000004b591ee7.wwn
-rw------- 1 root root 361 Sep 7 11:57 600a0b800047dd34000000004b612dbb.wwn
-rw------- 1 root root 15 Sep 7 12:17 devicemapping
--------------------------------------------------------------------------------------------------------
[2] # cat ./600a0b800047dfb8000000004b591ee7.wwn
--------------------------------------------------------------------------------------------------------
H2C0T0 :Active :Active :DS5300-POLUS
maxLun256:CAP 0 :CAP 1 :CAP 2 :CAP 3 :CBP 0 :CBP 1 :CBP 2 :CBP 3 :
L001 :H0C0T0:01H1C0T0:01---------:---------:H0C0T1:01H1C0T1:01---------:---------:
L002 :H0C0T0:02H1C0T0:02---------:---------:H0C0T1:02H1C0T1:02---------:---------:
--------------------------------------------------------------------------------------------------------
[2] # cat ./600a0b800047dd34000000004b612dbb.wwn
--------------------------------------------------------------------------------------------------------
H2C0T1 :Active :Active :DS5300-SITEL
maxLun256:CAP 0 :CAP 1 :CAP 2 :CAP 3 :CBP 0 :CBP 1 :CBP 2 :CBP 3 :
L001 :H1C0T2:01H0C0T3:01---------:---------:H1C0T3:01H0C0T2:01---------:---------:
L002 :H1C0T2:02H0C0T3:02---------:---------:H1C0T3:02H0C0T2:02---------:---------:
--------------------------------------------------------------------------------------------------------
[2] # cat /var/mpp/devicemapping
--------------------------------------------------------------------------------------------------------
0:DS5300-POLUS
1:DS5300-SITEL
[3] # ls -lR /proc/mpp
--------------------------------------------------------------------------------------------------------
...
[3] Every piece of runtime information the RDAC driver holds is in the "/proc" filesystem
--------------------------------------------------------------------------------------------------------
/proc/mpp/*
[3] Runtime informacie virtualneho MPP adaptera
Cislo adaptera moze byt ine na roznych systemoch (cislo je pridelovane SCSI medzivrstvou [middle layer])
--------------------------------------------------------------------------------------------------------
/proc/scsi/mpp/<cislo_adaptera>
[3] The entries for every disk array visible to the server
--------------------------------------------------------------------------------------------------------
/proc/mpp/<nazov_disk_pola>
[3] The entries for every disk controller of that array
--------------------------------------------------------------------------------------------------------
/proc/mpp/<nazov_disk_pola>/controllerA
/proc/mpp/<nazov_disk_pola>/controllerB
[3] Nizkourovnovy ovladac moze byt (qlogic, emulex)
HCT znamena:
Host# - the number of the low-level driver, assigned by the SCSI mid-layer
Channel# -
Target# -
--------------------------------------------------------------------------------------------------------
/proc/mpp/<nazov_disk_pola>/controller<A/B>/<low-level-driver><HCT#>Instalacia
Rozbalime potrebny ovladac
--------------------------------------------------------------------------------------------------------
# tar -xvzf ./rdac-LINUX-09.03.0B05.0331-source.tar.gz
Prekompilujeme ovladac
--------------------------------------------------------------------------------------------------------
# cd linuxrdac-09.03.0B05.0331/
# make clean
# make
# make install
Upravime GRUB a pridame zaznam pre "nove" jadro + initrd
--------------------------------------------------------------------------------------------------------
# chattr -i /boot/grub/menu.lst
# nano /boot/grub/menu.lst
--------------------------------------------------------------------------------------------------------
title Red Hat Enterprise Linux AS (2.6.9-89.0.9.ELsmp) with MPP support
root (hd0,0)
kernel /vmlinuz-2.6.9-89.0.9.ELsmp ro root=/dev/VG_ROOT/LV_ROOT rhgb quiet
initrd /mpp-2.6.9-89.0.9.ELsmp.img
--------------------------------------------------------------------------------------------------------
# chattr +i /boot/grub/menu.lst
Rebootneme server
--------------------------------------------------------------------------------------------------------
# reboot
Check that the MPP kernel module is loaded
--------------------------------------------------------------------------------------------------------
# lsmod | grep mpp
--------------------------------------------------------------------------------------------------------
mppVhba 138656 0
mppUpper 139132 1 mppVhba
scsi_mod 145297 8 mppVhba,usb_storage,cciss,qla2xxx,scsi_transport_fc,mppUpper,sg,sd_mod
Check that the LUN is mapped
--------------------------------------------------------------------------------------------------------
# ls -lR /proc/mpp
--------------------------------------------------------------------------------------------------------
/proc/mpp:
total 0
dr-xr-xr-x 4 root root 0 Aug 13 15:56 DS5300-POLUS
/proc/mpp/DS5300-POLUS:
total 0
dr-xr-xr-x 4 root root 0 Aug 13 15:56 controllerA
dr-xr-xr-x 4 root root 0 Aug 13 15:56 controllerB
-rw-r--r-- 1 root root 0 Aug 13 15:56 virtualLun1
-rw-r--r-- 1 root root 0 Aug 13 15:56 virtualLun2
-rw-r--r-- 1 root root 0 Aug 13 15:56 virtualLun3
/proc/mpp/DS5300-POLUS/controllerA:
total 0
dr-xr-xr-x 2 root root 0 Aug 13 15:56 qla2xxx_h0c0t0
dr-xr-xr-x 2 root root 0 Aug 13 15:56 qla2xxx_h1c0t0
/proc/mpp/DS5300-POLUS/controllerA/qla2xxx_h0c0t0:
total 0
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN1
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN2
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN3
/proc/mpp/DS5300-POLUS/controllerA/qla2xxx_h1c0t0:
total 0
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN1
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN2
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN3
/proc/mpp/DS5300-POLUS/controllerB:
total 0
dr-xr-xr-x 2 root root 0 Aug 13 15:56 qla2xxx_h0c0t1
dr-xr-xr-x 2 root root 0 Aug 13 15:56 qla2xxx_h1c0t1
/proc/mpp/DS5300-POLUS/controllerB/qla2xxx_h0c0t1:
total 0
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN1
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN2
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN3
/proc/mpp/DS5300-POLUS/controllerB/qla2xxx_h1c0t1:
total 0
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN1
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN2
-rw-r--r-- 1 root root 0 Aug 13 15:56 LUN3
FINITO
--------------------------------------------------------------------------------------------------------
;-)Pouzitie (priklady)
mppUtil
================================================================================================================
Showing every debug message for synchronous I/O activity, logging to the console
--------------------------------------------------------------------------------------------------------
# mppUtil -d 0x80200008
Vypnutie vsetkych debug sprav
--------------------------------------------------------------------------------------------------------
# mppUtil -d 0x0
Zobrazenie failoveru kontrolera a fatalne chyby
--------------------------------------------------------------------------------------------------------
# mppUtil -e 4
Showing the RDAC driver's internal information for the target with ID 1
--------------------------------------------------------------------------------------------------------
# mppUtil -g 1
Finding the failed controller that has come back online
--------------------------------------------------------------------------------------------------------
# mppUtil -s failback
A listing of every disk array this server can see
--------------------------------------------------------------------------------------------------------
# mppUtil -a
Showing the RDAC driver's internal information for the disk array named "pole"
--------------------------------------------------------------------------------------------------------
# mppUtil -a "array_name"
Showing every available special option (feature_options)
--------------------------------------------------------------------------------------------------------
# mppUtil -o
Setting the special option "ErrorLevel" to "0x2"
--------------------------------------------------------------------------------------------------------
# mppUtil -o "ErrorLevel=0x2"
Zobrazenie behovych (run-time) informacii kontrolera a ciest k serveru
--------------------------------------------------------------------------------------------------------
# mppUtil -S
mppUpdate
================================================================================================================
Odstranenie disk. pola s nazvom 'DS5300-SITEL' zo suboru '/var/mpp/devicemapping'
--------------------------------------------------------------------------------------------------------
# mppUpdate -d DS5300-SITEL
Clearing every disk array entry out of "/var/mpp/devicemapping"
--------------------------------------------------------------------------------------------------------
# mppUpdate -c
mppBusRescan (hot_add)
================================================================================================================
Najdenie (scan) noveho diskoveho pola na fyzickej zbernici (SAN)
--------------------------------------------------------------------------------------------------------
# mppBusRescan
# mppBusRescan -s
# mppBusRescan --scan
alebo
# hot_add
# hot_add -s
# hot_add --scan
Odstranenie (delete) odmapovaneho (nenamapovaneho) alebo odpojeneho zariadenia
--------------------------------------------------------------------------------------------------------
# hot_add -d
# hot_add --delete
alebo
# mppBusRescan -d
# mppBusRescan --delete
Kontrola ci sa zmenilo zariadenie/LUN. Ak nastala zmena aktualizuje sa RDAC s novym remapovanym
zariadenim alebo LUNom
--------------------------------------------------------------------------------------------------------
# mppBusRescan -u
# mppBusRescan --update
Kontrola ci sa zmenila kapacita (capacity) zariadenia/LUNu. Ak nastala zmena aktualizuje sa kapacita zariadenia/LUNu
--------------------------------------------------------------------------------------------------------
# mppBusRescan -c
# mppBusRescan --capacity
lsvdev
================================================================================================================
Show the basic LUN mapping
--------------------------------------------------------------------------------------------------------
# /opt/mpp/lsvdev
--------------------------------------------------------------------------------------------------------
Array Name Lun sd device
-------------------------------------
DS5300-POLUS 1 -> /dev/sda
DS5300-POLUS 2 -> /dev/sdb
DS5300-SITEL 1 -> /dev/sdc
DS5300-SITEL 2 -> /dev/sddPouzitie (priklady s vystupmi)
mppUtil
================================================================================================================
Showing every disk array this machine can see
--------------------------------------------------------------------------------------------------------
# mppUtil -a
--------------------------------------------------------------------------------------------------------
Hostname = simspcea
Domainname = (none)
Time = GMT 09/07/2010 08:54:48
---------------------------------------------------------------
Info of Array Module's seen by this Host.
---------------------------------------------------------------
ID WWN Type Name
---------------------------------------------------------------
0 600a0b800047dfb8000000004b591ee7 FC DS5300-POLUS
1 600a0b800047dd34000000004b612dbb FC DS5300-SITEL
---------------------------------------------------------------
Showing every special option
--------------------------------------------------------------------------------------------------------
# mppUtil -o
--------------------------------------------------------------------------------------------------------
Current Default Minimum Maximum
Variable Options Value Value Value Value
---------------------------------------- ---------- ---------- ---------- ----------
DebugLevel 0x0 0x0 0x0 0xffffffff
ErrorLevel 0x3 0x3 0x0 0x5
DisableLUNRebalance 0x0 0x0 0x0 0x4
LoadBalancePolicy 0x1 0x1 0x0 0x2
ClassicModeFailover 0x0 0x0 0x0 0x1
BusResetTimeout 0x96 0x0 0x0 0x12c
Action Options
---------------------------------------
No actions defined
--------------------------------------------------------------------------------------------------------
Setting the variable "ErrorLevel" to 0x2 (not a lasting change, in memory only; after a restart the value
goes back to what it was)
--------------------------------------------------------------------------------------------------------
# mppUtil -o ErrorLevel=0x2
Setting the variable "ErrorLevel" to 0x2 and saving it (a lasting change [SaveSettings])
--------------------------------------------------------------------------------------------------------
# mppUtil -o ErrorLevel=0x2,SaveSettings
# mppUpdate
Showing the RDAC driver's internal information for one disk array (named by ID [mppUtil -a])
--------------------------------------------------------------------------------------------------------
# mppUtil -g 0
--------------------------------------------------------------------------------------------------------
Hostname = simspcea
Domainname = (none)
Time = GMT 09/07/2010 08:58:08
MPP Information:
----------------
ModuleName: DS5300-POLUS SingleController: N
VirtualTargetID: 0x000 ScanTriggered: N
ObjectCount: 0x000 AVTEnabled: Y
WWN: 600a0b800047dfb8000000004b591ee7 RestoreCfg: N
ModuleHandle: none Page2CSubPage: Y
FirmwareVersion: 7.60.28.xx FailoverMethod: C
ScanTaskState: 0x00000000
LBPolicy: LeastQueueDepth
Controller 'A' Status:
-----------------------
ControllerHandle: none ControllerPresent: Y
UTMLunExists: N Failed: N
NumberOfPaths: 2 FailoverInProg: N
ServiceMode: N
Path #1
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77000000 (hostId: 0, channelId: 0, targetId: 0)
Path #2
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77010000 (hostId: 1, channelId: 0, targetId: 0)
Controller 'B' Status:
-----------------------
ControllerHandle: none ControllerPresent: Y
UTMLunExists: N Failed: N
NumberOfPaths: 2 FailoverInProg: N
ServiceMode: N
Path #1
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77000001 (hostId: 0, channelId: 0, targetId: 1)
Path #2
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77010001 (hostId: 1, channelId: 0, targetId: 1)
Lun Information
---------------
Lun #1 - WWN: 600a0b800047dfb80000abba4c2c3c1c
----------------
LunObject: present CurrentOwningPath: B
RemoveEligible: N BootOwningPath: B
NotConfigured: N PreferredPath: B
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: N
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 2 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Path #2: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 2 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Path #2: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Lun #2 - WWN: 600a0b800047dd6c0000ae344c6b9649
----------------
LunObject: present CurrentOwningPath: A
RemoveEligible: N BootOwningPath: A
NotConfigured: N PreferredPath: A
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: N
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 2 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Path #2: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 2 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Path #2: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
--------------------------------------------------------------------------------------------------------
A live report on the state of the controllers and paths for each LUN
--------------------------------------------------------------------------------------------------------
# mppUtil -S
--------------------------------------------------------------------------------------------------------
H2C0T0 Active Active DS5300-POLUS
H0C0T0L001 Up H0C0T1L001 Up
H1C0T0L001 Up H1C0T1L001 Up
H0C0T0L002 Up H0C0T1L002 Up
H1C0T0L002 Up H1C0T1L002 Up
Missing Arrays
There are no missing arrays
--------------------------------------------------------------------------------------------------------
Najdenie noveho diskoveho pola
--------------------------------------------------------------------------------------------------------
# mppBusRescan
--------------------------------------------------------------------------------------------------------
scan qla2 HBA host /sys/class/scsi_host/host1...
found 1:0:2:1
found 1:0:2:2
found 1:0:3:1
found 1:0:3:2
scan qla2 HBA host /sys/class/scsi_host/host0...
found 0:0:2:1
found 0:0:2:2
found 0:0:3:1
found 0:0:3:2
run /usr/sbin/mppUtil -s busscan...
scan mpp virtual host /sys/class/scsi_host/host2...
found 2:0:1:1->/dev/sdc
found 2:0:1:2->/dev/sdd
--------------------------------------------------------------------------------------------------------
A live report on the state of the controllers and paths for each LUN (after a new disk array has been found)
--------------------------------------------------------------------------------------------------------
# mppUtil -S
--------------------------------------------------------------------------------------------------------
H2C0T0 Active Active DS5300-POLUS
H0C0T0L001 Up H0C0T1L001 Up
H1C0T0L001 Up H1C0T1L001 Up
H0C0T0L002 Up H0C0T1L002 Up
H1C0T0L002 Up H1C0T1L002 Up
H2C0T1 Active Active DS5300-SITEL
H1C0T2L001 Up H1C0T3L001 Up
H0C0T3L001 Up H0C0T2L001 Up
H1C0T2L002 Up H1C0T3L002 Up
H0C0T3L002 Up H0C0T2L002 Up
Missing Arrays
There are no missing arrays
--------------------------------------------------------------------------------------------------------
mppUpdate
================================================================================================================
Vymazanie disk. pola s nazvom 'DS5300-SITEL' zo suboru /var/mpp/devicemapping
--------------------------------------------------------------------------------------------------------
# mppUpdate -d DS5300-SITEL
--------------------------------------------------------------------------------------------------------
The module DS5300-SITEL was removed from the MPP persistence mapping list (/var/mpp/devicemapping).
Detected 2 QLogic Host Adapter Port(s) on the system
Creating new MPP initrd image...
--------------------------------------------------------------------------------------------------------
mppBusRescan (hot_add)
================================================================================================================
Najdenie (scan) noveho diskoveho pola na fyzickej zbernici (SAN)
--------------------------------------------------------------------------------------------------------
# mppBusRescan -s
--------------------------------------------------------------------------------------------------------
Starting new devices re-scan...
scan mptscsih HBA host /sys/class/scsi_host/host5...
no new device found
scan mptscsih HBA host /sys/class/scsi_host/host4...
no new device found
scan mptscsih HBA host /sys/class/scsi_host/host3...
no new device found
scan mptscsih HBA host /sys/class/scsi_host/host2...
found 2:0:0:7
found 2:0:0:9
found 2:0:1:7
found 2:0:1:9
found 2:0:2:7
found 2:0:2:9
found 2:0:3:7
found 2:0:3:9
scan mptscsih HBA host /sys/class/scsi_host/host1...
no new device found
scan mptscsih HBA host /sys/class/scsi_host/host0...
no new device found
run /usr/sbin/mppUtil -s busscan...
scan mpp virtual host /sys/class/scsi_host/host6...
found 6:0:0:7->/dev/sdo
found 6:0:0:9->/dev/sdp
--------------------------------------------------------------------------------------------------------
Odstranenie (delete) odmapovaneho (nenamapovaneho) alebo odpojeneho zariadenia
--------------------------------------------------------------------------------------------------------
# mppBusRescan -d
--------------------------------------------------------------------------------------------------------
Searching for unmapped or disconnected MPP devices.
Found 6:0:0:7/sdo as unmapped/disconnected device. Device removed.
Found 6:0:0:9/sdp as unmapped/disconnected device. Device removed.
--------------------------------------------------------------------------------------------------------
Kontrola ci sa zmenilo zariadenie/LUN. Ak nastala zmena aktualizuje sa RDAC z novym remapovanym
zariadenim alebo LUNom
--------------------------------------------------------------------------------------------------------
# mppBusRescan -u
--------------------------------------------------------------------------------------------------------
Searching for remapped MPP devices.
Found 6:0:0:7/sdo as remapped device. Device updated.
Found 6:0:0:9/sdp as remapped device. Device updated.
--------------------------------------------------------------------------------------------------------
Kontrola ci sa zmenila kapacita (capacity) zariadenia/LUNu. Ak nastala zmena aktualizuje sa kapacita
zariadenia/LUNu
--------------------------------------------------------------------------------------------------------
# mppBusRescan -c
--------------------------------------------------------------------------------------------------------
Searching for MPP device with size modified
Found 6:0:0:7/sdo with size modified. Device size updated.
Found 6:0:0:9/sdp with size modified. Device size updated.
--------------------------------------------------------------------------------------------------------PROBLEMY
[1] Problem
The server sees two disk arrays, with the "main" one mirrored to the "standby" one (a problem mainly if LVM is in use,
where devlabel cannot be used to identify the device being attached)
================================================================================================================
The faulty state looks like this
--------------------------------------------------------------------------------------------------------
# mppUtil -a
--------------------------------------------------------------------------------------------------------
Hostname = simspcea
Domainname = (none)
Time = GMT 09/08/2010 10:50:35
---------------------------------------------------------------
Info of Array Module's seen by this Host.
---------------------------------------------------------------
ID WWN Type Name
---------------------------------------------------------------
0 600a0b800047dfb8000000004b591ee7 FC DS5300-POLUS
1 600a0b800047dd34000000004b612dbb FC DS5300-SITEL
---------------------------------------------------------------
The faulty state looks like this
--------------------------------------------------------------------------------------------------------
# cat ./devicemapping
--------------------------------------------------------------------------------------------------------
0:DS5300-POLUS
1:DS5300-SITEL
The faulty state looks like this
--------------------------------------------------------------------------------------------------------
# ls -l
--------------------------------------------------------------------------------------------------------
-rw------- 1 root root 361 Sep 7 11:57 600a0b800047dd34000000004b612dbb.wwn
-rw------- 1 root root 361 Sep 7 11:57 600a0b800047dfb8000000004b591ee7.wwn
-rw------- 1 root root 15 Sep 8 13:26 devicemapping
The state we want looks like this
--------------------------------------------------------------------------------------------------------
# mppUtil -a
--------------------------------------------------------------------------------------------------------
Hostname = simspcea
Domainname = (none)
Time = GMT 09/08/2010 10:50:35
---------------------------------------------------------------
Info of Array Module's seen by this Host.
---------------------------------------------------------------
ID WWN Type Name
---------------------------------------------------------------
0 600a0b800047dfb8000000004b591ee7 FC DS5300-POLUS
---------------------------------------------------------------
The state we want looks like this
--------------------------------------------------------------------------------------------------------
# cat ./devicemapping
--------------------------------------------------------------------------------------------------------
0:DS5300-POLUS
The state we want looks like this
--------------------------------------------------------------------------------------------------------
# ls -l
--------------------------------------------------------------------------------------------------------
-rw------- 1 root root 361 Sep 7 11:57 600a0b800047dfb8000000004b591ee7.wwn
-rw------- 1 root root 15 Sep 8 13:26 devicemapping
[1] Riesenie
================================================================================================================
[1] Vypneme pokusy o objavenie novych poli pri boote
[2] Check that mpp is turned off
--------------------------------------------------------------------------------------------------------
[1]# chkconfig mpp off
[2]# chkconfig --list | grep mpp
Odstranime diskove pole s nazvom 'DS5300-SITEL' z persistentneho mapovania
--------------------------------------------------------------------------------------------------------
# mppUpdate -d DS5300-SITEL
--------------------------------------------------------------------------------------------------------
The module DS5300-SITEL was removed from the MPP persistence mapping list (/var/mpp/devicemapping).
Detected 2 QLogic Host Adapter Port(s) on the system
Creating new MPP initrd image...
Odstranime WWN zaznam pre disk. pole 'DS5300-SITEL'
--------------------------------------------------------------------------------------------------------
# rm ./600a0b800047dd34000000004b612dbb.wwn
Aktualizujeme initrd obraz
--------------------------------------------------------------------------------------------------------
# mppUpdate
Restart the machine so the settings take effect
--------------------------------------------------------------------------------------------------------
# init 6
Check that the server sees only the one disk array it should
--------------------------------------------------------------------------------------------------------
# mppUtil -a
# cat ./devicemapping
[2] Problem
Potreba odstranenia LUNu (hot remove LUN)
================================================================================================================
[2] Riesenie
================================================================================================================
Odstranienie LUNu (hot remove LUN)
--------------------------------------------------------------------------------------------------------
# echo "1" > /sys/block/sdc/device/delete
# echo "1" > /sys/block/sdd/device/delete
where sd* is the disk device we want to remove from the system.
Odstranienie LUNu (hot remove LUN)
--------------------------------------------------------------------------------------------------------
# echo 1 > /sys/class/scsi_device/h:c:t:l/device/delete
h is the HBA number
c is the channel on the HBA
t is the SCSI target ID
l is the LUN
The device name, the HBA number, the HBA channel, the SCSI target and the LUN are found like this
--------------------------------------------------------------------------------------------------------
# ls -l /dev/disk/by-*
ENGLISH POPIS
--------------------------------------------------------------------------------------------------------
Take the path offline using echo offline > /sys/block/sda/device/state.
This will cause any subsequent IO sent to the device on this path to be failed immediately.
Device-mapper-multipath will continue to use the remaining paths to the device.
Kto pouziva co? (miniscript)
--------------------------------------------------------------------------------------------------------
for DSK in `ls /sys/block/`
do
if [ -e "/sys/block/${DSK}/device/vendor" ]
then
VENDOR=`cat /sys/block/${DSK}/device/vendor`
MODEL=`cat /sys/block/${DSK}/device/model`
echo “${DSK} ${VENDOR} ${MODEL}
fi
done
Zoznam ciest veducich k disk. polu s ID 1 (DS5300-SITEL)
--------------------------------------------------------------------------------------------------------
# mppUtil -g 1 | grep PathId
--------------------------------------------------------------------------------------------------------
PathId: 77000002 (hostId: 0, channelId: 0, targetId: 2)
PathId: 77010002 (hostId: 1, channelId: 0, targetId: 2)
PathId: 77000003 (hostId: 0, channelId: 0, targetId: 3)
PathId: 77010003 (hostId: 1, channelId: 0, targetId: 3)
Zmazeme vsetky cesty k disk. polu s ID 1 (DS5300-SITEL)
--------------------------------------------------------------------------------------------------------
# echo 1>/sys/bus/scsi/devices/0\:0\:2\:1/delete
# echo 1>/sys/bus/scsi/devices/0\:0\:2\:2/delete
# echo 1>/sys/bus/scsi/devices/1\:0\:2\:1/delete
# echo 1>/sys/bus/scsi/devices/1\:0\:2\:2/delete
# echo 1>/sys/bus/scsi/devices/0\:0\:3\:1/delete
# echo 1>/sys/bus/scsi/devices/0\:0\:3\:2/delete
# echo 1>/sys/bus/scsi/devices/1\:0\:3\:1/delete
# echo 1>/sys/bus/scsi/devices/1\:0\:3\:2/deleteLINKz *****
****** Sun StorageTek RDAC Multipath Failover Driver for Linux OS
--------------------------------------------------------------------------------------------------------
http://docs.sun.com/source/820-4738-13/chapsing.html
****** SCSI - Hot add, remove, rescan of SCSI devices
--------------------------------------------------------------------------------------------------------
http://www.ibm.com/developerworks/wikis/display/linuxp/scsi+-+hot+add,+remove,+rescan+of+scsi+devices
****** Online Storage Guide
--------------------------------------------------------------------------------------------------------
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/html/Online_Storage_Reconfiguration_Guide/index.html
****** LVM, Sysfs and Me
--------------------------------------------------------------------------------------------------------
http://ukuug.org/events/spring2009/programme/lvm-sysfs-me.pdf
XenServer und storage
--------------------------------------------------------------------------------------------------------
http://www.slideshare.net/ADannbacher/xenserver-und-storage
IBM System Storage DS4000/DS5000 Linux RDAC multipath driver package version 09.03.0B05.0331 FOR USE IN
LINUX 2.6 KERNEL
--------------------------------------------------------------------------------------------------------
http://www.lsi.com/rdac/ibm_sw_ds4krdac_09.03.0B05.0331_linux_anycpu.txt
Precitat Readme.txt v 'rdac-LINUX-09.03.0B05.0331-source.tar.gz'
--------------------------------------------------------------------------------------------------------
Readme.txt
DS4000 Implementation Cookbook
--------------------------------------------------------------------------------------------------------
http://utcc.utoronto.ca/ftp/docs/IBM/Storage%20Servers/DS4000%20Implementation%20Cookbook%20v1.0.pdf
Post-Installation Tasks
Dell™ PowerVault™ Modular Disk 3000 Systems Installation Guide
--------------------------------------------------------------------------------------------------------
http://support.dell.com/support/edocs/systems/md3000/en/2ndgen/ig/html/postinst.htm
Engenio Linux RDAC Driver Package for IBM DS4000/DS5000 Storage Subsystems
--------------------------------------------------------------------------
http://www.lsi.com/rdac/ds4000.html
Engenio Linux RDAC Driver Package for IBM DS3000 Storage Subsystems
--------------------------------------------------------------------------------------------------------
http://www.lsi.com/rdac/ds3000.html
multipath-tools
--------------------------------------------------------------------------------------------------------
http://christophe.varoqui.free.fr/
The Linux SCSI Generic (sg) Driver
--------------------------------------------------------------------------------------------------------
http://sg.danny.cz/sg/
Guide to the implementation of SCSI in Linux 2.6.X kernel
--------------------------------------------------------------------------------------------------------
http://devresources.linuxfoundation.org/dev/opendrivers/documentation/scsi_design_analysis_2_6_v2.pdfLINKz ***
Blades and external storage: Set up a fault-tolerant environment
How to set up an IBM BladeCenter HS21/LS21 server to boot from external DS3200 SAS storage
--------------------------------------------------------------------------------------------------------
http://www.ibm.com/developerworks/linux/library/l-sasboot/
Multipath Implementations in XenServer
--------------------------------------------------------------------------------------------------------
http://support.citrix.com/article/CTX121364
RDAC capable of hot removing unmapped LUNs?
--------------------------------------------------------------------------------------------------------
http://arstechnica.com/civis/viewtopic.php?f=21&t=34163
Using RDAC multipath driver
--------------------------------------------------------------------------------------------------------
http://www.redbooks.ibm.com/redbooks/SG246700/wwhelp/wwhimpl/js/html/wwhelp.htmLINKz - NEW
Linux on System z - Device Drivers, Features, and Commands - Linux Kernel 2.6 - Development stream
--------------------------------------------------------------------------------------------------------
http://public.dhe.ibm.com/software/dw/linux390/docu/l26ddd02.pdf