Linux Storage 09 - RDAC installation and operation
Linux Storage Solution · Previous: RDAC multipath driver basics · Next: RDAC problems and LUN removal
The RDAC driver did not come with the distribution. It was a source tarball from the array vendor, built against the running kernel on mppsrv01, and it needed its own initrd and a boot entry of its own, because the whole point of mppUpper is to be loaded before the QLogic driver. This Article follows my notes through that installation on Red Hat Enterprise Linux AS 4 and then through the everyday commands: listing arrays, reading the driver's internal state, finding a new array on the SAN, and keeping the persistent mapping in order. Everything ran as root on mppsrv01. The complete command sets with their outputs are five Config documents, linked where they belong.
Building the driver
The package was rdac-LINUX-09.03.0B05.0331-source.tar.gz. It was unpacked, and in the directory linuxrdac-09.03.0B05.0331/ the usual three steps ran:
$ make clean $ make $ make install
The notes keep no output of the build, and no Readme.txt of the package. The build needs the headers of the running kernel, as any out-of-tree module does; that the host had them installed is implied by the build succeeding, not written down. What make install left behind is visible in what follows: the two modules, the tools mppUtil, mppUpdate, mppBusRescan and /opt/mpp/lsvdev, the init script /etc/init.d/mpp, and an initrd /boot/mpp-2.6.9-89.0.9.ELsmp.img. That the initrd came from make install is my inference; the notes only show that it existed when GRUB was edited, and that every later mppUpdate prints "Creating new MPP initrd image...". The commands are a Config document: RDAC installation commands.
A boot entry with the MPP initrd
The stock initrd of Red Hat loads qla2xxx and the HBAs publish their LUNs. The MPP initrd, as I understand it, loads mppUpper first, so the LUNs the HBAs find are held back and only the virtual adapter's disks reach the system. The kernel stays the same; only the initrd changes. A new stanza was therefore added to /boot/grub/menu.lst, titled "Red Hat Enterprise Linux AS (2.6.9-89.0.9.ELsmp) with MPP support", whose one line that matters is:
initrd /mpp-2.6.9-89.0.9.ELsmp.img
The notes call it the entry for the "new" kernel, but the kernel line is the stock one down to rhgb quiet and the root on the LVM volume LV_ROOT; the "new" part is the last line. menu.lst was immutable on this host, so the edit was wrapped in chattr -i /boot/grub/menu.lst and chattr +i /boot/grub/menu.lst, with nano in between. Whether the entry became GRUB's default is not in the notes; what is in them is that after reboot the modules were loaded, so this entry is what booted. The stanza is a Config document: GRUB menu.lst stanza for the MPP initrd.
flowchart TB a["unpack rdac-LINUX-09.03.0B05.0331-source.tar.gz"] b["make clean, make, make install"] c["chattr -i menu.lst"] d["add the stanza with mpp-2.6.9-89.0.9.ELsmp.img"] e["chattr +i menu.lst"] f["reboot"] g["lsmod shows mppVhba and mppUpper"] h["ls -lR /proc/mpp shows the array, controllers, paths and LUNs"] a --> b --> c --> d --> e --> f --> g --> h
The two checks after the boot
The first check was that the modules were there:
$ lsmod | grep mpp
output 3 lines
mppVhba 138656 0 mppUpper 139132 1 mppVhba scsi_mod 145297 8 mppVhba,usb_storage,cciss,qla2xxx,scsi_transport_fc,mppUpper,sg,sd_mod
mppUpper is used by mppVhba, and scsi_mod is used by both of them next to qla2xxx, sd_mod and sg, which is the stack of the previous Article in one line. The second check was ls -lR /proc/mpp, which showed one array, DS5300-SITEA, with controllerA and controllerB, under controller A the path directories qla2xxx_h0c0t0 and qla2xxx_h1c0t0, under controller B qla2xxx_h0c0t1 and qla2xxx_h1c0t1, and in each of the four path directories the entries LUN1, LUN2 and LUN3; the array directory itself held virtualLun1, virtualLun2 and virtualLun3. The notes end the installation there with the author's FINITO and ;-).
Two things in that listing. The standby array is absent, which fits the later mppBusRescan that finds it, though the notes do not say whether it was not zoned yet or simply not scanned. And there are three LUNs, where the .wwn files, lsvdev, mppUtil -S and mppUtil -g 0 all have two per array. The listing is dated 13 August, everything else 7 and 8 September, so a LUN may have been unmapped in between; the notes do not say, and I report the difference rather than explain it.
mppUtil
mppUtil is the driver's control tool. The notes list its options first without output and then run five of them with output; both lists are in mppUtil commands. mppUtil -a lists the arrays the host sees, with an ID, the array WWN, the transport and the name. The output opens with the host name, the domain name and a time stamp, GMT 09/07/2010 08:54:48, and frames the table with rule lines; here is the table alone:
output 4 lines
ID WWN Type Name --------------------------------------------------------------- 0 600a0b800012345600000000aabbcc01 FC DS5300-SITEA 1 600a0b800012345700000000aabbcc02 FC DS5300-SITEB
600a0b80…) on this page are made-up values of the right shape.mppUtil -o lists the six driver variables that can be changed at run time, each with its current, default, minimum and maximum value:
| Variable | Current | Default | Minimum | Maximum |
|---|---|---|---|---|
DebugLevel | 0x0 | 0x0 | 0x0 | 0xffffffff |
ErrorLevel | 0x3 | 0x3 | 0x0 | 0x5 |
DisableLUNRebalance | 0x0 | 0x0 | 0x0 | 0x4 |
LoadBalancePolicy | 0x1 | 0x1 | 0x0 | 0x2 |
ClassicModeFailover | 0x0 | 0x0 | 0x0 | 0x1 |
BusResetTimeout | 0x96 | 0x0 | 0x0 | 0x12c |
All six are directives of /etc/mpp.conf, and only BusResetTimeout is off its default, at 150 against a default of 0, which is what the file says too. Setting one of them has two forms. mppUtil -o ErrorLevel=0x2 changes the value in memory only; after a restart it is back to what it was. mppUtil -o ErrorLevel=0x2,SaveSettings saves it, and mppUpdate must follow to rebuild the initrd with the saved value. The notes give both as examples, without output.
mppUtil -g <ID> dumps the driver's internal information for one array, named by the ID from -a. The usage list says -g 1 shows "the target with ID 1"; the example runs -g 0, the main array. The output has three parts. "MPP Information" describes the array as a module: name, VirtualTargetID: 0x000, WWN, FirmwareVersion: 7.60.28.xx, FailoverMethod: C, AVTEnabled: Y, LBPolicy: LeastQueueDepth. The AVTEnabled: Y is worth a second look, because the vendor text in the previous Article says AVT must be disabled for RDAC; the notes do not comment on it and I cannot resolve it. "Controller 'A' Status" and "Controller 'B' Status" each report ControllerPresent: Y, Failed: N, NumberOfPaths: 2 and then the two paths, each PathState: OPTIMAL with a path identifier that decodes itself:
output 5 lines
Path #1
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77000000 (hostId: 0, channelId: 0, targetId: 0)The second byte of the PathId is the host and the last byte the target, so controller A's paths are 77000000 and 77010000 (hosts 0 and 1, target 0) and controller B's 77000001 and 77010001 (target 1); what the leading 77 means the notes do not say. "Lun Information" then lists each LUN with its WWN, CurrentOwningPath, BootOwningPath and PreferredPath, DevState: OPTIMAL, and under each controller the two LUN path objects. LUN 1 is owned by controller B and LUN 2 by controller A, all three ownership fields agreeing, which I read as the array's preferred-controller assignment being honoured, though the notes do not say so.
mppUtil -S is the short live report: one line per array with its virtual address, the two controller states and the name, then one line per LUN and path pair, H0C0T0L001 Up. The first report in the notes shows H2C0T0 Active Active DS5300-SITEA alone, with eight Up entries, four paths times two LUNs. The notes place this report before the mppBusRescan of the next section; the mppUtil -a above it, time-stamped 08:54 GMT, already lists both arrays, so the examples were not kept in the order they were run.
Finding the second array
mppBusRescan, or hot_add, which is a link to it, finds a new array or LUN without a reboot. The notes run it when the standby array appears on the SAN, and its output tells the whole story in fourteen lines. The first five:
output 5 lines
scan qla2 HBA host /sys/class/scsi_host/host1...
found 1:0:2:1
found 1:0:2:2
found 1:0:3:1
found 1:0:3:2Then the same five lines for host0, with the addresses 0:0:2:1 to 0:0:3:2, and the last four:
output 4 lines
run /usr/sbin/mppUtil -s busscan...
scan mpp virtual host /sys/class/scsi_host/host2...
found 2:0:1:1->/dev/sdc
found 2:0:1:2->/dev/sddBoth HBAs find two new targets, 2 and 3, the two controllers of DS5300-SITEB, each with LUNs 1 and 2: eight physical devices. Then the tool runs mppUtil -s busscan, which has the driver build the virtual LUNs, and scans the virtual host host2, where the new array is target 1 and its two LUNs become /dev/sdc and /dev/sdd. A second mppUtil -S afterwards shows both arrays, H2C0T1 Active Active DS5300-SITEB with its eight Up entries under the first array's. The tool also has -d to remove unmapped or disconnected devices, -u to pick up a remapped LUN and -c to refresh a changed capacity, and the notes keep an output for each. Those four outputs, though, are not from this host: they show six mptscsih adapters host0 to host5, a virtual host host6 and disks sdo and sdp, where mppsrv01 has two qla2xxx adapters and host2. They were kept from another machine or from documentation, the notes do not say which, and I show them for the shape of the output only. All of it is in mppBusRescan commands.
lsvdev and mppUpdate
/opt/mpp/lsvdev prints the mapping that matters day to day, array and LUN to disk name:
output 6 lines
Array Name Lun sd device ------------------------------------- DS5300-SITEA 1 -> /dev/sda DS5300-SITEA 2 -> /dev/sdb DS5300-SITEB 1 -> /dev/sdc DS5300-SITEB 2 -> /dev/sdd
The virtual disks carry the plain sd names and the four physical paths behind each virtual disk have none, which is the visible result of mppUpper holding the HBAs back. mppUpdate maintains the persistent mapping: mppUpdate -d DS5300-SITEB removes that array from /var/mpp/devicemapping, mppUpdate -c clears every array from it, and mppUpdate alone rebuilds the initrd. The output of the first is three lines, "The module DS5300-SITEB was removed from the MPP persistence mapping list (/var/mpp/devicemapping).", "Detected 2 QLogic Host Adapter Port(s) on the system" and "Creating new MPP initrd image...". It is the heart of the fix in the next Article, and both outputs are in mppUpdate and lsvdev.
What the notes do not hold
No build output, no Readme.txt, no listing of what make install put where. No record of whether the GRUB default was moved, and no /proc/mpp listing after the second array was found. No failover test: every path in every output is OPTIMAL or Up. And no explanation of the three LUNs of 13 August against the two of September.
Two of the claims in this Article the research for this write-up could confirm from the package readme of the same build: hot_add is a symbolic link to mppBusRescan, and a change to /etc/mpp.conf or /var/mpp/devicemapping needs mppUpdate to rebuild the initrd. The readme also names the initrd mpp-<uname -r>.img, so the file name in the GRUB stanza was the driver's convention. The same readme says the driver "doesn't support LUN deletion", which is the background of the next Article. Everything else about the driver is history: see the "Checked against" sections of the five Config documents.