Linux Storage 08 - RDAC multipath driver basics

Linux Storage Solution · Previous: LUNs seen and SCSI bus rescan · Next: RDAC installation and operation

The last three Articles of this Solution leave the 2013 FC SAN behind. They are my notes from September 2010 on a different system: the host mppsrv01 running Red Hat Enterprise Linux AS 4 with kernel 2.6.9-89.0.9.ELsmp, two QLogic HBAs, and two IBM DS5300 arrays, DS5300-SITEA as the main array and DS5300-SITEB as a standby that mirrors it, each with two controllers and two LUNs. Nothing of the 3PAR, the Cisco switches or the blades of Articles 1 to 7 appears here, and the notes hold no cabling schema and no switch configuration for this older system. What they hold is the vendor's multipath driver for those arrays, RDAC, also called MPP: what it is, which files and commands it has, how it was built and installed, and two problems solved with it. This Article is the basics; RDAC installation and operation and RDAC problems and LUN removal follow. I keep them here because the problem they solve is the one Articles 6 and 7 stop short of: a host that sees every LUN four times and needs one device per LUN.

The system

The notes do not say how the second array came to be a mirror of the first, nor how the host was zoned to both; that was somebody else's work and it is missing here.

What RDAC is

The RDAC driver (Redundant Disk Array Controller, also known as the Multi-Path Proxy driver, MPP) adds a new layer to the Linux driver stack, and that layer is what provides multiple paths to the disk array. It is a combination of two drivers. The first, the high-level driver, has the same role as sd (the SCSI disk driver) or sg (the SCSI generic driver): it serves the disk arrays, gives user space an ioctl interface and is responsible for managing the several paths to an array. The second, the virtual HBA driver, is a Linux low-level driver and sits at the same level of the stack as any driver for a QLogic or LSI Fibre Channel HBA.

The notes then quote the vendor documentation, which puts it in terms of modules. When a host has more than one HBA and all of them reach one or more IBM TotalStorage DS4xxx arrays, a multipath driver is required, and for that family it is the RDAC/MPP driver; on AIX the base system already holds it as MPIO, on Linux it has to be downloaded and installed by hand. The two modules are mppUpper, which is loaded before any HBA driver and prevents the HBA from publishing its LUNs to the system, and mppVhba, a virtual HBA that sits on top of the real HBAs, bundles the different paths to the same LUN, organises the traffic to and from it, and presents it to the operating system as a single entity. The same documentation says that RDAC needs a non-failover HBA driver and that auto volume transfer (AVT) must be disabled at the array, which is done by changing the host type from Linux to LNXCL, and it points to the Readme.txt of the package for the details. The notes stop there: the host type change on the arrays is not recorded, the Readme.txt is not kept, and, as the next Article shows, mppUtil -g 0 later reports AVTEnabled: Y without the notes remarking on it.

flowchart TB
  app["File systems, LVM"]
  sd["sd, the SCSI disk driver"]
  upper["mppUpper, high-level driver, loaded before any HBA"]
  mid["SCSI mid-layer, scsi_mod"]
  vhba["mppVhba, virtual HBA, SCSI host2"]
  qla["qla2xxx, QLogic low-level driver"]
  h0["HBA host0"]
  h1["HBA host1"]
  app --> sd
  sd --> mid
  upper --> mid
  mid --> vhba
  mid --> qla
  vhba -. "sits on top of the real HBAs" .-> qla
  qla --> h0
  qla --> h1

The picture is my reading of the two descriptions and of the lsmod output in the next Article, where mppUpper is used by mppVhba and both, together with qla2xxx, sd_mod and sg, sit on scsi_mod: the high-level driver mppUpper beside sd above the mid-layer, the virtual HBA mppVhba beside qla2xxx below it, and the dotted line for the vendor's sentence that the virtual HBA sits on top of the real HBAs. What it comes down to for the operator: the eight physical devices the two HBAs find (two controllers times two LUNs, on each HBA) never become sd disks, because mppUpper holds them back; the only disks the system gets are the virtual ones of host2, one per LUN, and these carry the plain names /dev/sda onwards.

Commands and files

The driver comes with a handful of tools. The notes list them with one line each; the next Article shows them with output.

Three places hold the driver's state. /etc/mpp.conf is the configuration file with the driver's directives; the notes keep it as installed and refer to man RDAC for the meaning of each directive. It is a Config document: mpp.conf. /var/mpp/ holds the persistent configuration of arrays and LUNs, kept by the driver (who writes the files is not in the notes), and /proc/mpp/ the run-time information. Two of the directives matter for reading the rest: MaxLunsPerArray=256 and MaxPathsPerController=4, which fix the shape of the path tables below. One more, S2ToS3Key, holds a key value; the one in the Config document is a placeholder.

The persistent state in /var/mpp

The directory holds one file per array the driver has ever seen, named after the array's WWN with the suffix .wwn, and the file devicemapping, which gives each array its number. The whole listing is a Config document: /var/mpp files. The file of the main array is four lines:

H2C0T0   :Active               :Active               :DS5300-SITEA
maxLun256:CAP 0    :CAP 1    :CAP 2    :CAP 3    :CBP 0    :CBP 1    :CBP 2    :CBP 3    :
L001     :H0C0T0:01H1C0T0:01---------:---------:H0C0T1:01H1C0T1:01---------:---------:
L002     :H0C0T0:02H1C0T0:02---------:---------:H0C0T1:02H1C0T1:02---------:---------:

The notes show the file without a legend, so most of this is my reading, checked against the mppUtil -S and mppUtil -g 0 outputs that use the same notation. The first line is clear from those outputs: H2C0T0 is the array's address on the virtual adapter (host 2, channel 0, target 0), the two Active are the states of controllers A and B, and the last field is the array name. maxLun256 is MaxLunsPerArray. The eight columns CAP 0 to CBP 3 I read as four path slots for controller A and four for controller B, which is MaxPathsPerController=4; the -g 0 output supports that, because there controller A has the paths with target 0 and controller B those with target 1. The rows L001 and L002 are the two LUNs. A cell such as H0C0T0:01 is one path to that LUN: physical host 0, channel 0, target 0, and LUN 01 on that path; the cells are written without a separator, so H0C0T0:01H1C0T0:01 is two of them, and --------- is an empty slot. For the main array that gives the clean picture of four paths per LUN. For the standby array the file has H2C0T1 in the first line and targets 2 and 3 in the cells, in a crossed order that I discuss in the Config document and do not fully understand.

devicemapping is the shortest file of the three, two lines, 0:DS5300-SITEA and 1:DS5300-SITEB. The number is the array ID that mppUtil -a prints and that mppUtil -g takes as its argument, and the file is what mppUpdate -d and mppUpdate -c edit.

flowchart LR
  subgraph host["mppsrv01"]
    h0["host0, QLogic HBA"]
    h1["host1, QLogic HBA"]
    v["host2, mppVhba virtual LUN 1, /dev/sda"]
  end
  subgraph arr["DS5300-SITEA, LUN 1"]
    ca["controller A, target 0"]
    cb["controller B, target 1"]
  end
  h0 -- "H0C0T0" --> ca
  h1 -- "H1C0T0" --> ca
  h0 -- "H0C0T1" --> cb
  h1 -- "H1C0T1" --> cb
  v -.-> h0
  v -.-> h1

The four solid lines are the four paths of LUN 1 as the .wwn file lists them; mppUtil -g 0 shows that controller B is the current and preferred owner of LUN 1 and controller A of LUN 2, so each LUN is normally served through two of its four paths. That /dev/sda is LUN 1 of the main array comes from the lsvdev output in the next Article.

The run-time state in /proc

Every piece of run-time information the driver holds is in the /proc file system. The virtual adapter has an entry under /proc/scsi/mpp/<adapter number>, and the notes warn that the number can differ from system to system, because the SCSI mid-layer assigns it; on this host it was 2. The driver's own tree is /proc/mpp, with one directory per array visible to the server, under it one directory per controller, under that one directory per physical path, and in each of those one entry per LUN. The notes give the paths as four entries, each under its own title; gathered into one list, with the Slovak placeholder for the array name translated:

/proc/mpp/<array name>
/proc/mpp/<array name>/controllerA
/proc/mpp/<array name>/controllerB
/proc/mpp/<array name>/controller<A/B>/<low-level-driver><HCT#>

The low-level driver can be QLogic or Emulex; here it is qla2xxx. HCT is host, channel and target: the host is the number of the low-level driver's SCSI host as the mid-layer assigned it, and the notes leave channel and target without a description. A real path directory on this host is therefore /proc/mpp/DS5300-SITEA/controllerA/qla2xxx_h0c0t0, and it holds LUN1, LUN2 and so on. The array directory also holds one virtualLun<n> entry per LUN the virtual adapter presents. The full listing taken after the installation is in the next Article and in RDAC installation commands, and it has a surprise in it: three LUNs, where every other listing in the notes has two.

What the notes do not hold

The array side is missing entirely: how the LUNs were created and mapped, the host type change to LNXCL, the mirror between the two arrays. There is no multipath.conf, because this host used RDAC and not DM-Multipath, and no comparison between the two. There is no failover test; the paths are all OPTIMAL and Up in every output, and nothing in the notes shows a controller being pulled. And the package Readme.txt, which the vendor text says to read first, was read at the time and not kept.

RDAC today

None of this exists any more. The driver build here, 09.03.0B05.0331, was released on 15 June 2010 for Red Hat Enterprise Linux 4 and SLES 9 and 10; later builds followed (0439 in December 2010, 0455 in June 2011, 0642 named in a 2015 readme), and NetApp, who took over the Engenio array business from LSI in 2011, discontinued RDAC with SANtricity OS 11.25. Its job is done by DM-Multipath with the kernel device handler scsi_dh_rdac, which attaches itself to IBM 1818 devices, the DS5000 family, and the DS5300 itself reached end of support on 31 January 2018. The "Checked against" sections of the Config documents hold the details.