Linux Storage 06 - Linux HBA and SCSI inventory

Linux Storage Solution · Previous: Port security · Next: LUNs seen and SCSI bus rescan

The switch side of the SAN ends with the zoneset active and port security on. This part moves to the host: before trusting a single LUN, I wanted to see the Fibre Channel HBA of HV01 the way Linux sees it, and to know which of its numbers match the switch. Everything the kernel knows about an HBA port is in sysfs, and the notes read it twice, once with cat on the attribute files and once with systool, which reads the same files for every port at once. The point of the exercise is that the identities on the host (port_name, port_id, the remote port's port_name) are the same strings as in the switch's flogi database, the FC alias and the zone of the earlier parts, so a mismatch anywhere in the chain shows up here first. The inventory of the LUNs themselves and the rescans are the next part, LUNs seen and SCSI bus rescan.

The host and its SCSI hosts

HV01 is an HP BL660c Gen8 blade in bay1 of the c7000 enclosure, meant to become a XenServer host; the notes call its task "configure the XenServer server HV01". Its FC adapter is one QLogic QMH2572, a dual-port 8 Gb mezzanine card: one PCI device with two functions, 0000:03:00.0 and 0000:03:00.1, each function a port, each port a SCSI host of its own to the kernel. The driver is qla2xxx 8.05.00.03.55.6-k and the firmware 5.09.00, both printed in the symbolic_name attribute. The blade also has the HP Smart Array P220i controller for its local disks, which the hpsa driver presents as the first SCSI host.

The notes say that the HBAs are usually the last two SCSI hosts, and here they are: ls -ld /sys/class/scsi_host/host/ shows host0 to host2, and ls -ld /sys/class/fc_host/ shows only host1 and host2, the two that are Fibre Channel. Everything in this part ran as root on HV01 and only reads; nothing is configured.

The sysfs classes

Four classes under /sys/class/ describe one FC path, and they are views of one tree under /sys/devices/. For host1 the tree reads: the PCI function, then the SCSI host host1, then the remote port rport-1:0-0 that the port logged into, then the SCSI target target1:0:0 the kernel made out of that remote port, then the SCSI devices 1:0:0:0 to 1:0:0:5 and 1:0:0:254.

flowchart TB
  pci["PCI function 0000:03:00.0"]
  sh["scsi_host/host1"]
  fh["fc_host/host1, WWPN 50:01:43:80:12:34:56:c0"]
  rp["fc_remote_ports/rport-1:0-0, 3PAR port 21:12:00:02:ac:00:ab:cd"]
  tg["fc_transport/target1:0:0"]
  d0["1:0:0:0 VV"]
  d1["1:0:0:1 VV"]
  d2["1:0:0:2 VV"]
  d3["1:0:0:3 VV"]
  d4["1:0:0:4 VV"]
  d5["1:0:0:5 VV"]
  d254["1:0:0:254 SES"]
  pci --> sh
  sh --> fh
  fh --> rp
  rp --> tg
  tg --> d0
  tg --> d1
  tg --> d2
  tg --> d3
  tg --> d4
  tg --> d5
  tg --> d254

The naming is regular: rport-1:0-0 is remote port 0 on channel 0 of host 1, target1:0:0 is host 1, channel 0, target 0, and the devices add the LUN as the fourth number. That the classes are views of one device tree is my reading of the Device path lines systool prints; the notes show the paths and do not comment on them.

What the attributes tell

The cat series reads twelve attributes of host1, four under scsi_host/host1/ and the rest under scsi_host/host1/device/fc_host:host1/. The full systool -c fc_host -v adds the ones that were not read by hand. The whole command set with its output is a Config document, HBA sysfs attributes; the table has every attribute the notes looked at.

Three of these close the loop to the switch parts: port_name is the member of the FC alias HV01 and therefore of the zone HV01-Storage1 (FC aliases, zones and zonesets), port_id 0x2a0000 is the FCID the active zoneset shows next to that pwwn, and node_name is the NODE NAME column of the flogi database. The fabric_name has 0b in its third and fourth hex digits, 11 in decimal, and host2's has 15, which is 21; those are the VSAN numbers of the schema for SW1 and SW2. That the VSAN is encoded there is my inference from the two values, not something the notes say. What tgtid_bind_type and max_npiv_vports mean is my understanding of the FC transport class; the notes read them without comment. The author's one explanatory note is the hex-to-colon conversion of the WWPN.

systool, the second way

systool, from the sysfsutils package, prints a class with all its devices and attributes, so one command answers for both ports what the cat series answered for one. The notes use it in three forms. systool -c fc_host -A <attribute> prints one attribute for host1 and host2, and the notes run it for port_state, port_type, port_name, speed, supported_classes, supported_speeds and symbolic_name. systool -c fc_host -v prints everything, and is where fabric_name, node_name, port_id and tgtid_bind_type come from. The third form takes a class other than fc_host: -c fc_transport -v for the targets and -c fc_remote_ports -v for the array ports.

$ systool -c fc_host -A port_name

Class = "fc_host"
  Class Device = "host1"
    port_name           = "0x50014380123456c0"
    Device = "host1"
  Class Device = "host2"
    port_name           = "0x50014380123456c2"
    Device = "host2"

The -A series and the full fc_host listing are one Config document, systool fc_host; the other two classes are another, systool fc_transport and fc_remote_ports. One more systool form looks at the driver module, systool -m qla2xxx -v, and the notes do not type the module name: they find it with a nested command that runs systool -c fc_transport -v, keeps the PHYSDEVDRIVER lines, cuts after the =, takes the last line and strips the closing quote. It is a long way to spell qla2xxx, but it works on any host with any FC driver, which I suppose was the point.

The remote ports

fc_remote_ports is the host's record of the array. Each HBA port sees exactly one remote port, the 3PAR port it is zoned to, and rport-1:0-0 carries port_name 0x21120002ac00abcd, which is 21:12:00:02:ac:00:ab:cd: the pwwn behind the alias Storage1-C0P0, 3PAR port 0:1:1 by the schema and the alias name, logged into ext1 of SW1 with FCID 0x2a0300, and the port_id of the remote port is that same 0x2a0300. Behind host2 is 0x20110002ac00abcd with port_id 0x580000; that port is on SW2, whose configuration the notes do not hold, and by the schema it is 3PAR port 1:1:1 (the port names themselves decode the other way round, see SCSI addressing and FC names). Both remote ports share the node_name 0x2ff70002ac00abcd, the WWNN of the one array, and both report roles FCP Target, port_state Online, scsi_target_id 0 and supported_classes Class 3.

Two attributes on the remote port decide what happens when a path drops. dev_loss_tmo is 16 and fast_io_fail_tmo is off on both. As I understand the FC transport class, dev_loss_tmo is the number of seconds the kernel waits for a vanished remote port to come back before it deletes the SCSI devices behind it, and fast_io_fail_tmo, when set, fails outstanding I/O after that many seconds so that a multipath layer above can switch paths without waiting for dev_loss_tmo. The notes record the two values and nothing else: they do not say whether 16 and off were defaults or had been set, and no failover was tested. The driver source answers the first half today: the FC transport class defaults to 60 seconds, but qla2xxx overrides it with the HBA's port-down retry count, which with qlport_down_retry 0 comes out of the HBA's NVRAM, so the 16 was most likely written into the card at the factory and nobody on the host chose it. And off is what a host without a multipath daemon shows; Red Hat sets 5 seconds only when multipathd runs.

The qla2xxx module

The systool -m qla2xxx -v listing shows the module live with refcnt 11, its 24 parameters, and the addresses of its loaded sections. The whole listing is a Config document, qla2xxx module listing. No parameter was changed and the notes show no modprobe.d file, so I read the values as the driver's defaults, but the notes do not say so. The meanings below are my understanding of the driver's own parameter descriptions, not something the notes explain.

Three numbers in the listing are worth a second look later. ql2xmaxqdepth 32 is the per-LUN queue depth that a 3PAR host guide would normally have an opinion on; the notes have none. qlport_down_retry 0 and the remote port's dev_loss_tmo 16 are one number seen twice, as explained above, and nobody tested what they do. And the section addresses have eight hex digits, so the control domain kernel was 32-bit, which as I remember it is how XenServer was built at the time; the notes do not say.

What is not in the notes

The inventory stops at the host. There is nothing about the 3PAR side (the host definition, its persona, the virtual volumes and their exports), nothing about HBA BIOS settings, nothing about the second switch, and the XenServer layer above (storage repository, multipathing) is not touched at all; the notes only read what the kernel presents. The LUNs that the two ports see, and how to make the kernel look again, are the next part.

What I would do differently

• Read the vendor's host guide before the inventory. HP's RHEL guide of 2013 already asked for options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=10 ql2xloginretrycount=30, and HPE's current 3PAR and Primera guides ask for qlport_down_retry=14 ql2xmaxqdepth=16. The host ran the driver's defaults, and the notes do not show that anyone looked.
• Know where the timeouts come from. dev_loss_tmo 16 came, most likely, from the card's NVRAM; whether anyone had set it there is not in the notes. Today HPE puts fast_io_fail_tmo 10 and dev_loss_tmo infinity into the multipath configuration for 3PAR, and Red Hat says to ask the vendor before changing either on a host with multipath software.
• Expect fewer tools. The sysfs attributes are unchanged in Linux 7.2 and the QMH2572 is still in the driver's PCI table, but systool has had no upstream release since 2022 and no current vendor guide uses it; cat on the sysfs files and lsscsi are what the RHEL and SLES guides show. Of the 24 module parameters, five are gone (ql2xmaxqueues, ql2xmultique_tag, ql2xtargetreset, ql2xiidmaenable, ql2xetsenable) and ql2xmaxqdepth defaults to 64. Details are in the "Checked against" sections of the Config documents.