Linux Storage 02 - SCSI addressing and FC names

Linux Storage Solution · Previous: Overview and design · Next: Cisco MDS base configuration

A Fibre Channel SAN names the same thing several times over. The HBA port of HV01 is a port_name in sysfs, a PORT NAME in the switch's login database, a pwwn in an alias, a member of a zone, and a scsi1 in /proc/scsi/scsi. The notes take the identities one at a time and never draw the line between them; this Article does, with the definitions the notes give, the values they print, and one chain of identities from host1 on HV01 to port 0:1:1 of the array. The Articles that follow use these names without explaining them again.

The SCSI address

The host-side notes open with the definition of a SCSI address, which I translate as the notes have it:

Written host:bus:target:lun, this is the name under which the kernel keeps every SCSI device in /sys/bus/scsi/devices, and the four numbers that /proc/scsi/scsi prints as Host, Channel, Id and Lun. The scheme is unchanged today: Red Hat's current storage guide defines the same H, B, T and L and adds R for the remote port, as in /sys/class/fc_remote_ports/rport-H:B-R/. /proc/scsi/scsi itself is still in the kernel, kept for old tools and superseded by sysfs; lsscsi is what replaces reading it. On HV01 the hosts are host0 to host2. The notes remark that the HBAs are usually the last two, here host1 for the first HBA port and host2 for the second; host0 is the local HP P220i RAID controller. The notes list the SCSI hosts first, as root on HV01:

$ ls -ld /sys/class/scsi_host/host*/

drwxr-xr-x 3 root root 0 Dec  9 01:50 /sys/class/scsi_host/host0/
drwxr-xr-x 3 root root 0 Dec  9 01:50 /sys/class/scsi_host/host1/
drwxr-xr-x 3 root root 0 Dec  9 01:50 /sys/class/scsi_host/host2/

Only host1 and host2 are also in /sys/class/fc_host, because only they are Fibre Channel ports; the next listing of the notes shows that:

$ ls -ld /sys/class/fc_host/*

drwxr-xr-x 4 root root 0 Dec  9 01:50 /sys/class/fc_host/host1
drwxr-xr-x 4 root root 0 Dec  9 01:50 /sys/class/fc_host/host2

The cat /proc/scsi/scsi listing of the notes gives every device HV01 saw. Rearranged as addresses, with the two HBA paths to the same LUN side by side:

The local controller adds 0:0:0:0, an HP LOGICAL VOLUME of revision 4.68, and 0:3:0:0, the HP P220i controller itself, reported as type RAID on channel 3. Both HBAs see the array on bus 0 as target 0, and the same six LUN numbers plus LUN 254, a SCSI enclosure services device. HPE's host documentation ties that device to the host persona (the SESLun element of personas 1 and 2) and the 2014 XenServer guide warns that a LUN exported at ID 254 needs a host reboot before it can be used; the notes themselves only list it. Which virtual volume is behind which LUN is not in the notes. The whole listing is a Config document: /proc/scsi/scsi. Taking the first LUN apart (the FCID is the address the fabric gave the port; it is explained below):

flowchart LR
  h["host1, the first HBA port, qla2xxx"] --> b["bus 0, channel 0"]
  b --> t["target 0, the array port 21:12:00:02:ac:00:ab:cd at FCID 0x2a0300"]
  t --> l["LUN 0, a 3PARdata VV"]
  l --> a["SCSI address 1:0:0:0"]

As I understand it, the target number is not a property of the array port but something the kernel hands out. The fc_host attribute tgtid_bind_type on both ports reads wwpn (World Wide Port Name), which is still the FC transport's default binding type, and my reading of it is that the driver remembers the target ID it gave to a WWPN and gives it the same one again after a rescan. With one array port per fabric, each HBA has exactly one target, number 0.

WWNN and WWPN

Every Fibre Channel node has a World Wide Node Name and every port of it a World Wide Port Name, both 64 bits. Linux prints them as a hex number, the switch as eight colon-separated bytes; the notes make the point with the first port of HV01:

$ cat /sys/class/scsi_host/host1/device/fc_host:host1/port_name

0x50014380123456c0

and write beside it 0x50014380123456c0 -> 50:01:43:80:12:34:56:c0. The same value appears in the switch's login database as the PORT NAME of bay1. The names HV01 reports, in systool -c fc_host -v and in systool -c fc_remote_ports -v:

Two things the table shows. The QLogic HBA gives each port its own node name, one above the port name, so HV01 has two node names. The 3PAR does the opposite: both array ports, on two different fabrics, report the same node name 2f:f7:00:02:ac:00:ab:cd, and the flogi database on FCSwitch1 shows the same node name for ext1 and ext2 too. The array is one node with four ports; the host is, to the fabric, two nodes with one port each. The prefixes are the vendors' registered identifiers, 50:01:43:80 for HP and 00:02:ac for 3PAR; the serial parts in this Solution are made up and nothing should be read from them. The sysfs attributes themselves, port_name, node_name, port_id, fabric_name and the rest of fc_host and fc_remote_ports, are unchanged in the current kernel; they are defined only in the FC transport's source, there is no ABI document for them.

N_Port, F_Port and the fabric login

A few attributes of host1 describe how the port is attached; the systool -c fc_host -A series of the notes shows the same values on host2, except link_state, which the notes read only on host1:

As I understand the terms, an N_Port is an end device port, a host or an array, and an F_Port is the switch port it plugs into; the HBA reports both sides, itself as NPort and its link partner as F_Port. The switch side agrees: on FCSwitch1 every bay and ext interface was set to switchport mode F, with the notes' own remark that this is the default anyway. Class 3 is, as I understand it, the Fibre Channel service class without acknowledgements, the one SCSI over FC uses; the notes print the value without comment. What is documented today: the qla2xxx driver still enables only Class 3 unless its parameter ql2xenableclass2 is set, and Cisco's F ports support Class 2 and Class 3.

When an N_Port comes up it logs into the fabric, and the switch records the login. On FCSwitch1, in privileged EXEC mode, the datacenter half of the login database (the VSAN 12 rows, the dash rules around the header and the closing count are left out here):

$ sh flogi database

INTERFACE        VSAN    FCID           PORT NAME               NODE NAME
bay1             11    0x2a0000  50:01:43:80:12:34:56:c0 50:01:43:80:12:34:56:c1
bay2             11    0x2a0100  50:01:43:80:12:34:56:d4 50:01:43:80:12:34:56:d5
bay5             11    0x2a0200  50:01:43:80:12:34:40:2c 50:01:43:80:12:34:40:2d
ext1             11    0x2a0300  21:12:00:02:ac:00:ab:cd 2f:f7:00:02:ac:00:ab:cd

The whole database, with the DMZ VSAN and the fcns name server that adds initiator or target to each entry, is a Config document: flogi and fcns database.

FCID, fabric name and VSAN

The third column of the login database is the FCID, the address the fabric assigns to a port at login (24 bits wide, as I understand it). Linux calls it port_id: host1 reports port_id 0x2a0000, which is the FCID of bay1 in VSAN 11, and its remote port reports port_id 0x2a0300, the FCID of ext1. Everything in VSAN 11 starts with 0x2a, everything in VSAN 12 with 0x66, and the two ports host2 sees on the other switch with 0x58. As I understand the format, the first byte is the domain ID of the switch inside that VSAN, so each VSAN has its own domain; the notes do not say so and the values are all they show.

The fc_host attribute fabric_name is the identity of the fabric the port logged into:

Two different fabric names, as two separate switches should give. My inference, not a statement of the notes: the second byte of each fabric name is the VSAN number in hex, 0x0b is 11 and 0x15 is 21, exactly the two VSANs the schema puts the two HBA ports in. The middle bytes 54:7f:ee are the Cisco prefix, the same one the switch WWN 20:00:54:7f:ee:ab:cd:18 in the old interface-based zones carries. As I understand it, the fabric name is the WWN of the principal switch of that VSAN, and on a single switch per fabric that is the switch itself with the VSAN folded into it; the notes do not decode the value.

A VSAN is a virtual fabric inside one physical switch. On FCSwitch1 the notes create two, vsan 11 name MGMT_DC and vsan 12 name DMZ, and put each interface into one of them; the fcns database is printed per VSAN, four entries in 11 and three in 12, and the zoning and port security below are all per VSAN. The schema numbers the second switch's VSANs 21 and 22. The fabric_name of host2 is the only trace of VSAN 21 in the notes, and it is a trace only by the inference above.

The chain of identities

Put together from the three sources, the host's sysfs, the switch's login database and the switch's zoning, one path of HV01 to the array looks like this:

The zone activation output closes the loop: show zoneset active vsan 11 lists HV01-Storage1 with fcid 0x2a0000 [pwwn 50:01:43:80:12:34:56:c0] and fcid 0x2a0300 [pwwn 21:12:00:02:ac:00:ab:cd], the two FCIDs the host reports as port_id of itself and of its remote port.

flowchart LR
  subgraph host["HV01, Linux sysfs"]
    h1["fc_host host1, port_name 0x50014380123456c0, port_id 0x2a0000"]
    rp["fc_remote_ports rport-1:0-0, port_name 0x21120002ac00abcd, port_id 0x2a0300"]
  end
  subgraph sw["FCSwitch1, VSAN 11 MGMT_DC"]
    fl1["flogi bay1, FCID 0x2a0000, 50:01:43:80:12:34:56:c0"]
    al1["fcalias HV01"]
    z["zone HV01-Storage1, zoneset PROD-DC"]
    al2["fcalias Storage1-C0P0"]
    fl2["flogi ext1, FCID 0x2a0300, 21:12:00:02:ac:00:ab:cd"]
  end
  arr["3PAR Storage1, port 0:1:1, C0P0"]
  h1 -- "same WWPN" --> fl1
  fl1 --> al1
  al1 --> z
  z --> al2
  al2 --> fl2
  fl2 -- "same WWPN" --> rp
  fl2 -- "cable, per the schema" --> arr

The remote port's scsi_target_id 0 is where the chain meets the SCSI address of the first section: rport-1:0-0 is host 1, bus 0, remote port 0 in the rport-H:B-R form Red Hat documents, and the devices behind it are 1:0:0:0 to 1:0:0:5 and 1:0:0:254. The fc_transport class shows the same remote port once more as target1:0:0. Both listings are a Config document: systool fc_transport and fc_remote_ports; the fc_host attributes of both ports are in systool fc_host.

The second fabric, as far as the notes go

For host2 the notes hold only the host side, because SW2 was never recorded.

Across both fabrics the notes thus show three distinct storage WWPNs: 21:12:00:02:ac:00:ab:cd on ext1 of SW1 (VSAN 11, alias Storage1-C0P0), 21:11:00:02:ac:00:ab:cd on ext2 of SW1 (VSAN 12, alias Storage1-C0P1), and 20:11:00:02:ac:00:ab:cd, which host2 sees at FCID 0x580000 and which appears nowhere on FCSwitch1. That is consistent with the schema: four array ports, one per switch and VSAN, of which HV01 in the datacenter VSANs reaches two, and the DMZ port of SW2 is the fourth, never seen in the notes. Which of the four physical ports carries 20:11:… the notes do not say beyond the schema. HPE's documentation does describe how a 3PAR names its ports: 2N:SP: followed by the 3PAR prefix and the array's serial, where N is the node, S the slot and P the port, and one node name of the form 2f:f7:… for the whole array. The first two bytes of the three port names are as in my notes, only the serial part was replaced for publication, so the scheme can be applied to them. It gives 21:12 on ext1 of SW1 (alias Storage1-C0P0, VSAN 11) as node 1, slot 1, port 2, 21:11 on ext2 of SW1 (alias Storage1-C0P1, VSAN 12) as 1:1:1, and 20:11, the port host2 sees on SW2, as 0:1:1. That does not match the cabling schema, which puts 0:1:1 and 0:1:2 on SW1 and 1:1:1 and 1:1:2 on SW2. The aliases C0P0 and C0P1 follow the schema's naming, not the port names. The notes give no way to tell whether the schema or the cabling was wrong; I report it and do not resolve it.

Where to look next

The switch side of these names, in the order it was built, is in Cisco MDS base configuration, FC aliases, zones and zonesets and Port security. The host side, attribute by attribute, is in Linux HBA and SCSI inventory, and what happens to the SCSI addresses on a rescan in LUNs seen and SCSI bus rescan.