Linux Storage 01 - Overview and design

Linux Storage Solution · Next: SCSI addressing and FC names

In December 2013 I connected a blade enclosure full of XenServer hosts to an HP 3PAR array over two small Fibre Channel fabrics, and I wrote down what I did on the switch and what the Linux side of one host saw afterwards. This Solution is written from those working notes, two text files of commands and outputs, plus a third file from 2010 about the IBM/LSI RDAC multipath driver on a different system. This first Article says what the system was, how it was wired, which decisions are visible in the notes, where the notes contradict themselves, what they do not contain, and how the ten Articles are ordered.

What was built

One HP 3PAR 7400 storage array, Storage1, with two controller nodes and two FC ports per node. HP writes a port as node:slot:port, so the four ports are 0:1:1, 0:1:2, 1:1:1 and 1:1:2; the notes also call them C0P0, C0P1, C1P0 and C1P1 in the generic controller-and-port way. An HP c7000 blade enclosure holds the servers and, in its interconnect bays, two Cisco MDS 9124e blade FC switches, SW1 and SW2. The schema shows no link between the two switches and host1 and host2 on HV01 report different fabric names, so I take them as two fabrics of their own; each carries two VSANs, one for the datacenter servers and one for the DMZ servers. The notes configure only SW1, under the switch name FCSwitch1. The task written at the top of both 2013 notes was "configure the XenServer server HV01".

flowchart TB
  subgraph arr["HP 3PAR 7400, Storage1"]
    subgraph n0["Node 0"]
      p011["0:1:1, C0P0"]
      p012["0:1:2, C0P1"]
    end
    subgraph n1["Node 1"]
      p111["1:1:1, C1P0"]
      p112["1:1:2, C1P1"]
    end
  end
  subgraph enc["HP c7000 enclosure"]
    subgraph sw1["SW1, FCSwitch1, fabric 1"]
      v11["VSAN 11 MGMT_DC, DC in the schema"]
      v12["VSAN 12 DMZ"]
    end
    subgraph sw2["SW2, fabric 2, not in the notes"]
      v21["VSAN 21, DC"]
      v22["VSAN 22, DMZ"]
    end
    hv01["bay1 HV01, BL660c Gen8, XenServer"]
    hv02["bay2 HV02, BL660c Gen8, XenServer"]
    hv03["bay3 HV03, BL660c Gen8, XenServer"]
    hv04["bay4 HV04, BL660c Gen8, XenServer"]
    mg01["bay5 MGMT01, BL460c Gen8"]
  end
  p011 -- "ext1" --> v11
  p012 -- "ext2" --> v12
  p111 -- "ext1" --> v21
  p112 -- "ext2" --> v22
  v11 -- "bay1, host1" --- hv01
  v11 -- "bay2" --- hv02
  v11 -- "bay5" --- mg01
  v12 -- "bay3" --- hv03
  v12 -- "bay4" --- hv04
  v21 -- "bay1, host2" --- hv01
  v21 -- "bay2" --- hv02
  v21 -- "bay5" --- mg01
  v22 -- "bay3" --- hv03
  v22 -- "bay4" --- hv04

The diagram is the two ASCII cabling schemas of the notes redrawn as one. Each 3PAR node gives one port to each switch, and on each switch the first external port ext1 is the datacenter VSAN and the second ext2 the DMZ VSAN. Every blade has a dual-port mezzanine HBA; the first port reaches SW1 through the blade's bay number, the second port reaches SW2 through the same bay number. On HV01 the Linux side names these two ports host1 and host2. The schema shows SW2 with the same layout as SW1, numbered VSAN 21 and 22 and fed by the ports 1:1:1 and 1:1:2; nothing else about SW2 is in the notes.

Hardware inventory

The enclosure's interconnect bay mapping is not in the notes beyond what the schema shows: blade bay N logs into switch interface bayN on both switches.

Software inventory

The versions are the ones the notes print in command output; the notes carry no version list of their own.

The only dates in the notes are in the output: the !Time: lines of the two show running-config zone listings on the switch say Friday, 6 December 2013, at 04:31:15 and 05:28:58, and the ls -ld of /sys/class/scsi_host/host*/ on HV01 shows directory timestamps of Dec 9 01:50. The notes do not say when the listing was taken; a sysfs directory time is not a time of running a command.

Checked against today

Everything in this build is out of support, and I checked the current state of each part against the vendors' own pages in October 2026. The comparison tables for individual commands and attributes are in the Articles and Config documents that hold them; this is the summary.

Multipathing is where the gap is widest: the notes show no multipathing on the host (the remote ports' fast_io_fail_tmo is off, which on Red Hat means no multipathd; whether XenServer had it enabled is not recorded), and today HPE and the upstream multipath-tools hardware table both carry a ready 3PAR entry (ALUA, no_path_retry 18, fast_io_fail_tmo 10, dev_loss_tmo infinity), so a current RHEL or XenServer needs no 3PARdata section of its own. That is the subject the notes do not touch, and the reason the RDAC Articles from 2010 are here at all.

Design choices visible in the notes

The notes are commands, not a design document. The decisions below can be read off them; where I say why, that is my understanding, not something the notes state.

Two separate fabrics. The schema draws no link between SW1 and SW2, host1 and host2 report different fabric names, and each blade and each 3PAR node has one port in each, so I take them as two fabrics; no note states it. As I understand it this is the usual dual-fabric SAN: a switch, a cable, an HBA port or a whole fabric can fail and the host still has a path through the other fabric. The notes show the result on HV01, where the same six LUNs appear once through host1 and once through host2; how XenServer joined the two paths is not recorded.

Datacenter and DMZ separated by VSAN. On each switch the datacenter servers (HV01, HV02, MGMT01) and the DMZ servers (HV03, HV04) sit in different VSANs, and the array gives each VSAN its own port. A VSAN is, as I understand it, a separate fabric inside the switch with its own name server and its own zoning, so a DMZ host cannot even see the datacenter port of the array, zoning or no zoning. The names MGMT_DC and DMZ are the notes' own.

One array port per VSAN and switch. Each 3PAR node feeds both VSANs of one switch, with ext1 always datacenter and ext2 always DMZ. The result is four array ports, four fabric-and-VSAN combinations, and two paths per host to the array (one per fabric).

Single-initiator zoning with aliases. Each host gets an FC alias named after it (HV01, HV02, ...), each array port gets one (Storage1-C0P0 in VSAN 11, Storage1-C0P1 in VSAN 12), and each zone holds exactly one host alias and the storage alias of its VSAN: HV01-Storage1, HV02-Storage1, MGMT01-Storage1 in the zoneset PROD-DC, HV03-Storage1 and HV04-Storage1 in PROD-DMZ. Before that, the switch ran interface-based zonesets ZONESET_V11 and ZONESET_V12, whose members were switch ports rather than WWPNs; the new zonesets replaced them. The zoning is in FC aliases, zones and zonesets.

Port security without learning. After zoning, every WWPN was bound to its interface in a port-security database per VSAN and the database was activated with no-auto-learn, so the switch accepts a given HBA only on the bay it was bound to. The switch had no ENTERPRISE_PKG licence and announced a grace period of about 119 days. That is Port security.

Where the notes contradict themselves

The two 2013 notes describe the same system and do not agree on everything. I report, I do not resolve.

Three or four XenServer hosts, then. The host-side schema may simply predate the fourth blade; the notes do not say. The 3PAR port names are the odd one out: the first two bytes are as in the notes, and read with the port-naming scheme HPE documents they put node 1 on SW1 and node 0 on SW2, the reverse of the schema. Whether the schema or the cabling was wrong the notes do not tell; the details are in SCSI addressing and FC names. The switch model is a smaller puzzle of the same kind: both notes call the switches MDS 9124e, a name that, as far as I can tell from Cisco's documents, belongs to its 4-Gbps blade switch for the c-Class, yet both HBA ports of HV01 linked at 8 Gbit. The notes name no other model, so I keep the name as they wrote it and state no link speed for the switch. The remaining contradictions belong to the Articles where they appear: the port-security headings that say "VSAN 21" and "VSAN 22" over commands that say vsan 11 and vsan 12, and the FCwitch1 prompt, in Port security; the order in which scsi-rescan scans the LUNs in LUNs seen and SCSI bus rescan. The RDAC notes are titled 2010 but were filed with the 2013 notes; they are dated 2010 in this Solution.

What the notes do not hold

A reader who wants to rebuild this will look for the following and not find it in the Source material.

• The 3PAR side: CPGs, virtual volumes, the host definition for HV01, its persona, the exports, and which virtual volume became which LUN. The host simply sees 3PARdata VV at LUN 0 to 5 and a 3PARdata SES enclosure device at LUN 254.
• The configuration of SW2. Only the schema and the host2 attributes on HV01 say anything about the second fabric.
• The XenServer side: the storage repository, XenServer multipathing, and the XenServer version. The notes stop at the Linux SCSI layer.
• HBA BIOS settings and anything about boot from SAN.
• Any multipath.conf, any failover test, and how long anything took.
• The interconnect bay mapping of the c7000 beyond bay number equals interface number.

How the Solution is organised

The notes are three files, and the Articles follow them: switch first, host second, the 2010 driver notes last.

flowchart LR
  a3["03 switch base configuration"] --> a4["04 aliases, zones, zonesets"]
  a4 --> a5["05 port security"]
  a5 --> a6["06 host HBA and SCSI inventory"]
  a6 --> a7["07 LUNs seen and rescan"]

Before the switch comes SCSI addressing and FC names, which explains the identifiers every later Article uses; after the host come the three RDAC Articles. The full list with one line per Article is on the Solution page. Each Article links the Config documents that hold the whole command sets, listings and files it quotes from; an Article itself holds only short excerpts.

Articles 8 to 10 describe a different system from September 2010: the host mppsrv01 on Red Hat Enterprise Linux AS 4 with two QLogic HBAs, two IBM DS5300 arrays, DS5300-SITEA and a standby mirror DS5300-SITEB, and the RDAC driver 09.03.0B05.0331 built from source. Nothing in them applies to the 3PAR system of Articles 1 to 7 except the SCSI addressing of Article 2 and the rescan mechanics of Article 7. They are here because they are my only notes on Linux multipathing from that period, and because the 2013 notes say nothing about multipathing at all.

Secrets appear in this Solution only as the placeholders <ADMIN_PASSWORD> and <ADMIN_PASSWORD_HASH>. The World Wide Names keep their vendor prefixes and carry made-up serial parts; the array and LUN identifiers of the RDAC notes are made-up values of the right shape.