Linux Storage - a Fibre Channel SAN for Linux hosts

In December 2013 I connected a blade enclosure full of Linux hypervisors to an HP 3PAR 7400 array through two Cisco MDS 9124e Fibre Channel blade switches. The array side was not part of my notes; they begin where its ports met the switches: the fabrics, the VSANs that kept the datacenter and the DMZ apart on the same hardware, the aliases and single-initiator zones that let each host see the storage and nothing else, port security so that no other adapter could take a host's place, and then the Linux side, where the operating system has to be asked, through sysfs and systool, what it sees through its HBAs before any LUN can be used.

This Solution is that work, written up from my working notes, in two parts. Articles 1 to 7 are the 2013 SAN: one array, two fabrics, two VSANs per switch, four or five blades with dual-port QLogic HBAs, one switch configured step by step, and one host inventoried down to the driver parameters. Articles 8 to 10 are older notes from 2010 on a different system that I kept with the SAN notes because they answer the question the 2013 notes stop at: what happens when a Linux host sees the same LUN down four paths. There the answer was the IBM/LSI RDAC multipath driver on Red Hat Enterprise Linux AS 4 in front of two IBM DS5300 arrays, built from source, with its own initrd, its own tools, and two problems that had to be solved by hand.

The system in one picture

flowchart TB
  subgraph storage1["Storage1, HP 3PAR 7400, ports as the schema names them"]
    n0["node 0: ports 0:1:1 and 0:1:2"]
    n1["node 1: ports 1:1:1 and 1:1:2"]
  end
  subgraph c7000["HP c7000 blade enclosure"]
    subgraph sw1["SW1, FCSwitch1, Cisco MDS 9124e"]
      v11["VSAN 11 MGMT_DC: ext1, bay1, bay2, bay5"]
      v12["VSAN 12 DMZ: ext2, bay3, bay4"]
    end
    subgraph sw2["SW2, per the schema only"]
      v21["VSAN 21: ext1, bay1, bay2, bay5"]
      v22["VSAN 22: ext2, bay3, bay4"]
    end
    hv01["bay1: HV01, BL660c Gen8, XenServer"]
    hv02["bay2: HV02, BL660c Gen8"]
    hv03["bay3: HV03, BL660c Gen8"]
    hv04["bay4: HV04, BL660c Gen8"]
    mgmt01["bay5: MGMT01, BL460c Gen8"]
  end
  n0 -- "0:1:1, Storage1-C0P0" --> v11
  n0 -- "0:1:2, Storage1-C0P1" --> v12
  n1 -- "1:1:1" --> v21
  n1 -- "1:1:2" --> v22
  v11 --- hv01
  v11 --- hv02
  v11 --- mgmt01
  v12 --- hv03
  v12 --- hv04
  v21 --- hv01
  v21 --- hv02
  v21 --- mgmt01
  v22 --- hv03
  v22 --- hv04

The fictional environment

Every Article and every Config document uses the same names, addresses and World Wide Names.

Host, switch, alias, zone and VSAN names are role names and were kept. The vendor parts of the World Wide Names are real, their serial parts are not; the array and LUN identifiers of the 2010 system are made up.

Articles

Read in this order; it is the order the work was done in.

Configuration

Each document holds one file, one command set or one listing as it was used, with comments and a check against the current release. The switch and the multipath driver leave few files to show, so most documents are command sets with the output they produced.

Cisco MDS 9124e

Linux host

RDAC multipath driver, 2010

What the write-up found

Reading the notes again turned up things I did not see when I wrote them. Each is told in its Article.