Linux Storage 03 - Cisco MDS base configuration

Linux Storage Solution · Previous: SCSI addressing and FC names · Next: FC aliases, zones and zonesets

Before a Linux host can see a single LUN, the switch between the HBA and the array has to let the two talk. This Article is the first of three about the Cisco MDS 9124e blade switch FCSwitch1: the reset to factory state, the setup dialogue, management access, and the two VSANs with their ports. Everything else on the switch, the aliases and zoning in FC aliases, zones and zonesets and the port security in Port security, builds on what is done here. The whole command set is a Config document: MDS base configuration commands.

The switch and what the notes cover

The HP c7000 enclosure holds two Cisco MDS 9124e blade switches, SW1 and SW2 in the cabling schema of Overview and design. I take each as a fabric of its own: the schema shows no link between them, host1 and host2 on HV01 report different fabric names, and every blade has one HBA port in each; no note states it outright. The switch has internal ports named bayN towards the blade slots and external SFP ports named extN towards the outside world; this installation uses five bays and two external ports.

My notes configure only SW1, which became FCSwitch1. The second switch is missing from them: no reset, no address, no VSANs. The schema shows it with the same layout as SW1 and the VSANs numbered 21 and 22 instead of 11 and 12, and the 3PAR ports 1:1:1 and 1:1:2 instead of 0:1:1 and 0:1:2, so presumably it was configured the same way with those numbers; but that is the schema, not a configuration I can show. The NX-OS version appears once, as version 5.2(8) in the running configuration taken in the next Article.

Starting from nothing

The switch was wiped first: write erase deletes the startup configuration, reload reboots, and because there is no configuration to boot from, the switch runs its setup utility. Both ran at the exec prompt of FCSwitch1 as admin and each asked once; the switch name in the prompt is from before the wipe. The notes do not say how the switch was reached for this; a wipe takes the management address away, so as I understand it this was the serial console or the enclosure's management path.

$ write erase
$ reload

The setup utility asked three things. Whether to enforce the secure password standard: answered y. The password for admin, twice. And whether to enter the basic configuration dialogue, which would have asked for the management address, the gateway, SNMP and the rest: answered no. The author's note above the first question reads "password complexity". Everything the dialogue would have set was then typed by hand, which gives a record of exactly what was set and nothing else. The end of the dialogue as it appeared:

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime
to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): no

Management access

The first configuration lines set the admin user again, this time by hash, and the author remarks that this was not strictly needed since the wizard had just set the password. The line is still useful: it records the role, network-admin, which as I understand NX-OS is the full administrative role.

$ username admin password 5 <ADMIN_PASSWORD_HASH>  role network-admin

Then the management interface, the default gateway, the SSH server and the switch name. The notes write all of these at the FCSwitch1# prompt; they are configuration commands, so as I understand it configure terminal was entered and not written down. On FCSwitch1 as admin:

$ interface mgmt0
$ ip address 10.50.10.14 255.255.255.0
$ ip default-gateway 10.50.10.254
$ feature ssh
$ switchname FCSwitch1

mgmt0 is the Ethernet management port of the switch, out of band of the Fibre Channel side; on a blade switch it is reached through the enclosure. feature ssh is the NX-OS way of switching a service on. There is no feature telnet in the notes, and nothing about SNMP, NTP, syslog or a central login, which is where a reader would look for them. The notes close the base configuration with ip domain-lookup and ip host FCSwitch1 10.50.10.14, which the author describes as an alternative to /etc/hosts: a local name-to-address entry for the switch itself.

Two VSANs

A VSAN, as I understand it, is Cisco's virtual fabric: one physical switch is carved into several logical fabrics, each with its own fabric services, its own name server, its own zoning and its own FCID address space. A port belongs to exactly one VSAN, and a device in VSAN 11 cannot see a device in VSAN 12 at all, whatever the zoning says. It is the SAN equivalent of a VLAN.

The notes create two, and give them the names of the two network zones of the installation:

$ vsan database
$ vsan 11 name MGMT_DC
$ vsan 12 name DMZ

Why two is not written down, and the reason is my inference from the design: HV03 and HV04 are the DMZ hypervisors and HV01, HV02 and MGMT01 the datacenter ones, and the two groups were to share nothing, not even a fabric. The 3PAR presents one port to each VSAN (0:1:1 to VSAN 11 and 0:1:2 to VSAN 12 by the cabling schema; the port names themselves decode to node 1, see SCSI addressing and FC names), so the array is the only device present in both, and it is present there with different ports. Zoning alone could have kept the servers apart; two VSANs make the separation structural, so that a zoning mistake cannot cross it. The schema calls the first VSAN "DC" and the switch names it MGMT_DC; I use the switch's name.

The interfaces were then assigned, again in the vsan database sub-mode:

$ vsan database
$ vsan 11 interface ext1
$ vsan 12 interface ext2
$ vsan 11 interface bay1
$ vsan 11 interface bay2
$ vsan 12 interface bay3
$ vsan 12 interface bay4
$ vsan 11 interface bay5

The device column is from the schema and from the aliases of the next Article; the switch at this point knows only port numbers, and the 3PAR port numbers are the schema's, not something the switch reports. The same as a picture:

flowchart LR
  subgraph sw1["SW1, FCSwitch1, Cisco MDS 9124e"]
    subgraph v11["VSAN 11, MGMT_DC"]
      b1["bay1"]
      b2["bay2"]
      b5["bay5"]
      e1["ext1"]
    end
    subgraph v12["VSAN 12, DMZ"]
      b3["bay3"]
      b4["bay4"]
      e2["ext2"]
    end
  end
  hv01["HV01"] --- b1
  hv02["HV02"] --- b2
  mgmt01["MGMT01"] --- b5
  hv03["HV03"] --- b3
  hv04["HV04"] --- b4
  e1 --- p1["3PAR port 0:1:1 by the schema, Storage1-C0P0"]
  e2 --- p2["3PAR port 0:1:2 by the schema, Storage1-C0P1"]

The second switch, by the schema, repeats the picture with VSANs 21 and 22 and the 3PAR ports 1:1:1 and 1:1:2, which is how each blade gets a second, independent path to the array. On the Linux side this is what makes host1 and host2 of HV01 report different fabric names, as SCSI addressing and FC names shows.

Port mode

The seven interfaces were then set to switchport mode F and enabled with no shutdown. The author's own remark on this step is that it was not strictly needed, because the ports are of type F by default. The excerpt for the first two ports; the Config document has all seven.

$ interface ext1
$ switchport mode F
$ no shutdown
$ interface ext2
$ switchport mode F
$ no shutdown

An F_Port, as I understand the terms, is a fabric port that faces an end device, an N_Port; the alternative modes (E for a link to another switch, auto for letting the switch decide) are not wanted on a blade switch whose ports all face HBAs or array ports. Fixing the mode removes one thing the switch would otherwise negotiate. The HBA side of the same link shows up on Linux as link_state Link Up - F_Port and port_type NPort (fabric via point-to-point), so both ends agree.

What is missing

The notes hold the base configuration of one switch and nothing of the other. They do not hold any check of the result at this stage: no show interface brief, no show vsan membership, no show version. The first evidence that the ports came up is the flogi database of the next Article, which lists all seven interfaces logged in with the expected VSANs. There is also no SNMP, NTP, syslog, banner or AAA, and no copy running-config startup-config; whether the configuration was ever saved is not in the notes.

Today

Checked against the current MDS NX-OS documentation, release 9.4(5a): the MDS 9124 reached its last date of support on 31 January 2019 and 5.2(8i) of late 2016 was its last release, so the switch itself is history. The commands are not. write erase, the setup utility with its secure-password question, username … password 5, feature ssh, interface mgmt0, switchname, vsan database and switchport mode F are all in the 9.x guides with the same syntax; SSH is now on by default and Telnet off. Two things changed. ip host is not in the 9.x command reference any more. And the author's remark that the ports are F by default has an explanation the notes did not have: the setup utility that runs after a write erase sets system default switchport mode F, while the documented default port mode is otherwise auto. Whether the blade switch's bay ports had a default of their own is not documented anywhere I could find. The table of differences is at the end of the Config document.

What I would do differently

Little; the notes are a clean minimal configuration. I would write configure terminal into the notes where it was typed, so that the mode of every line is beyond doubt, and I would take show version and show interface brief right after the port mode step, so that the version and the port states are on record before the zoning starts.