Balabit 06 - IPMI out-of-band management

Balabit SCB Solution · Previous: High availability · Next: Basic settings, logging and monitoring

The SCB is an appliance: when its web interface does not answer, its SSH console is locked, or an upgrade leaves it half booted, the only way in is the console in front of the rack. Each T-10 has an IPMI module (a SuperMicro BMC) with its own network port, which gives that console remotely, together with power control and hardware health, "independently of the operating system of SCB", as the design puts it. This part covers the out-of-band network the two modules of site 1 sit in, how they got their addresses, every setting of their web interface as I recorded it, the certificate story, and how administrators reach them.

The out-of-band network

The IPMI port of each appliance is cabled to port Gi1/0/16 of the out-of-band switch of its datacenter, DC1-A-SOOB003 or DC1-B-SOOB003, an access port in VLAN 12. The module itself has VLAN tagging disabled and uses its dedicated port, not a shared LAN port.

The IPv6 prefixes do not match the design's own network table, which gives 2001:db8:b1:ff3::/64 to datacenter A and 2001:db8:a1:ff3::/64 to B. One of the two is swapped; the material does not tell which, and nothing in the design used IPv6 to reach the modules.

The design's table of out-of-band communications lists what the modules talk to and who talks to them:

flowchart LR
  any["Any source"]
  subgraph oob["Out-of-band network, VLAN 12"]
    ia["IPMI node A 10.11.15.29"]
    ib["IPMI node B 10.11.23.29"]
  end
  dns["Infoblox 10.11.16.145 and 10.11.18.145"]
  ad1["AD server 1, 10.11.16.209"]
  ad2["AD server 2, written as 10.11.18.210"]
  mail["Internal Email server 10.11.19.33"]
  any -- "HTTPS 443, iKVM 5900" --> ia
  any -- "HTTPS 443, iKVM 5900" --> ib
  ia -- "DNS 53, NTP 123" --> dns
  ib -- "DNS 53, NTP 123" --> dns
  ia -- "LDAPS 636" --> ad1
  ib -- "LDAPS 636" --> ad1
  ia -- "LDAPS 636" --> ad2
  ib -- "LDAPS 636" --> ad2
  ia -- "SMTPS 465" --> mail
  ib -- "SMTPS 465" --> mail

Four rows of that table are wrong or at least doubtful. "AD server 2" is given as 10.11.18.210; everywhere else the second domain controller dc1-a-vcad002 is 10.11.16.210, and 10.11.18.210 is node B's redundant-heartbeat address. And the two mail rows open SMTPS on 465, while the SMTP page of the modules says port 25 with "SMTP SSL AUTH" switched on. The mail address 10.11.19.33 is the virtual address of the mail pair (see Network, DNS and firewall). Who may reach the out-of-band network at all is not in my material; "Any" in the table means the modules themselves do not filter.

Bringing the modules up

A new module has no useful address, so the first configuration runs on the appliance. The how-to in my notes: log in on the local console (or over SSH) as root, go to Shells > Boot shell, and use ipmitool: lan print to see the settings, lan set 1 ipsrc static and the address, netmask and gateway, then the vendor's raw command for the T-10 that makes the module use its dedicated port, and lan print 1 again. The command set is a Config document: IPMI configuration with ipmitool. The line that matters most, as the how-to gives it for the T-10, to be run as root in the boot shell of each appliance:

$ ipmitool raw 0x30 0x70 0xc 1 0

That one line is the easy one to miss. As I understand it, it is a SuperMicro OEM command that sets the LAN interface to dedicated; without it, the module may answer on a shared port instead of its own. The web interface shows the result as LAN INTERFACE Dedicate. After that the module's web interface answers in a browser, and the how-to's last step is to change the factory password of the ADMIN user under Configure > Users. The new password is not in my notes, which is as it should be.

The settings, page by page

The design records every page of node A "as is configured", and for node B only the three pages that differ (Alerts, Network, SSL Certification). The full answer sets are Config documents: IPMI of node A and IPMI of node B. In short:

Two things in that list need more than a line.

Who can log in. The design's access table says local users log in as account, with only ADMIN enabled, and directory users as account@ad.example.net. The Active Directory page maps the same administrator and operator groups as the SCB itself (see Active Directory and access control), and the communication table opens LDAPS from the modules to the domain controllers. But the LDAP page says ENABLE LDAP AUTHENTICATION = No. On the SuperMicro BMC, as far as I know, Active Directory has its own enable switch on its own page, separate from LDAP; the design does not record that switch, so whether directory logins to the modules ever worked is not in the material. If they did not, ADMIN with its changed password was the only way in.

The certificate. Constraint C3 of the design says that only the IPMI certificates are signed by the organisation's CA; everything else on the SCB uses the SCB's own CA, because the organisation's CA could not issue a timestamping certificate. The design gives the IPMI certificates a validity of 2017-07-19 to 2018-07-20. The certificates in my certificate folder are newer: issued by the organisation's "Internal CA" to dc1-a-ablb001m.mgmt.example.net and dc1-b-ablb001m.mgmt.example.net, RSA 2048 bits, SHA-256, with the name repeated as a DNS subject alternative name, valid from 2018-01-08 to 2019-01-08. So the certificates were replaced in January 2018, apparently together with the renaming of the modules. My notes have an earlier round of requests for the old names dc1-a-ablm001.mgmt.example.net with 4096-bit keys; the how-to kept beside the final certificates generates 2048-bit keys for the new names, and the key size of the issued certificates matches the how-to. The requests are a Config document of the certificates part: IPMI certificate requests, and the whole certificate picture is in Certificates and keys. The analysis noted that the organisation's CA issues certificates for 365 days, so the modules needed new certificates every year; the material ends before the next renewal was due.

Getting to the modules

Chapter 8 of the design gives the way in:

SSH to the modules is possible in principle and "not used in this solution"; the port is off. The design's DNS records still name the modules dc1-a-ablm001.mgmt.example.net and dc1-b-ablm001.mgmt.example.net (see DNS records (site 1)), so the URL works only if the records were renamed with the certificates; the design's interface-summary picture marks only the CL1 names as not in the Infoblox. What an administrator does once on the appliance's console, in the boot and core shells, is in Operations, upgrades and troubleshooting.

Site 2 has IPMI addresses in its L3 sheet, 10.12.15.29 and 10.12.23.29, and nothing else: no IPMI pages, no certificates.

Loose ends

• "AD server 2" at 10.11.18.210 in the out-of-band communication table, which is a heartbeat address of the SCB.
• SMTPS 465 in the communication table against port 25 on the SMTP page.
• LDAP authentication off while Active Directory role groups are configured; the Active Directory switch itself is not recorded.
• The IPv6 prefixes of the two out-of-band networks are swapped between the network table and the IPMI pages.
• The design's certificate dates (2017-07-19 to 2018-07-20) are, as I read them, those of an earlier pair of certificates than the ones in the folder; the organisation-CA certificate for the cluster's web name in the folder was issued the same day, 2017-07-19.
• The DNS server of both modules is only the Infoblox of datacenter A; the one in datacenter B is used for NTP but not for DNS.

What I would do differently

I would restrict who can reach the modules instead of leaving "Any" in the table and IP access control off: the vendor's current documentation warns that the IPMI "has known vulnerabilities that One Identity cannot fix" and should be connected only to "well-protected, separated management networks with restricted accessibility". Switching off virtual media (port 623) and SSH was right, as later SuperMicro advisories about Virtual Media showed. And I would decide between the LDAP and Active Directory login on the modules, test it, and record the result, so that the ADMIN password is not the only way in by accident.