Oracle RAC 01 - Overview and design

Oracle RAC Solution · Next: Network and DNS plan

In autumn 2016 I built a two-node Oracle Real Application Clusters database, version 11g Release 2, on two SUSE Linux virtual machines running on Hyper-V. This Solution is written from my working notes of that build: six text files with numbered steps, commands, outputs and installer answers. This first Article says what was built and from what, in which order, where the notes contradict themselves, and what they do not contain at all.

What was built

Two nodes, oradb01 and oradb02, run Oracle Grid Infrastructure (Clusterware and ASM) and on top of it one RAC database with one instance per node. The database files live on five shared iSCSI LUNs managed by ASM. A small Debian server, dns1, is the authoritative DNS server for every name of the installation and the time source of the nodes. A Windows server with Symantec Backup Exec, mng-backupsrv01, backs up the operating systems and the database through an agent on each node.

flowchart TB
  subgraph hv["Hyper-V virtual machines"]
    n1["oradb01, SLES 11 SP3"]
    n2["oradb02, SLES 11 SP3"]
  end
  pub(["Public 10.30.10.x, with VIPs and SCAN"])
  int(["Interconnect 10.30.20.x"])
  bck(["Backup 10.30.30.x"])
  mng(["Management 10.30.40.x"])
  san(["iSCSI SAN 10.30.50.x"])
  dns["dns1, Debian 8.5, Knot DNS and NTP"]
  bsrv["mng-backupsrv01, Backup Exec server"]
  lun[("Five shared LUNs, ASM disk groups OCR, DATA, FRA, ONTRLG")]
  n1 ---|"eth1"| pub
  n2 ---|"eth1"| pub
  n1 ---|"eth3 alias INT"| int
  n2 ---|"eth3 alias INT"| int
  n1 ---|"eth0"| bck
  n2 ---|"eth0"| bck
  n1 ---|"eth3"| mng
  n2 ---|"eth3"| mng
  n1 ---|"eth4"| san
  n2 ---|"eth4"| san
  dns --- mng
  bsrv --- mng
  bsrv --- bck
  lun --- san

Each node has five network interfaces and five roles for them, but the roles do not map one to one: eth2 is unused and the interconnect is an alias on the management interface. That is the trace of the largest problem of the whole build, a virtual switch that passes neither multicast nor broadcast. The plan is in Network and DNS plan, the story in Grid patches and the multicast problem.

Software inventory

The versions are the ones my notes give, in file names, in installer output or in the titles of the notes. Where the notes give none, the table says so.

The kernel version is not in the preparation notes at all. It is printed by the Backup Exec agent installer when it checks the system.

Homes, owners and groups

Everything Oracle is installed under /data, a 99 GB ext3 file system on a logical volume in the volume group DATA, one per node. The two products have separate owners and separate homes, the layout the Oracle 11.2 installation guide calls role separation: grid owns the cluster software and ASM, oracle owns the database software.

The Grid home stands beside the Grid base, not below it, while the database home is below the database base. The notes give the paths without comment.

The numeric identifiers are set explicitly when the groups and users are created, so both nodes have the same ones.

The header blocks of the notes say "Permissions: 755" for both homes, while the command that prepares the directory tree is chmod -R 775 /data/u01. The details are in Operating system preparation.

The names inside the cluster:

A remark in the notes next to the cluster name is worth repeating: the cluster name must not contain certain characters, - among them, and a single word is the safest choice. The remark is stricter than Oracle's documentation, which today allows hyphens in a cluster name and forbids underscores.

Build order

The notes are six files, and the Articles follow their order with two exceptions: the network plan and the DNS server come first here, because every later step resolves names, and the Grid Infrastructure notes are split into the installation and the trouble with it.

flowchart TB
  a2["02 Network and DNS plan"] --> a3["03 DNS server Knot"]
  a3 --> a4["04 Operating system preparation"]
  a4 --> a5["05 Storage and ASM disks"]
  a5 --> a6["06 Grid Infrastructure installation"]
  a6 -->|"root.sh fails"| a7["07 Grid patches and the multicast problem"]
  a7 -->|"root.sh passes on both nodes"| a8["08 Database software, listener and disk groups"]
  a8 --> a9["09 Database creation and user environments"]
  a9 --> a10["10 Backup Exec agent"]
  a10 --> a11["11 Backup Exec server and RAC backup"]
  a11 -.->|"records of the virtual RAC node"| a3

The order inside the notes is not always the order in which things were done. The ASMLib disks are created on device names like /dev/ORADB-CRS, and those names exist only after the udev rules of a later chapter are in place. The forward DNS zone was edited again when Backup Exec needed two more records, which is the dotted arrow in the diagram.

The notes carry no dates of their own, but a few can be read off what they quote.

So the shared storage was attached in the first days of October 2016, DNS followed within days, and the last change I can date is the DNS zone on 26 October, after the backup agent was installed.

Names that do not agree

The notes do not use one name for the database. I report the variants and do not pick a winner beyond what the command outputs show.

csldb in the first header is most likely a typing mistake. clsopdb is not: it is in the agent's instance list, in the connect strings, in the account names on the backup server and in the DNS name of the virtual RAC node, rac-clsopdb-1234567890. Yet the one SQL query in those notes, run over the connect string clsopdb, prints the database name CLSDB, and the comment above the DNS records also says CLSDB. The notes do not explain it, and they contain no tnsnames.ora that would show what the alias clsopdb pointed to. The Articles quote each name as the notes have it in that place.

The listener has the same problem on a smaller scale: both headers announce a listener opdb on TCP 1525, and the listener that NETCA created on that port is called CLSDB.

Other contradictions are reported where they belong: the patch number written two ways and the patch sets a step announces but does not show in Grid patches and the multicast problem, the disk sizes that swap places between two listings in Storage and ASM disks, the mistakes in the reverse zones in DNS server: Knot. One more observation of this kind: the disk group ONTRLG was created for the online transaction logs, the DBCA answers leave the storage at its defaults, and srvctl lists only DATA and FRA as disk groups of the database. The notes show no step that moves the logs to ONTRLG.

What the notes do not contain

The notes are a record of what I typed on the Linux and Oracle side. A reader who wants to rebuild the system will look for the following and not find it.

• Anything about the hypervisor: the Hyper-V version, the virtual switches, the virtual hardware of the nodes. Memory and CPU are not stated; only indirect figures exist, kernel.shmmax = 4124030976, a memlock limit of 3500000 and the DBCA memory answer "6307 MB, 80 %".
• The storage side: the iSCSI target, its configuration, and the iSCSI initiator setup on the nodes. The LUNs simply appear as sdc to sdg.
• The interface configuration files of the nodes, and the commands that moved the interconnect from eth2 to the alias on eth3.
• listener.ora and tnsnames.ora, the response file grid.rsp that one step uses, and the scripts DBCA generated.
• The configuration of dns1 beyond the added blocks of knot.conf and the zone files, and its NTP service, of which only the client side on the nodes is shown.
• Any backup job, schedule or restore test. The backup notes end when the server can see the database.
• Failover tests, and how long anything took.

Passwords appear in this Solution only as named placeholders such as <GRID_OS_PASSWORD>. The SCSI identifiers of the disks, the DBID in the name of the virtual RAC node and the voting file identifier are made-up values of the right shape.