Oracle RAC 11 - Backup Exec server and RAC backup

Oracle RAC Solution · Previous: Backup Exec agent

With the agent on both nodes, the Backup Exec server had to be taught what a RAC database is. Backup Exec 15 does not address a RAC database through one of its nodes: it wants a virtual node with a name made of the database name and the database identifier, resolvable to the nodes, and it wants accounts for the operating system and for the database. This part is the most trial and error of the whole build. The product set accounts I had not chosen, listed the nodes under names I did not want, and refused a database account that worked everywhere else. The notes record the order in which it finally worked, and that order is what is described here.

The backup paths

All backup traffic was meant for the backup network: eth0 of the nodes (10.30.30.11, 10.30.30.12) and the second interface of the server (mng-backupsrv01-bck, 10.30.30.14). The notes give the reason as a condition: if the production network with the public, VIP and SCAN addresses is in effect a back-end network for the application servers and so not directly reachable, the backup server has to be given addresses it can reach.

flowchart LR
  subgraph srv["mng-backupsrv01"]
    be["Backup Exec 15 server, 10.30.30.14"]
    hosts["hosts file: oradb01 and oradb02 to 10.30.30.x"]
  end
  subgraph dns["dns1"]
    zone["example.net: rac-clsopdb-1234567890, two A records"]
  end
  subgraph n1["oradb01, 10.30.30.11"]
    a1["agent beremote"]
    os1["file systems"]
    i1["instance clsopdb1"]
  end
  subgraph n2["oradb02, 10.30.30.12"]
    a2["agent beremote"]
    os2["file systems"]
    i2["instance clsopdb2"]
  end
  rac["virtual node RAC-clsopdb-1234567890"]
  be -- "OS backup, logon local/be" --> a1
  be -- "OS backup, logon local/be" --> a2
  be -- "database backup, logon local/oracle" --> rac
  rac -. "10.30.30.11" .-> a1
  rac -. "10.30.30.12" .-> a2
  zone -. "resolves the virtual node" .-> rac
  hosts -. "resolves the node names" .-> be
  a1 --> os1
  a2 --> os2
  a1 -- "clsopdb/sys as SYSDBA" --> i1
  a2 -- "clsopdb/sys as SYSDBA" --> i2

The DBID and the virtual node

The virtual node is named RAC-<database>-<DBID>. The DBID was read on a node as oracle, in SQL*Plus, connected as system with the connect identifier clsopdb.

select DBID, NAME from v$database;

      DBID NAME
---------- ---------
1234567890 CLSDB

The DBID printed here is a made-up value of the right shape; the real one is not published. With it the virtual node is RAC-clsopdb-1234567890.

The same output shows a contradiction I can only report. Everything in the Backup Exec notes calls the database clsopdb, with the instances clsopdb1 and clsopdb2, and the virtual node carries that name. The query, run in a session connected to clsopdb, prints the database name CLSDB, which is the name in the installer answers and the srvctl output of Database creation and user environments, and the comment above the DNS records below also says CLSDB. The notes hold no tnsnames.ora and no statement that would say whether clsopdb was an alias, a service or a second database, so I leave both names as the notes have them.

The database account BACKUP_EXEC

The plan was to run the database side of the backup under an account of its own instead of sys. It was created in two steps, on a node as oracle in SQL*Plus: the account and its ordinary grants as system, and the SYSDBA privilege as sys.

As system it got UNLIMITED TABLESPACE, the roles AQ_ADMINISTRATOR_ROLE and CONNECT, all roles as default roles and SYSTEM as its default tablespace. GRANT SYSDBA needs a SYSDBA session and was given as sys.

The statements as they were typed are a Config document: SQL for the DBID and the BACKUP_EXEC account. The password in it is the placeholder <BACKUP_EXEC_DB_PASSWORD>.

Names: DNS and the hosts file

The Backup Exec server has to resolve the virtual node. The notes name two ways: the hosts file of the server, or records on the DNS server. I took the second. Two A records with the same name were added to the forward zone on dns1, pointing at the backup-network addresses of the two nodes.

; Oracle RAC - records for Symantec Backup Exec (the Oracle RAC database named "CLSDB", DBID "1234567890")
rac-clsopdb-1234567890    A    10.30.30.11
rac-clsopdb-1234567890    A    10.30.30.12

The whole zone file is a Config document, example.net zone, and the server is described in DNS server: Knot.

The node names needed the other way. In DNS oradb01.example.net is the public address 10.30.10.11, which under that condition the backup server cannot reach directly, and the agents announce themselves under exactly that name. So on the Backup Exec server, and only there, the plain node names were pointed at the backup network in C:\Windows\System32\drivers\etc\hosts.

10.30.30.11 oradb01.example.net oradb01
10.30.30.12 oradb02.example.net oradb02

It is a Config document of its own: hosts file on the Backup Exec server.

Logon accounts and the Oracle server list

In the console, under Configuration and Settings, Logon Accounts, Manage Logon Accounts, four accounts were added: every account the server would need except root. Two are operating system accounts of the nodes from the group beoper (local/be, local/oracle), two are database accounts (clsopdb/sys, clsopdb/BACKUP_EXEC). None was marked as a restricted logon account or as the default account.

Then the list of Oracle servers, under Configuration and Settings, Backup Exec Settings, Oracle. It holds the Oracle nodes and the logon account for each. Three entries went in: the virtual node RAC-clsopdb-1234567890, ORADB01.example.net and ORADB02.example.net, all with the logon account local/oracle. Note that the nodes stand here under their names without -BCK. Everything that was set in the console is listed in one Config document: Backup Exec server settings.

What appeared after the agent restart

The agents were restarted on both nodes (as root, /etc/init.d/VRTSralus.init restart), and the server then showed five machines it had learned about from the agents.

The virtual node was there, detected as what it is. The nodes were there twice, and the entries with the backup-network names, the ones I wanted, were detected as Windows machines. The notes record the fact and no explanation, and I have none.

Three corrections followed. First, the two entries without -BCK were removed, because only the backup network was to be used. Second, they came back: at every restart the agent tells the server about itself again. Advertising was therefore switched off on the nodes with Advertising Disabled=1 in ralus.cfg, as described in Backup Exec agent; the notes add that with advertising on, logging in to the database was troublesome or did not work at all and the tablespaces and archive logs could not even be listed. Third, since nothing announces the nodes any more, they were added by hand: Backup and Restore, Add Server, Linux Computer, ORADB01-BCK.example.net and ORADB02-BCK.example.net, both with local/oracle. The notes do not say what happened to the two entries detected as Windows before the same names were added as Linux computers.

Correcting the logon account

The virtual node did not keep the account it was given. In the Oracle server list it had local/oracle; the object the server created for it had "System Logon Account". This had to be corrected in the properties of the virtual node, and the same on the node entry. After that both entries were refreshed with the Refresh button, so that the server read the nodes and the database again with the right account.

The database credential: BACKUP_EXEC, then sys

After the refresh the virtual node showed the database clsopdb under Credentials, with the setting "use server's logon account". That is the operating system account, and a database needs a database account. The notes hold two steps for the same field.

The remark between them is one of the few places where the notes lose their temper. The account BACKUP_EXEC was correct and worked in the database, with SYSDBA, and Backup Exec still could not use it, "because the Backup Exec server handles credentials very, very oddly". With sys it worked. So the dedicated account exists in the database, holds SYSDBA and unlimited quota, and does nothing; the backup runs as sys on both ends, in AgentConfig on the nodes and in the console.

The last step, [9.5], is the operating system backup. On both node entries the logon account and the credentials were changed from "use server's logon account" to local/be, the account made for this purpose in the previous part.

Contradictions in this part

• The database is clsopdb in every step here and CLSDB in the output of the query and in the comment of the zone file, as described above.
• Steps [9.1], [9.2] and [9.5] name the nodes ORADB01.example.net and ORADB02.example.net, although the entries without -BCK were removed in [8.2] and the servers added by hand in [9.0] are the ones with -BCK. Either the plain names in chapter 9 are shorthand for the -BCK entries, or the plain entries existed again at that point. The notes do not say.
• Step [9.1] corrects the logon account of ORADB01.example.net only; ORADB02 is not mentioned. Step [9.2] likewise refreshes only the first node and the virtual node.
• Step [9.1] sets the logon account of the node to local/oracle and step [9.5] sets it to local/be. The second is the later one.
• Step [9.5] says "after operation [9.3]", the step whose setting was replaced by [9.4].

What is missing

The notes stop where the configuration of the server ends. They hold no backup job: no selection list, no schedule, no storage or media set, no retention. They hold no RMAN settings and no job template other than the answer "no customized job template" in AgentConfig, no recovery catalog (the answer was no), no log of a finished backup and no restore test, neither of a file nor of the database. That sys had to be used is stated; that a backup ran to its end or that a restore was ever proven is not. For a write-up of a backup this is the largest gap, and I cannot fill it from memory.

What I found on the internet

The notes end with four links and a remark about them: the information in them does not explain the details and some of the configurations are plainly misleading, and I found nothing better about backing up Oracle RAC with Symantec Backup Exec. They are listed as they were then.

• Create Oracle backup job in Backup Exec
• Veritas support article TECH49969
• Veritas community: backup of Oracle RAC cluster is displayed as
• Oracle RAC backup with Backup Exec

Checked against Backup Exec 25.1

Backup Exec 15 has been out of support since 7 November 2019; the current release is 25.1. Its Administrator's Guide describes the RAC backup in the same terms, and it explains the one thing that cost me most.

So what the notes call very odd handling of credentials was the documented behaviour for an 11.2 database: for database versions before 12c the guide names SYS with SYSDBA as the user and privilege of the agent's RMAN connection. The support articles under veritas.com now redirect to a page that asks for a login, so the two Veritas links above should be taken as dead.

What I would do differently

• Remove what does not work. BACKUP_EXEC was left in the database with SYSDBA, unlimited quota and the default tablespace SYSTEM, and with a logon account on the server, although nothing uses it. An unused account with SYSDBA is a risk without a benefit.
• Write down the job and test a restore. The configuration was found by trial and error and is recorded step by step; the job that uses it and the proof that it restores are not recorded at all.
• One name per address. The hosts file on the server makes oradb01.example.net mean the backup address there and the public address everywhere else. Using the -bck names consistently, also in the Oracle server list, would have needed no override, if the product had accepted it; the notes do not show that this was tried.