Balabit 10 - Audit trails: encryption and replay

Balabit SCB Solution · Previous: Certificates and keys · Next: Backup, archive and retention

An audit trail is the reason the SCB was bought: the recording of what one user did in one channel of one session, which an auditor can replay later like a film. This part follows a trail from the policy that protects it to the auditor's screen: how it was encrypted, timestamped and signed, why there are encryption keys per auditor, how an auditor found and downloaded a trail, and how many certificates the Desktop Player needed before it reported the trail as intact.

What gets recorded

Recording is switched on in the channel policies. In the site 1 design every rule of every RDP and SSH channel policy has AUDIT set to Yes and FOUR-EYES to No, and every one of the seven connections names the audit policy default and has LOG AUDIT TRAILS DOWNLOADS set to Yes. The site 2 cluster's exported config.xml shows the same: all eight connections reference the one audit policy, and every rule of its own channel policies has <audit enabled="yes"/>. Auditing is off only in rules of built-in channel policies. The policies themselves are the subject of Connection and channel policies.

After a connection closes, the internal indexer processes its trail so that its content can be searched in the web interface.

The audit policy

The default audit policy does three things to every trail: it encrypts it with the certificate SCB-AUDIT-ENCRYPT, timestamps it, and signs it with the private key of SCB-AUDIT-SIGN. The four-eyes certificate stays empty, and the upstream traffic (what the user sends) is not encrypted with a separate certificate. The global options of each protocol complete the policy: timestamping Local, that is by the SCB's own TSA certificate, a signing interval of 30 seconds, and the cleanup of the channel database after 180 days. The whole page is a Config document: Default audit policy (site 1).

The design's own summary of why is one sentence in its introduction: all data is stored in encrypted, timestamped and signed files, preventing any modification or manipulation. My reading of the three parts is this. Encryption means that whoever runs the appliance, or reads the backups on the NAS, cannot watch the sessions; only the holder of a private encryption key can. The timestamp, made with the TSA certificate that cost the CA discussion of Certificates and keys, fixes when the trail was written. The signature, made on the box with a key that only the SCB uses, shows that the SCB wrote the trail and that nobody changed it afterwards.

The introduction of the design also describes the option that was not used: the two directions of the traffic can be encrypted with different keys, so that passwords typed by the user are shown only when necessary. With ENCRYPT UPSTREAM TRAFFIC WITH DIFFERENT CERTIFICATES = No, the auditor who opens an RDP trail sees what the user typed, including a password typed on a Windows login screen.

The trails leave the SCB in the nightly backups and archives on the NetApp exports scb__backup and scb__archive, and are removed from the box after 90 days; that is in Backup, archive and retention. The design's backup tables repeat the encryption certificate and the signing key for each partner's backup, so what lands on the NAS is the trail in its encrypted and signed form.

flowchart TB
  u["User session through a connection"] --> ch["Channel policy rule with audit on"]
  ch --> ap["Audit policy default"]
  ap --> e["Encrypt to the SCB-AUDIT-ENCRYPT certificates"]
  ap --> t["Timestamp by the local TSA"]
  ap --> s["Sign with SCB-AUDIT-SIGN, interval 30 s"]
  e --> z["Audit trail file .zat on the SCB"]
  t --> z
  s --> z
  z --> idx["Internal indexer"]
  z --> bck["Nightly backup to the NAS"]
  z --> arc["Archive and cleanup, 90 days"]
  z --> dl["Search and download in the web interface"]
  dl --> pl["Desktop Player on the ORG RDP jump server"]
  key["Auditor's private key from the auditor's PC"] --> pl

One certificate or six

The site 1 design of December 2017 has one encryption certificate in the audit policy, SCB-AUDIT-ENCRYPT. My XCA notes add one encryption key per auditor, SCB-AUDIT-ENCRYPT-admin01 and so on for admin01, admin05, admin07, admin08 and admin09, at both sites; the one per-auditor certificate in my folder, admin01's for site 1, was made on 22 May 2018. The list is in XCA and GPG keys.

How these were used together is written down nowhere, but the site 2 export of September 2018 shows the result. Its default audit policy has six certificate groups with one certificate each, and my notes list exactly six encryption keys for site 2: the shared SCB-AUDIT-ENCRYPT-DC2 and the five per-auditor ones. The export replaced the certificates by placeholders, so the match of the count is my inference. The element is a Config document: config.xml audit policy (site 2). As I understand the policy, each group is an alternative: the trail is encrypted so that the key of any one group opens it, and two certificates in the same group would both be needed. The decryption how-to fits that reading, because one auditor's key was enough to open a trail. Whether site 1 was changed the same way after May 2018 the documents do not show.

Finding and downloading a trail

The decryption how-to describes the auditor's way on the site 2 cluster and says that site 1 differs only in the URL. Its example is a session of test66, the test account of partner 4 in my notes, on 28 May 2018.

The file arrived as rdp-180528T1015-test66-test66-10.12.16.65.zat. I read the name as the protocol, the start of the session, the user name twice (once as given to the SCB and once on the server) and the address the client connected to, which is the site 2 SCB's production address; the documents do not explain the pattern. Who may search: the access control of the design gives SCB_ADM read and write on everything and SCB_OPS read on everything, see Active Directory and access control. Every download is logged, because each connection has LOG AUDIT TRAILS DOWNLOADS on.

Replay in the browser was possible in principle. The design warns that it needs Internet Explorer 11 with Google's WebM plugin and names the Audit Player or the Balabit Desktop Player as the way otherwise. The how-to chose the Desktop Player, installed only on the two ORG RDP jump servers, dc1-a-vcrdp001 and dc2-a-vcrdp001.

Opening it in the Desktop Player

The player started with a request to confirm the fallback to software rendering. Opening the file gave the first error, "The file could not be decrypted", with the hint to import the appropriate certificates. The screenshots kept with the how-to show the rest of the way:

sequenceDiagram
  participant D as Auditor's desktop PC
  participant P as Desktop Player on the jump server
  P->>P: Open the downloaded .zat file
  P-->>D: The file could not be decrypted
  D->>P: Auditor's private encryption key over the mapped drive, store temporarily only
  P-->>D: Downstream and upstream decrypted, timestamp and signature not verified
  P-->>D: The certificate could not be verified
  D->>P: CA certificate of the site 2 SCB
  P-->>D: Timestamp verified, signature still not verified
  D->>P: TSA certificate, then the server certificate
  D->>P: SCB-AUDIT-SIGN-DC2.pem, shown as a private key, store temporarily only
  P-->>D: Downstream, upstream, timestamp and signature OK

The last message comes from the how-to's text; no screenshot shows the final state.

With the key loaded, the player shows the session: an RDP connection PARTNER4_RDP_JumpServer, client 10.12.20.29, server 10.12.19.209 port 3389 (the partner 4 jump server dc2-a-vcrdp004), target 10.12.16.65, channels Drawing and Clipboard. The timestamp turned green only after the CA certificate of the SCB was imported: the timestamp is made with the TSA certificate, which the SCB's own CA issued, and the player trusts neither until it has the CA. That is the practical reason why the SCB's CA certificate has to reach the auditors; the decryption how-to shows the import only in its screenshots, and the CA section of the operation how-to reads "TODO". The signature turned green after the signing certificate was imported; that certificate is self-signed in XCA, see the Certificate inventory.

Two things in the how-to do not agree with its own screenshots. The text says the timestamp shows OK as soon as the private key is imported; the screenshots show it unverified until the CA certificate was imported. And the text says to import the public certificate of the signing key, while the last screenshot shows the file SCB-AUDIT-SIGN-DC2.pem from the auditor's copy of the certificate folder loaded as "Key type: Private key". If that file held the private signing key, the key that proves the SCB wrote a trail existed on an auditor's PC as well, which weakens the signature; verifying needs only the certificate. The documents do not settle which it was.

The private keys

The how-to names admin01, admin05, admin08 and admin07 as the persons authorised to decrypt; admin09 has keys in XCA but is not in that list. Its advice for the private key is not to store it on the jump server at all, but to load it from the auditor's own PC over the local disk mapped into the RDP session, and the screenshots show the player's choice "Store temporarily only". As I understand RDP drive mapping, the key file is read over the RDP connection and is not copied to the jump server's disk unless the player stores it. My notes do not say whether the keys exported from XCA carried a passphrase.

Loose ends

• The how-to's site 2 section lists the SCB as "SCB Cluster - Site 1" next to the site 2 URL and address, a copy of the site 1 section.
• The how-to says the timestamp is OK after the private key; the screenshots need the CA certificate for it.
• The how-to imports the signing certificate; the screenshot loads the signing PEM as a private key.
• The site 1 design's audit policy has one encryption certificate; the per-auditor keys of 2018 and the six groups of site 2 have no site 1 counterpart in the documents.
• What the documents do not hold: the content of any audit trail, a replay of an archived trail from the NAS, and how a removed auditor's certificate was handled.

Today

Checked against One Identity Safeguard for Privileged Sessions (SPS) 9.0, the audit model is the same: "Encrypt with a single certificate", "Encrypt separately with multiple certificates", "Encrypt jointly with two certificates" for four-eyes, up to 8 lines per policy, and SPS still cannot create the encryption certificates itself. Its security checklist, like the SCB 5 one, recommends a separate certificate for the upstream traffic because RDP login passwords are visible in the trail when typed on the Windows login screen. Timestamping is still Local or remote, with one interval of 10 to 100 000 seconds for timestamping and signing; the signing certificate now needs the code-signing usage. The player is now the One Identity Session Audit Player; trails are still .zat, keys and certificates must be PEM, and on Windows CA certificates cannot be imported from a shared drive. Browser replay no longer needs the WebM plugin, and Internet Explorer 11 has not been supported since 6.13.0. Both feature tables of the 9.0 documents claim replay for trails recorded with SPS 5 F4 and newer, which is later than the 5.0.x LTS this cluster ran; whether its trails still replay is not stated, so an old player should be kept until it has been tried.

What I would do differently

I would encrypt the upstream traffic with a separate certificate from the start, as the vendor recommended then and now; any password a user typed on a Windows login screen inside a session would be readable with an ordinary auditor's key. With NLA on and the credentials saved in MSTSC that should, as I understand it, be the exception, but nothing in my material shows that the trails were checked for it. I would give the auditors the signing certificate as a file that cannot be anything but a certificate, and write the missing section on importing the SCB's CA, so that the player's steps are the same for every auditor.