Proxy 01 - Overview and design
Proxy Solution · Next: Interfaces and firewalld zones
This Solution is a Squid forward proxy with SSL interception on Red Hat Enterprise Linux 7.5, behind firewalld, built in a lab in 2018 and then rolled out as one proxy per datacenter, where it ran through 2019. The servers of two datacenters reach the internet through it, and because the proxy opens the TLS connections it carries, every one of those servers has to trust the proxy's own certificate authority. This first part says what was built and why, describes the fictional environment the write-up uses, draws the three pictures the rest refers to, and lists what my working notes hold, what they do not hold, and what the write-up found wrong in them.
What was built and why
A forward proxy is the one place a server with no route to the internet may go to fetch something from it. The clients here are servers, not people: the only client the notes show is pip on Python 2.7 fetching from PyPI. As I understand the reason, instead of a firewall rule per server and per destination, the servers get one destination, TCP 3128 on the proxy's internal address, and the proxy has the one interface that leads out. The zones and rules themselves the notes state directly; the reasoning is mine.
SSL interception is the part the notes explain least. A plain forward proxy carries HTTPS as an opaque tunnel (CONNECT) and sees nothing of it. With ssl-bump Squid terminates the client's TLS connection with a certificate it generates on the fly, signed by its own CA, and opens its own TLS connection to the server. As I understand it, that is done to see and filter the content of HTTPS traffic; the notes do not say what the proxy was meant to look for, and no filtering rule beyond the standard http_access lines appears in the lab's squid.conf. What the notes do show is the mechanism: peek at the TLS hello first, let the connections to excluded server names and destinations through untouched (splice), and bump the rest, with localhost always spliced. That is in Squid with SSL interception.
The design has four parts that repeat in every note. One virtual machine per datacenter, with two interfaces: ens3 towards the clients, ens4 towards the internet. One firewalld zone per interface, internal and external, both with target DROP, with the default services removed and masquerading switched off because, as the step title in the notes says, "we are not firewall, but proxy". Rich rules on internal only, for the proxy port, ICMP, SSH from the Ansible hosts and the management ranges, and SNMP from the Sensu hosts. And Squid with a self-signed CA certificate, valid 365 days and renewed by hand when it ran out, whose public half a colleague's automation distributes to the servers.
The environment
The names and addresses below are the fictional ones of this site, shared with the Email and NetApp Solutions; the RFC 1918 ranges in the rules are what the notes used.
| Host | Interface | Zone | IPv4 | IPv6 |
|---|---|---|---|---|
dc1-a-vcprx001.adm.example.net, datacenter 1 | ens3, internal | internal | 10.11.16.113/28 | 2001:db8:a1:b96::f:1/64 |
dc1-a-vcprx001.adm.example.net | ens4, external | external | 10.11.17.145/28 | 2001:db8:a3:b95::f:1/64 |
dc2-a-vcprx001.adm.example.net, datacenter 2 | ens3, internal | internal | 10.12.16.113/28 | 2001:db8:a2:b96::f:1/64 |
dc2-a-vcprx001.adm.example.net | ens4, external | external | 10.12.17.145/28 | 2001:db8:a4:b95::f:1/64 |
proxy.lab.example.net, lab | ens32, external | external | 10.90.0.102/24, gateway 10.90.0.1 | link-local only, IPV6INIT=no |
proxy.lab.example.net | ens33, internal | internal | 10.90.114.114/24 | link-local only, IPV6INIT=no |
The IPv6 addresses of the two ens4 interfaces appear in the ifconfig output of the notes and nowhere else; no rule refers to them. The ifconfig output of both production hosts shows the same MAC addresses (96:ea:30:12:34:56 on ens3, fc:c4:a1:12:34:57 on ens4) and therefore the same link-local addresses on two different virtual machines; the notes do not explain it, and I do not derive anything from the hardware addresses.
On the internal side three groups of hosts reach the proxy, and the rich rules of firewalld rich rules are written for exactly these.
| Who | Datacenter 1 | Datacenter 2 | May reach |
|---|---|---|---|
| The servers, the proxy's clients | 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8, 2001:db8::/32 | the same ranges | TCP 3128, TCP 22, ICMP |
| Two Ansible hosts | 10.11.17.241, 10.11.17.242, 2001:db8:a1:b89::f:1, 2001:db8:a1:b89::f:2 | 10.12.17.241, 10.12.17.242, 2001:db8:a2:b89::f:1, 2001:db8:a2:b89::f:2 | TCP 22 |
| Three Sensu monitoring hosts | 10.11.16.129 to 10.11.16.131, 2001:db8:a1:bb1::f:1 to ::f:3 | 10.12.16.129 to 10.12.16.131, 2001:db8:a2:bb1::f:1 to ::f:3 | UDP 161 |
Who the servers are beyond "hosts in the three private ranges and the organisation's IPv6 /32" the notes do not say. The two datacenter notes draw the same ASCII schema of the host, so one Mermaid diagram stands for both.
flowchart LR inet["Internet"] subgraph host["dc1-a-vcprx001 or dc2-a-vcprx001, RHEL 7.5"] ens4["ens4 external, 10.11.17.145 or 10.12.17.145, zone external, target DROP"] squid["Squid on TCP 3128 with ssl-bump"] ens3["ens3 internal, 10.11.16.113 or 10.12.16.113, zone internal, target DROP"] end servers["Servers in 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8, 2001:db8::/32"] ansible["Two Ansible hosts per datacenter"] sensu["Three Sensu hosts per datacenter"] servers -- "TCP 3128, TCP 22, ICMP" --> ens3 ansible -- "TCP 22" --> ens3 sensu -- "UDP 161" --> ens3 ens3 --- squid squid --- ens4 ens4 --> inet
The lab was the same design on a VMware guest with other interface names and one client.
flowchart LR lan["Lab network 10.90.0.0/24, gateway 10.90.0.1"] subgraph lab["proxy.lab.example.net, RHEL 7.5"] ens32["ens32 external, 10.90.0.102, zone external, target DROP"] squid["Squid on 10.90.114.114 TCP 3128 with ssl-bump"] ens33["ens33 internal, 10.90.114.114, zone internal, target DROP"] end client["client 10.90.114.1"] client -- "TCP 3128, ICMP" --> ens33 ens33 --- squid squid --- ens32 ens32 --> lan
The software
| Component | As built | Where in the notes |
|---|---|---|
| Operating system | Red Hat Enterprise Linux 7.5 | the title of every note |
| Firewall | firewalld as shipped with RHEL 7.5, firewalld-0.4.4.4-14.el7, driven with firewall-cmd | the three firewalld notes; the package version is from the RHEL 7.5 repository, not from the notes |
| Proxy | Squid from the RHEL 7 repository, installed with yum install squid, which on RHEL 7.5 was squid-3.5.20-12.el7 | the lab note; no version is printed anywhere in the notes, the package version is from the RHEL 7.5 repository |
| Certificate | self-signed with openssl, 3652 days in the lab, 365 days in production | the lab and the two certificate notes |
| Certificate cache | /var/lib/ssl_db, created with /usr/lib64/squid/ssl_crtd -c -s /var/lib/ssl_db | all three Squid notes |
| SELinux | blocking ssl_crtd; a local module squid-local built with audit2allow | the datacenter 2 certificate note |
| Test client | pip on Python 2.7 from EPEL | the lab note |
The order of the work
The order below is the write-up's, not the notes': the datacenter 2 certificate note, for one, holds the SELinux problem after the renewal. The Articles follow this order.
flowchart TB s1["Interfaces and zones, lab first, then one host per datacenter"] s2["Rich rules on zone internal"] s3["Self-signed CA certificate, Squid installed, ssl_crtd cache, ssl-bump in squid.conf"] s4["SELinux module squid-local for ssl_crtd"] s5["Client test with pip and REQUESTS_CA_BUNDLE"] s6["Renewal of the 365-day certificate and the cache"] s1 --> s2 --> s3 --> s4 --> s5 --> s6
| Step | Article |
|---|---|
| Interfaces and zones | Interfaces and firewalld zones |
| Rich rules | firewalld rich rules |
| Certificate, Squid, SSL bump | Squid with SSL interception |
| Certificate renewal | Certificate renewal |
| SELinux and the client test | SELinux and client trust |
What the notes hold and what they do not
The Source material is eight files of my working notes: the lab build of firewalld and Squid, the firewalld configuration of each production proxy, the certificate replacement on each of them (the datacenter 2 one with the SELinux problem), the lab's squid.conf and its two exclusion lists. Each note is a sequence of step titles, the commands run as root and their output, with NOTE: remarks. They are what was typed, not a design document, and a reader who comes for the following will not find it.
| Missing | What the notes have instead |
|---|---|
The production squid.conf | two port lines, as grep ssl-bump printed them, both without a comment sign, so as printed both were active; how Squid ran with both the notes do not say |
| The production exclusion lists | the lab's two files, whose entries (.example.com, .example.org, 8.8.8.8) look like example entries |
| The Ansible roles and how the certificate reaches the servers | one sentence: the certificate file is sent to the colleague who runs the certificate deployment |
| The Sensu checks | the SNMP rule that lets the Sensu hosts in |
DNS, routing and the ifcfg files of the production hosts | ifconfig output; the lab has its two ifcfg files |
| Access log, cache settings, log retention | nothing; the lab cache_dir is commented out |
| Proxy settings of the clients | the pip test with http_proxy, https_proxy and REQUESTS_CA_BUNDLE |
| The IPv6 side of the lab | nothing beyond the link-local addresses in ifconfig; IPV6INIT=no |
The reason for version=1 and the options= list on the production ports | nothing |
What the write-up found
The notes contradict themselves in places, as notes typed during the work do. I report the contradictions where they belong and resolve none of them; each row names the Article that describes it.
| Found | Article |
|---|---|
After the ifcfg ZONE= method the lab's active zones list internal: ens32 and external: ens33, the reverse of the files, and the note under it says it is fine | Interfaces and firewalld zones |
Both production hosts start with the same leftover bindings, eth0 in internal with sources: 192.168.0.0/16 and eth1 in external, interfaces the machines do not have; only the datacenter 2 note removes them, and its sources: line stays to the end | Interfaces and firewalld zones |
Datacenter 2 shows target: default right after --permanent --set-target=DROP, the lab and datacenter 1 show target: DROP without a reload in between | Interfaces and firewalld zones |
external keeps the ssh service in the lab and in datacenter 1; public keeps 3128/tcp in datacenter 2 | Interfaces and firewalld zones |
| The datacenter 2 IPv6 rule for port 3128 is entered with the host's own address and listed with datacenter 1's | firewalld rich rules |
The datacenter 2 ICMP rule is entered with family=ipv4 and listed without a family | firewalld rich rules |
| Step titles name the wrong port, address family or datacenter: the "OS Management" IPv6 title says port 3128 for a port 22 rule, a datacenter 2 Sensu title lists a datacenter 1 address, a datacenter 1 Sensu title says IPv4 for an IPv6 rule | firewalld rich rules |
| Datacenter 1 admits SSH and SNMP from its own Ansible and Sensu hosts only, datacenter 2 from both datacenters'; the datacenter 1 note's final rule listing has no output | firewalld rich rules |
acl step3 is defined and never used; http_port 3128 and https_port 3128 … intercept on the same port in production | Squid with SSL interception |
chown -R squid:squid /var/lib/ssl_db/* leaves the directory itself to root, which is my reading of the command | Squid with SSL interception |
The datacenter 1 note names the certificate file …crt_1t, a typo for …crt_1 | Certificate renewal |
| Both production hosts show the same MAC and link-local addresses | this Article |
Most of these are slips in titles and listings that left the running configuration alone. One is not: if the datacenter 2 listing is right, the IPv6 proxy rule on dc2-a-vcprx001 accepted TCP 3128 to datacenter 1's address, which that host does not have, and IPv6 clients of datacenter 2 had no rule for their proxy port. The listing after the reload is the one piece of evidence of what the datacenter 2 host enforced, and it is a Config document of firewalld rich rules; the datacenter 1 listing is empty in the notes.
Today
Checked against the current versions on 2026-10-03. RHEL 7 reached the end of its maintenance support on 2024-06-30, and Extended Life Cycle Support covers only the last minor release, 7.9, until 2029-05-31; RHEL 10 has been out since 2025-05-20. RHEL 7.5 shipped squid-3.5.20-12.el7 and firewalld-0.4.4.4-14.el7; the Squid 3.5 series ended with 3.5.27 in August 2018, the Squid project supports only its 7.x series (7.7 at the time of checking), and RHEL 10 ships Squid 6.10 with firewalld 2.3 and later 2.4, against an upstream firewalld 2.5.2. What that means for each part of the build, from the renamed certificate helper to the removed ifcfg files, is in the "Checked against" sections of the Config documents and in the Articles that follow.