modules/proxy/mod_proxy_beacon-guide.md

# A Practical Guide to `mod_proxy_beacon`

*Self-registering, self-healing reverse-proxy balancer membership over UDP.*

---

## 1. What it does

Normally, adding a backend to an httpd reverse-proxy load balancer means editing
the proxy config (or clicking around in `balancer-manager`) every time the fleet
changes. `mod_proxy_beacon` flips that around: **each backend announces itself**
to the proxy, and the proxy automatically:

- **adds and enables** the backend as a live `BalancerMember` when it first
  announces, and
- **disables it** (takes it out of rotation) when it stops announcing — then
  **re-enables it** when it comes back.

The result is a balancer whose membership tracks the running fleet with no
operator action. A backend added this way is a normal balancer member: it shows
up in `balancer-manager` and behaves exactly like a statically configured
`BalancerMember`.

```
   backend-1 ──┐  (periodic UDP "BEACON" datagrams,
   backend-2 ──┤   each advertising its own URL)
   backend-3 ──┘            │
                            ▼
                    ┌──────────────┐
                    │ reverse proxy│  ProxyBeaconListen :5555
                    │  balancer://  │  → add / enable / evict members
                    │   cluster     │
                    └──────────────┘
                            │
                       clients
```

---

## 2. How it works (the short version)

- **Transport:** plain **unicast UDP** datagrams (not multicast — multicast is
  filtered on most networks and never crosses the public Internet). Backends
  `sendto` the proxy; the proxy binds one socket and receives. No external
  library — APR sockets for transport, APR-util SipHash for auth.
- **Fire-and-forget:** a lost datagram is simply recovered by the next periodic
  announcement; reordered/replayed datagrams are rejected by a per-backend
  timestamp check. No connection, reconnect, or framing layer.
- **Where it runs:** all the work happens in a single `mod_watchdog` **singleton**
  child process, which owns the socket and applies membership changes. The change
  propagates fleet-wide through the balancer's shared-memory “updated” timestamp,
  the same way `balancer-manager` edits do.

---

## 3. Requirements

| Need | Why |
|------|-----|
| `mod_proxy` + `mod_proxy_balancer` | the balancer the members are added to |
| `mod_watchdog` | runs the background send/receive/evict loop |
| A **non-prefork** MPM (`event`, `worker`, …) | the watchdog singleton can't run under `prefork`; the module is silently inactive there |
| `mod_proxy_http` (or your backend protocol module) | to actually proxy the traffic |

Build it with `--enable-proxy-beacon` (or `--enable-mods-shared=all`). Confirm
it's present:

```sh
httpd -M 2>&1 | grep proxy_beacon      # shared build
# or
httpd -l   | grep mod_proxy_beacon     # static build
```

---

## 4. Quick start

### On the reverse proxy

```apache
# Receive backend announcements on the cluster-facing interface (UDP/5555).
ProxyBeaconListen   0.0.0.0:5555
ProxyBeaconSecret   "hex:00112233445566778899aabbccddeeff"
ProxyBeaconBalancer cluster

# Drop a backend from rotation if it goes silent for 30s; re-add on next beacon.
ProxyBeaconTimeout  30

# An initially EMPTY balancer with spare slots for the dynamic members.
<Proxy balancer://cluster>
    ProxySet growth=16
</Proxy>

ProxyPass        "/" "balancer://cluster/"
ProxyPassReverse "/" "balancer://cluster/"
```

### On each backend

```apache
# Announce this backend's routable origin to the proxy every 10s (UDP).
ProxyBeaconAddress   proxy.example.com:5555
ProxyBeaconAdvertise http://10.0.0.5:8080
ProxyBeaconSecret    "hex:00112233445566778899aabbccddeeff"
ProxyBeaconInterval  10
```

That's it. Start the backends; within one announcement interval each one appears
as an enabled member of `balancer://cluster` on the proxy. Stop one, and after
`ProxyBeaconTimeout` it drops out of rotation; start it again and it returns.

> **`ProxyBeaconAdvertise` must be the URL the *proxy* can reach** — the
> backend's routable origin, not `127.0.0.1`. It's validated at config-parse
> time and must be a full `scheme://host[:port]` **with no path** (see
> [Member identity](#member-identity)).

---

## 5. Directive reference

### Proxy side (the receiver)

| Directive | Default | Meaning |
|-----------|---------|---------|
| `ProxyBeaconListen [addr][:port]` | — | **Marks this server as the receiver.** Binds a UDP socket. `addr`/`port` are optional and inherited from the server's own `Listen`/`ServerName` when omitted (see note below). |
| `ProxyBeaconBalancer name` | — | Balancer that announced backends are added to. Bare name (`cluster`); a leading `balancer://` is stripped. Must already exist with spare `growth`. |
| `ProxyBeaconTimeout interval` | `0` (no eviction) | Seconds of silence before a member is disabled. `0` = add-only, never auto-remove. Set to a small multiple of the backends' interval to get self-healing. |
| `ProxyBeaconSecret secret` | — (**required**) | Shared cluster secret; **same value on proxy and all backends**, either `hex:` + 32 hex digits (a random key: `openssl rand -hex 16`) or a passphrase of at least 16 characters (weaker: guessable offline from one captured packet). The server fails to start if a participating server omits it or it is invalid. |
| `ProxyBeaconMaxSkew interval` | `30` | Anti-replay freshness window: reject announcements whose signed timestamp is more than this far from now (either direction). Sub-second values (`500ms`) are honoured. |

**Inheriting the listen address:** because UDP and TCP are separate port spaces,
`ProxyBeaconListen` can share the server's own service port without colliding
with the TCP listener. With no argument it binds the server's own address:port;
given just an address it inherits the port; etc. (The socket binds in an
unprivileged watchdog child, so you **can't** share a privileged port like 80/443
this way — use an explicit high port there.)

### Backend side (the sender)

| Directive | Default | Meaning |
|-----------|---------|---------|
| `ProxyBeaconAddress addr:port` | — | **Marks this server as a sender.** UDP target = the proxy's `ProxyBeaconListen` address. |
| `ProxyBeaconAdvertise url` | — | The routable `scheme://host[:port]` origin the proxy adds as a member — no path, query, fragment or userinfo (rejected at config time). Omit it and the backend beacons but advertises nothing (logged, never added). |
| `ProxyBeaconInterval interval` | `5` | How often this backend announces. Must be **meaningfully smaller** than the proxy's `ProxyBeaconTimeout`. |
| `ProxyBeaconSecret secret` | — (**required**) | Same shared secret as the proxy. |

> `ProxyBeaconListen` and `ProxyBeaconAddress` are **mutually exclusive** on the
> same server — a server is either a receiver or a sender, not both.

All directives accept `server config` and `virtual host` context. Interval-style
directives use the standard httpd
[time-interval syntax](../../docs/manual/mod/directive-dict.html#Syntax) and
default to seconds (e.g. `ProxyBeaconInterval 10` or `... 500ms`).

### Member identity

An announced URL names a backend **origin** — `scheme://host[:port]`, nothing
more. The proxy canonicalizes it and uses that one string as the member's
identity everywhere: the key for its lease and replay state, the worker name it
creates, and the name used for every later enable or evict.

- **Canonical form:** scheme and host lower-cased, the scheme's default port
  dropped, a single trailing `/` ignored. `HTTP://App.Example.com:80`,
  `http://app.example.com/` and `http://app.example.com` are **one** member.
- **Rejected:** a path (other than `/`), query, fragment, userinfo, an invalid
  port, or the `unix:/path|http://…` socket form. These are dropped and logged as
  `unacceptable url`; the backend is not added. `ProxyBeaconAdvertise` enforces
  the same rule when the config is parsed, so a backend can't be misconfigured
  into announcing one.

**Why no paths?** The balancer finds a member by *longest-prefix* match.
`http://10.0.0.5:8080/one` and `http://10.0.0.5:8080/one/two` would be tracked as
two members with separate leases but resolve to one worker — so when one lease
lapsed, the proxy would disable a worker whose other lease was still current. If
you need different paths to go to different places, do that on the proxy with
`ProxyPass`; advertise only where the backend lives.

### Ownership and hand-off

Beacon marks every worker it creates with the **`Bcn`** status (flag `B`, visible
in the balancer-manager). It only ever enables or evicts workers carrying that
mark: a `BalancerMember` from your configuration, or one an administrator added,
is never touched, even if a backend announces exactly its URL.

The announcements and leases are handled by **one** watchdog child. If that child
dies, the watchdog moves to another child, which starts with no memory of the
members. On taking over it **adopts every `Bcn` member** in the balancer and
starts their leases afresh:

- a member that has since gone silent is evicted `ProxyBeaconTimeout` after the
  hand-off, instead of staying enabled forever;
- a member that was already disabled is treated as evicted and is re-enabled by
  its next announcement.

Two things to know: a dead backend can stay in rotation for up to one
`ProxyBeaconTimeout` after a hand-off, and the per-member replay high-water mark
is not carried over — during a hand-off a captured announcement still inside
`ProxyBeaconMaxSkew` can be replayed once per member, refreshing that lease for one
timeout (it cannot add a member without the secret).

**Holding a member out of rotation.** Beacon owns a member's **Disabled** flag
(lease expiry sets it, the next announcement clears it), so a manual "Disable" in
the balancer-manager is overridden. To keep a member out regardless of what it
announces, set it **Stopped** or **Drain** — beacon never touches those, nor the
health-check failure flag.

**Capacity.** At most 256 announced URLs are tracked. Records of evicted members
and of failed adds are released after being silent for more than twice
`ProxyBeaconMaxSkew` (a replay of their last message would be stale by then), so
they don't lock out new backends; a member that comes back is re-adopted. A
balancer itself accepts at most 100 runtime members (`growth`). A holder of the
secret can still fill the table with distinct URLs until they fall silent.

**Socket problems.** If the UDP socket can't be opened (unresolvable address, port
in use) it is retried with back-off, 30 s doubling to 10 min. Name lookups are
synchronous on the single watchdog thread shared by every beacon configuration of
the server, so an unresolvable name can stall them all for as long as the resolver
takes to give up — **use IP addresses** in `ProxyBeaconAddress` / `ProxyBeaconListen`.

> A new owner can only start once the old one's `watchdog-callback` mutex is
> released. If the default mechanism isn't released when its holder is killed on
> your platform, pick one that is: `Mutex sysvsem watchdog-callback`.

---

## 6. Security — read this before production

The control channel decides where the proxy sends client traffic, so an
unauthenticated channel is a route-hijack / SSRF risk: anyone who can reach the
receive port could announce an arbitrary backend URL, and **UDP source addresses
are trivially spoofable.**

`ProxyBeaconSecret` is therefore **required** (identical on proxy and every
backend) — `httpd -t` and startup both fail if a participating server omits it, so
there is no unauthenticated mode to fall into. With it:

- Each announcement is signed with a **SipHash-2-4 MAC** plus a timestamp; the
  proxy recomputes the MAC (constant-time compare) and drops anything forged or
  tampered **before parsing the URL**.
- **Replay protection is two-layered:** (1) the `ProxyBeaconMaxSkew` freshness
  window rejects stale timestamps, and (2) a per-backend high-water mark rejects
  any timestamp that doesn't strictly advance — so capturing and re-sending a
  dead backend's last datagram (to keep it from being evicted) is rejected even
  inside the freshness window.

**What the secret is.** A valid MAC proves only that the sender holds the
secret — not that it owns the URL it announces, may speak for another backend, or
is ready to serve. Every holder can register any origin the proxy can reach,
keep other members' leases alive, and use up growth slots and tracking records.
Treat the secret as an **administrator credential for the balancer** and share it
only among hosts you'd trust with that; it is not a boundary between partially
trusted workloads. Because the key is derived from the passphrase with a fast
hash (and the tag is 64 bits), a captured announcement allows offline guessing:
**use a long random secret** (at least 16 characters is enforced).

**What is checked.** Every datagram is dropped unless it is ≤ 2048 bytes with no
NUL, carries a valid MAC (even when no `ProxyBeaconBalancer` is set, so log-only
mode logs only authenticated announcements), has the exact `BEACON key=value … mac=…`
shape with no repeated field and a plain decimal `ts=`, and names a printable-ASCII
origin.

Operational notes:

- **Clocks must be roughly in sync** (NTP). The timestamp check compares the
  announcement's time against the proxy's clock; widen `ProxyBeaconMaxSkew` if
  your hosts drift.
- The secret lives in your config file — **restrict its permissions like a
  private key.**
- Announcements are **authenticated, not encrypted.** The payload is operational
  metadata (URLs), not secrets. There's no transport confidentiality (DTLS would
  be a separate future layer).

---

## 7. Tuning

- **Interval vs. timeout.** Keep `ProxyBeaconInterval` well under
  `ProxyBeaconTimeout` so an occasional dropped datagram doesn't evict a healthy
  backend. A common ratio is timeout ≈ 3–6× interval (e.g. interval 10s, timeout
  30–60s).
- **Size `growth` for your peak fleet.** A runtime-added member occupies one
  growth slot **for the life of the proxy process** — it's *disabled* on
  eviction, not removed (httpd can't remove workers at runtime, mirroring
  `balancer-manager`). So `growth` must cover the maximum number of distinct
  backend URLs you ever expect, not just the count live at one moment.
- **One proxy, many backends.** A single receiver serves the whole fleet. For
  multiple proxies, run `ProxyBeaconListen` on each and point backends at all of
  them (one `ProxyBeaconAddress` server per proxy, or accept that only the
  configured proxy learns about the backend).

---

## 8. Verifying & troubleshooting

**See members appear:** enable `balancer-manager` on the proxy and watch the
worker list:

```apache
<Location "/balancer-manager">
    SetHandler balancer-manager
    Require host localhost
</Location>
```

**Watch the log.** Bump verbosity for the module and tail the error log:

```apache
LogLevel proxy_beacon:info
```

Useful log signals (grep the error log):

| Log fragment | Means |
|--------------|-------|
| `received: BEACON ... url=...` | proxy is receiving announcements |
| `added backend ... balancer://cluster` | member created + enabled |
| `evicted backend ...` | a backend went silent past the timeout |
| `re-enabled backend ...` | a previously-evicted backend came back |
| `dropped ... mac mismatch` | wrong/missing secret on a sender (or a forged datagram) |
| `replayed/reordered ts` | a stale/duplicate datagram was rejected |
| `dropped ... oversize datagram` / `embedded NUL` / `bad frame` / `duplicate field` / `bad ts` | a datagram that isn't a well-formed announcement (see "What is checked") |
| `dropped ... member table full` | 256 urls are already tracked; records free up as silent ones age out (see Capacity) |
| `dropped ... unacceptable url` | an announcement's `url=` is not a plain origin (it has a path, query, userinfo, a bad port, or is a `unix:` URL) |
| `took over N existing member(s)` | a new watchdog owner adopted the beacon-created members already in the balancer (see [Ownership and hand-off](#ownership-and-hand-off)) |
| `dropped ... member not managed by beacon` | a backend announced the URL of a worker beacon did not create (configured or admin-added); it is left alone |
| `dropped ... balancer add not confirmed` | the balancer reported success but no such member exists; retried after the back-off |
| `ProxyBeaconSecret is required` (startup) | a participating server has no `ProxyBeaconSecret` — startup aborts |

**Common gotchas:**

- *Backend never joins.* Usually a **secret mismatch** (look for `mac mismatch`)
  or a firewall blocking **UDP** on the beacon port (it's UDP, not TCP — separate
  rule). Confirm the proxy is logging `received: BEACON` at all.
- *Nothing happens, no errors.* You're probably on the **prefork MPM** — the
  watchdog singleton doesn't run there, so the module is inactive. Switch to
  `event`/`worker`.
- *Traffic 503s to a backend that's “up”.* Check that `ProxyBeaconAdvertise` is
  the URL the **proxy** can reach (not the backend's loopback) and that the
  balancer has free `growth` slots.
- *Healthy backend keeps getting evicted.* Interval too close to timeout, or
  clock skew larger than `ProxyBeaconMaxSkew`. Widen the timeout/skew or sync
  clocks.

---

## 9. The wire format (for the curious)

One ASCII, space-separated, key=value datagram per announcement:

```
BEACON url=http://host:port host=<h> pid=<n> seq=<n> ts=<usec> mac=<hex>
```

- `url=` — the routable origin added as a `BalancerMember`; `scheme://host[:port]`
  only (see [Member identity](#member-identity)).
- `ts=` (microseconds since epoch) and `mac=` — present only when a secret is set;
  `ts` drives replay protection, `mac` is the SipHash signature.
- `host=` / `pid=` / `seq=` — informational. (`seq` resets on backend restart, so
  it's *not* used for replay detection — the wall-clock `ts` is.)

---

## 10. See also

- `mod_proxy`, `mod_proxy_balancer` — the balancer these members live in
- `mod_proxy_hcheck` — active health checking, complementary to beacon liveness
- `mod_watchdog` — the background runner this module rides on
- `modules/proxy/README.beacon` — the architecture/internals companion to this guide
- `docs/manual/mod/mod_proxy_beacon.xml` — the formal directive reference