Zero-downtime deploys for a support desk: what we did and what it took

A support desk is a bad place for a deploy to be visible. An agent mid-reply loses a draft; a visitor’s chat socket drops and the “someone is online” dot goes grey; a webhook arrives at a process that is shutting down. For most of Kommio Cloud’s life a deploy did exactly that: it recreated the one app container, a few seconds of dropped requests while the new version migrated and started, and before every deploy it dumped the whole database (19 GB) as insurance. This week we fixed it. Here is what the deploy looks like now and what it took in the product to make it safe.

Two slots behind Caddy

The app now runs in two slots, kommio-a and kommio-b, with a file naming the one that is serving. Caddy, the TLS proxy in front, health-checks both slots on /api/health/ready every second, sends traffic only to the ready ones, and retries a request it could not deliver on the other slot for up to fifteen seconds. A deploy is a swap:

1. Preflight the new image against the running database. The same check a self-hosted operator runs before an upgrade; it exits non-zero on anything that would make the start fail, and then nothing has changed.
2. Dump the database only if the image has migrations pending. Most deploys do not, and a 19 GB dump before each of them was the slowest part of the old routine.
3. Migrate first, while the active slot keeps serving, with --migrate-only. This is the step that needs the most from the product; see below.
4. Start the idle slot on the new image, probe it directly until it reports ready with the new version, then stop the old slot.

If the new slot never becomes ready, the script stops it and the old one, which never stopped, keeps serving. A rollback is a deploy of the previous tag through the same path, so it is also without downtime. In a local simulation the very first swap dropped one request (Caddy itself had to be recreated once for a new mount); later swaps dropped none.

What the product had to guarantee

A proxy and a script get you most of the way. The rest is three properties of Kommio that the swap relies on, each of which is now checked by CI rather than remembered by a person.

The old version runs on the new schema. Between step 3 and step 4 the old slot serves on a database the new version has migrated. That only works if migrations are additive: add tables, add nullable columns or columns with defaults, add indexes; never drop or rename what the previous release reads or writes. We had written that rule down before; now CI enforces it. On every push, and from each of the newest two release tags, the upgrade rehearsal seeds a database with the older build, migrates with the new one, and then starts the old build next to the new one on the migrated schema and proves it still answers, signs in and lists conversations, and that its own migrate step is a no-op. A migration that cannot be additive says so on a line of its own; CI skips the check for it, the release notes tell self-hosters to stop the old version first, and the deploy script does not attempt a swap.

A stopping process takes itself out before it closes. Caddy decides where to send traffic from the health check, so a process that closed its listener first would take dropped requests for up to a second. Instead, on SIGTERM an API process answers ready with 503 at once and keeps serving for a drain period (five seconds by default), then stops accepting connections and lets in-flight requests and jobs finish, within a grace period of 45 seconds. Give the proxy’s check an interval shorter than the drain and the container a stop grace longer than the drain plus the request timeout, and no request lands on a process that is going away.

Two processes can run every role at once. For the seconds the slots overlap, both carry the API, the workers and the scheduler. Jobs and the event outbox lock their rows, so a job runs once whichever process picks it up; the periodic schedules are leader-elected, so they fire once; a report rebuild the new version asked for is left alone by an old worker that picks it up. None of this was added for the deploy: it is what already made separate API and worker processes safe.

The one that was missing: live updates

There was a fourth property we thought we had. Agents’ and visitors’ browsers hold a socket to the API process they happen to be connected to, and that process’s realtime hub pushes frames: a new message, a status change, typing. The listener that turns an event into frames ran on whichever worker took the job and published to that process’s hub. With one process, that is the same hub. With two slots overlapping, a browser on slot A got nothing from a listener on slot B until its next fetch. Nobody had noticed, because nothing had ever run two API processes in anger.

The fix stays on Postgres, like the rest of the product. The listener still publishes to its own hub, then announces the event’s id on a Postgres notification channel. Every API process listens there, skips its own announcements, loads the event and runs the same listener against its own hub. Only the id travels, so nothing is duplicated; a single process is unchanged. It is best effort across processes: an announcement lost during a reconnect means those browsers catch up on their next fetch, while the local hub always has the frame. The same change made “is someone online” correct across replicas: each API process writes its connected agents to a shared table every 30 seconds, and the widget reads that as well as its own hub.

That one paragraph is also what makes several API replicas fine in general, on Kubernetes or anywhere else. One process must carry the scheduler; the rest can be as many as you need, behind any proxy that health-checks readiness and a database pooler that passes LISTEN through.

Doing the same on your own install

The self-hosted edition stays one container by default, because one container is the right size for most teams and a restart during an upgrade takes seconds. If seconds matter to you, an install with two replicas runs exactly this routine: back up, migrate once with the new version while the old replicas serve, start a new replica and wait until it reports ready with the new version, stop one old replica, start the second new one, stop the last old one. It needs a proxy that health-checks /api/health/ready more often than the drain period, and it must not be used for a release whose notes call a migration breaking; those are rare and say so. The operator guide that comes with the edition has the steps; the edition itself is not public yet, and how it is built says how to ask for access.

What it took

Most of the work was in CI, not in the deploy script: the rehearsal from the last two release tags, the drain on shutdown, and the realtime relay. The lesson is the ordinary one: the deploy was only ever as safe as the properties the product could prove, and a rule in a guide is not the same as a job that fails when someone breaks it.