# Bean & Bark: the bad Friday deploy Friday, 4:44pm: revision 4 ships the new wholesale tier pricing. Friday, 4:52pm: Nadia's phone lights up — every order total on the storefront has **doubled**. The deploy itself was flawless; the code is wrong. This is the chapter the ledger has been preparing you for: the worst-day path, done calmly, in two commands — and the one thing a rollback will *not* fix for you. **You'll learn** - Read the ledger under pressure with `kanject aws deployments list` - Roll a stage back with `rollback --to previous-stable` — and know the four forms `--to` accepts - Say exactly what a rollback changes on AWS (one thing) and what it never touches - Mark a revision `broken` so no future rollback lands on it > **Friday, 4:52pm — the ticket:** **Nadia**: *"TOTALS ARE DOUBLED. wholesale buyers are screenshotting carts. whatever just shipped — un-ship it, then figure out why. in that order 🙏"* ## Read the ledger first Resist the urge to hotfix. First question on a bad deploy is never *"what's wrong with the code?"* — it's *"what was the last version that worked?"* The ledger has been answering that all week: ```bash kanject aws deployments list --env dev --limit 5 ``` ```text Deployments stage dev ───────────────────────────────────────── Revision Target Deployed Commit Status By 4 Lambda 2026-07-11 16:44 d8e91f0 ✓ stable you@beanandbark 3 Lambda 2026-07-09 09:12 2b41c77 ✓ stable you@beanandbark 2 Lambda 2026-07-08 11:37 6c03aa1 ✓ stable you@beanandbark 1 Lambda 2026-07-07 09:26 9f3c1ab ✓ stable you@beanandbark ``` There's the week in four rows: Monday's first deploy, Tuesday's config wiring, Wednesday's connection-string fix, and tonight's revision 4. Revision 3 ran clean for two days — that's the target. ## Un-ship it ```bash kanject aws rollback --env dev --to previous-stable --yes ``` Before you run it — a guess. How long does this take? There's no build step in the answer: **rollback doesn't rebuild anything.** Every deployed revision is an immutable, already-published Lambda version sitting in AWS; a rollback just flips the `live` alias from version 4 to version 3. Seconds, not minutes — which is exactly what you want at 4:53pm on a Friday. `--to` takes four forms: `previous` (one back), `previous-stable` (the newest revision *not* marked broken — the one to reach for by default), a commit-SHA prefix like `2b41c77`, or a Lambda version number. In a script or CI it must be explicit — a non-interactive run refuses to guess. And the ledger stays honest: the rollback **appends a new entry** recording what happened (its `By` column carries a `(rollback)` suffix); revision 4 is not erased, because history that can be rewritten isn't history. > **A rollback flips the alias — nothing else:** Code is now back to revision 3, but the world revision 4 touched is not: **database rows, migrations, S3 objects, and queue contracts stay exactly as the bad deploy left them.** The eight minutes of doubled totals are still sitting in the orders table — and fixing bad *data* is a different discipline from fixing bad *code*. (It's also a chapter of the other Bean & Bark series: [Change prod without fear](https://www.kanject.com/docs/dynostudio-bean-bark-safe-writes/) repairs rows behind a preview and a backup.) ## Leave a warning for your future self Revision 4 still says `✓ stable` in the ledger — and some future 4:52pm, someone running `rollback --to previous-stable` could land right back on it. Fix the record while the context is fresh: ```bash kanject aws deployments mark 4 --status broken --env dev ``` ```text ✓ Marked revision 4 as broken. ``` From now on `previous-stable` **skips revision 4 forever**. The status column is the institutional memory of which deploys betrayed you — two keystrokes now, one avoided incident later. ## Rehearse it while nothing is on fire The reason tonight took two commands and no adrenaline is that you've *done it before*. Make that true: after any healthy deploy to `dev`, spend sixty seconds flipping back and forward once. `--dry-run` works here too if you'd rather see the plan than run it. ```bash # The 60-second drill — run it on dev while nothing is on fire kanject aws rollback --env dev --to previous --yes # flip back one version kanject aws rollback --env dev --to 4 --yes # flip forward again ``` > **🎯 Un-shipped in under a minute:** 4:53pm: totals are correct again. 4:54pm: revision 4 is marked `broken`. The bug fix can now happen on Monday, with coffee, because rolling back and rolling *forward* are the same cheap alias flip. The ledger you've been feeding all week just paid for itself — recovery is a read of history, not an act of heroism. **Recap** - Start every incident at the ledger: `deployments list` shows what shipped, when, by whom, and what's still trusted. - `rollback --to previous-stable` flips the `live` alias to an already-published version — no rebuild, seconds to run — and appends a new ledger entry with a `(rollback)` suffix. - A rollback changes **only the alias**: database rows, migrations, S3 objects, and queue contracts stay as the bad deploy left them. - `deployments mark --status broken` makes `previous-stable` skip that revision forever. Mark while the context is fresh. **Before Monday's post-mortem — check yourself** **1. After the rollback, what does the ledger contain?** - Everything it had, plus a new entry recording the rollback ✓ — the ledger is append-only. The rollback is itself a recorded event (with a `(rollback)`-suffixed `By`), and revision 4 stays in history — that's what makes the record trustworthy in an audit. - Revision 4's entry is deleted — nothing is ever deleted from the ledger by a rollback. Erasable history is no history at all — revision 4 stays, and you *annotate* it with `mark` instead. - Revision 4 is automatically flagged `broken` — the CLI doesn't guess why you rolled back — maybe revision 4 was fine and you're rehearsing. Marking a revision broken is a deliberate, separate command. **2. Later, revision 5 misbehaves too. With 4 marked `broken`, where does `rollback --to previous-stable` land?** - Revision 3 — the newest revision not marked broken ✓ — `previous-stable` walks back through the ledger skipping every `broken` entry. Friday's two-keystroke `mark` just saved you from rolling back onto the doubled-totals bug. - Revision 4 — it's the most recent previous one — that's what plain `previous` would do — which is why `previous-stable` exists. The broken mark is exactly the signal it consults. - Nowhere — it fails because a broken revision blocks the walk — broken entries are skipped, not blocking. It only fails if *no* stable prior revision exists at all. **Try it yourself** **1. Run the 60-second drill** On your own dev stage (with at least two deploys in the ledger), rehearse the worst day: flip back one version, confirm with `deployments list`, flip forward again. _Hint:_ Two rollbacks — the second one's `--to` can name the version you just left. _Solution:_ Flip back with `--to previous`, list to watch the new `(rollback)` entry appear, then flip forward by version number. Total cost: two alias flips and two ledger entries. The day this is muscle memory is the day a real incident is boring. ```bash kanject aws rollback --env dev --to previous --yes kanject aws deployments list --env dev --limit 3 kanject aws rollback --env dev --to 4 --yes ``` **2. What's still broken?** The rollback restored revision 3, but list two things from revision 4's eight bad minutes that are still wrong — and say why rollback couldn't fix them. _Hint:_ Reread the pitfall callout: what does the alias flip actually touch? _Solution:_ The doubled totals written to the orders table, and anything downstream that consumed them (an emailed receipt, a queued fulfilment message). Rollback moves the *code pointer* only — data the bad code wrote is a separate repair, done deliberately, ideally behind a preview and a backup. > **Where next?:** Monday's fix goes smoother with help — and help is coming. In [A teammate joins](https://www.kanject.com/docs/cli-bean-bark-teammate/), Tunde joins the repo full-time, the pricing logic moves to a shared library in *another* git repository, and you meet the machinery that keeps two people's builds — and CI's — byte-identical: `kanject add lib`, `sync`, and the lockfile. The full reference is [Dependency Sources & Resolution](https://www.kanject.com/docs/cli-dependencies/). --- _Source: https://www.kanject.com/docs/cli-bean-bark-rollback/ · Kanject Docs_