Bean & Bark: the bad Friday deploy

View .md

Friday, 4:44pm: revision 4 ships the new wholesale tier pricing. Friday, 4:52pm: Nadia's phone lights up — every order total on the storefront has doubled. The deploy itself was flawless; the code is wrong. This is the chapter the ledger has been preparing you for: the worst-day path, done calmly, in two commands — and the one thing a rollback will not fix for you.

You'll learn
  • Read the ledger under pressure with kanject aws deployments list
  • Roll a stage back with rollback --to previous-stable — and know the four forms --to accepts
  • Say exactly what a rollback changes on AWS (one thing) and what it never touches
  • Mark a revision broken so no future rollback lands on it

Read the ledger first

Resist the urge to hotfix. First question on a bad deploy is never "what's wrong with the code?" — it's "what was the last version that worked?" The ledger has been answering that all week:

bash 
kanject aws deployments list --env dev --limit 5
text 
Deployments  stage dev ─────────────────────────────────────────​  Revision   Target   Deployed           Commit    Status     By  4          Lambda   2026-07-11 16:44   d8e91f0   ✓ stable   you@beanandbark  3          Lambda   2026-07-09 09:12   2b41c77   ✓ stable   you@beanandbark  2          Lambda   2026-07-08 11:37   6c03aa1   ✓ stable   you@beanandbark  1          Lambda   2026-07-07 09:26   9f3c1ab   ✓ stable   you@beanandbark

There's the week in four rows: Monday's first deploy, Tuesday's config wiring, Wednesday's connection-string fix, and tonight's revision 4. Revision 3 ran clean for two days — that's the target.

Un-ship it

bash 
kanject aws rollback --env dev --to previous-stable --yes

Before you run it — a guess. How long does this take? There's no build step in the answer: rollback doesn't rebuild anything. Every deployed revision is an immutable, already-published Lambda version sitting in AWS; a rollback just flips the live alias from version 4 to version 3. Seconds, not minutes — which is exactly what you want at 4:53pm on a Friday.

--to takes four forms: previous (one back), previous-stable (the newest revision not marked broken — the one to reach for by default), a commit-SHA prefix like 2b41c77, or a Lambda version number. In a script or CI it must be explicit — a non-interactive run refuses to guess. And the ledger stays honest: the rollback appends a new entry recording what happened (its By column carries a (rollback) suffix); revision 4 is not erased, because history that can be rewritten isn't history.

Leave a warning for your future self

Revision 4 still says ✓ stable in the ledger — and some future 4:52pm, someone running rollback --to previous-stable could land right back on it. Fix the record while the context is fresh:

bash 
kanject aws deployments mark 4 --status broken --env dev
text 
  ✓  Marked revision 4 as broken.

From now on previous-stable skips revision 4 forever. The status column is the institutional memory of which deploys betrayed you — two keystrokes now, one avoided incident later.

Rehearse it while nothing is on fire

The reason tonight took two commands and no adrenaline is that you've done it before. Make that true: after any healthy deploy to dev, spend sixty seconds flipping back and forward once. --dry-run works here too if you'd rather see the plan than run it.

bash 
# The 60-second drill — run it on dev while nothing is on firekanject aws rollback --env dev --to previous --yes   # flip back one versionkanject aws rollback --env dev --to 4 --yes          # flip forward again
Recap
  • Start every incident at the ledger: deployments list shows what shipped, when, by whom, and what's still trusted.
  • rollback --to previous-stable flips the live alias to an already-published version — no rebuild, seconds to run — and appends a new ledger entry with a (rollback) suffix.
  • A rollback changes only the alias: database rows, migrations, S3 objects, and queue contracts stay as the bad deploy left them.
  • deployments mark <rev> --status broken makes previous-stable skip that revision forever. Mark while the context is fresh.
Before Monday's post-mortem — check yourself2
After the rollback, what does the ledger contain?
Later, revision 5 misbehaves too. With 4 marked broken, where does rollback --to previous-stable land?
Try it yourself 2
1 Run the 60-second drill
On your own dev stage (with at least two deploys in the ledger), rehearse the worst day: flip back one version, confirm with deployments list, flip forward again.
Two rollbacks — the second one's --to can name the version you just left.
Show solution
Flip back with --to previous, list to watch the new (rollback) entry appear, then flip forward by version number. Total cost: two alias flips and two ledger entries. The day this is muscle memory is the day a real incident is boring.
bash 
kanject aws rollback --env dev --to previous --yeskanject aws deployments list --env dev --limit 3kanject aws rollback --env dev --to 4 --yes
2 What's still broken?
The rollback restored revision 3, but list two things from revision 4's eight bad minutes that are still wrong — and say why rollback couldn't fix them.
Reread the pitfall callout: what does the alias flip actually touch?
Show solution
The doubled totals written to the orders table, and anything downstream that consumed them (an emailed receipt, a queued fulfilment message). Rollback moves the code pointer only — data the bad code wrote is a separate repair, done deliberately, ideally behind a preview and a backup.
Was this page helpful?