When a build wedges
kpack does NOT auto-retry once an Image's latest Build is in Succeeded=False. The Image stays wedged on the failed build — even if you push new code, even if the underlying cause is fixed externally, even if you set the kpack.io/rebuild annotation. The only thing that unwedges kpack is deleting the failed Build resource.
BuildRepairAgent (15-minute cadence) detects every wedged Image cluster-wide and emits OTel alerts to the /agents panel on the web console and the Alerts tab on iOS. Each alert tells you exactly what to run.
The four patterns
| Pattern | What you'll see in the alert | What it means | Fix |
|---|---|---|---|
missing-sa | serviceaccounts "system-default" not found | New project namespace skipped the kpack ServiceAccount bootstrap | Apply the SA yaml + kubectl delete build.kpack.io ... |
git-auth-dead-pat | Invalid username or token. Password authentication is not supported | Your git credential PAT expired or was revoked | c2a creds rotate ... --from-gh then kubectl delete build.kpack.io ... |
git-auth-other | authentication required / Repository not found | SA missing the per-app git secret, or wrong creds | Patch the SA's secrets[], then delete the failed Build |
transient-or-unknown | OOM / network glitch / other | Probably a one-off | Diagnose with kubectl describe build.kpack.io ..., then delete the failed Build |
Every fix ends with kubectl delete build.kpack.io <namespace>/<latest-build-name> because that's the missing step.
Why c2a app rebuild alone isn't enough in the wedged state
The rebuild API sets the kpack.io/rebuild annotation. In the non-wedged state (latest Build succeeded), kpack creates a fresh Build in response. In the wedged state (latest Build is Succeeded=False), kpack sees the annotation, sets its internal additionalBuildNeeded status, but does NOT schedule a new Build.
This is a footgun the CLI used to swallow — it returned ✅ Rebuild triggered successfully! even when no Build was scheduled. After clue2app-cli#67, the CLI verifies by default:
c2a app rebuild my-app
# Triggering rebuild for: my-app
# Verifying kpack actually scheduled a new Build (polling up to 30s)…
# ✅ New build scheduled: my-app-image-build-7 (3.2s ago)
If the verification finds no new Build, the CLI prints the unstick recipe instead of pretending success:
⚠️ Rebuild API returned success but kpack did NOT schedule a new Build.
This is the wedged-Image state...
Unstick recipe:
1. Inspect the failure: kubectl describe build.kpack.io <ns>/<latest>
2. Fix the underlying cause (SA missing, git-secret unbound, dead PAT — see `c2a creds check`)
3. Delete the failed Build to let kpack reschedule: kubectl delete build.kpack.io <ns>/<latest>
--no-verify opts out (for CI fire-and-forget); --verify-timeout=N adjusts the poll budget.
The "kpack reuses the build name after delete" gotcha
When you delete image-build-4, kpack might create a new image-build-4 (same number) rather than image-build-5. The image's status.buildCounter doesn't advance on a manual delete.
So if your watcher script looks for "a build with a higher number" as proof your fix worked, it'll never see it. Compare metadata.creationTimestamp instead, not the number.
Where this fits in the shared responsibility model
- The platform (automated) detects wedged builds and surfaces the recipe. v2 of
BuildRepairAgent(rolling out next, behindBUILD_REPAIR_ACTIVE=true) will apply the recipe itself for the known-safe patterns (missing-sa, git-auth-other) after a one-week dry-run validation period. - You apply the recipe today, or wait for v2.
- The platform (manual) rotates platform-owned credentials when those are the root cause (e.g. the shared
control-services-git-credsPAT).
See the Shared Responsibility Model for the broader split.
Related
- Pinning apps warm — separate verb, doesn't trigger any reconciliation that could wedge a build
- Daari Secrets — secret rotation here doesn't touch builds; rotation propagation is via ksvc restart