Skip to main content

Shared Responsibility Model

Running an app on Clue2App is a partnership. Some things you do, some things the platform does on your behalf — automatically via the agent layer, and some things the platform does manually via a human operator (us). This page is the explicit boundary.

If a failure happens in the middle of the night and you can't tell whose problem it is, look here.

The three lanes

You (developer / app owner)Platform (automated) — agents heal + detect 24/7Platform (manual) — operator action when escalated
App codeWrite it. Push to git.n/an/a
App secretsCreate via c2a daari secrets create. Rotate via UI / CLI when you decide.Detects when secrets are missing on a build (BuildRepairAgent Pattern A/B).Provides AWS Secrets Manager backing on request (Phase 2).
Git credentialsProvide initial PAT. Authorize SAML SSO for the org.GitCredsHealthAgent scans every 6h, alerts on dead / expiring PATs.We rotate the platform-owned PAT pool quarterly.
Builds (kpack)Push code. c2a app rebuild.BuildRepairAgent (15min) classifies wedged builds + emits OTel alerts with the exact kubectl recipe. v2 (active mode) auto-applies the repair after a dry-run window.We unwedge by hand if you ping us (or wait for v2).
Service bindingsc2a daari secrets bind <app> <secret>. Coordinator merges into the C2A_SYSTEM_ENV the runtime reads.Agents validate the binding lifecycle didn't get clobbered by a re-deploy.n/a
Knative service healthCode reads C2A_SYSTEM_ENV correctly.SurfaceDriftAgent (15min) probes /health, /version, /tools on every ksvc, alerts on regressions.Operator-on-call investigates critical alerts when paged.
Certificates (TLS)n/a (cert-manager owns it).CertHealthAgent (6h) alerts on Ready=False or notAfter within 14d.We rotate when cert-manager renewal fails.
Namespace bootstrapCreate a project.ProjectBootstrapDriftAgent (6h) catches namespaces missing the system-default ServiceAccount kpack needs.Coordinator now auto-bootstraps new project namespaces (coordinator#146). Legacy namespaces healed on-detection.
AWS / GCP / Groq upstream healthn/aAwsIamHealthAgent, LlmUpstreamAgent, etc. probe every upstream we depend on. Alerts on key-rejected / rate-limited.We rotate the platform-owned credentials.
GitHub App migrationn/a (Phase 3, not active yet)Future: GhAppTokenRotator mints installation tokens, replaces per-user PATs.We register the App in the org and store the private key.
OTel pipelinen/aAgents emit alerts → otel-collector → coordinator → persisted in realtime_notifications → surfaced on /agents console panel + iOS Alerts tab.We tune the collector pipeline as alert volume grows.
Cost trackingView on /costs.CostTrackerAgent snapshots CCU usage hourly; computes monthly trend.We adjust the platform markup if needed (currently 200% over infra base).

What each lane means in practice

Your lane (you do this)

These are the things only you can do because they encode your intent:

  • Decide what your app is. Code, dependencies, environment variables, secrets values.
  • Decide who can touch it. Project membership, RBAC roles.
  • Decide when to rotate. A secret rotation is a decision about trust; the platform never auto-rotates your application credentials.
  • Decide when to bind. Which apps consume which Daari Secret / LLM Gateway instance / Postgres instance.

Platform (automated, "agentic")

These are things the platform watches continuously. You shouldn't have to think about them in steady state. When something goes wrong, an OTel alert lands on:

  • The /agents panel in the web console
  • The Alerts tab in the iOS app
  • (Future) Slack when wired

Each alert carries a suggested_fix field that is the exact recipe to resolve. In active mode (rolling out over the next quarter, behind a per-agent feature flag), the agent applies the fix itself and emits a confirmation alert instead.

The agents shipped today:

AgentCadenceWhat it catchesRepair mode
cert-health6hCerts not Ready, expiring within 14dAlert only
git-creds-health6hDead / expiring PATs across all namespacesAlert only
buildpack-stack6hPaketo stack drift (would have caught libatomic regression)Alert only
github-ratelimit1hGitHub API rate limit < 100 remainingAlert only
postgres-health6hProject Postgres instances unreachable / slowAlert only
aws-iam-health1hSTS + ECR write checks; partial failures = policy driftAlert only
llm-upstream30mGroq / OpenAI / Anthropic reachability per gatewayAlert only
project-bootstrap-drift6hNamespaces missing the kpack system-default SAAlert only
build-repair15mWedged kpack Images that won't auto-retryDry-run / alert only — v2 active
surface-drift15m/health breaking, /version regressing, /tools count droppingAlert only

Platform (manual, "operator")

These require human judgement or have a high enough blast radius that we run them ourselves:

  • AWS IAM credentials that the platform uses (the c2s-iam-admin user). Quarterly rotation, by us.
  • Cluster-wide control-plane patches (coordinator, core, console, agents redeploys).
  • The Paketo BP_NODE_VERSION=22.*.* pin that goes on every new kpack Image (k8s-core-service auto-applies; we'll remove the pin when Paketo ships fixed run images).
  • AWS Secrets Manager backing for high-stakes secrets when you tag a Daari Secret source=aws-secrets-manager (Phase 2 — not yet live).
  • GitHub App registration in the clue2solve org (Phase 3 of the per-user PAT migration).
  • OTel collector tuning as alert volume changes.

Recent capability inventory

What we shipped between 2026-04 and 2026-06. Each capability is in one or more of the lanes above:

Platform reliability sweeps (agent layer)

Pattern: paired core REST endpoint + Python agent. The agent runs on a cadence, calls the core endpoint, emits OTel alerts. See agentic-architecture.

  • cert-health + 7 other detection agents (see table above)
  • OTel pipeline end-to-end: agents → otel-collector → coordinator /api/notifications/ingest → JSONB realtime_notifications rows → /agents panel + iOS Alerts tab

Build experience

  • c2a app rebuild verifies by default. After the API call, polls for ~30s to confirm kpack actually scheduled a new Build. When wedged, prints the exact kubectl delete build.kpack.io ... recipe instead of pretending success.
  • PreBuild Audit endpoint (POST /api/v1/k8s/code/audit) — pre-build static scan for committed secrets, hardcoded AWS keys, PEM private keys. Surfaced via c2a audit <git-url> and /audit console page.
  • Bootstrap drift detection. Every project namespace now auto-gets the system-default ServiceAccount kpack needs at project-create time (coordinator#146).

Daari Secrets (service catalog category)

Project-scoped key/value secrets, bindable to apps via the existing service-binding lifecycle. At runtime, every bound value lands in a single C2A_SYSTEM_ENV env var whose JSON contains three lanes: variables (plain env), secrets (masked env), and binding (decoded service references keyed by display name). Apps parse it once at startup and read by service name.

  • Coordinator endpoints: GET /api/v1/daari/secrets/ (list + detail + bindings), POST create, PUT /values rotate, DELETE
  • CLI: c2a daari secrets list / show / create / rotate / delete / bindings / bind
  • Console: pinned Secrets tab inside /daari with deep links to /daari/secrets/<k8sName>
  • Values are NEVER echoed back from any surface — only keys + metadata + bindings
  • cert-manager-managed TLS secrets surface read-only with a "managed by cert-manager" badge
  • Quota: 50 per project (decision #5 of the locked design)

Build wedge repair

  • BuildRepairAgent (15min) calls GET /api/v1/k8s/builds/wedged. Classifies failures into 4 patterns (missing-sa, git-auth-dead-pat, git-auth-other, transient-or-unknown) with copy-paste fixes that always end with the required kubectl delete build.kpack.io ... step (kpack won't auto-retry once latest Build is in Succeeded=False — confirmed during the expense-mcp incident 2026-06-08).

Surface drift

  • SurfaceDriftAgent (15min) probes well-known endpoints (/health, /version, /tools) on every ksvc cluster-wide. Tracks last snapshot in memory; alerts on regressions:
    • /health was 200, now isn't (critical)
    • /version went backwards by semver (warning)
    • /tools count dropped by >20% (warning)
    • Endpoint was reachable, now unreachable (critical)
  • Honest limitation: doesn't catch always-wrong cases (baseline is what was there last scan). v2 will add per-app canonical-source manifests.

Where to look when something breaks

SymptomFirst check
App not buildingc2a build list <app>, then c2a app rebuild <app> (verifies by default; prints unstick recipe when wedged)
Build fails on git authc2a creds check -A --only-dead
c2a heal says 500c2a heal endpoint was never implemented; use specific verbs (c2a app rebuild, c2a creds rotate) instead
Knative pod won't start/agents panel — surface drift or build-repair alerts
Daari Secret rotate didn't take effect on appRestart the ksvc (auto-restart on rotate is decision #3 of the spec, ships in v2)
You hit a cap (project limit, secrets limit)The error message names the limit; if it's blocking you, ping us

Why this matters

Without this boundary, every failure ends up in the same Slack channel labelled "broken." Drawing the line means:

  • You spend zero time investigating cluster-internal issues.
  • We spend zero time guessing whether a failure is your config or our infra.
  • The agent layer takes over the boring repeatable repairs so you (and we) stay on the hard problems.

If you find a failure mode where this table is wrong or silent, tell us — that's the highest-leverage feedback you can give us.