Investigate · root cause

risk mediumno prod write

CrashLoopBackOff on payments-auth — sealed Stage-1 RCA

Incident inc-clb-01 · owner Platform Engineering — Nordic Federated Bank. Each verified class includes attached logs, probe output, and artefact links.

Organisation · client scope

Tenant / org
Nordic Federated Bank
tn-nordic · eu-north-1 · EU
Client / customer
FS Core Banking Platform
cu-fsprod · production · platinum
Back to workspaceEvidence

Confidence

94%

live

Risk

medium

posture

Prod write

none

required

Evidence

3

classes

Rejected

2

hypotheses

Focus

off

⌘\

Analysis package

Verification agent confidence, supporting evidence with attached logs/output, and rejected hypotheses.

Read-only Agent OS
  • No shell execution
  • No cluster-admin
  • No secret reads
  • No database writes
  • No firewall changes
  • No autonomous remediation

Root cause

Verification agent confidence score

94%

Deployment payments-auth v4.21 introduced a hard dependency on AUTH_JWKS_URI. The companion ConfigMap auth-runtime was not updated in the same ArgoCD revision, so the container panics on boot and Kubernetes restarts it into CrashLoopBackOff.

Supporting evidence

3 verified classes · expand for logs & output

  • host pay-auth-a3.payments.corpip 10.33.12.44captured Aug 9, 2026, 8:12:18 AM

    Check · k8s-read · get pod

    Logs · attached excerpt

    STATUS=CrashLoopBackOff RESTARTS=14

    Output · verified details

    { "phase": "Running", "containerStatuses": [{ "ready": false, "restartCount": 14, "state": { "waiting": { "reason": "CrashLoopBackOff" } } }] }
  • host pay-auth-a3.payments.corpip 10.33.12.44captured Aug 9, 2026, 8:12:41 AM

    Check · log-reader · previous

    Logs · attached excerpt

    panic: required env AUTH_JWKS_URI not set

    Output · verified details

    exitCode=1 reason=Error
  • host argocd.fsprod.corpip 10.33.1.20captured Aug 9, 2026, 8:13:02 AM

    Check · change-calendar · ArgoCD

    Logs · attached excerpt

    syncedAt=2026-08-09T08:01:12Z configMap unchanged

    Output · verified details

    CHG-slice-01

Rejected hypotheses

2 ruled out · with counter-evidence

Recommendation

Requires human execution — the Agent OS never remediates

Approve apr-clb-01 to patch ConfigMap auth-runtime with AUTH_JWKS_URI (from vault metadata path only), then trigger a controlled rollout restart via Workflow Studio / StackStorm. Verify ready=1/1 and no CrashLoop before closing ITSM.