fs-prod-cs-tool2 NotReady — registry egress interruption

Incident inc-4821 · owner Network Operations — Nordic Federated Bank. Each verified class includes attached logs, probe output, and artefact links.

Back to workspace Evidence

Analysis package

Verification agent confidence, supporting evidence with attached logs/output, and rejected hypotheses.

Read-only Agent OS

No shell execution
No cluster-admin
No secret reads
No database writes
No firewall changes
No autonomous remediation

Root cause

Verification agent confidence score

88%

Registry egress traffic from fs-prod-cs-tool2 is being reset mid-transfer, most likely by SSL inspection on the outbound path introduced in change CHG-20482. Container image layers for the CNI plugin cannot complete, so the container runtime network never becomes ready and the node reports NotReady.

Supporting evidence

4 verified classes · expand for logs & output

host fs-prod-cs-tool2ip 10.42.6.21captured Aug 2, 2026, 6:43:29 AM

Check · journalctl -u kubelet -u containerd · PullImage / layer transfer

Logs · attached excerpt

2026-08-02T06:38:11.204Z kubelet[1184]: E  failed to pull image "registry.corp.internal/cni/calico-node:v3.27.2"
2026-08-02T06:38:11.204Z kubelet[1184]: E  read tcp 10.42.6.21:52344->198.51.100.44:443: read: connection reset by peer
2026-08-02T06:38:14.228Z containerd: transfer aborted at 5.2MB/38.4MB — connection reset by peer
2026-08-02T06:39:01.884Z containerd: retry 2 · aborted at 4.9MB/38.4MB — connection reset by peer
2026-08-02T06:39:57.103Z containerd: retry 3 · aborted at 5.1MB/38.4MB — connection reset by peer
2026-08-02T06:42:18.440Z kubelet[1184]: E  ErrImagePull (9 occurrences in 10m window 06:32–06:42)

Output · verified details

{
  "image": "registry.corp.internal/cni/calico-node:v3.27.2",
  "destination": "198.51.100.44:443",
  "source": "10.42.6.21",
  "failures": "9/9",
  "abortBoundaryMb": [5.2, 4.9, 5.1, 5.0],
  "layerExpectedMb": 38.4,
  "internalMirror": "pause:3.9 SUCCEEDED (control — external only fails)"
}

host fs-prod-cs-tool2ip 10.42.6.21captured Aug 2, 2026, 6:42:18 AM

Check · kubectl get events --field-selector involvedObject.name=calico-node-7xk2m

Logs · attached excerpt

2026-08-02T06:38:11Z  Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox for calico-node-7xk2m_kube-system
2026-08-02T06:38:46Z  Warning  FailedCreatePodSandBox  kubelet  rpc error: network plugin is not ready: cni config uninitialized
2026-08-02T06:39:21Z  Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox: failed to setup network for sandbox
2026-08-02T06:39:56Z  Warning  FailedCreatePodSandBox  kubelet  … repeat (~35s interval)
… (14 additional FailedCreatePodSandBox through 06:42:18Z)

Output · verified details

{
  "pod": "calico-node-7xk2m",
  "namespace": "kube-system",
  "phase": "ContainerCreating",
  "node": "fs-prod-cs-tool2",
  "FailedCreatePodSandBox": 17,
  "ErrImagePull": 9,
  "intervalSec": 35,
  "window": "2026-08-02T06:32Z→06:42Z"
}

host fs-prod-cs-tool2ip 10.42.6.21captured Aug 2, 2026, 6:45:33 AM

Check · tcp probe 198.51.100.44:443 from 10.42.6.21 (read-only net-diag)

ev-4

Logs · attached excerpt

probe tcp 198.51.100.44:443 from 10.42.6.21
  dns:       registry.corp.internal → 198.51.100.44 (ok, 4ms)
  handshake: ok (TLSv1.3, 118ms)
  stream:    RST after 6.4s, 5.1MB transferred (layer expected 38.4MB)
egress path: node-subnet -> fw-core-02 -> ssl-inspect-appliance-03 -> internet-edge
change correlation: CHG-20482 (SSL inspection policy update)

Output · verified details

{
  "fqdn": "registry.corp.internal",
  "resolvedIp": "198.51.100.44",
  "port": 443,
  "dnsMs": 4,
  "tls": "TLSv1.3 OK (118ms)",
  "streamRstAfterSec": 6.4,
  "bytesBeforeRstMb": 5.1,
  "change": "CHG-20482",
  "exclusionPresent": false
}

host fs-prod-cs-tool2ip 10.42.6.21captured Aug 2, 2026, 6:44:10 AM

Check · node conditions + PromQL utilisation snapshot

Logs · attached excerpt

conditions:
  MemoryPressure=False  DiskPressure=False  PIDPressure=False  NetworkUnavailable=False
  Ready=False reason=KubeletNotReady (CNI / runtime network — not resource pressure)

node_cpu_utilisation{node="fs-prod-cs-tool2"}        0.21
node_memory_utilisation{node="fs-prod-cs-tool2"}     0.48
node_filesystem_used_ratio{node="fs-prod-cs-tool2"}  0.39

Output · verified details

{
  "cpuPct": 21,
  "memPct": 48,
  "diskPct": 39,
  "pressureConditions": "none",
  "Ready": false,
  "Ready.reason": "KubeletNotReady",
  "interpretation": "Host load flat; NotReady is CNI/image-pull, not exhaustion"
}

Rejected hypotheses

3 ruled out · with counter-evidence

Recommendation

Requires human execution — the Agent OS never remediates

Validate outbound TCP 443 connectivity from the node subnet to the registry egress range and confirm SSL-inspection exclusions cover registry.corp.internal and the upstream mirror. Re-run the image pull after the exclusion is verified.