Platform engineering

Platform engineering: turn intent into a supported service

A platform gives users a predictable way to request capabilities. It validates the request, reconciles the desired state, reports useful status and makes failures diagnosable. The core skill is connecting those responsibilities without requiring every user to understand the implementation.

Domains and scope

Weights follow the CNPE overview ↗. The exercises use Argo CD, Kyverno and Prometheus on a disposable kind cluster. They provide concrete examples, not complete coverage of every platform tool or competency. The supplemental concepts below cover lifecycle/deletion, secrets, tracing, cost analysis, and user-facing status; those explanations do not replace hands-on validation. Broader backup/recovery and supported-tool practice remain necessary.

Build the command-to-evidence habit

Coach Caz: A platform should make the right thing repeatable. A form that says success while nothing exists is motivational fiction.

Use each Practice loop below as a short rehearsal. Read its starting state, predict the result of the first command, then compare that prediction with the evidence. The outputs are illustrative: names, timestamps, addresses, and counts vary. Commands use the example's names; substitute the actual task's namespace, resource, host, and file names when transferring the pattern to a drill.

1. Recognize: say which symptom puts you on this path.
2. Inspect: run the smallest check that separates two plausible causes.
3. Act: change the field or configuration supported by that evidence.
4. Prove: repeat the failed operation and check a constraint that must still hold.
5. Repeat: hide the commands, change one input, and rebuild the sequence from memory.

Use a disposable practice environment for changes. A case that assumes an installed controller, tool, or prepared resource says so; it does not install those prerequisites for you. Linux host commands belong inside a practice Linux VM or the specified lab node. These examples teach the investigation pattern; use the CLI's assigned task and grader for recorded reps.

Architecture: defaults and limits serve different purposes

A LimitRange can fill in a missing container request or limit and reject values outside bounds. A ResourceQuota limits aggregate consumption in a namespace. Defaults make ordinary workloads easier to submit; quotas constrain the combined result. Verify the admitted Pod, because admission may change the submitted template. LimitRange ↗, ResourceQuota ↗.

A PriorityClass affects scheduling and potential preemption; it does not create extra cluster capacity. Namespace separation also needs identity and network controls if it is intended to protect tenants. The multitenancy exercise checks resource governance, not full hostile-tenant isolation.

The two admission plugins run in a fixed order: LimitRanger fills in missing requests and limits first, then ResourceQuota checks whether current usage plus the new request fits within hard. That order is why a quota on requests.cpu needs a LimitRange beside it in any namespace where developers omit resources — without defaults every such Pod is refused with must specify requests.cpu. Recognise which guard said no from the message: maximum cpu usage per Container is 1 is the LimitRange max; exceeded quota is the ResourceQuota. Proof of success is the admitted Pod read back from the API with the defaults filled in, plus the rejection text for an oversized one. Cap a tenant with quota and LimitRange (kubefit drill cnpe-13-tenant-quota-limitrange) grades exactly those two facts; Set tenant resource defaults and ceilings (kubefit drill cnpe-04-multitenancy) adds a PriorityClass to the same namespace.

Design a platform contract and its failure boundaries

Start with the developer's operation: request an environment, deploy a revision, obtain a credential, or retire a service. Define the supported inputs, defaults, status, ownership, and deletion policy. A form or CRD that accepts a request is only the front door; the platform must report whether provisioning converged and how the requester can recover.

Compute placement, network reachability, and storage topology determine where that contract can succeed. Namespace quotas help share resources but are not sufficient isolation between hostile tenants. Decide which identities, networks, nodes, and data need stronger boundaries. Prove it: a permitted request succeeds, another tenant cannot access its resources, and a dependency failure produces an actionable status. Kubernetes multi-tenancy ↗

Right-size capacity and attribute its cost

Requests reserve scheduling capacity; measured usage shows consumption. Neither alone is a bill. Attribute shared and idle capacity using a stated allocation model, then compare utilization, throttling, OOMs, latency, and workload growth before reducing resources. An HPA changes replicas; it cannot provide node capacity that the cluster does not have.

Use a tool such as OpenCost to connect namespace/workload ownership with allocated compute, memory, and storage costs. Prove it: explain the allocation window and units, make a bounded sizing change, and show that service objectives hold under load. The current quota/HPA drills do not grade cost attribution or node autoscaling. OpenCost allocation ↗

Practice loop: a Deployment exists but quota blocked its Pods

Starting state: namespace concepts-platform has a Deployment web, a CPU request quota, and no resource defaults. Its Pod template omits CPU requests.

kubectl -n concepts-platform get deployment,replicaset,pods
kubectl -n concepts-platform describe replicaset WEB_REPLICASET
kubectl -n concepts-platform get resourcequota,limitrange -o yaml

Replace WEB_REPLICASET with the name owned by web. Look for: Deployment and ReplicaSet exist, no Pod was admitted, and FailedCreate includes must specify requests.cpu. That differs from a Pending Pod with Insufficient cpu: admission never handed this case to scheduling.

Act: if the workload contract requires explicit resources, add suitable requests/limits to its source manifest. In an unmanaged practice Deployment, this example supplies measured practice values:

kubectl -n concepts-platform set resources deployment/web --requests=cpu=100m,memory=64Mi --limits=cpu=200m,memory=128Mi
kubectl -n concepts-platform rollout status deployment/web --timeout=90s
kubectl -n concepts-platform describe resourcequota

For a platform-defaulting requirement, implement the appropriate LimitRange instead. Do not evade governance by deleting the quota. In GitOps, change the declared source so reconciliation retains the repair.

Prove: new Pods contain the intended resources, quota usage stays within hard, and an oversized request is still refused. If the error changes to exceeded quota, calculate current usage plus the new request before choosing a new action.

Repeat: explain why a ReplicaSet event can be more useful than get pods.

Coach Caz: The membership desk said no. The scheduler has not even seen your training plan yet. ResourceQuota ↗

GitOps: reconcile the declared state

An Argo CD Application links a source revision and path to a destination. Sync describes whether desired manifests match managed resources; health describes resource condition. A synced application can still be unhealthy. Pruning removes resources no longer declared; self-heal corrects drift for managed resources that remain. Argo CD automated sync ↗.

Test reconciliation by introducing the specific permitted drift and watching it converge. Do not use manual edits as the final fix for a resource that Git immediately resets. In production, immutable revisions and review controls make changes traceable; this fixture deliberately uses the task's requested HEAD revision. Reconcile an application with Argo CD (kubefit drill cnpe-01-gitops-argocd) is the automated form of that loop.

Without syncPolicy.automated the controller still diffs on every refresh and every watch event, so status.sync.status flips to OutOfSync seconds after someone scales a managed Deployment by hand — but nothing changes until a human sets .operation.sync on the Application (the argocd app sync command does the same thing through the API). Inspect status.operationState for the actual result: an operation can fail. A successful sync can add a history entry; correlate its revision and time with the corrected workload, because history alone does not explain why an operation occurred. Repair drift with a manual sync (kubefit drill cnpe-20-argocd-drift-detection) grades the history and the replica count together for that reason.

An AppProject is the trust boundary: sourceRepos, destinations and clusterResourceWhitelist are allow-lists, and the default project wildcards all three. An Application that points outside its project is admitted by the API server (Kubernetes knows nothing about projects) and refused by the controller with an InvalidSpecError condition — the condition, not an apply error, is the evidence. An empty cluster-resource whitelist also means CreateNamespace=true cannot work, which is why the platform provisions tenant namespaces first. Fence a team with an Argo CD AppProject (kubefit drill cnpe-07-argocd-appproject) has you draw that boundary and prove it holds. Argo CD projects ↗.

Argo CD can render Kustomize and Helm, with customization stored in the source or configured on the Application. spec.source.kustomize adds a namePrefix and commonLabels at render time, which is why the cluster carries prod-guestbook-ui while the repo's base is still guestbook-ui; Overlay a Kustomize app from Argo CD (kubefit drill cnpe-08-argocd-kustomize-overlay) checks the prefixed name and the label. spec.source.helm carries parameters (Helm's --set, applied last) and a values or valuesObject block (an inline values file); no Helm release object exists in the cluster, and the release name defaults to the Application name, which the chart's fullname helper turns into the Deployment and Service names. Prove a Helm-sourced Application by reading replica count, image and Service type back from the rendered objects — Deploy a Helm chart through Argo CD (kubefit drill cnpe-16-argocd-helm-app). Argo CD Helm ↗.

Delivery also includes the pipeline that produces what Git points at. Init containers give a Job its stage order and a shared emptyDir carries results between stages; Build a gated in-cluster pipeline (kubefit drill cnpe-06-cicd-pipeline) practices that shape with the same identity and security-context expectations as any other workload.

Progressive delivery and a reproducible rollback

CI builds and tests an artifact; delivery promotes a particular immutable revision into an environment. Record the source revision, image digest, test results, and policy evidence together. A pipeline Job that succeeds without producing a usable artifact is not a complete delivery system.

A canary gradually changes traffic or capacity and evaluates a success criterion; blue/green switches between prepared versions. Define the observation window, minimum sample, and abort condition before promotion. Restore the declarative source when rolling back a GitOps-managed workload, or reconciliation may reapply the bad version. Database migrations and external side effects need their own compatibility plan. Prove it: route representative traffic, observe errors and latency, and rehearse abort as well as success. Argo Rollouts concepts ↗

Practice loop: repair drift through the declared source

Starting state: Argo CD Application practice-web manages Deployment web. The Argo CD CLI is authenticated to the practice instance. Git declares two replicas; a manual edit changed the live Deployment to four. Automated self-heal is disabled for this rehearsal.

argocd app get practice-web
argocd app diff practice-web
kubectl -n argocd get application practice-web -o yaml

Look for: OutOfSync plus a replica diff. argocd app diff normally exits nonzero when differences exist; read the output before treating that as a tool failure. Healthy and Synced answer different questions: four healthy replicas can still disagree with Git.

Act: when the declared two replicas are the intended state, sync the application:

argocd app sync practice-web
argocd app wait practice-web --sync --health --timeout 120
argocd app get practice-web

If four replicas are the newly approved requirement, update the source first, then sync that revision. Repeatedly scaling the live Deployment is not a durable fix for an unwanted declared value.

Prove: inspect the source revision, operation result, live replica count, and application response. A completed sync operation can leave an unhealthy application; wait on both conditions and investigate failures rather than suppressing them.

Repeat: switch the changed field from replicas to image digest. Identify whether to revert drift or promote new desired state.

Coach Caz: Git is the training plan. Arguing with the mirror does not edit the plan. Argo CD diff ↗

Self-service APIs: schema plus reconciliation

A CRD stores and validates a new API shape. It does not provision a namespace by itself. A controller or generation policy watches that request and creates the supporting resources. Its service identity needs appropriate permissions, and failures must be visible. CustomResourceDefinitions ↗, Kyverno generation rules ↗.

For the tenant task, submit one TenantSpace and inspect the resulting namespace, quota and NetworkPolicy. Manually creating the expected outputs would hide a broken provisioning path. Next-level practice should test an invalid request, a retry, an update, a deletion and a partial failure. Build a tenant self-service API (kubefit drill cnpe-02-platform-api-selfservice) is that task.

An ApplicationSet is a template plus a generator, and the generated Applications carry an ownerReference to the set: that reference is what separates a self-service platform from a folder of hand-written Applications, because editing the set rewrites every child and removing an element can delete its generated Application, subject to the ApplicationSet synchronization policy. Adding - env: qa to a list generator is the whole onboarding step for a third environment. Whether the child's workloads disappear with it depends on the resources-finalizer.argocd.argoproj.io finalizer in the template. Stamp out environments with an ApplicationSet (kubefit drill cnpe-15-argocd-applicationset) grades the owner references and both environments' Deployments. ApplicationSet ↗.

Self-service also needs a permission model the team can use without becoming cluster admin. Argo CD has two RBAC layers — the global argocd-rbac-cm and spec.roles on each AppProject — and both use Casbin lines of the form p, <subject>, <resource>, <action>, <object>, <effect>. A project role's subject is proj:<project>:<role> and its object must begin with <project>/, so the role cannot reach beyond its own project even if written carelessly; applications, sync, team-b/* lets a team press the button without letting it rewrite or delete the Application. Tokens and SSO groups bind people to the role later; the policy lines are the interface. Scope a deployer role to an AppProject (kubefit drill cnpe-19-argocd-project-roles) checks the lines, the fence and a Synced Application inside it. Argo CD RBAC ↗.

Reconcile updates, retries, and deletion

A controller can observe the same request repeatedly. Reconciliation must converge without duplicating external resources. Use stable ownership and report conditions with reasons; where supported, compare observedGeneration with the current generation so an old Ready condition cannot masquerade as success for a new request.

Owner references support Kubernetes garbage collection; finalizers let a controller finish required cleanup before deletion. External resources need explicit ownership and retention rules. A stuck finalizer is a lifecycle failure to investigate, not permission to discard data. Prove it: retry, update, delete, and recreate a request; inspect both Kubernetes and external resources for leaks. Schema acceptance and one successful creation do not prove this lifecycle. Controllers ↗ · Finalizers ↗

Practice loop: accepted custom resources still need a controller

Starting state: a prepared platform has namespaced CRD environments.training.example.com and a controller. Request demo in concepts-platform is stored but has not produced its expected workload. This is an illustrative API; use the real installed type and schema in a task.

kubectl api-resources --api-group=training.example.com
kubectl explain environments.spec --api-version=training.example.com/v1
kubectl -n concepts-platform get environments.training.example.com demo -o yaml
kubectl -n concepts-platform describe environments.training.example.com demo

Look for: the API version and required fields, then conditions, reason/message, and generation. If metadata.generation is 3 while a supported status.observedGeneration is 2, a previous Ready status may not describe this request. Not every CRD defines those status fields; inspect its contract rather than assuming them.

Act: follow the reported boundary. Schema errors require request changes; a controller's Forbidden error requires its narrow RBAC repair; a missing external dependency requires restoring that dependency. Inspect the controller's logs and ServiceAccount from the prepared installation. Manually creating the expected output can conceal a broken self-service system.

Prove: the current request is reconciled, the expected owned resources appear, and the user can perform the requested operation. Submit one invalid request as a negative control and confirm it produces a useful rejection instead of partial resources.

Repeat: rehearse an update and a deletion as well as initial creation. Predict what should remain under the platform's retention policy.

Coach Caz: A request accepted at reception is not a completed workout. Find the controller doing the lifting. Custom resource controllers ↗

Observability: collect, query, alert

Discovery selects scrape targets. Successful scraping collects samples. A query selects a series, and an alert adds an expression and duration. Verify each stage separately. The task's Prometheus is configured to interpret scrape annotations; those annotations are not a universal Kubernetes behavior. Prometheus configuration ↗.

An up value of zero represents a failed scrape for a discovered target. A missing series can instead mean discovery failed or the application has not emitted that metric yet. Loading an alert rule does not prove it fires correctly under failure. Check the rules API, then plan a controlled firing/recovery test for broader operational practice. Alerting rules ↗. Expose metrics and load an alert rule (kubefit drill cnpe-03-observability) walks all three stages once.

In this annotation-based fixture, the Pod template is the contract: prometheus.io/scrape, prometheus.io/port and prometheus.io/path must be on the running Pod, not only on the Deployment, and /api/v1/targets tells you which of the three facts you are missing — no target means discovery failed, health unknown means the first scrape has not happened yet (inspect the configured interval), down with a lastError means the scrape ran and failed. A query may return older samples even after scraping fails; check the target health and sample timestamps together. Get a workload scraped by Prometheus (kubefit drill cnpe-12-prometheus-scrape-target) grades the target's health and the series value separately.

In the community prometheus chart there is no operator, so rules live in the prometheus-server ConfigMap under alerting_rules.yml and a config-reload sidecar POSTs /-/reload once the kubelet refreshes the volume. The rules API (/api/v1/rules) is the proof that the file loaded: it reports duration in seconds, the labels Alertmanager will route on, and a health of ok, unknown (not yet evaluated — normal for a freshly loaded group) or err (loaded, but the expression fails). severity belongs under labels, never annotations. Alert on pods restarting too often (kubefit drill cnpe-11-prometheus-alert-rule) grades that API result.

Alertmanager turns those labels into a decision about who is told. Its routing tree is traversed in order; by default a matching sibling stops further sibling matching, while continue: true permits more matches. The root route must name a receiver; a child route with matchers: ['severity="critical"'] sends the page-worthy alerts to a receiver such as a webhook. The chart stores alertmanager.yml in ConfigMap prometheus-alertmanager, mounted into a StatefulSet with no reload sidecar, so editing the ConfigMap changes nothing until the StatefulSet is restarted. /api/v2/status returns config.original, the configuration as the running process parsed it, with secrets (including webhook URLs) masked — read the URL from the ConfigMap and the routes from the API. Route critical alerts to a pager receiver (kubefit drill cnpe-18-alertmanager-routing) grades both. Alertmanager configuration ↗.

Operating on metrics closes the loop: a HorizontalPodAutoscaler needs CPU requests on the target, a metrics-server that reports usage, and a couple of scrape cycles before status.currentMetrics holds a number. The TARGETS column showing <unknown> or a ScalingActive condition of False with a reason is the finding; the replica count on its own proves nothing. Autoscale a deployment on CPU (kubefit drill cnpe-14-hpa-metrics) grades the condition and the current metric, not just the object. HorizontalPodAutoscaler ↗.

SLOs, deployment metrics, logs, and traces

An SLI measures a user-visible outcome; an SLO sets a target over a defined window. For a request-based 99.9% success objective across one million eligible requests, the allowance is 1,000 unsuccessful requests—not an arbitrary CPU threshold. Define the denominator, exclusions, and alert action. Deployment frequency, lead time, recovery time, and failed changes answer delivery questions distinct from service availability. Implementing SLOs ↗

Metrics show aggregate behavior, logs record events, and traces connect spans of one request. Trace context must propagate across service boundaries; installing a collector alone does not instrument applications. Correlate a failed request with its trace and logs before naming a bottleneck. Prove it: a known test request appears end to end, a controlled failure alerts the correct receiver, and recovery resolves the alert. The supplied Prometheus drills do not establish a complete logging/tracing platform. OpenTelemetry signals ↗ · Context propagation ↗

Practice loop: missing metrics and failed scrapes are different

Starting state: an authenticated practice Prometheus endpoint is available through a localhost port-forward on 9090; jq is installed. Use the provided access method and keep that forwarding terminal open.

curl -fsS http://127.0.0.1:9090/api/v1/targets | jq '.data.activeTargets[] | {url:.scrapeUrl,health,lastError}'
curl -fsSG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up' | jq '.data.result'

Look for:

Act: repair the discovered field in the responsible source: Pod annotations for an annotation-based setup, or the appropriate monitor/configuration for that installation. Prometheus does not universally interpret scrape annotations without configuration.

Prove: the target is healthy, the intended metric has fresh samples, and the query selects the intended workload. An old graph can remain visible while current scrapes fail. For an alert, verify its expression, duration, firing under a controlled failure, and recovery notification.

Repeat: name what up=0 proves and what an absent up series does not prove.

Coach Caz: Yesterday's personal best is not today's heart rate. Check the timestamp. Prometheus HTTP API ↗

Policy: prevent invalid requests and explain why

Validation rejects a request; mutation can supply defaults. Inspect match scope and controller handling so the policy governs the intended workloads. Test both a rejected request and a compliant Deployment that becomes Ready. A rule that rejects everything is not a usable guardrail. Kyverno validation ↗. Enforce and default workload policy (kubefit drill cnpe-05-policy-governance) pairs one validate and one mutate policy on a labelled namespace.

A validate rule has two outcomes to choose from, rule by rule: Audit admits the resource and records a fail in a per-resource policy report (named by the resource UID, living beside the resource, kind PolicyReport or reports.openreports.io depending on how Kyverno was installed), while Enforce refuses it with an admission error that names the policy. The policy-level validationFailureAction is the default and validate.failureAction overrides it per rule, so one ClusterPolicy can report a new requirement while blocking an obvious one such as a :latest tag (image: "!*:latest"). Rolling a guardrail out as Audit first gives you the list of who would break before anyone does. Proof is three admissions: one admitted and reported, one refused with the error text, one in an unlabelled namespace left alone. Audit owner labels, enforce image tags (kubefit drill cnpe-17-kyverno-validate-audit) grades all three. Policy reports ↗.

For this defaulting task, mutation should fill gaps without replacing explicitly supplied values. Kyverno's +(field) add-if-absent anchor and the (name): "*" conditional anchor let one rule default requests on every container without cutting a value the author set; scope it with namespaceSelector (namespace labels), not selector (pod labels). kubectl run -o json prints the object as the API server stored it, so you can see the mutation without the Pod ever scheduling. Because a Pod rule is auto-generated for Deployments and the other controllers, render/admission differences can appear as GitOps drift. Inspect the diff and use the tool's supported comparison strategy; do not broadly ignore security-relevant fields. Default pod labels and requests with Kyverno (kubefit drill cnpe-09-kyverno-mutate) grades the added values, the preserved one and the untouched namespace. Kyverno mutation ↗.

A generate rule turns one event into the objects every tenant must have: a Namespace created with tenant=true gets a default-deny NetworkPolicy and a ResourceQuota within seconds, and an unlabelled one gets nothing. The background controller needs RBAC to create what the rule generates — a policy that is Ready but produces nothing usually means that ClusterRole is missing. Generate tenant namespace guardrails (kubefit drill cnpe-10-kyverno-generate) creates a labelled and an unlabelled namespace and reads the generated objects' fields back.

Pipeline workloads need the same identity and runtime controls as other workloads. Init containers provide ordering and a shared volume passes results forward. The sample root-user scanner is an intentionally limited exercise gate; production scanning must parse real build semantics and preserve artifacts outside temporary pod storage. Kubernetes init containers ↗.

Deliver secrets and secure service-to-service calls

Keep plaintext credentials out of Git and build output. An external-secret controller authenticates to a provider and reconciles the selected data into a workload-facing Secret; provider access, controller access, namespace access, and application consumption are separate permissions. A successful refresh does not prove an application using environment variables has reloaded the new value. Prove it: rotate a dummy credential, verify the consumer changes over, and check that the old credential no longer works without printing either value. External Secrets status ↗

mTLS protects transport and authenticates peers; authorization decides which operations those identities may perform. Scope identities to workloads, validate trust and rotation, and test both an allowed and a denied caller. Preserve scan results, SBOMs, and admission/audit evidence against the exact deployed digest. A tag allow-list alone cannot establish signer identity or artifact safety. These are supplemental workflows beyond the current platform drills. Istio security model ↗

For every platform change, ask what the requester sees on success and on failure. Capture status and an actionable reason, then prove that a retry or corrected request converges without manual cleanup.

Practice loop: prove a guardrail with an allowed and denied request

Starting state: Kyverno is installed in a disposable cluster. A prepared policy requires label owner on Pods in namespace concepts-platform. The policy must enforce there while leaving an unrelated namespace outside its scope.

kubectl get clusterpolicy -o yaml
kubectl -n concepts-platform run policy-check --image=busybox:1.36 --restart=Never --dry-run=server -o yaml --command -- sleep 60
kubectl -n concepts-platform run policy-check --image=busybox:1.36 --restart=Never --labels=owner=training --dry-run=server -o yaml --command -- sleep 60

Look for: the unlabeled request is rejected with the intended policy name; the labeled request is accepted. If both pass, inspect namespace selection, match rules, enforcement mode, and policy readiness. An Audit report is useful evidence but is not an admission rejection.

Act: repair the prepared policy's scope or enforcement field according to its installed Kyverno version. Do not broaden a namespace-scoped requirement into a cluster-wide block. Use server-side dry-run to exercise admission; client-side dry-run stops before the policy sees the object.

Prove: run the two cases again, then the unlabeled case in the designated out-of-scope namespace. Expected outcomes are reject, accept, accept. If another admission policy rejects the last case, attribute that error instead of claiming this policy is mis-scoped.

Repeat: change the rule to a required image registry in a fresh exercise and keep the same three-case test matrix.

Coach Caz: A guardrail that blocks everyone is a locked gym. Test that the right people can still train. Kyverno validation ↗

Further applied practice

Validate a self-service API before provisioning

Schema validation is an API contract, not a controller. Required fields, enums, bounds and defaults reduce invalid requests before provisioning starts. Establish the CRD before creating instances; a missing API is not the same as an invalid request.

custom resource definitions ↗ custom resources ↗

Scope a release bot to batch submissions

Separate the automation identity from an administrator’s identity. Grant the API operations the workflow actually performs and test them with impersonation. Job creation is still powerful within the allowed namespace, so it belongs in a disposable, scoped environment.

rbac ↗ service accounts ↗

Give stateful replicas separate retained claims

A StatefulSet gives each ordinal its own claim and stable name. The retention policy controls claim lifecycle; it is separate from a PV’s reclaim policy. Reading each marker establishes current mounted identity, not a backup or a completed disaster-recovery test.

statefulset ↗ persistent volumes ↗

Recover a failed release check with bounded retries

A Job is an execution record with an immutable Pod template. Inspect the failure, then replace it deliberately with bounded retries and lifetime. A completed object and its output together provide stronger evidence than a renamed or simply deleted failure.

job ↗ ttlafterfinished ↗

Roll forward with immutable configuration versions

Immutable configuration encourages explicit versions and reproducible rollouts. Changing the template’s reference replaces Pods; retaining the prior version leaves a clear rollback target. Inspect both the declared reference and the process environment.

configmap ↗ configure pod configmap ↗