Operations workshop: make recovery evidence useful
This workshop extends the certification paths with operational practice: it began as a focused etcd snapshot exercise and now covers 25 timed drills across node maintenance, the control plane, access, workloads, storage and diagnostics. It is supplemental operational practice, not an assertion that a particular live exam includes these exact tasks.
Every drill runs in your terminal through the kubefit CLI against a three-node kind cluster (muscle-control-plane, muscle-worker, muscle-worker2). Node shells are docker exec -it <node> bash; on a real cluster the same steps run over ssh. The thread through all of them is the one the title names: do the change, then produce evidence that the cluster, not you, says it is done.
Build the command-to-evidence habit
Coach Caz: Recovery has a finish line: the original operation works again. A quiet error log is just the warm-up.
Use each Practice loop below as a short rehearsal. Read its starting state, predict the result of the first command, then compare that prediction with the evidence. The outputs are illustrative: names, timestamps, addresses, and counts vary. Commands use the example's names; substitute the actual task's namespace, resource, host, and file names when transferring the pattern to a drill.
- Recognize: say which symptom puts you on this path.
- Inspect: run the smallest check that separates two plausible causes.
- Act: change the field or configuration supported by that evidence.
- Prove: repeat the failed operation and check a constraint that must still hold.
- Repeat: hide the commands, change one input, and rebuild the sequence from memory.
Use a disposable practice environment for changes. A case that assumes an installed controller, tool, or prepared resource says so; it does not install those prerequisites for you. Linux host commands belong inside a practice Linux VM or the specified lab node. These examples teach the investigation pattern; use the CLI's assigned task and grader for recorded reps.
Backup and recovery
Understand the recovery chain
Kubernetes stores cluster state in etcd. A snapshot captures database state at a point in time. It does not back up every external volume, application database or file on a node. Name the recovery scope before treating a snapshot as a complete system backup. etcd disaster recovery ↗.
The exercise asks for a snapshot inside the control-plane node and evidence that it is a valid snapshot. Endpoint, certificate and key paths come from the running static-pod configuration. Guessing familiar paths can connect to the wrong endpoint or fail authentication.
Three checkpoints
First, establish that the client can reach and authenticate to the intended etcd member. Second, take the snapshot and inspect its metadata with tools compatible with that etcd version. Third, preserve the file at the exact requested path and record its revision, hash and size.
In current etcd workflows, etcdctl saves the snapshot and etcdutl inspects or restores it. Older fixtures may expose snapshot status through a different command. Inspect the installed tools and their help rather than substituting a command blindly. etcd Kubernetes operations ↗.
What this exercise proves
A successful snapshot and metadata check show that a recoverable-looking backup artifact was captured. They do not establish a successful restore, acceptable recovery time or complete application recovery. The current task does not ask you to restore the cluster despite the legacy scenario identifier containing backup-restore. Practice it as Take and verify an etcd snapshot (kubefit drill ops-01-etcd-backup-restore).
For a later recovery rehearsal, use another disposable cluster, follow the version-specific restore procedure, test API health and reconcile restored Kubernetes state with controllers. Keep that extension separate from the timed snapshot task so the expected result remains clear.
Moving one application instead of the whole cluster
An etcd snapshot recovers stored cluster API state; an application export selects resource definitions. Neither automatically includes persistent-volume bytes or external dependencies. kubectl get -o yaml output is not portable until the fields the API server owns are gone: metadata.uid, resourceVersion, creationTimestamp, managedFields, the whole status block, and allocated Service addresses where the destination must allocate new ones. Preserve intentional headless clusterIP: None semantics rather than deleting them blindly. Leave metadata.namespace out too, so -n decides where the copy lands. Recognition cue: kubectl apply into the new namespace refuses with "the namespace from the provided object does not match", or a Service fails because its ClusterIP is already allocated. Proof: the files grep clean for those fields, and jsonpath comparisons of image, replicas, ports and data keys match between the source and the copy while the source keeps running. Export a namespace and rebuild it elsewhere (kubefit drill ops-14-namespace-export).
Recovery objectives and restore proof
A recovery point objective bounds tolerable data loss; a recovery time objective bounds tolerable outage. Snapshot age, off-cluster retention, encryption keys, volume backups, and external services all affect whether those targets can be met. A backup on the same failed node is not an independent recovery copy.
Rehearse restore into an isolated destination with version-compatible tools. Verify database/API state, controller convergence, application data, credentials, and a real read/write transaction. Record what was not restored and the measured times. Never confuse a snapshot integrity check with that end-to-end result. etcd disaster recovery ↗
Practice loop: distinguish a saved snapshot from a tested recovery
Starting state: inside a prepared disposable recovery environment, you have etcd snapshot snapshot.db and a compatible etcdutl. This is an example artifact supplied by the exercise, not an existing file on your host.
etcdutl snapshot status snapshot.db -w table
sha256sum snapshot.db
Look for: a readable snapshot with hash/revision/key-count/size information. Record its time and source separately. A checksum establishes the bytes you retained; it does not establish that the snapshot contains the right cluster or that applications can recover from it.
Act: follow the environment's restore procedure on a designated recovery target, including membership and revision/watch handling appropriate to the installed etcd version. Do not overwrite a live data directory to demonstrate this concept. Keep the original artifact and recovery configuration available.
Prove: after the prepared restore, the API is ready, expected objects exist, new writes work, and controllers converge. Restore and verify application volume data independently: it is not inside the etcd snapshot. Record the recovery time and the newest recoverable data so RTO/RPO claims have evidence.
Repeat: name one check before restore, one after API recovery, and one for application data.
Coach Caz: A photo of your training plan is not proof you can finish the workout. Restore is the rep. etcd recovery ↗
Node maintenance
Draining under a disruption budget
A drain cordons the node and normally uses the Eviction API for eligible Pods, respecting PodDisruptionBudgets. Mirror Pods and DaemonSet Pods need separate handling; bypassing eviction changes the protection you are testing. The budget has to exist before the drain starts; minAvailable: 2 on a three-replica Deployment means the eviction of the second pod waits until its replacement is Ready elsewhere. DaemonSet pods need --ignore-daemonsets and emptyDir data needs --delete-emptydir-data, otherwise the drain refuses to run at all. Proof: kubectl get pdb -o yaml shows healthy replicas meeting the required floor, with disruptionsAllowed reflecting the currently available eviction budget, the node reads Ready,SchedulingDisabled, and no application pod is left on it. Drain a Node Under a Disruption Budget (kubefit drill ops-02-node-drain).
The kubelet's configuration file
The kubelet reads /var/lib/kubelet/config.yaml once, at start (systemctl cat kubelet shows the --config flag). Effective behavior combines configuration, command-line flags, runtime and hardware observations. Node status is reported state; patching a status value does not repair its cause. Labels, taints, and scheduling controls are separate administrator-managed fields.
Capacity drift looks like Pending pods on a Ready node with Too many pods in the FailedScheduling event, while allocatable.pods differs between twin workers. The fix is the file (maxPods), a systemctl restart kubelet, and watching allocatable.pods return to 110 while the scheduler retries on its own. Repair Kubelet Config Drift (kubefit drill ops-07-kubelet-config-drift).
Eviction thresholds live in the same file. evictionHard signals (memory.available, nodefs.available, imagefs.available, nodefs.inodesFree) evict immediately; evictionSoft signals wait out their evictionSoftGracePeriod, and every soft signal needs a grace period or the kubelet refuses to start. kind ships every disk signal at 0%, which is eviction switched off. After the edit and restart, the proof is not the file but the running kubelet: kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq .kubeletconfig.evictionHard shows what was loaded, and the node is Ready again. Set Kubelet Eviction Thresholds (kubefit drill ops-19-eviction-thresholds).
Reading the kubelet's journal
In this kubeadm fixture the kubelet runs as a systemd unit, so inspect journalctl -u kubelet; other distributions may collect its logs differently. Static-pod manifests are one of the kubelet's disk inputs; a file in /etc/kubernetes/manifests that does not parse produces no pod, no event and no node condition, only a Could not process manifest file line repeated every fileCheckFrequency (20 seconds). The kubelet ignores filenames beginning with a dot but parses other regular files regardless of extension, so renaming pod.yaml to pod.yaml.bak keeps the error going; the fix is removing the file, and no restart is needed. Proof: the file is gone, the directory is empty on a worker, and a --since window of the journal no longer mentions it. Triage Kubelet Logs on a Node (kubefit drill ops-16-node-log-triage).
Practice loop: understand a blocked drain before bypassing it
Starting state: a disposable cluster has three replicas protected by PDB web with minAvailable: 2; one replica is already unready. A worker needs maintenance.
kubectl -n concepts-ops get pods -o wide
kubectl -n concepts-ops get pdb web -o yaml
kubectl describe node WORKER
Look for: two healthy replicas, desiredHealthy: 2, and disruptionsAllowed: 0. That is a functioning guardrail, not evidence that the PDB is broken. Inspect the unready replica and whether other nodes can accept replacements.
Act: restore health or add the required capacity under the task's constraints. Only then drain the intended worker using the needed options:
kubectl drain WORKER --ignore-daemonsets --timeout=120s
Replace WORKER. Do not automatically add --delete-emptydir-data: decide whether the task permits losing those local contents. Do not bypass the Eviction API to make the command finish.
Prove: the node is unschedulable, eligible application Pods moved, and the required healthy replicas remain available. After maintenance, use kubectl uncordon WORKER and verify readiness and scheduling again. DaemonSet and mirror Pods are not proof a normal drain failed.
Repeat: calculate the budget with all three replicas healthy, then with only two.
Coach Caz: The safety bar is doing its job. Fix the failed rep before removing the bar. Disruptions ↗
Control plane
Static pods and the manifest directory
On a kubeadm cluster the API server, controller manager, scheduler and etcd are static pods: the kubelet on the control-plane node runs whatever is in /etc/kubernetes/manifests, and the pods you see in kube-system are read-only mirrors. Changing a flag means editing the file; the kubelet notices within seconds, stops the old container and starts a new one, and the API can be unavailable during restart on a single-control-plane cluster. Recovery time and availability depend on the topology and health. Watch from the node with crictl ps -a and crictl logs; a bad flag shows as an Exited container with unknown flag in its log, and a bad indentation shows as no new container at all with the parse error in the kubelet journal. Keep backups outside the manifests directory, because non-hidden backup files are still scanned and can create conflicting definitions. Proof: kubectl get --raw /readyz prints ok and the mirror pod's .spec.containers[0].command carries the new flags. Edit the API Server Manifest (kubefit drill ops-08-apiserver-manifest).
Audit logging
Audit logging is the API server recording who did what. It takes three pieces that fail independently: a Policy file (audit.k8s.io/v1, rules with a level of None, Metadata, Request or RequestResponse), the flags --audit-policy-file, --audit-log-path and the rotation flags (--audit-log-maxage, --audit-log-maxsize, --audit-log-maxbackup), and hostPath volumes so the container can read the policy and write the log on the node. A missing policy mount stops the API server; a missing log mount lets it start while the log vanishes with the container. Proof: the flags in the mirror pod, both mounts in its spec, and one JSON event line in the file on the node for an object you just created. Enable API Server Audit Logging (kubefit drill ops-20-enable-audit-logging).
Certificates
kubeadm's PKI lives in /etc/kubernetes/pki. Expiry is invisible to kubectl get until the day the API server stops accepting connections, so the check is openssl x509 -noout -enddate -in apiserver.crt (the serving certificate, not the CA, not etcd/server.crt, not a client certificate) cross-checked with kubeadm certs check-expiration. Recording the notAfter string and the whole days remaining is the shape of every renewal runbook's first line. Read the API Server Certificate Expiry (kubefit drill ops-03-cert-expiry).
Cluster DNS
CoreDNS is a Deployment like any other and dies like one: two replicas on the same node are one drain away from a cluster without name resolution. Three separate controls cover three separate failures: replicas for capacity, a topologySpreadConstraint (or preferred pod anti-affinity) on kubernetes.io/hostname for placement, and a PodDisruptionBudget with minAvailable: 1 on the k8s-app: kube-dns label for voluntary disruptions. Proof: 3/3 Ready across at least two nodes, a PDB whose status shows currentHealthy (a wrong selector shows 0), and a name resolved from a pod. Spread CoreDNS across nodes (kubefit drill ops-09-coredns-resilience).
Practice loop: renewed on disk does not mean served by the API
Starting state: the prepared kubeadm control-plane certificate was renewed, but a client still reports the old expiry. Use node access and the actual API hostname/endpoint from the task. Certificate files here are public certificate material; never print private keys.
On the practice control-plane node:
sudo kubeadm certs check-expiration
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates -serial
From an allowed client, inspect what the endpoint serves:
openssl s_client -connect API_HOST:6443 -servername API_HOST </dev/null 2>/dev/null | openssl x509 -noout -dates -serial
Replace API_HOST with the task's DNS name. Look for: a new serial/expiry on disk but an old served serial. The serving process may not have reloaded, or a load balancer may be reaching another control-plane member. This inspection command does not, by itself, validate trust or hostname correctness.
Act: use the supported activation/restart procedure for the prepared topology, one member at a time where availability requires it. Never assume restarting kubelet automatically recreates every static Pod. Preserve manifest backups outside the watched directory.
Prove: the intended endpoints serve the renewed certificate, clients connect with their normal trust configuration, and API readiness remains healthy. If the error is instead unknown CA or hostname mismatch, renewal alone may not address it.
Repeat: compare “file is new,” “process serves new,” and “client trusts new.”
Coach Caz: New shoes in the locker do not improve the shoes you are still wearing. Check the running process. Kubeadm certificate management ↗
Access
Users are certificates
Kubernetes has no built-in API object for an ordinary human user. It supports several authentication mechanisms. In this certificate-based exercise, a trusted client certificate supplies the username from its CN and groups from its O fields; OIDC or other configured authenticators can supply identities differently. Onboarding is three artefacts: a CertificateSigningRequest with signerName: kubernetes.io/kube-apiserver-client and client auth usage, approved and issued; a Role and RoleBinding in the one namespace the person needs; and a kubeconfig that embeds the certificate and key as -data fields with a current context selected. Proof: kubectl get csr reads Approved,Issued, kubectl auth can-i --as=jane answers yes for pods in dev and no for secrets or other namespaces, and kubectl --kubeconfig jane.kubeconfig auth whoami names her. Onboard a User with a CSR (kubefit drill ops-04-csr-user-kubeconfig).
ServiceAccount tokens
Modern Kubernetes does not automatically create a long-lived token Secret for every ServiceAccount. kubectl create token requests a time-limited token; binding to a particular object requires the relevant bound-object options. Prefer short-lived credentials when the client can refresh them. A legacy appliance that needs a static bearer token gets a Secret of type kubernetes.io/service-account-token annotated with kubernetes.io/service-account.name; the token controller fills data.token only if the ServiceAccount already exists. Grant it the narrowest ClusterRole that fits (nodes are cluster-scoped, so a Role cannot help) and test as the account with --as=system:serviceaccount:<ns>:<name>, then with the token itself against the API server from an empty kubeconfig so your admin certificate does not answer for it. Issue a long-lived ServiceAccount token (kubefit drill ops-13-sa-legacy-token).
Reviewing who holds cluster-admin
Start an access review with bindings whose roleRef.name is cluster-admin, then inspect their subjects and owners. That is not a complete privilege audit: custom roles, wildcard grants, escalation permissions, and other bindings may provide equivalent access. Remove only the unauthorized grant and retest both required and prohibited operations. Grants are additive, so deleting one binding may leave access through another. For group tests, impersonate the intended user with --as and supply the relevant --as-group values; a user-only test does not infer external group membership. The system:masters group bypasses normal authorization: deleting a ClusterRoleBinding does not revoke those credentials. Distinguish ordinary RBAC grants from break-glass certificate access and follow a credential-replacement plan. Authorization ↗ Audit Cluster-Admin Bindings (kubefit drill ops-17-rbac-audit).
Workloads
Priority and preemption
Pod priority decides who waits and who is preempted when a node is full. The value lives on a PriorityClass, the class name lives in the pod template, and admission resolves its value into spec.priority when a Pod is created; a pod created before the change keeps its previously assigned priority until it is replaced. globalDefault should stay false unless every unlabelled pod in the cluster is meant to get it. Proof: kubectl get pods -o custom-columns=NAME:.metadata.name,PRI:.spec.priority shows the class value on every current pod in both namespaces. Set Pod Priority with PriorityClasses (kubefit drill ops-05-priorityclass).
Rollout strategy and history
A Deployment's strategy sets the rollout's intended availability and surge bounds. maxUnavailable: 0 and maxSurge: 1 permit one extra Pod while preserving the configured availability target; they do not prevent unrelated application failures. minReadySeconds sets how long a Pod must remain Ready before it counts as available. revisionHistoryLimit controls retained ReplicaSets: zero allows old revisions to be cleaned up after a successful rollout, removing those rollback targets. Set these before the rollout you care about. Prove it: observe availability throughout the change, inspect kubectl rollout history, and confirm the intended prior revision remains recoverable. A change cause is an annotation, not an automatic record of every command. Terminating Pods can temporarily make the total process count exceed the nominal surge allowance. Tune a Deployment Rollout Strategy (kubefit drill ops-06-pdb-rollout-strategy).
Image pulls and registry credentials
ImagePullBackOff has two common roots that kubectl describe pod separates for you: a tag or repository that does not exist (not found) and a registry that wants credentials (unauthorized). Fix the reference in the Deployment; put credentials in a kubernetes.io/dockerconfigjson Secret and add it to the namespace's default ServiceAccount's imagePullSecrets, so every future pod in the namespace inherits it without each manifest naming it. Fix image pulls and wire a registry secret (kubefit drill ops-12-imagepull-fix).
Batch hygiene
A CronJob creates a Job on every schedule, and a finished Job stays until something deletes it. successfulJobsHistoryLimit and failedJobsHistoryLimit bound what the CronJob keeps of the Jobs it owns; ttlSecondsAfterFinished on the Job template is the Job's own self-destruct timer and also covers Jobs nobody owns, as long as they were created with it; concurrencyPolicy: Forbid keeps a slow run from stacking up behind itself; suspend: true is the maintenance switch that stops new Jobs while you work. Jobs created by hand with kubectl create job --from=cronjob/... have no owner and are never pruned by the limits, so they are deleted by hand, oldest first with --sort-by=.metadata.creationTimestamp. Proof: the four fields on the CronJob, suspend: true, and at most the two newest completed Jobs left. Clean Up CronJob History (kubefit drill ops-18-job-hygiene).
Storage
Released volumes
A Retain PersistentVolume preserves backing storage after claim release, but it is not a backup. The PV becomes Released and retains its old spec.claimRef. Before making it available to another claim, confirm the original workload has stopped using it and the intended data may be reused. In this recovery drill, clear that reference, then bind a PVC with matching storage class, access mode, and sufficient capacity; volumeName selects the intended volume. Preserve the reclaim policy, backing path, and node affinity. Prove it: the PV binds to the new claim and a Pod reads the original file through the mount. A Bound phase alone does not prove data recovery. Rebind a Released PersistentVolume (kubefit drill ops-11-pv-reclaim-rebind).
Practice loop: a Released volume may still hold the data you need
Starting state: a prepared practice PV uses Retain, is Released, and contains a dummy recovery file. The original consumer has stopped. Discover the actual PV name and storage implementation before touching claim metadata.
kubectl get pv
kubectl get pv PRACTICE_PV -o yaml
kubectl get pvc -A -o wide
Look for: reclaim policy, old claimRef UID, capacity, class, volume mode, access modes, and node affinity. Released does not mean “empty.” A node-local backing directory is not automatically accessible from another node.
Act: follow the prepared recovery task: clear the stale claim reference only after ownership and data reuse are authorized, then create a compatible claim selecting the intended volume. Do not change backing paths or format storage to make binding succeed. On storage managed by external controllers, inspect their lifecycle requirements too.
Prove: the new claim binds to the intended PV, the correct consumer mounts it on an eligible node, and the original recovery file is readable. Also confirm new writes work if required. A fresh empty directory on the wrong node can make a mount appear successful while losing the recovery goal.
Repeat: explain why Retain, Released, Bound, and “original file read” are four different pieces of evidence.
Coach Caz: Reassigning the locker does not prove the old kit is still inside. Open it and check. Reclaiming persistent volumes ↗
Diagnostics
Finalizers and stuck deletions
A namespace stays Terminating until every object inside it is gone, and a finalizer on one leftover object holds it forever once the controller that owned that finalizer is gone. The namespace's own status.conditions name the kind and the finalizer (NamespaceContentRemaining, NamespaceFinalizersRemaining); kubectl api-resources --namespaced --verbs=list piped through kubectl get finds the object. Restore the responsible controller or complete and verify its cleanup first. Remove an orphaned finalizer only when that obligation is understood and satisfied; do not force the namespace through the finalize subresource, which makes the namespace vanish while the object stays in etcd as an orphan that reappears if the name is ever reused. Unblock a Terminating namespace (kubefit drill ops-10-stuck-namespace).
Support bundles
When you open a vendor ticket or hand an incident to the next shift, the cluster state at that moment is the evidence. A support bundle is a dated, self-contained directory: kubectl cluster-info dump --all-namespaces, every Node object, events from all namespaces sorted by time, and client and server versions, built where kubectl runs and copied to the control-plane node with docker cp (or scp) under a path you record in a ConfigMap so the next person can find it. Collect a support bundle on the control plane (kubefit drill ops-15-support-bundle).
Admission failures and incident evidence
A failed webhook can block writes before an object exists. Inspect the matching configuration, namespace/object selectors, backing Service endpoints, TLS trust, timeout, and failurePolicy. Repair availability or excessive scope; switching everything to fail-open is not equivalent to restoring the intended boundary. Admission webhooks ↗
Before remediation, retain a bounded timeline: affected context, UTC time, symptoms, recent changes, and relevant status/logs. Support bundles may contain Secret bodies, credentials in logs, or private configuration; minimize and redact before sharing. Prove it: the original operation recovers, a disallowed control still fails where expected, and the recorded evidence explains the change. Debug clusters ↗
Where the evidence is
The drills above rehearse a small set of places to look, in a fixed order: the object's events (kubectl describe), its status conditions (-o jsonpath='{.status.conditions}'), the controller's or component's log (kubectl logs for pods, journalctl -u kubelet for the node, crictl logs for a static pod while the API is down), the running configuration rather than the file you edited (configz for the kubelet, the mirror pod's command for the API server), and finally the audit log for who changed it. Write down which of those gave you the answer; that note is the runbook.
Practice loop: a finalizer points to unfinished cleanup
Starting state: a prepared namespaced custom resource is stuck deleting. Use its actual type, name, and namespace in place of RESOURCE, NAME, and NAMESPACE; jq is installed.
kubectl -n NAMESPACE get RESOURCE NAME -o json | jq '{deletionTimestamp:.metadata.deletionTimestamp,finalizers:.metadata.finalizers,owners:.metadata.ownerReferences,status}'
kubectl -n NAMESPACE describe RESOURCE NAME
Look for: a deletion timestamp, a remaining finalizer, and the responsible controller's condition/log evidence. The API is waiting for a lifecycle obligation; repeatedly issuing delete does not complete it. Identify any external resource the controller must clean up.
Act: restore the controller's health, permissions, or external dependency so it can finish. Remove a finalizer manually only when the task authorizes that recovery and you have independently completed or deliberately retained the external cleanup. An empty finalizer list is not the same as a clean system.
Prove: the Kubernetes object disappears and the external resources match the intended deletion/retention policy. Preserve timestamps and errors for the incident record; redact credentials from collected logs or objects.
Repeat: rehearse a controller permission failure and identify the narrow missing verb.
Coach Caz: Crossing “put the weights away” off the checklist does not put the weights away. Finalizers track unfinished work. Finalizers ↗
Further applied practice
Renew and activate an API server certificate
Renewing a certificate on disk does not reload it into a running API server. Compare the served certificate with the file, then check API readiness.
Rejoin a reset worker
Joining creates kubelet credentials and registers a real node. A Ready Node plus a running pinned workload is stronger evidence than a Node object alone.
Restore application data from an archive
A namespace export does not back up the bytes on a volume. Restore data into the correct mounted filesystem and verify contents, hidden files and metadata.
Recover a namespace from a mis-scoped webhook
A dead webhook can block API writes before a resource exists. Trace admission scope and repair the selector while preserving fail-closed behavior for the intended scope.
Resolve a layered application outage
An outage can have simultaneous faults in readiness, endpoint selection and port mapping. Follow the request path and validate each layer after the repair.
Practice domain weights
Operations is a KubeFit practice track, not a certification blueprint. Scheduling and Reports use Backup and recovery 25%, Cluster maintenance 25%, Access and policy 20%, Workload reliability 15%, and Incident response 15%. These are KubeFit training priorities.