Service mesh

Istio: follow the request through the mesh

A mesh applies traffic and security rules through its data plane. Your job is to connect a rule to the proxy that enforces it, then observe the request. Healthy pods and valid YAML are useful checkpoints, but neither proves routing or authorization works.

Domains and scope

The ICA domains are Installation, Upgrades and Configuration 20%, Traffic Management 35%, Securing Workloads 25%, and Troubleshooting 20%. The certification uses a hybrid format; the 25 KubeFit drills focus on hands-on behavior. Official ICA overview ↗.

This fixture uses ambient mode, with ztunnel for Layer 4 transport and waypoints for the Layer 7 behavior used in these exercises. Supplement it with sidecar operation, TLS origination to external hosts and multi-cluster practice. An exercise list is not a complete curriculum checklist.

Build the command-to-evidence habit

Coach Caz: Know which proxy is doing the work. Giving instructions to the wrong spotter rarely improves the lift.

Use each Practice loop below as a short rehearsal. Read its starting state, predict the result of the first command, then compare that prediction with the evidence. The outputs are illustrative: names, timestamps, addresses, and counts vary. Commands use the example's names; substitute the actual task's namespace, resource, host, and file names when transferring the pattern to a drill.

1. Recognize: say which symptom puts you on this path.
2. Inspect: run the smallest check that separates two plausible causes.
3. Act: change the field or configuration supported by that evidence.
4. Prove: repeat the failed operation and check a constraint that must still hold.
5. Repeat: hide the commands, change one input, and rebuild the sequence from memory.

Use a disposable practice environment for changes. A case that assumes an installed controller, tool, or prepared resource says so; it does not install those prerequisites for you. Linux host commands belong inside a practice Linux VM or the specified lab node. These examples teach the investigation pattern; use the CLI's assigned task and grader for recorded reps.

Installation and upgrades: know what is connected

The control plane distributes configuration; proxies process traffic. In ambient mode, enrollment and waypoint attachment determine which data-plane components a request traverses. Inspect the installed revision and proxy connections before changing installation settings. Ambient architecture ↗.

A second control-plane revision lets you stage an upgrade. Installing it does not prove existing proxies migrated safely. The local canary task intentionally stops before migration: keep the original connections working, create the new revision and tag, and record the evidence. Full upgrade practice should also include workload migration and rollback. Canary upgrades ↗.

Install the chosen mode and prove enrollment

In sidecar mode, each enrolled Pod receives an Envoy proxy through injection; labeling a namespace does not inject a proxy into Pods that already exist. Recreate the workload through its controller, then inspect containers, revision selection, and proxy synchronization. In ambient mode, the CNI and node ztunnel handle enrollment and transport; waypoints add supported L7 processing. Neither mode is established merely by creating a namespace label. Sidecar injection ↗ · Ambient architecture ↗

With Helm, distinguish base/CRD installation, istiod, gateways, and the ambient components required by the chosen release. With istioctl, inspect the selected profile and overrides. Keep component versions compatible and preserve the installation configuration used to reproduce the result. Prove it: controller health, data-plane connections, workload enrollment, and a known request. Helm installation ↗

Canary and in-place upgrades have different rollback paths

A canary control plane coexists with the original revision. Move a small workload set deliberately, verify its actual proxy version and connectivity, then expand. Revision tags are indirection: changing a tag affects future injection, not a magical replacement of every existing sidecar. Keep the old revision until migration is verified.

An in-place upgrade changes the existing installation and requires a compatible data-plane rollout. Check the release's supported version transitions and migration instructions; shared CRDs and node-level ambient components need their own lifecycle attention. Prove it: successful and denied traffic still behave correctly after migration, and the tested rollback returns the selected workloads to a working revision. The local canary drill does not grade full migration or in-place recovery. Canary upgrades ↗ · Upgrade guide ↗

Practice loop: prove which data plane the workload joined

Starting state: Istio is installed in the practice cluster. Namespace concepts-mesh was just labeled for enrollment. Identify whether this exercise uses sidecars or ambient before choosing a repair.

kubectl get namespace concepts-mesh --show-labels
kubectl -n concepts-mesh get pods
kubectl -n concepts-mesh get pod WEB_POD -o jsonpath='{.spec.containers[*].name}{"\n"}'
istioctl proxy-status

Look for: in sidecar mode, a pre-existing Pod may still lack istio-proxy; namespace labeling does not retrofit its containers. In ambient mode, absence of that container is expected, so “add a sidecar” would be the wrong conclusion. Check CNI/ztunnel enrollment and the installed version's ambient diagnostics instead.

Act: for the sidecar case, after confirming correct injection/revision labels and available capacity, replace controller-managed Pods:

kubectl -n concepts-mesh rollout restart deployment/web
kubectl -n concepts-mesh rollout status deployment/web --timeout=120s
kubectl -n concepts-mesh get pods
istioctl proxy-status

Prove: the new sidecar is present, its proxy connects and receives configuration, and the application still serves. For an ambient L7 task, additionally inspect the waypoint Gateway, attachment, and request path; healthy transport alone does not prove HTTP policy enforcement.

Repeat: explain what a 1/1 Pod means in each mode.

Coach Caz: A sidecar is a spotter, not a sticker. Labeling the room does not put one beside every lifter. Sidecar injection ↗ · Ambient verification ↗

Traffic management: route, destination and budget

A route chooses a destination and can apply matching, weights, timeouts or faults. Destination policies define subsets and upstream behavior. A subset name must match the route and select real endpoint labels. Valid resources can still point at nothing.

For ambient L7 exercises, verify that traffic traverses the waypoint. A transport-only path cannot apply the HTTP behavior merely because a VirtualService exists. Version support matters; inspect the installed Istio release and its supported API combinations. Ambient Layer 7 features ↗.

An 80/20 split is a probabilistic routing choice, not a promise that every batch of five requests contains exactly one v2 response. Sample enough requests and retain counts. A route timeout is the overall budget; retries and per-try timeouts must fit within it. Outlier detection temporarily removes failing upstream hosts, which can reduce available capacity. Traffic management concepts ↗.

Fault injection lets you observe caller behavior under controlled delay or failure. Apply it to the intended route, then measure both status and latency. Remove or reset the fault before evaluating an unrelated exercise.

The waypoint is the L7 enforcement point

In ambient, ztunnel provides authenticated, encrypted transport between participating mesh workloads and enforces supported L4 policy; it does not process HTTP. The service-directed L7 behavior practised here requires a waypoint on the request path. Deploy the waypoint Gateway with gatewayClassName: istio-waypoint and enroll the intended namespace or Service using istio.io/use-waypoint. A missing namespace label is only a clue: inspect Service-level attachment and the actual traffic path too. Prove it: the Gateway is Programmed, its proxy is ready, the intended destination uses it, and positive and negative requests produce the expected policy behavior with proxy evidence. An HTTP status alone does not identify its source. Add a waypoint and enforce L7 policy (kubefit drill istio-09-waypoint-l7).

Two routing APIs, one per Service

Gateway API is the native L7 configuration surface in ambient. An HTTPRoute can attach to a Service through parentRefs (kind Service, group ""); the destination's waypoint must handle that traffic. Header matches select a rule, and backendRefs identify its destinations. Some local exercises use VirtualService and DestinationRule instead: support depends on the installed release. Istio documents mixing those APIs with Gateway API on the same Service as undefined, so choose one approach per Service. Prove it: inspect the intended parent's Accepted and ResolvedRefs conditions, then test matching and nonmatching requests. Do not assume the first status entry represents the parent you meant. Route by request header in ambient (kubefit drill istio-10-header-routing).

Request budgets follow the same split: an HTTPRoute carries Gateway API timeouts.request, a VirtualService carries Istio retries (attempts, perTryTimeout, retryOn). A timeout is proved by timing a deliberately slow endpoint and reading 504 after about the budget, never by reading the YAML alone. Bound requests with a timeout and retries (kubefit drill istio-12-timeouts-retries).

Subsets, weights and mirrors

A DestinationRule turns Pod labels into named subsets; a VirtualService distributes traffic to those destinations. Match subset selectors to Pod labels and reference the intended host and subset in each destination. Use valid weights, conventionally totaling 100; each destination receives a share relative to the total. Prove it: send enough independent requests to compare observed proportions with the configured weights. A small random sample cannot guarantee an exact split or even that both versions appear. Split traffic between subsets by weight (kubefit drill istio-16-subset-weighted-routing).

Mirroring sends a copy of each request to a second host while the caller only ever sees the primary's answer. The VirtualService route names the primary (audit-v1), mirror names the shadow (audit-v2) and mirrorPercentage.value sets the share. Because the response never changes, the proof lives in the mirror target's logs: note the line count, send a known number of requests, and diff. Mirrored copies carry the authority <host>-shadow. Mirror live traffic to a second version (kubefit drill istio-19-request-mirroring).

Registering hosts outside the mesh

A ServiceEntry registers an external service so Istio can apply supported routing and policy to it. hosts identifies the service, resolution controls endpoint discovery, and endpoints can supply explicit addresses. Unknown-destination behavior depends on the mesh's outbound configuration; the registry alone is not an egress security boundary. MESH_EXTERNAL is not a plaintext switch. In this ambient fixture, DNS capture and address allocation let the registered name resolve; verify those features in the installed version. Prove it: resolve the name and receive the expected response using the required transport, without weakening unrelated egress controls. Register an external service with a ServiceEntry (kubefit drill istio-20-external-service-entry). ServiceEntry ↗ · External access ↗

Circuit breaking, outlier detection, and failover

Timeouts bound waiting; retries issue additional attempts; connection-pool limits constrain concurrent or queued work; outlier detection ejects unhealthy upstream hosts for a period. These controls interact: unbounded retries can increase load precisely when capacity is failing. Configure a budget and test a known failure rather than copying arbitrary thresholds.

Failover also needs reachable healthy alternatives and the intended locality/load-balancing policy. Ejecting every available host leaves nowhere useful to send traffic. Prove it: create bounded failures, inspect response details and upstream metrics, observe recovery, and confirm a healthy control request still works. A proxy-generated 503 is a symptom to attribute, not proof that a particular protection worked. Circuit breaking ↗ · Destination rules ↗

TLS termination, passthrough, and origination

Termination decrypts TLS at a gateway; passthrough forwards encrypted traffic to a backend; origination makes the proxy establish TLS toward an upstream. The host used for routing, SNI sent to the server, trusted CA, and certificate SAN must agree. MESH_EXTERNAL marks a service's location; it does not choose plaintext HTTP or configure upstream TLS.

If an application already sends HTTPS, blindly originating TLS again can create an unintended extra TLS layer. Decide which hop owns encryption and test that hop. Prove it: validate the served identity and trust chain, successful response, and failure for an incorrect hostname or untrusted certificate. Keep curl -k out of trust-validation evidence. Egress TLS origination ↗

Practice loop: a route can be valid and point at no useful destination

Starting state: the prepared mesh exercise uses VirtualService web and DestinationRule web in concepts-mesh. It requires routing header x-track: preview to subset preview. That subset should select Pods labeled version=v2.

kubectl -n concepts-mesh get virtualservice web -o yaml
kubectl -n concepts-mesh get destinationrule web -o yaml
kubectl -n concepts-mesh get pods --show-labels
istioctl analyze -n concepts-mesh

Look for: the same host referenced across routing objects, matching subset names, and subset selectors matching real Pod labels. If preview selects version=preview while the Pods carry version=v2, the rule can name a subset with no useful backend. An analyzer warning is a lead; a clean result does not certify live traffic.

Act: correct the prepared subset label under spec.subsets, preserving other subsets and policies. If the task instead uses Gateway API, inspect its HTTPRoute parent and backend conditions; do not add a VirtualService on top of it to force a result.

Prove: from the prepared in-mesh client, make one matching and one nonmatching request:

kubectl -n concepts-mesh exec client -- curl -fsS -H 'x-track: preview' http://web/version
kubectl -n concepts-mesh exec client -- curl -fsS http://web/version

This assumes the client has curl and the application exposes /version. Expect the preview response first and the configured default second. Verify repeated requests and the actual waypoint/sidecar path when applicable.

Repeat: change the header value, then the subset label, one at a time.

Coach Caz: “Preview” is the class name; version=v2 is the roster. Match the roster before blaming the room. Traffic routing ↗

Security: transport identity and request permission

Mutual TLS authenticates communicating workloads and encrypts their transport. Authorization determines whether a caller may perform a requested action. A namespace STRICT policy and a method/path authorization policy solve different problems. Test identity, method and path independently. Authentication ↗, AuthorizationPolicy reference ↗.

In this ambient fixture, HTTP authorization needs waypoint enforcement. Distinguish a transport rejection from a deliberate HTTP 403. A failed request is not automatically evidence that the intended policy enforced it.

Default-deny, then a precise allow

An AuthorizationPolicy bound to the waypoint with targetRefs and no rules matches nothing; because it is an ALLOW policy, requests in its scope are denied unless another applicable ALLOW policy permits them. A second ALLOW policy then opens exactly what you list, and its from (source principal) and to (methods, paths) are ANDed inside one rule. Bind both with targetRefs (kind Gateway, or kind Service for one Service). Using a workload selector for an L7 rule can send it to an enforcement point that cannot process those fields and deny traffic instead. Inspect the attachment and proxy evidence; do not depend on one universal error status. Proof is five status codes, not one: the allowed identity gets 200 on the listed paths, the same identity gets 403 for another method or path, and a pod in another namespace gets 403. Deny everything, then allow one caller two paths (kubefit drill istio-18-deny-then-allow).

Request authentication and JWT

RequestAuthentication validates a token from the issuer and jwksUri you name and turns it into a request principal; on its own it still lets requests with no token through. An AuthorizationPolicy that requires a requestPrincipals value is what makes the token mandatory. Test three cases: a valid token (200), an invalid token (401 from authentication) and no token (403 from authorization). Require a valid JWT at the waypoint (kubefit drill istio-11-jwt-auth).

Port-level mTLS exceptions

A workload-scoped PeerAuthentication (with a selector) can require STRICT mTLS for a pod while carving out one port with portLevelMtls, keyed by the container port. In ambient it is enforced by ztunnel with no waypoint involved. Prove it by sending plaintext to the exempt port and to a protected port from outside the mesh: one succeeds, the other is refused at the transport layer. Exempt a metrics port from STRICT mTLS (kubefit drill istio-14-port-level-mtls).

At the edge, Gateway API separates listener configuration from HTTPRoute rules. Certificate references must resolve, the listener must be programmed and the route must attach. Test with the intended hostname so certificate selection and host matching are exercised. A self-signed certificate with curl -k proves presentation and connectivity, not public trust. Gateway API with Istio ↗.

One Gateway can own the listener and the external address for several application namespaces: each HTTPRoute attaches to it across namespaces and claims a hostname, and the listener's allowedRoutes decides whether a route from another namespace may attach at all. Proof is the Gateway's Programmed condition and address, each route's Accepted condition, and three requests with different Host headers — two routed, one unknown host rejected. Route hostnames through one ingress Gateway (kubefit drill istio-13-gateway-hosts).

Practice loop: explain a 403 before widening access

Starting state: a prepared ambient exercise routes Service web through a waypoint. Policy allows ServiceAccount member to GET /health; client member-client should succeed, while guest-client must not. Both clients contain curl.

kubectl -n concepts-mesh get authorizationpolicy -o yaml
kubectl -n concepts-mesh get pod member-client guest-client -o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountName
kubectl -n concepts-mesh exec member-client -- curl -sS -o /dev/null -w '%{http_code}\n' http://web/health
kubectl -n concepts-mesh exec guest-client -- curl -sS -o /dev/null -w '%{http_code}\n' http://web/health

Look for: the actual account, policy targetRefs, permitted method/path, and applicable ALLOW/DENY rules. A namespace match is not equivalent to one permitted ServiceAccount. Correlate failures with the enforcing proxy's logs; the application can emit 403 too.

Act: if the caller is the right account but the policy names the wrong principal, correct that principal under the task's trust domain. Do not replace it with *. If policy is attached to the wrong Service or waypoint, fix attachment instead.

Prove: expected outcomes are member GET succeeds, guest GET is denied, and member access to a prohibited method/path is also denied. A successful TLS connection does not prove authorization; a failure without attribution does not prove the intended rule fired.

Repeat: build the three-case test matrix before writing the policy.

Coach Caz: Membership is not a master key to every cupboard. Identity, method, and path all count. Authorization policy ↗

Troubleshooting: configuration → proxy → application

Start with istioctl analyze, inspect route hosts, subset labels and ports, then inspect proxy configuration. Compare what the API stores with what the data plane received. Finally make a request from the relevant client and inspect the result. A clean analyzer report cannot establish application health or the expected traffic distribution.

istioctl analyze -n <namespace> checks supported configuration relationships against the resources it can see and names the field that points at nothing: IST0101 for a referenced host or subset that does not exist, IST0173 for a DestinationRule subset whose labels match no pod. Read each message literally, fix that field in place, and re-run until the output is clean; then send twenty requests, because a clean analysis does not prove that the live traffic path works. Find and fix two routing faults with istioctl analyze (kubefit drill istio-17-analyze-and-fix).

A 403 requires attribution: it may be a mesh denial or an application response. Read the denial in the waypoint's log (RBAC: access denied), read the principal the policy names, then read the identity the caller actually presents (cluster.local/ns/<namespace>/sa/<serviceaccount>), and change only what is wrong — never widen the rule with namespaces or wildcards to make the symptom go away. Diagnose a 403 from an AuthorizationPolicy (kubefit drill istio-15-authz-troubleshooting).

Control-plane delivery and data-plane configuration

A resource accepted by Kubernetes may never reach the intended proxy. In sidecar mode, istioctl proxy-status helps identify disconnected or unsynchronized proxies; istioctl proxy-config inspects listeners, routes, clusters, and endpoints. In ambient, use the installed version's ztunnel and waypoint diagnostics rather than assuming every Pod owns a sidecar.

Compare the desired API object, the configuration loaded by the actual enforcement point, and one request's access-log details. Missing endpoints, TLS verification failure, timeout, and authorization rejection require different repairs. The application itself can return 403 or 404 too: identify who produced the response. Prove it: configuration is synchronized and the original client request behaves as required. Proxy debugging ↗ · Ambient troubleshooting ↗

Record one successful and one intentionally denied or failed request for each security or resilience exercise. Explain which component produced the outcome. That explanation transfers to unfamiliar tasks better than memorizing a complete YAML example.

Practice loop: diagnose TLS with the intended hostname

Starting state: a practice gateway should serve practice.example.test over HTTPS. You know its reachable IP and have its test CA in practice-ca.crt. The file is supplied by the prepared lab; use your actual trusted CA and hostname.

curl -v --connect-timeout 5 --cacert practice-ca.crt --resolve practice.example.test:443:GATEWAY_IP https://practice.example.test/health

Replace GATEWAY_IP. --resolve chooses the connection address while preserving the URL hostname for SNI, HTTP Host, and certificate checks. Merely adding an HTTP Host header to an IP-based HTTPS URL does not do all three.

Look for:

Act: repair the identified certificate reference, trust chain, route, or backend. curl -k suppresses an important check and is not proof of a trust repair.

Prove: the original request succeeds with verification enabled, while a deliberate wrong hostname still fails verification. In a sidecar-specific backend investigation, use istioctl proxy-config to inspect effective routes/clusters/endpoints; choose waypoint diagnostics for the ambient path.

Repeat: state which name each layer uses before changing a certificate.

Coach Caz: Wearing someone else's name badge does not make the ID check pass. SNI and certificate identity must agree. Istio TLS configuration ↗

Further applied practice

Return an explicit canonical HTTPS redirect

Redirecting returns a Location to the client; rewriting changes a request before proxying it. Do not use a successful HTTPS follow-up as the only evidence: the redirect and the destination are separate systems.

gateway api ↗ #HTTPRedirect ↗

Authorize one cross-namespace backend reference

The backend owner grants permission in the destination namespace. A route in another namespace cannot grant itself that access. Limit the destination resource by name and inspect ResolvedRefs after changing the grant.

gateway api ↗ gateway ↗

Apply request and response headers at the gateway

Request and response filters operate on opposite sides of the proxy exchange. Test with a conflicting caller value to distinguish replacement from append behavior. Backend echo and response headers provide independent evidence.

gateway api ↗ #Headers ↗

Repair TLS hostname validation at a gateway

A reachable HTTPS listener can still fail certificate validation. Check the trust chain and SAN hostname independently of routing; disabling verification conceals the fault.

Istio secure ingress ↗ Gateway TLS ↗

Use gateway access logs to diagnose a route

Access logs connect a request to the proxy that handled it. A route mismatch can produce a proxy-generated 404 even while every backend Pod is healthy.

Istio access logs ↗ Telemetry API ↗