Cloud native networking

Kubernetes networking: trace the path and prove the result

Network troubleshooting gets simpler when each observation answers one question. Start with address allocation, follow routing and Service selection, inspect policy, then test the application protocol. Cilium and Hubble expose those layers, but the evidence still needs interpretation.

Published domains and course scope

The Linux Foundation's CKNE program page describes a certification under development. KubeFit uses its published domain map for study organization; this course does not assert a released exam, a confirmed beta window or an exact exam environment. Official CKNE program ↗.

25 exercises use a separate Cilium cluster. Additional work is needed for cross-cluster service discovery, broader load-balancing scenarios and the program's emerging AI/LLM traffic objectives. Match commands and CRD schemas to the installed Cilium release.

Build the command-to-evidence habit

Coach Caz: Follow one packet path. Sprinting between unrelated commands only improves your typing endurance.

Use each Practice loop below as a short rehearsal. Read its starting state, predict the result of the first command, then compare that prediction with the evidence. The outputs are illustrative: names, timestamps, addresses, and counts vary. Commands use the example's names; substitute the actual task's namespace, resource, host, and file names when transferring the pattern to a drill.

1. Recognize: say which symptom puts you on this path.
2. Inspect: run the smallest check that separates two plausible causes.
3. Act: change the field or configuration supported by that evidence.
4. Prove: repeat the failed operation and check a constraint that must still hold.
5. Repeat: hide the commands, change one input, and rebuild the sequence from memory.

Use a disposable practice environment for changes. A case that assumes an installed controller, tool, or prepared resource says so; it does not install those prerequisites for you. Linux host commands belong inside a practice Linux VM or the specified lab node. These examples teach the investigation pattern; use the CLI's assigned task and grader for recorded reps.

Infrastructure: separate allocation from forwarding

A CNI configures pod connectivity. IPAM chooses addresses. Kubernetes node PodCIDRs and the Service CIDR are different ranges. Read the controller configuration, node assignments and pod addresses before drawing conclusions about allocation. Three plausible-looking pod IPs are not proof of correct IPAM.

This cluster omits kube-proxy and uses Cilium for Service routing. Bootstrap therefore needs a reachable API-server address before Service translation is available. Inspect Cilium health and the eBPF service map after installation. Cilium without kube-proxy ↗.

Multus adds network attachments without replacing the primary interface. Check secondary addresses, node placement and L2 reachability, then prove normal DNS still works over the primary network. A successful secondary ping alone is incomplete. Multus usage ↗.

Routing, encapsulation, and MTU

A native-routing datapath forwards Pod traffic over routes understood by the surrounding network. An overlay encapsulates it between nodes; its headers reduce the effective payload MTU. Neither design removes the need for a return path. Inspect Pod and node addresses, route selection, tunnel configuration, and the actual interface carrying the flow.

A small ping can succeed while large TLS or HTTP transfers stall. Capture both ends, check retransmissions and ICMP feedback, and compare the configured MTU with the whole path. Prove it: transfer a representative payload in both directions after repair; merely lowering an application's request size conceals the fault. Cilium routing ↗

Practice loop: investigate the node where networking failed

Starting state: a Cilium practice cluster has a Pod stuck in ContainerCreating. The API is reachable. Find the failing node before reading a random network agent's logs.

kubectl -n concepts-network get pods -o wide
kubectl -n concepts-network describe pod CLIENT_POD
kubectl -n kube-system get pods -l k8s-app=cilium -o wide

Look for: FailedCreatePodSandBox and a CNI-related message on a particular node. Copy that node's Cilium Pod name into the next commands:

kubectl -n kube-system exec CILIUM_POD -- cilium-dbg status --verbose
kubectl -n kube-system logs CILIUM_POD --since=5m --tail=100

Act: follow the actual error. An unreachable Kubernetes endpoint during kube-proxy-free bootstrap requires a reachable bootstrap endpoint; exhausted Pod address allocation requires IPAM investigation. Neither is fixed by editing a Service selector. Check the agent's configuration source before changing it so a restart retains the repair.

Prove: the agent is healthy, the original Pod obtains networking, cluster DNS works, and a representative cross-node connection succeeds. A Running Cilium container alone does not establish those data-path properties.

Repeat: compare a failure on one node with the same failure on every node. Identify what scope makes you suspect node-local state versus shared configuration.

Coach Caz: Find the treadmill with the fault before unplugging the whole gym. Cilium troubleshooting ↗

Services and DNS: discovery is not readiness

A Service selects endpoints; its dataplane implementation forwards traffic to them. Inspect selectors, target ports and EndpointSlice conditions. An unready endpoint normally should not receive ordinary Service traffic. Publishing unready addresses is a deliberate peer-discovery choice for systems that need it during bootstrap. EndpointSlices ↗.

A LoadBalancer Service on a cluster without a cloud provider needs two separate things: an allocator that writes an address into status.loadBalancer.ingress, and something on the node network that answers for that address. Cilium's LB IPAM does the first as soon as a CiliumLoadBalancerIPPool exists (no Helm flag; a pool marked CONFLICTING overlaps another and allocates nothing). A CiliumL2AnnouncementPolicy does the second: it elects one node per Service to reply to ARP, recorded as a Lease named cilium-l2announce-<namespace>-<service> in Cilium's namespace, and it announces nothing until loadBalancerIPs: true or externalIPs: true is set. Recognition cues: a Service stuck at <pending> calls for allocator and selector checks; an assigned but unreachable address calls for announcement, route, policy, and backend checks. Proof of success is the pool status, the Service's external IP inside the pool, the lease holder being a worker, and an HTTP 200 against the address. Assign LoadBalancer IPs With L2 Announcements (kubefit drill ckne-17-lb-ipam-l2). LB IPAM ↗, L2 announcements ↗.

internalTrafficPolicy: Local restricts a ClusterIP to endpoints on the client's own node; without kube-proxy each Cilium agent simply programs only its local backends into the Service map, and a node with no backend gets no connection rather than a fallback. It is a Service field, so the workload and the EndpointSlice look unchanged. The cue that it is the right tool is a per-node DaemonSet that must answer for its own node; the proof is a client pinned with spec.nodeName receiving only that node's reply, repeated, from each node in turn. Distinguish it from externalTrafficPolicy (NodePort and LoadBalancer clients, source-IP preservation) and from trafficDistribution: PreferClose (a preference with fallback). Keep Service Traffic on the Local Node (kubefit drill ckne-20-internal-traffic-policy). Service internal traffic policy ↗.

For DNS, isolate whether a request reaches CoreDNS, matches the intended zone and receives the intended response. Hosts entries, forwarding and rewrites serve different purposes. After a change, retest normal cluster and external resolution as well as the new rule. CoreDNS hosts ↗, forward ↗, rewrite ↗.

Practice loop: an allocated LoadBalancer IP still needs a path

Starting state: the practice cluster uses Cilium LB IPAM and L2 announcements. Service web in concepts-network has type LoadBalancer. A client on the intended external network cannot reach it.

kubectl -n concepts-network get service web -o wide
kubectl get ciliumloadbalancerippools -o yaml
kubectl get ciliuml2announcementpolicies -o yaml
kubectl -n kube-system get leases
kubectl -n concepts-network get endpointslices -l kubernetes.io/service-name=web -o yaml

Look for: if EXTERNAL-IP is Pending, investigate allocator pool conditions, available addresses, class, and selectors first. If an address is assigned, check policy selection and the announcement lease before moving to routes, filtering, and backend health. L2 announcement depends on the client reaching the appropriate L2 network; a laptop outside that network may not be a valid test location.

Act: repair the specific pool or announcement selection in the prepared task. Do not change a correct Service target port merely because an external address is unreachable. Use the selected node and actual interface when inspecting ARP/NDP traffic.

Prove: a request from the intended external client reaches the expected backend. Also check a Service or namespace outside the policy's intended scope. Assigned address, elected announcer, ready endpoint, and successful request are separate checkpoints.

Repeat: rehearse one Pending-IP case and one assigned-but-unreachable case. Predict the first two commands for each.

Coach Caz: Booking a locker number does not build a corridor to it. Allocation is only the first step. LB IPAM ↗ · L2 announcements ↗

Traffic management: control plane, data plane, vantage point

A Gateway needs an address, a programmed listener and accepted routes. Test the exact hostname, path and headers. Weighted routing needs a sample of requests; one response cannot establish the distribution. Gateway API HTTP routing ↗.

An egress gateway changes the source address selected external systems observe. Select both source workloads and destinations precisely, then compare the partner's observation with the chosen gateway address. Verify unrelated traffic is unaffected. Cilium egress gateway ↗.

BGP session establishment, prefix advertisement, route installation and successful forwarding are four different checkpoints. Inspect the router as well as Cilium. An Established peer does not prove the Service address was advertised or can answer traffic. Cilium BGP control plane ↗.

Cross-cluster discovery is not cross-cluster reachability

Sharing a Service across clusters requires compatible addressing, inter-cluster connectivity, trusted control-plane connections, and the chosen discovery mechanism. A global Service name cannot repair overlapping Pod addresses or a blocked node-to-node path. Cilium Cluster Mesh can advertise selected services and choose local or remote backends according to its configuration.

Prove it: identify which cluster answered, test from both clusters, then remove a backend or connection in a controlled rehearsal and measure failover. Check that endpoints disappear when no longer healthy. This needs multiple clusters; the single CKNE practice cluster cannot grade it. Cilium multi-cluster services ↗

LLM traffic: queues and streams change the routing question

For inference workloads, response latency includes queuing and generation. Time to first token and time between subsequent tokens describe different user experiences. A connection-count balance can still send work to an overloaded model replica. Keep model identity, endpoint capacity, queue state, and long-lived streaming behavior in view.

Gateway API Inference Extension adds model-serving routing capabilities; inspect the installed release's APIs rather than assuming an ordinary HTTPRoute alone supplies inference-aware selection. Prove it: test stream completion, cancellation, concurrency, and overload while measuring token timing. This is emerging program-scope study, not a claim that KubeFit supplies an inference runtime or a released CKNE exam environment. Inference Extension ↗

Practice loop: read the route condition before sending more traffic

Starting state: Gateway API is installed. HTTPRoute web in concepts-network points to Gateway edge in that namespace and a backend Service. The request fails. jq is available.

kubectl -n concepts-network get gateway edge -o yaml
kubectl -n concepts-network get httproute web -o json | jq '.status.parents[]? | {parentRef,conditions}'
kubectl -n concepts-network get httproute web -o yaml

Look for: the parent reference you intended, its Accepted condition, and ResolvedRefs. ResolvedRefs=False with a backend-reference reason points to name, namespace, kind, port, or reference permission. A listener that rejects the route's namespace requires an attachment-policy repair, not a backend restart.

Act: if the intended Service is in another namespace, verify its existence and port and create the appropriately scoped ReferenceGrant in the backend's namespace when required. Do not grant every namespace access to solve one route. Also verify the Gateway's listener permits this Route's namespace.

Prove: conditions for the intended parent become healthy and a request with the route's exact hostname/path reaches the correct backend. Use the assigned address as GATEWAY_IP:

curl -v --connect-timeout 3 -H 'Host: practice.example.test' http://GATEWAY_IP/health

The hostname and path above are illustrative; they must match the prepared route. A default-backend response is not success for the intended rule.

Repeat: distinguish listener permission, backend ReferenceGrant, and backend readiness.

Coach Caz: Three doors, three keys. Shaking the last door harder will not unlock the first. HTTP routing ↗

Security: encryption and permission are different

WireGuard protects the configured cross-node path. Verify peers and capture the relevant underlay traffic. A capture without plaintext proves only what was observed on that interface during that test; it does not establish that all traffic everywhere is encrypted. Cilium WireGuard ↗.

At L3/L4, a denied connection may time out. At L7, an HTTP-aware policy can return a deliberate denial. Test method, path and source identity independently. FQDN policies also depend on allowed DNS observation; granting arbitrary DNS egress is not equivalent to granting arbitrary application egress. Cilium policy language ↗.

A toFQDNs rule allows destinations by the name the pod resolved, which only works when the lookup itself is allowed and passes through Cilium's DNS proxy (rules.dns with matchPattern on the kube-dns egress rule). Use endpoint or Service selectors for in-cluster backends rather than treating FQDN policy as a general substitute for workload identity; inspect the installed Cilium version's identity and DNS-policy behavior. The proof is success from the named destination, a policy-confirmed denial against a disallowed destination, names still resolving, and the name present in cilium-dbg fqdn cache list. Restrict Egress by DNS Name (kubefit drill ckne-15-dns-aware-egress). DNS-based policies ↗.

Two facts govern deny rules. Deny beats allow from any policy kind, and selection by a policy normally enables default-deny for the applicable direction, unless the policy's enableDefaultDeny settings change that behavior. A cluster-wide policy that must block one destination for many namespaces therefore selects on the namespace label (io.cilium.k8s.namespace.labels.<key>) in a CiliumClusterwideNetworkPolicy, carries an egressDeny with toCIDR, and either sets enableDefaultDeny: { egress: false } or adds an explicit allow-all egress rule so everything else keeps working. CIDR rules only match identities Cilium did not derive from labels, which is exactly what an off-cluster address like the metadata endpoint is. Proof is a timeout from a tenant pod, a 200 from the same pod to an in-cluster Service, working DNS, and an unlabelled namespace still reaching the address. Block Metadata Access Cluster-Wide (kubefit drill ckne-16-clusterwide-policy). Deny policies ↗.

When a freshly applied allow-list "breaks the application", the dropped flow is usually not the application flow. An egress list that names the backend but not kube-dns drops every lookup first, so curl reports "could not resolve" rather than a timeout. Read the verdict on the node where the policy is enforced (hubble observe --verdict DROPPED from the agent on the pod's node, or cilium-dbg monitor --type drop), note the real destination, and repair the policy in place by adding the missing rule with io.kubernetes.pod.namespace: kube-system and k8s-app: kube-dns on port 53. A broad world or CIDR allowance can weaken the intended boundary and may not even repair cluster DNS; allow the observed DNS path precisely. Proof is the saved dropped flow, a 200 to the intended backend, resolution working, and a still-blocked third destination. Restore DNS Under an Egress Policy (kubefit drill ckne-18-policy-troubleshooting). Policy troubleshooting ↗.

For gateway TLS, inspect both certificate references and listener status. A cross-namespace reference must be granted in the namespace holding the target. Then test HTTPS with the correct hostname and test HTTP redirection separately. Gateway API cross-namespace routing and references ↗.

Pod identity, authentication, and authorization

An IP address or namespace label is not a cryptographic identity. NetworkPolicy constrains selected traffic, TLS authenticates a peer under a trust model, and an authorization policy grants operations to an identity. Encryption alone does not decide whether the caller may access a particular HTTP path.

Trace the enforcement point: CNI for supported network policy, gateway for edge TLS, and an application or mesh for workload-level authentication and authorization. Prove it: valid allowed identity succeeds, valid unauthorized identity is denied, invalid credentials fail, and unrelated application paths remain available. Correlate the response with the enforcing component's evidence. Istio authentication and authorization ↗

Practice loop: use a dropped flow to identify the denied operation

Starting state: Hubble CLI is connected to the practice cluster's relay. Pod client in concepts-network should reach one backend and DNS, while unrelated egress stays denied. Reproduce the failing request in another terminal while observing:

hubble observe --from-pod concepts-network/client --verdict DROPPED --follow

Look for: source identity, destination, protocol/port, direction, and drop reason. A UDP/53 drop answers a different question from a TCP/8080 drop. Stop the live observation with Ctrl-C after collecting evidence, then inspect policy:

kubectl -n concepts-network get networkpolicy,ciliumnetworkpolicy -o yaml
kubectl get ciliumclusterwidenetworkpolicy -o yaml

Act: repair the missing allowed flow in the prepared policy. With Cilium, an explicit deny can override an allow; another allow rule will not cancel it. If no drops appear, confirm Hubble connectivity, filters, timing, and enforcement node before concluding that policy is uninvolved.

Prove: the original client can resolve DNS and reach the allowed application. A disallowed but healthy destination remains denied with corresponding flow evidence. Test from the original Pod; using an administrator's different identity can conceal the fault.

Repeat: say the destination port out loud before editing a rule.

Coach Caz: Read the failed rep on the log sheet. Do not invent a different exercise to make the number green. Hubble inspection ↗

Observability: correlate instead of guessing

A packet capture records traffic at a specific interface and time. Missing replies can have several causes. Hubble adds flow identity and verdict information; metrics aggregate events; exports retain history. Correlate these views before naming a policy as the cause. Hubble observability ↗.

A Hubble verdict is per flow and carries source, destination and reason. To answer "which pod is being denied", filter the flows to the destination with --verdict DROPPED and read source.pod_name and drop_reason_desc (POLICY_DENIED, printed as Policy denied). Ingress policy is enforced, and the drop recorded, on the destination pod's node, so query that node's agent or the relay; the client's node shows only its egress. Reasons that are not policy (CT: Map insertion failed, Stale or unroutable IP) point at the datapath, not at a rule. Proof of success is a finding whose pod name and reason match the flow log, cross-checked against the policy selector and the pod labels without changing either. Identify a Denied Client With Hubble (kubefit drill ckne-19-hubble-verdicts). Hubble CLI ↗.

Distinguish flow evidence from an application trace

A flow log can show addresses, identity, protocol, bytes, and a verdict at one observation point. It does not by itself explain time spent inside an application. Metrics reveal rates and saturation; a distributed trace follows the request through instrumented services. Request IDs or trace context let you correlate those records.

Prove it: locate a slow request, separate network transit from server/queue time, and compare before/after latency and errors from the same client. Include retransmissions, drops, and DNS/TLS setup when relevant. Do not infer complete end-to-end tracing from the presence of Hubble alone. Hubble observability ↗ · OpenTelemetry tracing ↗

For each attempt, record the client and server, source and destination address, expected path, failing layer and proof after repair. Preserve the capture or report requested by the task. Repeat from the original client so a successful test from another vantage point does not mask the failure.

Practice loop: small packets work but larger transfers stall

Starting state: a prepared Linux networking lab has two endpoints and a path-MTU fault. Small requests succeed; larger responses stall. Use the affected Linux network namespace and its actual destination, not your host's unrelated route.

ip route get DESTINATION_IP
ip link show
ping -4 -c 3 -M do -s 1200 DESTINATION_IP
ping -4 -c 3 -M do -s 1472 DESTINATION_IP

Look for: a size-dependent failure, fragmentation-needed feedback, or a route MTU. These sizes are IPv4 ICMP payloads: add IP/ICMP headers and account for tunnel overhead when reasoning about the path. ICMP filtering can also make ping fail, so pair this evidence with the application's transfer and captures.

Inspect further: on the relevant interface, a bounded capture of the controlled flow can reveal retransmissions and ICMP feedback:

sudo tcpdump -ni INTERFACE -c 30 'host DESTINATION_IP and (icmp or tcp)'

Replace both uppercase tokens. Act: correct the prepared path's MTU or permitted PMTU feedback according to the evidence and intended network design. Arbitrarily lowering every interface is not a diagnosis.

Prove: the representative large transfer works in both directions, normal small requests remain healthy, and the fix survives the configuration lifecycle required by the task.

Repeat: explain why a successful ping is not proof that a large encrypted transfer can work.

Coach Caz: Fitting one dumbbell through the door does not prove the rack will fit. Measure the whole load. Cilium routing and encapsulation ↗

Further applied practice

Keep DNS working while denying application egress

DNS is application traffic with its own destination and ports. Two selectors within one peer are an AND condition; separate peers would broaden access. Test DNS, the forbidden application request and an unaffected control path to distinguish isolation from a service outage.

network policies ↗ dns debugging resolution ↗

Repair a UDP Service transport mismatch

UDP and TCP with the same numeric port are different transports. Trace the listener, Service protocol, target port and actual datagram response.

Service protocols ↗ Debug Services ↗

Diagnose a path MTU black hole

Path MTU discovery needs ICMP feedback. A successful small ping does not prove that larger application packets can traverse a path.

Cilium routing concepts ↗ Troubleshooting connectivity ↗

Repair an asymmetric return path

Every request needs a return path. Inspect route selection at the destination as well as the source; forwarding in one direction is not enough.

Cilium routing concepts ↗ Troubleshooting connectivity ↗

Authorize one cross-namespace backend reference

The backend owner grants permission in the destination namespace. A route in another namespace cannot grant itself that access. Limit the destination resource by name and inspect ResolvedRefs after changing the grant.

Cross-namespace routing ↗ Cilium Gateway API ↗