OneLinersCommand workbench
Guides
Containers & Kubernetes / Networking & DNS / Security

Enforce Kubernetes NetworkPolicies with Cilium

Prove Cilium health and enforcement, build a portable default-deny baseline, restore exact DNS and application flows, observe allow and drop verdicts with Hubble, add one optional Cilium-specific FQDN rule, and roll out with reversible SLO gates.

210 min13 stepsChanges system stateRevision 2
Save or explore
Save to collectionCreate a collection in the sidebar first.
0 of 13 steps completed
Goal

Reduce Kubernetes workload network reachability to explicitly approved identities, directions, protocols, ports, and external destinations while preserving application availability, strong diagnosis, narrow rollback, and a clear portability boundary between upstream NetworkPolicy and Cilium extensions.

Supported environments
  • Kubernetes 1.36, current supported
  • Cilium 1.19+, current stable
  • Hubble matching Cilium release
Prerequisites
  • Supported Cilium platform Confirm Cilium is the active CNI and policy implementation, its release is compatible with Kubernetes, kernel, routing, kube-proxy, cloud, encryption, and service-mesh settings, and its deployment owner is known.cilium version && cilium status --wait --verbose
  • Observability Provide controlled Hubble or equivalent policy-verdict visibility with healthy relay coverage, bounded retention, and authorization appropriate for sensitive traffic metadata.hubble status
  • Stable identity labels Protect namespace and workload labels used by policy selectors through RBAC, GitOps, and admission; inventory every selected Pod before apply.kubectl get namespace --show-labels && kubectl get pod -A --show-labels
  • Dependency matrix Document DNS, services, ports, protocols, health probes, sidecars, monitoring, control-plane, external egress, dual-stack, host-network, and maintenance flows with owners and tests.
  • Canary and recovery Use a disposable namespace or narrow canary label, a second administrator, out-of-band cluster access, separate policy files, and a tested rollback command.
  • Application SLOs Provide deterministic allowed and denied probes plus latency, error, readiness, queue, and dependency signals so a forwarded packet is not mistaken for application health.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A measured, namespace-by-namespace zero-trust network baseline enforced by Cilium: stable workload and namespace labels, explicit ingress and egress isolation, DNS allowances, application-port grants, and tests from permitted and denied identities.
  • A policy observation and incident workflow using Cilium endpoint state, policy revisions, drop verdicts, Hubble flows, Kubernetes events, service and EndpointSlice inspection, and packet-path checks without weakening the policy globally.
  • A portable policy core written with upstream networking.k8s.io/v1 NetworkPolicy plus one clearly separated CiliumNetworkPolicy example for an optional Cilium-only FQDN use case, including migration notes for clusters using another conformant implementation.
Observable outcome
  • Selected application Pods become isolated in both directions, continue to resolve cluster DNS and reach their approved backend on one port, and fail deliberate attempts to reach unrelated Pods, namespaces, ports, metadata endpoints, and external destinations.
  • Cilium reports healthy agents and endpoints with realized policy revisions, Hubble shows forwarded and dropped flows matching the expected matrix, and application probes remain healthy during a staged rollout.
  • Operators can remove the narrow policy causing an incident, distinguish a policy drop from DNS, service discovery, endpoint readiness, routing, MTU, or application failure, and port the standard baseline to another NetworkPolicy implementation with documented behavioral retesting.

Architecture

How the parts fit together

Kubernetes NetworkPolicy is an additive allow-list API implemented by the cluster networking provider. The API server accepts NetworkPolicy objects even when no implementation enforces them, so a resource existing is not proof of isolation. Cilium implements the standard namespaced API with identity-aware eBPF policy enforcement and also offers CiliumNetworkPolicy extensions for features such as DNS-aware and layer-7 rules. In default enforcement mode, a selected endpoint becomes default-deny for the direction represented by policy, while unselected endpoints remain allowed. Reply traffic for an allowed stateful connection is permitted, but the reverse side does not automatically gain permission to initiate a connection. Multiple policies combine additively, selectors follow Kubernetes label semantics, and deny behavior often appears to applications as a timeout. The design first proves Cilium health and observes baseline flows, then applies standard portable default-deny and explicit DNS/application allowances to a canary namespace. A Cilium-specific FQDN example is optional and kept separate because it relies on Cilium's DNS proxy and cannot be assumed to work elsewhere.

Cilium agents and operatorProgram and coordinate endpoint identities, service load balancing, policy maps, DNS proxy state, and cluster networking. Agent health is a prerequisite for trustworthy enforcement evidence.
Kubernetes NetworkPolicyProvides the portable L3/L4 policy core: pod and namespace selection, ingress and egress directions, peer selectors, IP blocks, protocols, and ports.
CiliumNetworkPolicyProvides optional Cilium-specific identity, FQDN, and layer-7 controls. It is a deliberate extension, not a portable alias for standard NetworkPolicy.
Endpoint identity and labelsDetermine which policy selects each Pod and which peers match. Label governance is therefore part of the security boundary.
CoreDNS or cluster DNSResolves Services and external names. Egress-isolated Pods need explicit UDP and TCP DNS access, and Cilium FQDN policy relies on observing DNS responses.
Hubble observabilityShows flow verdicts, identities, protocols, ports, DNS and HTTP context where enabled, allowing policy diagnosis without changing an allow rule.
Canary client and backendProvide deterministic allowed and denied probes across labels, namespaces, ports, DNS, and external egress before production workloads are selected.
  1. Confirm that Cilium is the active CNI and NetworkPolicy implementation, every agent and endpoint is healthy, kube-proxy replacement and routing mode are understood, and Hubble or equivalent verdict visibility works.
  2. Inventory workload communication from service ownership, manifests, Hubble observations, and application telemetry. Observation describes current behavior but does not automatically make every observed destination legitimate.
  3. Apply and verify stable namespace and Pod labels. Selectors are authorization inputs and must be protected from arbitrary workload mutation.
  4. Deploy disposable canary client and backend identities, record successful baseline DNS and application flows, and create a matrix with both allowed and denied cases.
  5. Apply standard default-deny ingress and egress policies to the canary scope, expect probes to fail, then add the minimum DNS and application allowances one dependency at a time.
  6. Use standard NetworkPolicy for the portable baseline. Add a CiliumNetworkPolicy only when a required FQDN or L7 control cannot be represented upstream and document its Cilium dependency.
  7. Inspect endpoint realized policy, Cilium status, policy revisions, Hubble verdicts, service endpoints, DNS, and application response before labeling a timeout as a policy failure.
  8. Roll out by namespace and workload label with SLO monitoring, emergency policy rollback, ownership, drift checks, and a migration test suite for future CNI changes.

Assumptions

  • Cilium is already the installed and supported cluster CNI, deployed through a known Helm or platform owner. This guide does not replace another CNI in a running production cluster.
  • The cluster runs a current Cilium stable release compatible with its Kubernetes version, kernel, routing mode, kube-proxy behavior, cloud integration, and observability configuration.
  • The operator can run Cilium status and endpoint checks and has Hubble CLI or equivalent relay access where permitted. Policy testing does not require privileged shell access to application Pods.
  • Workload and namespace labels are controlled by reviewed deployment manifests and admission policy. Untrusted users cannot self-assign labels that represent a trusted peer.
  • Application owners can identify required service, DNS, port, protocol, and external egress dependencies and provide deterministic health probes.
  • The examples use IPv4 and standard TCP/UDP flows. IPv6, SCTP, hostNetwork, node-local services, service meshes, encrypted overlays, and host firewall policy need explicit additional tests when present.
  • Emergency cluster access does not depend solely on the workload network being changed. A second administrator and rollback manifest are available.

Key concepts

Endpoint selection
A policy applies to Pods matching its podSelector in the policy namespace. Selection is direction-specific based on policyTypes and rule sections.
Default deny
Once an endpoint is selected for ingress or egress isolation, traffic in that direction is denied unless at least one applicable policy allows it.
Additive allow
All matching allow rules are combined. One policy cannot subtract access granted by another standard NetworkPolicy.
Identity
Cilium derives security identities from relevant labels and uses them in enforcement. Pod IP changes do not necessarily change application identity.
Realized policy
The policy revision programmed into an endpoint's data path. Kubernetes object creation can precede endpoint realization.
Stateful reply
Return packets for an allowed connection are permitted. This does not authorize the responder to initiate a new connection in the opposite direction.
Namespace selector
A selector over namespace labels. Kubernetes exposes the stable `kubernetes.io/metadata.name` label for matching a namespace by name.
FQDN policy
A Cilium extension that learns IPs from observed DNS responses and applies egress rules to names or patterns. It depends on DNS proxy behavior and is not portable Kubernetes NetworkPolicy.
Policy verdict
An enforcement observation such as FORWARDED or DROPPED with source, destination, identity, protocol, and port context.
Policy portability
The ability to apply the upstream NetworkPolicy manifest on another conformant implementation, followed by behavioral tests because implementation details and unsupported edges may differ.

Before you copy

Values used in this guide

{{policyNamespace}}

Canary or application namespace where standard NetworkPolicies are applied.

Example: payments-policy-canary
{{clientLabel}}

Stable Pod label value selecting the approved client identity.

Example: checkout
{{backendLabel}}

Stable Pod label value selecting the protected backend identity.

Example: ledger
{{backendService}}

Service name exposing the approved backend inside {{policyNamespace}}.

Example: ledger
{{backendPort}}

Exact TCP port accepted by the backend and allowed from the approved client.

Example: 8080
{{dnsNamespace}}

Namespace containing the cluster DNS Pods and Service.

Example: kube-system
{{dnsLabelKey}}

Actual cluster DNS Pod label key verified from live objects.

Example: k8s-app
{{dnsLabelValue}}

Actual cluster DNS Pod label value verified from live objects.

Example: kube-dns
{{approvedExternalFqdn}}

Optional exact external hostname used only in the Cilium-specific FQDN example.

Example: status.example.net
{{approvedExternalPort}}

Optional external TCP port permitted with the FQDN rule.

Example: 443

Security and production boundaries

  • A default-deny policy is not effective merely because the API accepted it. Confirm the active CNI supports enforcement and prove expected drops from a selected endpoint.
  • Labels are authorization attributes. If a tenant can add a label matched by a trusted namespaceSelector or podSelector, it can bypass intent without editing the policy.
  • DNS access is infrastructure access. Permit only the actual cluster DNS identity and ports, test TCP as well as UDP, and avoid broad namespace allowances that accidentally include unrelated Pods.
  • IPBlock semantics around Pod, node, load-balancer, and source NAT addresses depend on where translation occurs. Prefer identity selectors for in-cluster peers and test the actual data path.
  • Hubble payload visibility depends on enabled parsers and policy. Protect observability access, retention, and exported metadata; it may reveal service names, domains, paths, or identity relationships.
  • Cilium-specific L7 policy can redirect traffic through proxies and change failure or latency behavior. Introduce it separately from L3/L4 isolation and monitor application SLOs.
  • Do not create a global allow policy as incident rollback. Delete or revert the smallest newly introduced selector or rule and preserve unaffected isolation.

Stop before continuing if

  • Stop if Cilium status, endpoint health, or policy realization is degraded before the change.
  • Stop if the cluster has another CNI or NetworkPolicy implementation and the Cilium-specific manifests are not actually supported.
  • Stop if workload and namespace labels are mutable by subjects that should not gain the selected trust.
  • Stop if required DNS, control-plane, sidecar, health, telemetry, or external dependencies are unknown.
  • Stop if a canary expected denial remains allowed; enforcement is absent, selection is wrong, or another policy grants access.
  • Stop if an expected allow works only after applying a broad namespace or IP range not present in approved requirements.
  • Stop if Hubble or equivalent evidence is unavailable and application probes cannot distinguish policy drops from other network failures.
  • Stop after any production SLO regression, endpoint policy error, agent restart loop, or unexpected cross-namespace reachability.
01

verification

Confirm Cilium ownership, compatibility, health, and observability

read-only

Identify the active CNI and Cilium release, wait for agent health, inspect endpoints and configuration, and prove Hubble can observe current flows before policy changes.

Why this step matters

Policy testing is invalid when agents, identities, endpoints, or observability are unhealthy. Establishing implementation ownership also prevents applying Cilium CRDs to a cluster where they are inert.

What to understand

Resolve every error and relevant warning from `cilium status --verbose`; the concrete example intentionally demonstrates that an apparent status summary can still contain unacceptable errors.

Compare the Cilium release with Kubernetes, kernel, routing, kube-proxy, encryption, and cloud compatibility documentation owned by the platform team.

Endpoint list should contain the canary workloads after deployment and show healthy policy realization. Missing endpoints indicate networking integration failure rather than policy denial.

Hubble status proves the query path, not complete visibility. Check relay coverage, flow capacity, dropped events, and permissions before relying on it for incident decisions.

Record the Helm or platform owner and ConfigMap settings, especially policy enforcement mode. Do not mutate Cilium configuration as part of an application policy rollout.

System changes

  • No cluster changes; captures CNI version, configuration, agent and operator health, endpoint inventory, and Hubble readiness.

Syntax explained

cilium status --wait --verbose
Waits for a bounded healthy state and exposes detailed component errors rather than a superficial summary.
cilium-dbg endpoint list
Shows Cilium-managed endpoints, identities, policy modes, and state needed to prove enforcement.
Command
cilium version && cilium status --wait --verbose && kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list && hubble status && kubectl -n kube-system get ds cilium -o wide && kubectl -n kube-system get configmap cilium-config -o yaml
Example output / evidence
cilium-cli: v0.18.5 compiled with go1.24.5 on linux/amd64
cilium image (default): v1.19.6
Cilium:             OK
Operator:           OK
Hubble Relay:       OK
Cluster health:     6/6 reachable
Current/Max Flows:  4095/4095 (100.00%)
Connected Nodes:    6/6

Checkpoint: Healthy enforcing data plane

cilium status --wait && hubble status

Continue whenCilium agents, operator, cluster health, and Hubble are healthy with no unresolved error; active CNI ownership is documented.

Stop whenAny agent, endpoint, policy subsystem, or observation path is degraded, or Cilium is not the active policy implementation.

If this step fails

Cilium status times out on one node.

Likely causeAgent crash, API connectivity, kernel feature, CNI initialization, identity, or node pressure is unhealthy before policy work.

Safe checks
  • kubectl -n kube-system get pod -l k8s-app=cilium -o wide
  • kubectl -n kube-system logs ds/cilium --tail=160

ResolutionRepair platform health and confirm existing workloads are stable; do not add policy as a diagnostic experiment.

Security notes

  • Access to Cilium agents and Hubble can reveal topology and traffic metadata. Limit CLI and relay permissions and sanitize exported support data.

Alternatives

  • If Hubble is not approved, use Cilium endpoint metrics, policy counters, application probes, and bounded agent logs, but require equivalent verdict evidence.

Stop conditions

  • No policy rollout starts while status contains unresolved errors like the example output.
02

verification

Inventory legitimate flows and create an allow-and-deny matrix

read-only

Observe representative application traffic, service and DNS dependencies, namespaces, ports, protocols, external destinations, and health paths before selecting endpoints.

Why this step matters

A policy must represent approved dependencies, not merely current packets. Observation finds real paths while an owner decides whether each path is legitimate, removable, or missing from tests.

What to understand

Capture DNS over UDP and TCP, service destinations, backend ports, health probes, sidecars, telemetry, certificate or secret retrieval, and external APIs across normal and maintenance periods.

Correlate flows with Service and EndpointSlice state. A Service name may map to changing Pod IPs and should not become a static IPBlock.

Record source and destination identities and label sets, not only IPs. Cilium policy follows identities while IPs are ephemeral.

Create explicit expected-deny cases: unrelated namespace, wrong client label, wrong backend port, metadata IP, arbitrary internet, and direct endpoint access where unsupported.

Do not automatically allow every observed flow. Compromise, misconfiguration, or obsolete behavior can be present in baseline traffic.

System changes

  • No changes; produces an approved dependency matrix and observed-flow evidence for policy design.

Syntax explained

--since 30m
Bounds observation to a relevant representative interval and avoids an unreviewable unlimited stream.
--output compact
Shows identities, ports, protocols, and verdicts in a concise review format.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

kubectl get namespace --show-labels && kubectl -n {{policyNamespace}} get pod,service,endpointslice --show-labels -o wide && hubble observe --namespace {{policyNamespace}} --since 30m --output compact
Example output / evidence
Jul 28 14:02:11.331: payments-policy-canary/checkout-7d8b:47544 (ID:32110) -> kube-system/coredns-6f6b:53 (ID:112) to-endpoint FORWARDED (UDP)
Jul 28 14:02:11.348: payments-policy-canary/checkout-7d8b:51122 (ID:32110) -> payments-policy-canary/ledger-9bd6:8080 (ID:45122) to-endpoint FORWARDED (TCP Flags: SYN)

Checkpoint: Dependency and denial matrix approved

Continue whenEach DNS, service, health, telemetry, and external path has an owner, direction, identity selector, protocol, port, and test.

Stop whenDependencies are unknown, observation is incomplete, or a required path can only be expressed as broad all-namespace or all-internet access.

If this step fails

Observed flows use raw IPs with no meaningful identity.

Likely causeDestination is external, host-networked, unmanaged, or missing stable Kubernetes labels.

Safe checks
  • hubble observe --namespace {{policyNamespace}} --last 200
  • kubectl get pod -A -o wide --show-labels

ResolutionIdentify ownership and choose an explicit CIDR, gateway, proxy, or Cilium entity rule only after the path is understood.

Security notes

  • Observed domains and paths can be sensitive. Limit Hubble access and redact exported evidence according to data policy.

Alternatives

  • Use application tracing and service catalog data to supplement Hubble when encrypted or proxied flows hide final destinations.

Stop conditions

  • Do not turn an incomplete observation window into an allow-all exception.
03

config

Establish protected namespace and workload identity labels

caution

Apply stable labels through desired state, verify selector matches, and protect trusted labels from arbitrary tenant mutation.

Why this step matters

Selectors are the policy's identity inputs. Stable, owned labels make rules readable and prevent unexpected Pods from entering trusted source or destination sets.

What to understand

Use built-in `kubernetes.io/metadata.name` for exact namespace matching when appropriate, and organization labels for lifecycle tiers only when admission protects them.

Select application labels controlled in Deployment templates, not generated pod-template hashes or mutable debugging labels.

Before policy apply, list every selected Pod and every intended peer. An empty selector match can create false confidence; an overly broad match can cause outage.

Namespace labels affect every policy that references them. Change through reviewed manifests and search existing selectors before overwriting a value.

Admission policy should reserve trust-bearing label prefixes for platform automation or authorized groups.

System changes

  • Adds or changes a namespace label that may affect current and future NetworkPolicy selectors.
  • Does not modify Pod labels in this command; workload labels remain owned by deployment manifests.

Syntax explained

--overwrite
Changes an existing label value and therefore requires review of every selector using that key.
kubernetes.io/metadata.name
Stable built-in namespace label useful for exact name selection without custom label drift.
Configuration
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

kubectl label namespace {{policyNamespace}} security.oneliners.guru/policy-tier=canary --overwrite && kubectl -n {{policyNamespace}} get pod --show-labels && kubectl get namespace {{policyNamespace}} --show-labels
Example output / evidence
namespace/payments-policy-canary labeled
NAME                         READY   STATUS    LABELS
checkout-7d8b7f6f9f-2sxgq   1/1     Running   app=checkout,pod-template-hash=7d8b7f6f9f
ledger-9bd6dd9cc-rm8bk       1/1     Running   app=ledger,pod-template-hash=9bd6dd9cc
payments-policy-canary   Active   31d   kubernetes.io/metadata.name=payments-policy-canary,security.oneliners.guru/policy-tier=canary

Checkpoint: Selectors match only intended identities

kubectl -n {{policyNamespace}} get pod -l app={{clientLabel}} --show-labels && kubectl -n {{policyNamespace}} get pod -l app={{backendLabel}} --show-labels

Continue whenEach selector returns only the reviewed client or backend Pods and namespace trust labels are protected.

Stop whenSelectors match unexpected Pods, required Pods lack stable labels, or untrusted subjects can mutate trusted labels.

If this step fails

A policy unexpectedly selects a debug or migration Pod.

Likely causeThe Pod inherited a broad app or environment label used as a security identity.

Safe checks
  • kubectl -n {{policyNamespace}} get pod --show-labels
  • kubectl -n {{policyNamespace}} get networkpolicy -o yaml

ResolutionIntroduce a purpose-specific protected identity label and update selectors through canary tests.

Security notes

  • Anyone who can create a Pod with a trusted source label may gain network access. Align workload creation and label mutation RBAC with policy trust.

Alternatives

  • Use ServiceAccount-based Cilium identity selectors for Cilium-specific policies when workload label governance cannot express the trust boundary, documenting portability loss.

Stop conditions

  • Do not deploy policy based on generated, mutable, or tenant-controlled trusted labels.
04

config

Deploy deterministic canary clients and backend probes

caution

Create approved and denied client identities plus a backend Service so every policy direction and port has repeatable evidence.

Why this step matters

Deterministic canaries turn policy intent into executable acceptance tests. Separate client labels prove selector boundaries instead of relying on one happy-path connection.

What to understand

Pin test images by digest in production repositories and remove tools after validation. The sample upstream image is illustrative and must undergo the same supply-chain review as any workload.

Disable service-account token automount because network canaries do not need Kubernetes API credentials.

Create one stable Service and named backend port, then confirm EndpointSlice readiness before policy. This separates service discovery failures from enforcement.

Run baseline probes for DNS, allowed backend port, wrong port, untrusted client, unrelated namespace, and external egress. Record exact expected transitions after each rule.

Apply restrictive Linux security context and resource limits in the complete manifest even though the excerpt focuses on networking.

System changes

  • Creates disposable Deployments and a Service in {{policyNamespace}} that generate controlled test traffic.
  • Introduces test image and Pod identities that must be removed after validation if not retained as continuous probes.

Syntax explained

automountServiceAccountToken:false
Keeps network testing from distributing an unrelated API credential.
targetPort:http
Maps Service traffic to the named container port while policy can allow the numeric or named port deliberately.
File policy-canary-workloads.yaml
Configuration
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ledger
  namespace: {{policyNamespace}}
spec:
  replicas: 1
  selector:
    matchLabels: { app: {{backendLabel}} }
  template:
    metadata:
      labels: { app: {{backendLabel}} }
    spec:
      automountServiceAccountToken: false
      containers:
        - name: server
          image: registry.k8s.io/e2e-test-images/agnhost:2.53
          args: ["netexec", "--http-port={{backendPort}}"]
          ports: [{ name: http, containerPort: {{backendPort}} }]
---
apiVersion: v1
kind: Service
metadata:
  name: {{backendService}}
  namespace: {{policyNamespace}}
spec:
  selector: { app: {{backendLabel}} }
  ports: [{ name: http, port: {{backendPort}}, targetPort: http }]
---
# Deploy separate checkout and untrusted client Pods with labels
# app={{clientLabel}} and app=untrusted using the same reviewed test image.
Example output / evidence
deployment.apps/ledger created
service/ledger created
deployment.apps/checkout created
deployment.apps/untrusted created
deployment.apps/ledger condition met
deployment.apps/checkout condition met
deployment.apps/untrusted condition met

Checkpoint: Healthy policy-free baseline

kubectl -n {{policyNamespace}} get deploy,pod,svc,endpointslice -o wide && kubectl -n {{policyNamespace}} exec deploy/checkout -- curl -fsS http://{{backendService}}:{{backendPort}}/hostname

Continue whenAll canaries are Ready, Service has one endpoint, DNS resolves, and baseline approved and unapproved probes are recorded before isolation.

Stop whenService has no endpoint, image is unreviewed, baseline allowed path fails, or tests cannot distinguish client identities.

If this step fails

Service probe fails before any policy exists.

Likely causeDeployment, selector, readiness, named port, DNS, Service, or application listener is wrong.

Safe checks
  • kubectl -n {{policyNamespace}} get service,endpointslice,pod -o wide --show-labels
  • kubectl -n {{policyNamespace}} logs deploy/ledger --tail=100

ResolutionRepair the canary and establish a green baseline; policy cannot explain a pre-existing failure.

Security notes

  • Do not leave a feature-rich debug image running broadly. Limit identity, egress, resources, lifetime, and RBAC.

Alternatives

  • Use purpose-built minimal probe images maintained by the organization rather than a general diagnostic image.

Stop conditions

  • No default deny is applied until every baseline probe has deterministic evidence.
05

config

Apply portable namespace-wide ingress and egress default deny

caution

Select every Pod in the canary namespace for both directions with an upstream NetworkPolicy and confirm all test traffic transitions to denied.

Why this step matters

An explicit empty allow set creates the boundary from which dependencies are added. Applying it to a canary namespace proves the implementation and selection before production.

What to understand

An empty podSelector selects all Pods in the policy namespace; it does not mean no Pods. Both policyTypes isolate both directions.

No ingress or egress arrays are present, so the policy allows nothing. Existing additive policies can still allow traffic and must be included in analysis.

Expect DNS and application probes to time out. That failure is the planned checkpoint proving enforcement, not an incident in a dedicated canary scope.

Observe drop verdicts and endpoint policy state. Application timeout alone could come from many unrelated failures.

Apply this as a separate object so rollback can delete one policy without changing future allow rules or creating a global allow.

System changes

  • Selects every Pod in {{policyNamespace}} for ingress and egress isolation.
  • Immediately denies traffic not allowed by another matching policy, including DNS and health dependencies.

Syntax explained

podSelector:{}
Selects all Pods in the policy namespace.
policyTypes:[Ingress,Egress]
Activates isolation for both incoming and outgoing connection initiation.
File default-deny-all.yaml
Configuration
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: {{policyNamespace}}
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
Example output / evidence
networkpolicy.networking.k8s.io/default-deny-all created
curl: (28) Failed to connect to ledger port 8080 after 3002 ms: Timeout was reached
;; communications error to 10.96.0.10#53: timed out

Checkpoint: Cilium demonstrably drops canary traffic

hubble observe --namespace {{policyNamespace}} --verdict DROPPED --last 50 && kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list

Continue whenDNS and backend probes fail with corresponding drops, and selected endpoints show ingress and egress policy enabled.

Stop whenTraffic remains allowed, agents report errors, or production workloads were selected accidentally.

If this step fails

Some canary traffic remains allowed.

Likely causeAnother additive policy grants it, endpoint is unselected, host or node traffic has special behavior, or enforcement is not active.

Safe checks
  • kubectl -n {{policyNamespace}} get networkpolicy -o yaml
  • kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
  • hubble observe --namespace {{policyNamespace}} --last 100

ResolutionTrace the matching identity and all policies; correct enforcement or selection before writing allow rules.

Security notes

  • Namespace-wide isolation can cause immediate outage. Restrict the first application to disposable canaries and maintain an out-of-band rollback path.

Alternatives

  • Select one canary label first when a dedicated namespace is unavailable, then expand only after evidence is complete.

Stop conditions

  • Stop if intended deny behavior cannot be proven with both endpoint state and flow verdicts.
06

config

Restore only the actual cluster DNS path

caution

Allow selected Pods to query the verified DNS identity over UDP and TCP 53 while retaining all other egress denial.

Why this step matters

Egress isolation commonly breaks service discovery first. A narrow DNS rule restores only verified DNS Pods and both transport modes instead of opening the entire system namespace.

What to understand

Inspect live CoreDNS or node-local DNS labels before substituting variables. Managed distributions may use different labels or an IP path outside ordinary Pods.

The namespaceSelector and podSelector in the same `to` entry are conjunctive: the peer must match both namespace and Pod labels.

Permit TCP 53 as well as UDP because large responses, truncation, DNSSEC, and resolver behavior can require TCP fallback.

This rule allows DNS queries but not every destination returned. Other egress remains denied until a separate policy permits it.

For Cilium FQDN policy, ensure DNS is observed through the proxy path documented by Cilium; bypass resolvers can undermine name-based enforcement.

System changes

  • Allows all selected Pods in {{policyNamespace}} to send DNS traffic to the exact cluster DNS Pod identity on UDP and TCP 53.

Syntax explained

namespaceSelector + podSelector
Matches DNS Pods only when both the namespace and Pod labels are correct.
UDP/TCP 53
Supports ordinary queries and legitimate TCP fallback without opening other DNS-Pod ports.
File allow-cluster-dns.yaml
Configuration
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-cluster-dns
  namespace: {{policyNamespace}}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: {{dnsNamespace}}
          podSelector:
            matchLabels:
              {{dnsLabelKey}}: {{dnsLabelValue}}
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
Example output / evidence
networkpolicy.networking.k8s.io/allow-cluster-dns created
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name:      ledger
Address 1: 10.244.3.81 ledger.payments-policy-canary.svc.cluster.local

Checkpoint: DNS works while backend remains denied

kubectl -n {{policyNamespace}} exec deploy/checkout -- nslookup {{backendService}} && kubectl -n {{policyNamespace}} exec deploy/checkout -- curl -m 3 http://{{backendService}}:{{backendPort}}/hostname

Continue whenService name resolves, but backend connection still times out until the application allowance is added.

Stop whenRule requires all kube-system Pods, DNS remains blocked, or unrelated egress becomes reachable.

If this step fails

UDP DNS works but some lookups still fail.

Likely causeTCP fallback is blocked, resolver labels differ, responses exceed UDP size, or node-local DNS changes the path.

Safe checks
  • hubble observe --namespace {{policyNamespace}} --protocol DNS --last 100
  • kubectl -n {{dnsNamespace}} get pod,svc --show-labels

ResolutionModel the actual resolver path and retain both protocols; do not allow the entire system namespace.

Security notes

  • DNS query metadata may expose sensitive service usage. Protect Hubble DNS visibility and resolver logs.

Alternatives

  • For node-local DNS, allow the documented local address or entity through implementation-specific policy after explicit platform review.

Stop conditions

  • Do not proceed if DNS only works through a broad temporary allow.
07

config

Allow only the approved client to reach the backend port

caution

Create separate portable egress and ingress policies so checkout can initiate TCP {{backendPort}} to ledger while untrusted clients and wrong ports remain denied.

Why this step matters

When both endpoints are isolated, the connection needs source egress and destination ingress permission. Separate policies make each selected endpoint and direction auditable.

What to understand

The podSelectors are namespaced because no namespaceSelector is present. They match only Pods in {{policyNamespace}}, preserving a simple identity boundary.

The egress rule permits checkout to ledger on one TCP port. The ingress rule permits ledger to receive only from checkout on the same port.

Stateful reply traffic returns automatically. Do not add a broad reverse initiation rule merely for responses.

Test untrusted-to-ledger, checkout-to-wrong-port, ledger-to-checkout initiation, and unrelated namespace paths as denied.

Named ports improve intent but their resolution has implementation nuances; the numeric example is explicit and must match the Service and container listener.

System changes

  • Allows {{clientLabel}} Pods to initiate TCP {{backendPort}} to {{backendLabel}} Pods and permits those backends to accept that flow.
  • Leaves all other ingress and egress denied by the namespace baseline unless another policy allows it.

Syntax explained

podSelector app={{clientLabel}}
Selects the source workload for egress and trusted peer for backend ingress.
port: {{backendPort}}, protocol: TCP
Restricts the allowance to the exact application listener.
File allow-checkout-ledger.yaml
Configuration
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-to-ledger-egress
  namespace: {{policyNamespace}}
spec:
  podSelector:
    matchLabels: { app: {{clientLabel}} }
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: {{backendLabel}} }
      ports:
        - { protocol: TCP, port: {{backendPort}} }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ledger-from-checkout-ingress
  namespace: {{policyNamespace}}
spec:
  podSelector:
    matchLabels: { app: {{backendLabel}} }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: {{clientLabel}} }
      ports:
        - { protocol: TCP, port: {{backendPort}} }
Example output / evidence
networkpolicy.networking.k8s.io/checkout-to-ledger-egress created
networkpolicy.networking.k8s.io/ledger-from-checkout-ingress created
ledger-9bd6dd9cc-rm8bk
curl: (28) Failed to connect to ledger port 9090 after 3002 ms: Timeout was reached

Checkpoint: One allowed path, adjacent denials intact

kubectl -n {{policyNamespace}} exec deploy/checkout -- curl -fsS http://{{backendService}}:{{backendPort}}/hostname && kubectl -n {{policyNamespace}} exec deploy/untrusted -- curl -m 3 http://{{backendService}}:{{backendPort}}/hostname

Continue whenCheckout succeeds; untrusted client, wrong port, reverse initiation, and unrelated namespace probes fail with Cilium drop verdicts.

Stop whenAny denied identity or port succeeds, or approved flow requires a broader selector.

If this step fails

Checkout remains blocked although both policies exist.

Likely causeLabels, endpoint readiness, service endpoints, port, protocol, or another sidecar path differs from the manifest.

Safe checks
  • kubectl -n {{policyNamespace}} get pod,service,endpointslice --show-labels
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED --last 100

ResolutionMatch the exact failing source, destination, and port from flow evidence; do not expand to all Pods.

Security notes

  • A user who can create a Pod labeled app={{clientLabel}} can use this path. Protect workload creation and trust labels.

Alternatives

  • Use Cilium ServiceAccount selectors in a CiliumNetworkPolicy when ServiceAccount identity is the approved trust anchor, accepting implementation coupling.

Stop conditions

  • Stop if selector governance cannot prevent unauthorized Pods from acquiring {{clientLabel}}.
08

config

Model one reviewed cross-namespace peer with conjunctive selectors

caution

If an ingress controller or platform client must reach the backend, match its exact namespace and Pod labels together instead of allowing a whole namespace.

Why this step matters

Cross-namespace traffic should match both a reviewed namespace and a reviewed workload identity. Separate peer entries would be ORed and could accidentally allow an entire namespace or same-namespace Pods.

What to understand

A namespaceSelector and podSelector within the same `from` item are ANDed. Two separate list items represent alternatives and therefore broaden access.

Verify the ingress controller's live labels and ownership. Distribution-specific labels may differ from the example and must not be guessed.

Use the built-in namespace-name label for exact namespace selection. Custom environment labels are useful only when protected and intentionally multi-namespace.

Test a controller Pod, a non-controller Pod in ingress-nginx, and a controller-like label in another namespace to prove conjunction.

Ingress controller source identity can change with host networking, proxies, or source NAT. Hubble evidence must show the identity actually seen at the backend.

System changes

  • Allows exact ingress-controller Pods in the ingress-nginx namespace to initiate TCP {{backendPort}} to ledger.

Syntax explained

namespaceSelector and podSelector in one item
Requires both namespace and workload labels, avoiding an unintended OR.
kubernetes.io/metadata.name
Uses Kubernetes' stable namespace-name label for an exact namespace.
File allow-ingress-controller.yaml
Configuration
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ledger-from-ingress-controller
  namespace: {{policyNamespace}}
spec:
  podSelector:
    matchLabels: { app: {{backendLabel}} }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/component: controller
      ports:
        - { protocol: TCP, port: {{backendPort}} }
Example output / evidence
networkpolicy.networking.k8s.io/ledger-from-ingress-controller created
Jul 28 14:31:02.774: ingress-nginx/controller-8f7c:43182 -> payments-policy-canary/ledger-9bd6:8080 to-endpoint FORWARDED (TCP Flags: SYN)

Checkpoint: Only the intended cross-namespace identity passes

hubble observe --to-namespace {{policyNamespace}} --to-label app={{backendLabel}} --last 100

Continue whenApproved controller flow is forwarded; other Pods in that namespace and lookalike labels elsewhere are dropped.

Stop whenObserved source is a node or unknown identity, labels are unprotected, or a broader peer is required.

If this step fails

Controller traffic appears with host identity rather than Pod labels.

Likely causehostNetwork, externalTrafficPolicy, proxying, or routing changes the policy-visible source.

Safe checks
  • kubectl -n ingress-nginx get pod -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork,NODE:.spec.nodeName
  • hubble observe --to-namespace {{policyNamespace}} --last 100

ResolutionModel the actual architecture with reviewed Cilium entities or host policy after separate threat review; do not allow every node casually.

Security notes

  • Namespace membership alone is rarely sufficient trust when many teams can create Pods there.

Alternatives

  • Terminate ingress at a dedicated gateway identity whose source is stable and protected.

Stop conditions

  • Do not apply an example selector before verifying the real ingress controller labels and visible source identity.
09

config

Add portable CIDR egress only for stable reviewed infrastructure

caution

Permit one documentation-safe external CIDR and port when the destination contract is IP-based, then test NAT and routing behavior explicitly.

Why this step matters

Standard NetworkPolicy can express stable IP-based egress, but external SaaS addresses and translated in-cluster traffic may not provide a durable IP contract. Use CIDRs only when ownership and change process are explicit.

What to understand

198.51.100.40 is a documentation address and will not represent a real service. Replace it only with a reviewed destination whose operator guarantees the address range.

A /32 is narrower than a provider-wide range, but availability may require multiple addresses. Each addition needs ownership and expiry review.

IPBlock behavior can interact with pre- or post-policy NAT. Test the actual cluster routing, cloud SNAT, service mesh, and egress gateway path.

Do not use 0.0.0.0/0 with exceptions as a shortcut unless broad internet egress is the approved requirement and metadata or private ranges are controlled elsewhere.

For dynamic hostnames, use the optional Cilium FQDN rule or a controlled proxy rather than continuously chasing IPs in portable manifests.

System changes

  • Allows {{clientLabel}} Pods to initiate TCP {{approvedExternalPort}} to one exact external IPv4 address.

Syntax explained

ipBlock.cidr /32
Matches one IPv4 address independently of Kubernetes Pod labels.
ports TCP {{approvedExternalPort}}
Limits connection initiation to the approved application port.
File allow-approved-cidr.yaml
Configuration
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-to-approved-api
  namespace: {{policyNamespace}}
spec:
  podSelector:
    matchLabels: { app: {{clientLabel}} }
  policyTypes: [Egress]
  egress:
    - to:
        - ipBlock:
            cidr: 198.51.100.40/32
      ports:
        - { protocol: TCP, port: {{approvedExternalPort}} }
Example output / evidence
networkpolicy.networking.k8s.io/checkout-to-approved-api created
Connection to 198.51.100.40 443 port [tcp/https] succeeded!
nc: connect to 198.51.100.41 port 443 (tcp) timed out: Operation now in progress

Checkpoint: Exact address and port only

Continue whenApproved address and port succeeds; neighboring address, wrong port, metadata endpoints, private ranges, and arbitrary internet remain denied.

Stop whenDestination addresses are dynamic, NAT makes identity ambiguous, or a broad range is requested without ownership.

If this step fails

The correct hostname resolves to an address outside the allowed CIDR.

Likely causeThe service uses dynamic DNS, CDN, failover, or provider address rotation incompatible with a static IP contract.

Safe checks
  • kubectl -n {{policyNamespace}} exec deploy/checkout -- nslookup {{approvedExternalFqdn}}
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED --last 100

ResolutionUse a Cilium FQDN policy or egress proxy with a stable contract; do not widen to the internet.

Security notes

  • An allowed IP can host multiple tenants or change ownership. IP allowlisting is not application authentication; retain TLS hostname verification.

Alternatives

  • Route external traffic through an authenticated egress proxy or gateway that provides stable policy and audit.

Stop conditions

  • Do not encode a provider's undocumented, changing address set as a permanent security control.
10

config

Optionally enforce one Cilium-specific FQDN egress rule

caution

When a dynamic external hostname is required, add a separate CiliumNetworkPolicy that allows DNS plus the exact name and port, and label it as non-portable.

Why this step matters

Cilium's DNS-aware policy tracks IPs returned for an approved name and can express a requirement that standard NetworkPolicy cannot. Keeping it separate makes implementation dependency and rollback obvious.

What to understand

This resource requires Cilium CRDs and DNS proxy behavior. Another NetworkPolicy provider will not enforce it and may not even serve the API.

The DNS rule permits queries to the verified cluster DNS identity and asks the proxy to observe names. A broad matchPattern for DNS queries does not itself grant connectivity to every resolved destination.

`matchName` is exact. Use matchPattern only when wildcard semantics are justified, and understand that CNAME chains and DNS behavior can expand the learned address set.

FQDN policy maps names to current IPs; it does not authenticate the remote service. TLS hostname and chain validation remain necessary.

Application DNS caching, alternate resolvers, encrypted DNS, or direct IP use can conflict with this model. Force all workload DNS through the observed, approved path.

Do not combine the first FQDN rollout with L7 HTTP rules. Measure name-based behavior first, then evaluate application-layer requirements separately.

System changes

  • Creates a Cilium-specific policy that redirects or observes DNS and allows current IPs learned for {{approvedExternalFqdn}} on TCP {{approvedExternalPort}}.
  • Introduces an implementation dependency that must be replaced during any CNI migration.

Syntax explained

apiVersion:cilium.io/v2
Selects Cilium's extension API; it is intentionally not portable Kubernetes NetworkPolicy.
toFQDNs.matchName
Allows IPs learned from DNS responses for exactly {{approvedExternalFqdn}}.
rules.dns matchPattern:*
Lets the selected client perform DNS queries through the approved resolver so Cilium can observe responses; destination egress remains separately constrained.
File cilium-fqdn-egress.yaml
Configuration
Fill variables0/7 ready

Values stay on this page and are never sent or saved.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-to-status-fqdn
  namespace: {{policyNamespace}}
spec:
  endpointSelector:
    matchLabels:
      app: {{clientLabel}}
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: {{dnsNamespace}}
            k8s:{{dnsLabelKey}}: {{dnsLabelValue}}
      toPorts:
        - ports:
            - { port: "53", protocol: UDP }
            - { port: "53", protocol: TCP }
          rules:
            dns:
              - matchPattern: "*"
    - toFQDNs:
        - matchName: {{approvedExternalFqdn}}
      toPorts:
        - ports:
            - { port: "{{approvedExternalPort}}", protocol: TCP }
Example output / evidence
ciliumnetworkpolicy.cilium.io/checkout-to-status-fqdn created
Jul 28 14:44:09.081: payments-policy-canary/checkout:52290 -> 203.0.113.55:443 policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Jul 28 14:44:12.214: payments-policy-canary/checkout:52544 -> 198.51.100.80:443 to-stack FORWARDED (TCP Flags: SYN)

Checkpoint: Exact hostname works; other names and direct IPs do not

kubectl -n {{policyNamespace}} exec deploy/checkout -- nslookup {{approvedExternalFqdn}} && hubble observe --namespace {{policyNamespace}} --protocol DNS --last 100

Continue whenApproved name resolves and TCP {{approvedExternalPort}} succeeds; unrelated names, ports, and direct IP destinations are dropped.

Stop whenDNS bypass exists, Cilium DNS proxy errors appear, CNAME expansion is unreviewed, or application SLO regresses.

If this step fails

FQDN resolves but connection is dropped.

Likely causeDNS response was not observed by Cilium, IP cache expired, application uses another resolver, CNAME behavior differs, or port rule is wrong.

Safe checks
  • hubble observe --namespace {{policyNamespace}} --protocol DNS --last 100
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED --last 100

ResolutionRestore the observed DNS path and inspect name-to-IP state; use an egress proxy if DNS dynamics cannot be controlled safely.

Security notes

  • A compromised DNS path can influence learned IPs. Protect resolver integrity and retain TLS authentication at the application layer.

Alternatives

  • Use a stable authenticated egress proxy and portable NetworkPolicy to its fixed identity.
  • Use static IPBlock only when the provider publishes and supports a stable address contract.

Stop conditions

  • Do not deploy this Cilium extension in a portability-required package without an alternative implementation and migration test.
11

verification

Run the full positive and negative behavior matrix with Cilium evidence

read-only

Test DNS, approved application flow, denied identities, wrong ports, cross-namespace paths, external egress, and endpoint policy realization after all intended policies apply.

Why this step matters

A complete matrix proves both availability and isolation. Combining application results, Hubble verdicts, and endpoint realization prevents a single tool from creating false confidence.

What to understand

Run each test from a named identity and record expected success or failure. A generic debug Pod with unexpected labels can invalidate source assumptions.

Test both directions because ingress and egress selection are independent. Allowed replies do not imply reverse initiation permission.

Include wrong port, untrusted label, unrelated namespace, direct Pod IP if relevant, arbitrary internet, approved FQDN, unapproved FQDN, and DNS TCP fallback.

Correlate timestamps among probe output, Hubble flows, policy revisions, Kubernetes events, and application logs.

A timeout with a DROPPED verdict is expected denial. A timeout with FORWARDED verdict requires service, routing, MTU, or application diagnosis.

Repeat after Pod replacement and node movement so identity and policy follow workload lifecycle rather than one cached endpoint.

System changes

  • No intentional changes; generates bounded canary traffic and reads flow and endpoint state.

Syntax explained

--since 5m
Aligns flow evidence with the exact test window.
FORWARDED versus DROPPED
Distinguishes enforcement decisions from failures after policy allowed a packet.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

kubectl -n {{policyNamespace}} exec deploy/checkout -- curl -fsS http://{{backendService}}:{{backendPort}}/hostname && kubectl -n {{policyNamespace}} exec deploy/untrusted -- curl -m 3 http://{{backendService}}:{{backendPort}}/hostname || true && hubble observe --namespace {{policyNamespace}} --since 5m --output compact && kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
Example output / evidence
ledger-9bd6dd9cc-rm8bk
curl: (28) Failed to connect to ledger port 8080 after 3002 ms: Timeout was reached
Jul 28 14:51:04.113: checkout:40718 -> ledger:8080 to-endpoint FORWARDED (TCP Flags: SYN)
Jul 28 14:51:07.224: untrusted:49112 -> ledger:8080 Policy denied DROPPED (TCP Flags: SYN)

Checkpoint: All allows and denials match intended identity graph

Continue whenEvery approved path is forwarded and healthy; every denied path has a matching drop; endpoints have realized current policy revisions.

Stop whenAny negative path succeeds, positive path lacks a forwarded verdict, or endpoint policy state is stale or error.

If this step fails

Hubble shows FORWARDED but curl times out.

Likely causePolicy allowed the packet; Service endpoints, listener, return route, MTU, proxy, TLS, or application behavior failed later.

Safe checks
  • kubectl -n {{policyNamespace}} get service,endpointslice,pod -o wide
  • kubectl -n {{policyNamespace}} logs deploy/ledger --tail=120

ResolutionDiagnose the application and network path without broadening policy that already forwarded the request.

Security notes

  • Do not use production customer payloads for probes. Test deterministic non-sensitive endpoints and control Hubble retention.

Alternatives

  • Automate the matrix as a continuous conformance Job with a minimal image and no Kubernetes API credential.

Stop conditions

  • No production label expansion occurs until every matrix row has authoritative evidence.
12

decision

Roll out by canary label and namespace with SLO gates

caution

Expand selectors in small reviewed stages, watch application and Cilium health, and keep each policy change independently reversible.

Why this step matters

Small selector expansion contains outages and makes causality clear. SLOs, policy drops, endpoint regeneration, and application probes form a release gate rather than an after-the-fact dashboard.

What to understand

Keep default deny, DNS, each application dependency, and optional Cilium extension in separate resources. One change should have one interpretable purpose and rollback target.

Use server-side dry run and diff with the same field owner as production. Ownership conflicts identify another controller and should not be forced.

Canary one workload or namespace, wait through normal traffic and maintenance cycles, then expand with explicit owner approval.

Monitor drop rate by source, destination, and reason, but do not treat zero drops as success if probes are absent or endpoints are unselected.

Track application latency, errors, readiness, queue depth, and external dependencies alongside Cilium agent CPU, endpoint regeneration, policy revision, and Hubble health.

Avoid relabeling many namespaces and applying policies in the same change. Identity expansion should be independently reviewable.

System changes

  • The shown command uses server-side dry run; the production process later applies reviewed policy resources through their owning repository.

Syntax explained

kubectl diff
Shows intended live changes and ownership before admission.
--dry-run=server
Exercises API validation and admission without making policy effective.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

kubectl diff -f policy-bundle.yaml && kubectl apply --server-side --dry-run=server -f policy-bundle.yaml && kubectl -n {{policyNamespace}} get networkpolicy,ciliumnetworkpolicy -o wide && hubble observe --namespace {{policyNamespace}} --verdict DROPPED --since 10m
Example output / evidence
networkpolicy.networking.k8s.io/default-deny-all serverside-applied (server dry run)
networkpolicy.networking.k8s.io/allow-cluster-dns serverside-applied (server dry run)
ciliumnetworkpolicy.cilium.io/checkout-to-status-fqdn serverside-applied (server dry run)
No flows found

Checkpoint: Canary SLO and policy health remain stable

Continue whenApplication probes and SLOs are healthy, expected denies occur, unexpected drops remain below the approved threshold, and endpoints realize policy.

Stop whenAny SLO regression, unexpected drop, policy error, agent pressure, or label expansion appears.

If this step fails

Drop count rises but no user-facing error appears.

Likely causeBlocked obsolete retries, scans, optional telemetry, or an untested dependency is generating traffic.

Safe checks
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED --since 10m
  • kubectl -n {{policyNamespace}} get pod --show-labels

ResolutionClassify each flow with the application owner; allow only legitimate dependencies and keep malicious or obsolete behavior denied.

Security notes

  • Do not temporarily grant broad egress during rollout. That removes the evidence needed to find missing dependencies.

Alternatives

  • Use a dedicated policy audit mode where supported and understood, followed by an explicit enforcement gate; audit mode alone is not protection.

Stop conditions

  • Pause selector expansion after the first unexplained drop affecting a required workflow.
13

instruction

Operate policy drift, incident diagnosis, rollback, and portability

read-only

Assign owners and review dates, monitor realized behavior, rehearse narrow deletion, and keep upstream and Cilium-specific packages separable for future migrations.

Why this step matters

Network policy changes as applications, labels, CNIs, DNS, and external providers evolve. Operations must detect stale allowances and support rollback without dismantling all isolation.

What to understand

Store owner, purpose, ticket, review date, and expected tests as repository metadata or annotations. An unexplained allow rule is technical debt with security impact.

Compare desired and live resources, selected endpoints, realized revisions, and flow evidence. Object drift and behavior drift are both material.

For incidents, delete or revert the smallest new allow or selector first. If availability requires removing default deny, do so only as a time-bounded, approved emergency with preserved evidence and compensating controls.

Classify resources into portable `networking.k8s.io/v1` and Cilium-specific `cilium.io/v2`. A future CNI migration must replace extensions before Cilium is removed.

Behavior-test standard policies on the target implementation because host traffic, NAT, dual stack, named ports, SCTP, and established connections can differ around the edges.

Upgrade Cilium only after policy conformance passes in a representative cluster, and keep agent, operator, CRD, CLI, Hubble, kernel, and Kubernetes compatibility in one matrix.

System changes

  • No direct changes; establishes review, drift detection, incident, rollback, upgrade, and CNI migration procedures.

Syntax explained

networking.k8s.io/v1
The portable upstream policy API that should form the baseline.
cilium.io/v2
Cilium extension API requiring replacement or equivalent controls during migration.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

kubectl -n {{policyNamespace}} get networkpolicy,ciliumnetworkpolicy -o yaml && cilium status --verbose && kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list && hubble observe --namespace {{policyNamespace}} --verdict DROPPED --since 15m
Example output / evidence
NAME                                                               POD-SELECTOR   AGE
networkpolicy.networking.k8s.io/default-deny-all                      <none>         42d
networkpolicy.networking.k8s.io/allow-cluster-dns                     <none>         42d
networkpolicy.networking.k8s.io/checkout-to-ledger-egress             app=checkout   42d
ciliumnetworkpolicy.cilium.io/checkout-to-status-fqdn                 app=checkout   21d
Cilium: OK
Hubble Relay: OK

Checkpoint: Owned, tested, reversible policy inventory

Continue whenEvery policy has an owner and behavior test; current agents and endpoints are healthy; rollback and portability classifications are current.

Stop whenA policy is orphaned, live drift exists, Cilium is unsupported, or no replacement exists for a required extension before migration.

If this step fails

Deleting one policy does not restore the expected path.

Likely causeAnother direction remains isolated, another policy or Cilium deny rule applies, service path is unhealthy, or endpoint realization lags.

Safe checks
  • kubectl -n {{policyNamespace}} get networkpolicy,ciliumnetworkpolicy -o yaml
  • hubble observe --namespace {{policyNamespace}} --last 150

ResolutionTrace both directions and all matching resources. Restore only the precise known-good rules and verify endpoint revisions.

Security notes

  • Emergency rollback permission is powerful. Audit who can mutate cluster-wide Cilium policy and namespace NetworkPolicy.

Alternatives

  • Maintain a vendor-neutral egress proxy and upstream NetworkPolicy boundary to reduce dependence on implementation-specific FQDN controls.

Stop conditions

  • Do not remove Cilium while any cilium.io policy is still a required control.

Finish line

Verification checklist

Cilium and Hubble healthcilium status --wait --verbose && hubble statusAll agents, operator, cluster health, policy subsystems, relay, and observation coverage are healthy with no unresolved errors.
Endpoint realizationkubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint listEvery selected canary and application Pod is a healthy Cilium endpoint with the expected ingress and egress policy modes and current revision.
Approved application flowkubectl -n {{policyNamespace}} exec deploy/checkout -- curl -fsS http://{{backendService}}:{{backendPort}}/hostnameApproved checkout-to-ledger connection succeeds with matching Hubble FORWARDED evidence.
Denied adjacencykubectl -n {{policyNamespace}} exec deploy/untrusted -- curl -m 3 http://{{backendService}}:{{backendPort}}/hostnameUntrusted identity, wrong port, reverse initiation, unrelated namespace, metadata, and arbitrary external probes fail with matching DROPPED evidence.
DNS boundarykubectl -n {{policyNamespace}} exec deploy/checkout -- nslookup {{backendService}}Cluster DNS works over the actual approved resolver path while DNS does not implicitly grant connectivity to every resolved destination.
Portable and extension inventorykubectl -n {{policyNamespace}} get networkpolicy,ciliumnetworkpolicy -o wideUpstream NetworkPolicies form the L3/L4 baseline; every cilium.io extension has an owner, justification, behavior test, rollback, and migration replacement.

Recovery guidance

Common problems and safe checks

A default-deny NetworkPolicy exists but all traffic still succeeds.

Likely causeThe policy selects no Pods, policyTypes omit the tested direction, Cilium enforcement is disabled or unhealthy, or the cluster uses another non-enforcing CNI.

Safe checks
  • kubectl -n {{policyNamespace}} get networkpolicy -o yaml
  • kubectl -n {{policyNamespace}} get pod --show-labels
  • cilium status --wait
  • kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list

ResolutionCorrect selectors and direction, restore Cilium health, and prove a canary drop. Do not proceed based on object existence.

DNS lookups time out after egress isolation.

Likely causeThe policy omits TCP or UDP 53, selects the wrong DNS labels or namespace, or the cluster uses a node-local DNS path not represented by the example.

Safe checks
  • kubectl -n {{dnsNamespace}} get pod,svc --show-labels
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED --protocol DNS
  • kubectl -n {{policyNamespace}} exec deploy/policy-client -- nslookup {{backendService}}

ResolutionModel the actual DNS path with exact selectors or approved IP path, allow UDP and TCP as required, and keep unrelated egress denied.

The Service resolves but the approved backend connection times out.

Likely causeIngress and egress rules are not both satisfied, the backend selector or port is wrong, EndpointSlices are empty, or the application is not listening.

Safe checks
  • kubectl -n {{policyNamespace}} get service,endpointslice,pod --show-labels
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED
  • kubectl -n {{policyNamespace}} describe networkpolicy

ResolutionVerify endpoints and listener first, then correct the exact source, destination, direction, and port rule rather than adding a broad allow.

An unapproved client can still reach the backend.

Likely causeAnother additive policy allows it, the backend is not selected, a label matches unexpectedly, or traffic reaches a different endpoint.

Safe checks
  • kubectl -n {{policyNamespace}} get networkpolicy -o yaml
  • kubectl -n {{policyNamespace}} get pod --show-labels
  • hubble observe --to-label app={{backendLabel}}

ResolutionTrace every matching allow and identity label, remove the unintended grant through its owner, and repeat the negative matrix.

Hubble shows no flows for a test that clearly reaches the application.

Likely causeRelay or observer filters are wrong, Hubble is disabled or unhealthy, traffic bypasses expected endpoints, or retention elapsed.

Safe checks
  • hubble status
  • cilium status
  • hubble observe --namespace {{policyNamespace}} --last 100

ResolutionRestore observability and verify endpoint identities. Use application and Cilium endpoint evidence until flow visibility is trustworthy.

Cilium endpoint policy revision does not catch up.

Likely causeAgent health, identity allocation, policy parsing, API watch, or node resource pressure prevents realization.

Safe checks
  • kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
  • cilium status --verbose
  • kubectl -n kube-system logs ds/cilium --tail=160

ResolutionStop rollout, repair the unhealthy agent or policy error, and wait for endpoints to realize the expected revision before testing.

FQDN egress works initially and later fails for some addresses.

Likely causeDNS responses, TTLs, alternate resolvers, CNAME chains, proxy visibility, or application IP caching diverge from the Cilium DNS policy state.

Safe checks
  • hubble observe --namespace {{policyNamespace}} --protocol DNS
  • kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
  • kubectl -n {{policyNamespace}} exec deploy/policy-client -- nslookup {{approvedExternalFqdn}}

ResolutionEnsure all DNS travels through the observed resolver, inspect CNAME and TTL behavior, and use an egress proxy or gateway if name-to-IP dynamics cannot meet the control requirement.

Connections through a Service work by ClusterIP but direct Pod-IP probes differ.

Likely causeService translation, endpoint selection, source identity, or policy hook placement changes the observed path.

Safe checks
  • kubectl -n {{policyNamespace}} get service,endpointslice -o wide
  • hubble observe --namespace {{policyNamespace}} --last 100
  • kubectl -n kube-system exec ds/cilium -- cilium-dbg service list

ResolutionTest the supported application path through the Service and use flow evidence to map translation; do not base policy on unstable Pod IPs.

A hostNetwork or node-local component ignores expected Pod policy.

Likely causeStandard NetworkPolicy applies to Pod connections and implementation behavior for host or node traffic differs.

Safe checks
  • kubectl -n {{policyNamespace}} get pod -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork,NODE:.spec.nodeName
  • kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list

ResolutionUse the documented Cilium host policy or node architecture after separate threat review; do not assume Pod NetworkPolicy covers host networking.

Application latency increases after adding a Cilium L7 rule.

Likely causeTraffic is redirected through a proxy, parser load is high, rules are too broad, or application behavior differs under L7 inspection.

Safe checks
  • hubble observe --namespace {{policyNamespace}} --protocol http
  • cilium status --verbose
  • kubectl -n kube-system top pod -l k8s-app=cilium

ResolutionRollback the optional L7 policy, preserve L3/L4 isolation, measure proxy impact, and reintroduce only with a justified capacity and SLO plan.

A policy works on Cilium but fails after moving to another implementation.

Likely causeThe manifest used a Cilium CRD, unsupported edge semantics, host or NAT assumptions, named-port behavior, or implementation-specific treatment.

Safe checks
  • kubectl api-resources | grep -E 'NetworkPolicy|CiliumNetworkPolicy'
  • kubectl -n {{policyNamespace}} get networkpolicy -o yaml

ResolutionSeparate upstream resources from Cilium extensions, remove or replace extensions, and rerun the complete behavior matrix on the target implementation.

Applying policy interrupts health probes and causes a rollout failure.

Likely causeProbe source identity, node traffic, ingress controller, service mesh, or monitoring flow was absent from requirements.

Safe checks
  • kubectl -n {{policyNamespace}} describe pod -l app={{backendLabel}}
  • hubble observe --namespace {{policyNamespace}} --verdict DROPPED --last 200
  • kubectl -n {{policyNamespace}} get events --sort-by=.lastTimestamp | tail -n 80

ResolutionRollback the smallest canary policy, identify the exact health source and port, add a narrow justified rule, and repeat canary acceptance before production.

External egress appears from an unexpected source address.

Likely causeNode SNAT, egress gateway, cloud NAT, service mesh, or routing mode changes source identity after policy evaluation.

Safe checks
  • cilium status --verbose
  • hubble observe --namespace {{policyNamespace}} --last 100
  • kubectl -n kube-system get configmap cilium-config -o yaml

ResolutionMap translation and enforce stable egress identity with a reviewed gateway or proxy; do not assume NetworkPolicy preserves source address.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Use only upstream NetworkPolicy when L3/L4 namespace, Pod, CIDR, and port controls meet requirements; this maximizes portability and reduces implementation coupling.
  • Use CiliumNetworkPolicy for Cilium-specific DNS-aware, entity, or L7 controls only when the security requirement justifies the dependency and testing burden.
  • Use a service mesh authorization layer for workload identity and application semantics that NetworkPolicy cannot express, while retaining network isolation as defense in depth.
  • Use egress gateways or proxies when external destinations require stable source identity, centralized audit, or controls that DNS-to-IP policy cannot reliably provide.
  • Use Cilium policy audit mode or a dedicated canary namespace to understand impact where supported, but never treat observation as enforcement.

Operate it safely

  • Add policy-as-code checks for empty selectors, broad namespace matches, missing DNS, wildcard CIDRs, unowned labels, and accidental Cilium-only APIs in portable packages.
  • Create a conformance suite that deploys allowed and denied clients for every critical namespace and runs after Cilium, Kubernetes, kernel, service-mesh, or policy changes.
  • Protect trusted labels with admission policy and restrict who can create or modify NetworkPolicy and CiliumNetworkPolicy objects.
  • Enable Hubble metrics and alerts for sustained policy drops, DNS errors, agent health, endpoint regeneration, policy revision lag, and relay availability while controlling sensitive metadata.
  • Review external egress through dedicated gateways or proxies where DNS-to-IP policy, NAT identity, audit, or compliance requirements exceed ordinary NetworkPolicy.
  • Document CNI migration by classifying each policy as upstream portable or Cilium extension, then behavior-test every path on the target implementation.

Reference

Frequently asked questions

Does an empty podSelector mean no Pods?

In a NetworkPolicy spec, an empty podSelector selects all Pods in that policy's namespace. This makes it useful and dangerous for namespace-wide default deny.

Why do replies work without an explicit reverse rule?

NetworkPolicy enforcement is stateful for established connections. Reply traffic is allowed, but the peer still needs its own policy to initiate a separate connection.

Are standard NetworkPolicies fully portable?

They are the upstream contract, but provider support, NAT, host traffic, named ports, dual stack, and edge behavior still require behavioral tests. Cilium CRDs are explicitly not portable.

Should we start with CiliumNetworkPolicy everywhere?

No. Start with standard NetworkPolicy for L3/L4 controls. Use Cilium extensions only for requirements the upstream API cannot represent and document the dependency.

Recovery

Rollback

Revert the smallest policy or selector change through its owner, keep the portable default-deny boundary whenever possible, prove endpoint revisions and application behavior recover, and preserve flow evidence before considering any broader emergency relaxation.

  1. Stop selector expansion and save bounded Hubble flows, policy YAML, endpoint state, Cilium status, Kubernetes events, Service and EndpointSlice state, and application probe results without collecting sensitive payloads.
  2. Revert or delete the most recent narrow allow or Cilium extension through GitOps. If that rule caused overexposure rather than outage, removal should tighten access immediately; rerun negative tests.
  3. For an availability incident caused by a new default-deny selector, remove only that canary policy or restore the prior selector. Do not apply a permanent allow-all policy.
  4. Wait for affected endpoints to realize the rollback policy revision, then repeat DNS, approved flow, health, wrong-port, untrusted-client, cross-namespace, and external-egress tests.
  5. If Cilium agent health is degraded, stop application policy changes and follow the platform's Cilium rollback or repair procedure for the pinned release; do not replace the CNI during an application incident.
  6. Remove optional `cilium.io/v2` FQDN or L7 resources independently while retaining upstream NetworkPolicy. Route required external traffic through the last known-good proxy or gateway if necessary.
  7. Reconcile live and desired state, document the missing dependency or selector defect, add a regression test, and only reintroduce policy through the canary sequence after SLOs and verdicts are green.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
Kubernetes Network PoliciesofficialKubernetes NetworkPolicy APIofficialCilium network policy overviewofficialCilium policy enforcement modesofficialCilium Kubernetes network policyofficialCilium Hubble observabilityofficialCilium troubleshootingofficial