Fix Kubernetes Metrics API not available: APIService, TLS and metrics-server
Trace Metrics API not available from kubectl top through API aggregation, Service endpoints, metrics-server and kubelet access. Repair the failing layer without disabling TLS verification or replacing a managed add-on blindly.
Restore current CPU and memory resource metrics for the intended cluster and application namespace, with a verified ownership boundary, a compatible metrics-server deployment, and an explicit rollback path for every approved change.
- kubectl client Compatible with the target Kubernetes API server; commands use kubectl, not a shell-specific scripting language
- Metrics Server compatibility reviewed 2026-09-09 0.9.x: Kubernetes 1.34+, 0.8.x: Kubernetes 1.31+, 0.7.x: Kubernetes 1.27+, Use the supported provider add-on version when the cluster is managed
- Known cluster and incident scope Identify the exact kubeconfig context, cluster owner and failing namespace before investigating. Record whether kubectl top fails for everyone, only your identity, all nodes, or one workload. A screenshot without the command, context and timestamp is insufficient to choose a repair.
kubectl config current-context - Authorized diagnostic access Use your normal approved identity. Some checks require reading cluster-scoped APIService and Node objects or logs in the add-on namespace. A Forbidden response is a reason to ask the owner for the relevant evidence, not to grant yourself cluster-admin. Do not export raw kubeconfig credentials.
- Add-on ownership and recovery Determine whether the provider, Helm, GitOps or an operator owns metrics-server. Have the previous approved release and configuration available before changing it. Establish who is responsible for capacity decisions if autoscaling is degraded; a metrics repair must not silently change replica policy.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A boundary-by-boundary diagnostic record that explains why resource metrics are unavailable, rather than a collection of speculative restarts.
- A verified, owner-managed recovery with current samples, preserved TLS validation and an acceptance check using the original caller.
- Distinguish a missing registration, unhealthy Service backend, failed node scrape and caller authorization problem.
- Explain why a healthy Prometheus dashboard or Running pod is not proof that kubectl top and resource-based HPA have recovered.
Architecture
How the parts fit together
Resource metrics pass through several independently failing boundaries. The workflow follows the request outward and checks collection inward; it does not assume that one successful hop proves the complete pipeline.
- Capture the original caller, cluster and literal error before changes.
- Resolve the APIService to its actual Service and ready backends.
- Correlate current add-on logs with one affected node and the failing trust or network boundary.
- Repair that boundary through its owner and verify fresh samples under the original access policy.
Assumptions
- You are troubleshooting the standard resource metrics API, not installing a complete historical monitoring platform or a custom-metrics adapter.
- The illustrative excerpts are source-reviewed teaching examples, not results from an execution against your cluster. No Kubernetes cluster was used to claim end-to-end reproduction of this guide.
- Cluster-specific networking, provider add-on ownership and certificate issuance differ. Names and namespaces are inputs, and mutation details must come from your reviewed deployment system.
Key concepts
- Available is not complete
- The aggregated service can be reachable while individual nodes still lack current data. A condition is a useful checkpoint, not a substitute for checking the affected sample.
- Two TLS directions
- The API server connects to the aggregated metrics service, while metrics-server separately connects to kubelets. Diagnose which connection rejected which certificate before changing a CA or address.
- Resource versus historical metrics
- This workflow targets current CPU/memory resource data consumed by kubectl top and resource-based autoscaling. Historical dashboards, retention and Prometheus queries have a different collection and storage path.
- Original identity acceptance
- Repeating a test as an administrator can distinguish authorization from availability, but completion requires the intended caller to succeed with its approved scope.
Fill these once. Every matching command and configuration block updates immediately; values stay in this page only.
Security and production boundaries
- Treat deployment YAML, node addresses and log excerpts as internal operational evidence. Redact credentials and customer identifiers before sharing; never attach raw kubeconfig or private keys.
- Keep kubelet authentication and serving-certificate verification enabled. Broad RBAC, anonymous kubelet access and internet-exposed node ports can turn a monitoring incident into a much larger security incident.
Stop before continuing if
- Stop if you cannot identify the current add-on owner or if the next proposed step changes control-plane configuration outside your authority.
- Stop and escalate capacity risk when autoscaling is degraded and no operator owns the workload's safe operating limits; do not improvise scaling policy as a side effect of this guide.
command
Confirm the cluster and reproduce the exact failure
Start with the client and server identity, then run one bounded request for the affected workload namespace. The literal error Metrics API not available indicates that kubectl could not obtain resource metrics; it does not identify whether the missing layer is discovery, aggregation, data collection or access. Preserve the error text before restarting anything.
Why this step matters
A cluster-context mismatch can make every subsequent observation internally consistent while still diagnosing the wrong environment entirely.
What to understand
Record the time of the failing request and the command that produced it. Compare with a successful request made by an already-authorized operator in the same context and namespace. Do not use a different cluster or a privileged impersonation session as proof that the original identity works.
Check client/server version compatibility against the Kubernetes version-skew policy and the provider's supported add-on matrix. The example compatibility table is a reviewed starting point, not permission to deploy an arbitrary newer release. A working workload API does not prove that its aggregated metrics API is healthy.
If the initial command reports Forbidden, preserve that distinction and proceed to the authorization check. If it times out before reaching the API server, repair ordinary kubectl connectivity first. Repeated retries with an unbounded timeout only obscure the incident timeline and put more requests into the failing path.
System changes
- These commands read context metadata, server version and resource metrics. They do not install an add-on or alter autoscaling.
Syntax explained
-n- Selects the workload namespace, which can differ from the namespace running metrics-server.
--request-timeout=10s- Bounds each API request so an unavailable service does not leave this diagnostic waiting indefinitely.
-o yaml- Displays client and server version metadata for the compatibility record.
Values stay on this page and are never sent or saved.
kubectl config current-context
kubectl version -o yaml
kubectl top pods -n '{{workloadNamespace}}' --request-timeout=10sIllustrative failure, not a measurement from your cluster: production-eu error: Metrics API not available
Checkpoint: Identity and symptom captured
Continue whenThe correct context, namespace, observed timestamp and exact error are recorded; ordinary authenticated API access works.
Stop whenThe context is uncertain, basic API access fails, or investigating this namespace is not authorized.
command
Locate the registered metrics API and its availability condition
Inspect the APIService registration before searching for a Deployment with an assumed name. The registration identifies the Service and namespace that receive metrics requests. A provider may use names different from the upstream manifest, so copy the observed namespace and Service name into the fields used by the next step.
Why this step matters
The APIService is the routing contract between the Kubernetes API server and the metrics implementation, independent of a particular deployment name.
What to understand
If the APIService does not exist, ask whether resource metrics are an enabled feature of this cluster and whether the owner intentionally removed the add-on. Review installation history before installing anything. An absent registration is different from a registered service with no healthy backend.
Read status.conditions as a whole: type, status, reason, message and lastTransitionTime. Treat the reason as a lead, not a final root cause. Missing service or endpoints suggests a backend wiring problem; failed discovery can also involve TLS or control-plane reachability. Record the actual message rather than mapping every False condition to a certificate failure.
If Available is True, continue: the aggregation endpoint can answer discovery while some node samples are unavailable. Also inspect the APIService TLS configuration as evidence. Existing insecure settings should be recorded for owner review, not copied to the kubelet connection or advertised as a universal fix.
System changes
- Only the selected APIService registration and condition are read.
Syntax explained
v1beta1.metrics.k8s.io- The APIService registration used by the resource metrics API; this is not the Prometheus query API.
-o yaml- Includes service routing, TLS settings and condition details rather than only the summary column.
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml --request-timeout=10sIllustrative excerpt; names and reasons vary:
spec:
service:
name: metrics-server
namespace: kube-system
status:
conditions:
- type: Available
status: "False"
reason: MissingEndpointsCheckpoint: Routing boundary identified
Continue whenThe metrics implementation owner, Service namespace/name and current Available condition are known, or the owner confirms no add-on is installed.
Stop whenA second metrics implementation already owns the registration or managed-provider policy forbids replacing it.
command
Check Service routing and ready backends
Use the names from the APIService, not the defaults shown in an example. Compare the Service selector and target port with the EndpointSlice addresses, ports and readiness conditions. Then use the pod and controller names actually returned to identify the deployment whose logs you are authorized to inspect.
Why this step matters
A reachable Service address cannot supply useful data when its selector, port mapping or ready backend set is wrong.
What to understand
An empty EndpointSlice result can mean a wrong namespace/name, a selector mismatch, a controller problem or an intentionally custom Service without selectors. Compare the installed release with its declared configuration. Do not patch random labels until the owning controller and desired state are understood.
A pod phase of Running is not the same as Ready. A failed readiness probe can correctly remove an otherwise running process from Service traffic. Describe the affected pod when you need scheduling, image-pull or probe evidence, and distinguish those deployment failures from kubelet scrape failures.
The Service port and the port reached on a backing pod need not be equal. Write down both before reviewing firewall rules. Control-plane-to-add-on traffic and add-on-to-node traffic use different source identities and network paths; a successful connection from your laptop tests neither path conclusively.
System changes
- Reads Service, EndpointSlice and pod metadata only; does not edit labels, ports, probes or network policies.
Syntax explained
kubernetes.io/service-name- Selects EndpointSlices associated with the observed Service rather than assuming a pod label.
-o wide- Adds useful pod placement and address information; treat the output as infrastructure metadata.
Values stay on this page and are never sent or saved.
kubectl -n '{{metricsNamespace}}' get service '{{metricsService}}' -o yaml --request-timeout=10s
kubectl -n '{{metricsNamespace}}' get endpointslices -l 'kubernetes.io/service-name={{metricsService}}' -o yaml --request-timeout=10s
kubectl -n '{{metricsNamespace}}' get pods -o wide --request-timeout=10sIllustrative decision record: Service selector: k8s-app=metrics-server Matching backend: metrics-server-example Endpoint ready: false Next check: pod readiness and recent logs, not a firewall rewrite
Checkpoint: Backend state explained
Continue whenService selector/ports and readiness agree with the owning release, or a precise discrepancy has been captured for a reviewed repair.
Stop whenThe proposed change would take ownership from another controller or affect unrelated Services.
command
Read bounded logs and classify the failing hop
Inspect the current metrics-server configuration and a short log window. Fill the controller and container fields from the deployed workload. Correlate each error with the APIService state and the affected node, rather than taking an old startup error as evidence of a current outage.
Why this step matters
Short time-correlated evidence separates transport, certificate and authorization failures that otherwise produce the same visible kubectl symptom.
What to understand
Look for the destination address and the action associated with the error. A kubelet scrape failure is downstream of metrics-server; a front-proxy authentication failure concerns a request arriving at the aggregated server. The text x509 alone does not tell you which trust configuration should change.
Inspect image tag, command arguments, service account and declared resources. Check whether a recent deployment changed preferred node address types, certificate authority mounts, secure port or resource limits. Compare against the previous approved manifest, not an unrelated internet snippet.
Deployment-scoped logs may select one matching pod. For multiple replicas, inspect the individual pod names from the previous step and compare them. If a container restarted, request the previous instance's logs only for that observed pod. Redact tokens, internal identifiers and customer workload names before sharing an incident excerpt.
System changes
- Reads workload configuration and a bounded log excerpt; it does not execute commands inside pods or create a diagnostic pod.
Syntax explained
-c- Chooses the observed metrics-server container; sidecar logs may not contain the scrape failure.
--since=10m --tail=200- Limits both the time interval and returned lines to keep evidence focused.
--timestamps=true- Allows correlation with deployment changes and APIService transitions.
Values stay on this page and are never sent or saved.
kubectl -n '{{metricsNamespace}}' get deployment '{{metricsDeployment}}' -o yaml --request-timeout=10s
kubectl -n '{{metricsNamespace}}' logs 'deployment/{{metricsDeployment}}' -c '{{metricsContainer}}' --since=10m --tail=200 --timestamps=true --request-timeout=10sIllustrative classification, not a cluster execution transcript: Scrape to node-a: x509: certificate signed by unknown authority Scrape to node-b: dial tcp 10.20.0.22:10250: i/o timeout These require different investigations.
command
Verify node addresses, kubelet TLS and network ownership
For an affected node, inspect the addresses and kubelet endpoint advertised by Kubernetes. Compare them with the destination in metrics-server logs and the configured preferred address types. Ask the cluster owner to verify the actual serving certificate and network path from the metrics-server placement.
Why this step matters
Selecting an address that is routable but absent from the serving certificate can turn a networking workaround into a persistent TLS failure.
What to understand
For a timeout, the owner should trace the exact source workload location, destination node address and advertised kubelet port through NetworkPolicy, host firewall, routing and provider security rules. Restrict any approved allowance to that path. Opening a kubelet port to the internet is not required for resource metrics.
For an issuer error, verify which authority signed the kubelet serving certificate and whether the metrics-server trust configuration contains the correct public CA chain. For a name or IP mismatch, compare subject alternative names with the selected node address. Request a valid serving certificate or a supported address configuration; do not suppress verification with --kubelet-insecure-tls.
For unauthorized or forbidden scrape responses, the owner should inspect kubelet webhook authentication/authorization and the add-on service account permissions expected by that exact supported release. Do not enable anonymous kubelet access. A node-proxy request under your own privileged identity is a different authentication path and cannot prove that metrics-server itself is authorized.
System changes
- Reads one Node object. Certificate, routing and authentication changes remain a separate owner-approved operation.
Syntax explained
nodeName- The exact node named by current failed scrape evidence; compare a healthy node only after documenting the failing one.
Values stay on this page and are never sent or saved.
kubectl get node '{{nodeName}}' -o yaml --request-timeout=10sIllustrative excerpt:
status:
addresses:
- type: InternalIP
address: 10.20.0.22
daemonEndpoints:
kubeletEndpoint:
Port: 10250
This metadata alone does not prove the TLS certificate matches.Checkpoint: Failure is assigned to a specific boundary
Continue whenThe evidence identifies transport reachability, serving-certificate trust/name, downstream authorization, or missing kubelet measurements as distinct possibilities.
Stop whenThe only proposed workaround disables authentication or certificate checks, or requires uncontrolled node/control-plane reconfiguration.
command
Separate caller permissions from add-on permissions
Check the permissions of the same identity that received the original error. Reading node metrics is cluster-scoped; reading pod metrics can be scoped to the application namespace. These checks do not impersonate metrics-server and do not establish its ability to authenticate to kubelets.
Why this step matters
Giving the human caller broader permissions cannot repair an unavailable backend and can conceal a separate least-privilege problem.
What to understand
If your namespace request is denied but an already-authorized operator confirms healthy metrics for the same namespace, request only the intended read access through the normal RBAC process. A node-metrics denial may be correct policy for a tenant. Do not treat every no result as a misconfiguration.
If can-i says yes but the metrics request still fails, retain both results. Authorization success does not imply that the aggregated API is available or that it holds a fresh sample. If discovery itself is broken, ask the owner to inspect the registration and relevant access review rather than granting wildcard API groups.
Review three identities separately in the incident: the caller, the API server's aggregation/front-proxy identity, and the metrics-server service account. Their certificates and permissions serve different purposes. Changing the caller's RoleBinding cannot fix an invalid front-proxy client certificate, and replacing a front-proxy CA cannot repair a node-address SAN mismatch.
System changes
- Submits authorization reviews for the current identity without modifying roles or bindings.
Syntax explained
pods.metrics.k8s.io- The pod resource in the metrics API group, not the ordinary core pods resource.
list- Checks the collection-read operation used when listing metrics; the namespace scope must match the request.
Values stay on this page and are never sent or saved.
kubectl auth can-i list pods.metrics.k8s.io -n '{{workloadNamespace}}' --request-timeout=10s
kubectl auth can-i list nodes.metrics.k8s.io --request-timeout=10sIllustrative result for a namespace-limited operator: yes no Pod metrics may be authorized even when node metrics are not.
decision
Apply only the repair supported by the evidence
Stop collecting unrelated output once the failing boundary is sufficiently narrow. Produce a short change proposal: observed failure, evidence, owner, exact configuration difference, blast radius, acceptance command and rollback revision. Apply the change through the existing provider, Helm or GitOps workflow, not through a second unmanaged installation.
Why this step matters
One scoped, reversible change preserves causal evidence and avoids masking several configuration errors with a broad privilege or TLS bypass.
What to understand
If the add-on is missing, select the provider-supported installation or a pinned upstream release compatible with the cluster. Review the release manifests and permissions before installation. If a managed add-on already exists, reconcile or repair that installation; two controllers competing over the same APIService are not a redundancy strategy.
If the backend selector or port is wrong, correct the declarative release and reconcile it using its owner. If the pod cannot start or is resource-starved, address the concrete scheduling, image or resource evidence. Do not remove readiness checks merely to make an endpoint appear healthy.
If certificates, network policy or front-proxy configuration are involved, require the responsible platform owner to review the boundary-specific change. A successful test with verification disabled is not an acceptable production endpoint. Preserve pre-change manifests and release identifiers privately; review them for credentials before attaching evidence to a ticket.
System changes
- Only the approved owner-managed add-on, node, network or RBAC change should be applied. This guide intentionally supplies no universal mutation command.
Example change record: Evidence: replacement node addresses no longer match serving-certificate SANs Owner: cluster platform team Change: approved kubelet serving-certificate repair Unchanged: metrics-server TLS verification and tenant permissions Acceptance: fresh metrics from replacement and existing nodes
Security notes
- Do not apply an unpinned latest manifest or copy cluster-admin bindings from a troubleshooting comment.
Alternatives
- For a managed cluster where control-plane configuration is unavailable, escalate the APIService and current log evidence to the provider.
Stop conditions
- Stop if there is no trustworthy rollback revision, the add-on owner is unknown, or the change would broaden tenant access without separate approval.
verification
Verify fresh samples and the actual autoscaling consumer
After the approved repair, repeat the original request with the original identity. Check actual sample timestamps and windows, not just an Available condition or one successful table. Compare a previously failing node with known-good nodes, and review affected HPA conditions separately from resource API health.
Why this step matters
A repaired transport path is useful only when the intended consumer receives timely data without losing the security properties of the original design.
What to understand
Allow the configured collection interval and a bounded observation period for samples to appear, then record two successive observations. Do not promise a fixed warm-up duration for every cluster. Compare timestamps to the observation time and check that the previously failing node has returned; a partial list can hide a residual node-specific failure.
CPU and memory measurements have different semantics, and CPU includes a measurement window. Do not compare a CPU sample directly with an instantaneous application chart and declare corruption solely because numbers differ. This guide establishes availability and freshness; it does not validate your entire capacity model.
If kubectl top now works but an HPA still cannot calculate utilization, inspect that specific HPA and the selected pods' resource requests. Custom or external metric adapters are separate APIs. Do not restart metrics-server repeatedly to fix a missing resource request or an application selector mismatch. Preserve the successful resource-API evidence while investigating that next layer.
System changes
- Reads resource metrics and autoscaler status only. It does not force scaling or generate load.
Syntax explained
--raw- Reads the specified aggregated endpoint so timestamp, window and usage remain visible.
get hpa- Provides a starting inventory of the namespace's autoscalers; describe the affected object for conditions and events.
Values stay on this page and are never sent or saved.
kubectl get apiservice v1beta1.metrics.k8s.io --request-timeout=10s
kubectl get --raw '/apis/metrics.k8s.io/v1beta1/nodes/{{nodeName}}' --request-timeout=10s
kubectl top pods -n '{{workloadNamespace}}' --request-timeout=10s
kubectl -n '{{workloadNamespace}}' get hpa --request-timeout=10sIllustrative acceptance record: APIService Available: True Previously failing node: current timestamp and non-empty usage Original namespace request: succeeds under original identity HPA: reviewed separately; missing workload resource requests remain an application configuration issue
Checkpoint: Incident acceptance
Continue whenThe original request succeeds, previously missing nodes provide current samples across successive observations, and affected autoscalers have a separately explained state.
Stop whenSamples remain stale or partial, the repair depends on insecure flags, or workload capacity risk remains unowned.
Finish line
Verification checklist
kubectl top pods -n '{{workloadNamespace}}' --request-timeout=10sThe original authorized caller receives metrics for the intended running workloads. Missing very recent pods are assessed against their age and the collection interval, not ignored indefinitely.kubectl get --raw '/apis/metrics.k8s.io/v1beta1/nodes/{{nodeName}}' --request-timeout=10sThe node sample exists, contains usage, and has a current timestamp/window on repeated bounded observations.kubectl get apiservice v1beta1.metrics.k8s.io -o yaml --request-timeout=10sAvailable is True without adding a new insecure workaround; the registered Service still belongs to the approved installation.Recovery guidance
Common problems and safe checks
Metrics API not available, with no metrics APIService registration.
Likely causeThe add-on is absent, disabled by policy, or its registration was removed during a failed reconciliation.
Confirm the current context and inspect v1beta1.metrics.k8s.io.Ask the provider or release owner whether resource metrics are intended to be installed.
ResolutionRepair or enable the supported owner-managed add-on using a compatible pinned version; do not create a second independent installation.
APIService is False or metrics-server is Running but not Ready.
Likely causeService/backend wiring, readiness or collection failure prevents a usable aggregated endpoint.
Compare Service selectors and ports with EndpointSlices and pod readiness.Read recent logs and probe events for the actual backend, including other replicas when relevant.
ResolutionRepair the evidenced configuration or collection boundary through the owning release. Do not delete readiness checks to hide missing samples.
Only some nodes have metrics, with x509 or timeout errors.
Likely causeA node-specific advertised address, serving certificate, network path or kubelet configuration differs from working nodes.
Compare the failing destination with that Node's status.addresses and kubeletEndpoint.Have the owner verify the exact source-to-destination path and serving-certificate SAN/CA.
ResolutionCorrect the specific address, trust or network discrepancy while retaining authentication and certificate validation.
kubectl top works but HPA still reports unavailable utilization.
Likely causeThe autoscaler may lack required workload resource requests or depend on a different metric API.
Inspect the affected HPA conditions and selected pod specifications.Confirm which metric type the HPA requests.
ResolutionFix the application-level request or adapter configuration in its own reviewed change; retain the now-successful resource API acceptance evidence.
Kubernetes monitoring with Prometheus and GrafanaReference
Frequently asked questions
Does Metrics API not available mean metrics-server must be reinstalled?
No. Confirm context, registration, backend readiness, current scrape errors and caller permissions first. Reinstallation can overwrite provider-managed settings while leaving a node network or certificate problem unchanged.
Can --kubelet-insecure-tls fix this permanently?
It suppresses a security check instead of repairing the trust or address mismatch. Verify the actual kubelet serving CA and SANs and use a supported certificate/address configuration. Do not present a test with verification disabled as production recovery.
Why does Prometheus work while kubectl top fails?
They may use different collection, API, authorization and storage paths. A successful Prometheus query does not demonstrate that metrics.k8s.io is registered, reachable or supplying fresh resource samples.
Is Available=True enough to close the incident?
No. Check the originally failing namespace and nodes, current sample timestamps across successive observations, the intended caller and the affected autoscaler. Discovery health can coexist with incomplete node data.
Should I install the latest Metrics Server release?
Choose the version supported by your Kubernetes release and deployment owner. The reviewed compatibility table is not a promise for future releases; check the current upstream or provider matrix before changing a pinned release.
Recovery
Rollback
Read-only diagnosis requires no rollback. Any repair must be reverted through the system that owns the changed configuration, using the exact previously approved version; this guide does not define a universal cluster rollback command.
- If the repaired add-on or network change causes regression, stop further changes and preserve current condition/log evidence. Restore only the approved release values, manifest revision or narrowly scoped network rule that changed, through its existing owner.
- For certificate or control-plane changes, use the platform's documented recovery procedure and the responsible operator. Do not restore expired certificates or overwrite unrelated provider-managed state just to match an old file.
- Repeat the original metrics request, APIService check and workload-capacity review after rollback. Record whether service is restored or the incident remains open, and maintain a named owner for degraded autoscaling.
Evidence