Fix kubelet port 10250 errors on Ubuntu

What port 10250 is

Port 10250 is the kubelet API, and every error that names it is one of two opposite problems. Either something already holds the port, so kubeadm init refuses to run. Or nothing can reach the port, so kubectl logs and kubectl exec fail against a node that otherwise looks perfectly healthy.

The kubelet is the agent Kubernetes runs on every node. It starts containers and reports their state back to the control plane. It also listens on TCP 10250 and serves an HTTPS API that the control plane calls. The API server opens a connection to that port when you run kubectl logs, kubectl exec, kubectl attach or kubectl port-forward. metrics-server scrapes /metrics/resource on the same port, which is what makes kubectl top node work.

That API is authenticated. kubeadm turns anonymous access off and points the kubelet at the cluster CA (certificate authority), so a request with no credentials is answered with Unauthorized rather than a shell inside one of your containers. Remember that detail, because it is also the quickest way to prove the port is reachable. If ports in general are new to you, what a port actually is on Linux covers the model this guide assumes.

Both failure modes follow from one requirement. Port 10250 must be free before the kubelet starts, and reachable from the control plane once it is running.

Which of the two problems do you have

Run these on the node in question. Every command below is one you run yourself, on your own server.

sudo ss -lntp | grep 10250
sudo systemctl status kubelet --no-pager

ss -lntp lists listening TCP sockets with the process behind each one. -l means listening, -n keeps ports numeric, -t limits it to TCP, -p shows the owning process. That last flag needs root, or the process column comes back empty and you learn nothing.

A line ending in users:(("kubelet",pid=1043,fd=23)) means the kubelet is running and holding the port. If you expected the port to be free, that is your answer. If ss prints nothing at all and the control plane still cannot reach this node, then no firewall is involved yet, because nothing is serving the port in the first place. Find out why the kubelet is down before you touch any rules.

systemctl status kubelet gives the other half of the picture. active (running) with a start time from minutes ago is normal. A kubelet restarting every few seconds before you have run kubeadm init or kubeadm join is also normal: the packaged unit starts at install time, finds no config, and exits. Upstream documents that crash loop as expected behaviour while the kubelet waits for kubeadm to tell it what to do. If systemd restart behaviour is unfamiliar, how systemd service types and restart policies work is the background for this section.

Why is port 10250 already in use when kubeadm init runs

kubeadm init runs preflight checks before it writes anything to disk. One of those checks tries to bind each port the control plane needs, and it stops with an error naming port 10250 when that bind fails. This is not a bug. It is kubeadm refusing to build a second cluster on top of the remains of a first one.

Four things cause it in practice:

• An earlier kubeadm init or kubeadm join that failed part way through. The kubelet already got a config, so it is running and holding the port.
• A kubeadm reset that was started but not finished. Reset stops the kubelet, it does not disable the unit, so the next reboot brings the listener back.
• k3s or another Kubernetes distribution installed on the same server. k3s embeds a kubelet, and that kubelet binds 10250 too.
• The kubelet package pulled in by apt and started by its own systemd unit, on a box where you have not run kubeadm yet.

Work out which one it is before changing anything:

sudo ss -lntp 'sport = :10250'
systemctl list-units --type=service --state=running | grep -Ei 'kubelet|k3s|k0s'

If the listener belongs to k3s, stop and decide which cluster you actually want. k3s and kubeadm cannot share one server, because they claim the same ports and the same CNI (container network interface) directory. The k3s installer leaves an uninstall script at /usr/local/bin/k3s-uninstall.sh on a server node, and k3s-agent-uninstall.sh on an agent node.

Why killing the kubelet does not free the port

sudo pkill kubelet frees port 10250 for about ten seconds. The packaged unit sets a restart policy, so systemd starts a fresh kubelet, and it binds the same port again. You can see the policy for yourself:

systemctl show kubelet -p Restart -p RestartSec
sudo systemctl stop kubelet
sudo ss -lntp | grep 10250

Restart=always with RestartSec=10 is what the unit ships with, which is exactly why kill looks like it worked and then does not. systemctl stop is the correct way to free the port, because systemd stops restarting a unit you told it to stop.

A free port is still not enough on a node that carries half a cluster. /var/lib/kubelet/config.yaml, the certificates under /etc/kubernetes/pki and any static pod manifests in /etc/kubernetes/manifests are all still there. Later preflight checks trip over those files, and forcing past the checks gives you a cluster whose certificates do not match its config. Reset the node properly instead.

Reset the node cleanly

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d
rm -rf $HOME/.kube
sudo systemctl stop kubelet
sudo ss -lntp | grep -E '10250|6443|2379'

-f skips the confirmation prompt. Reset makes a best effort revert of what init or join did. It removes the local files and configuration, removes the local etcd member on a control plane node, cleans the certificates in /etc/kubernetes/pki, and removes the kubelet configuration and manifests.

The documentation is explicit about what reset leaves behind, and each item catches people out. It does not clean /etc/cni/net.d, so the old CNI plugin config survives and your new cluster reads it. It does not clean any iptables, nftables or IPVS rules that kube-proxy applied to the host. It does not touch $HOME/.kube, so kubectl keeps talking to a cluster that no longer exists and returns certificate errors that look like a new problem.

The leftover packet rules are the awkward part. Flushing the tables by hand also wipes the rules ufw installed, because ufw on Ubuntu writes through the same backend, which leaves the server unfiltered until you run sudo ufw reload. On a node you are rebuilding anyway, reboot after the reset. A reboot clears the runtime rules kube-proxy added and costs less time than untangling a half flushed ruleset. Why iptables rules and nftables rules show up in each other's output explains what is going on underneath.

The last ss command should print nothing. No listener on 10250, 6443 or 2379 means the node is ready for a fresh kubeadm init.

Why kubectl logs and kubectl exec time out on port 10250

This is the opposite complaint, and it does not announce itself as a port problem. The cluster comes up. Nodes are Ready. Pods run. Then one command fails:

Error from server: Get "https://10.0.0.12:10250/containerLogs/default/web-0/web": dial tcp 10.0.0.12:10250: i/o timeout

Read that message from the end. The API server tried to open a TCP connection to the node on port 10250 and got no answer. i/o timeout means the packets were dropped in silence, so something is filtering them: the host firewall on the node, or your provider's separate network firewall in the control panel. connect: connection refused in the same position means the opposite. The packet arrived and nothing was listening, so the kubelet is down. That is the same pair of causes described in connection refused versus connection timed out, read here on a different port.

Nodes stay Ready through all of this because node status travels the other way. The kubelet connects out to the API server on port 6443 and posts its own heartbeat, and that needs nothing inbound on 10250. So a blocked 10250 gives you a cluster that schedules pods normally and fails only on logs, exec, port-forward and metrics.

kubectl top node answering error: Metrics API not available is the same fault seen through metrics-server, whose log names the node and the port:

unable to fully scrape metrics from node worker-1: unable to fetch metrics from node worker-1: Get "https://10.0.0.12:10250/metrics/resource": dial tcp 10.0.0.12:10250: i/o timeout

Test the path before you edit any firewall rule

Run this from a control plane node, against the worker's address:

nc -zv 10.0.0.12 10250
curl -sk -o /dev/null -w '%{http_code}\n' https://10.0.0.12:10250/healthz

nc -z opens a connection, closes it, and prints succeeded! when the port accepts it. The curl line is the better test, because it proves the kubelet is serving rather than proving a port is merely open. It prints 401. That is the healthy result: the TLS (transport layer security) handshake completed, then the kubelet rejected an unauthenticated request, which is exactly what it should do. -k skips certificate checking, which is fine because you are testing the path, not the trust chain.

A long pause ending in a timeout means packets are being dropped. curl: (7) Failed to connect returned instantly means the port is closed on a reachable host. Test from the control plane node, not from your laptop, because the control plane is the only machine whose access matters here.

Which ports a control plane and a worker each need

These are the inbound ports upstream lists. On a control plane node, TCP 6443 for the API server, open to everything that runs kubectl. TCP 2379 to 2380 for the etcd client and peer API, used by the API server and etcd itself. TCP 10250 for the kubelet API, used by the node itself and the control plane. TCP 10259 for kube-scheduler and TCP 10257 for kube-controller-manager, both used only by the node itself.

On a worker node, TCP 10250 for the kubelet API, used by the node itself and the control plane. TCP 10256 for kube-proxy, used by the node itself and by load balancers doing health checks. TCP and UDP 30000 to 32767 for NodePort services, which is the default range and is reachable by whoever needs those services.

Your CNI plugin adds its own ports on top of that list, and they are not in it. Flannel and Calico in VXLAN mode need UDP 4789 between nodes. Calico with BGP needs TCP 179. Check your plugin's documentation and open those between nodes, or pods on different nodes will not talk to each other while every port in this section is open.

Open 10250 without exposing it to the internet

The kubelet API can start a process inside any container on that node. Treat an open 10250 as root access to the node, and restrict it by source address. Never allow it from anywhere.

sudo ufw allow from 10.0.0.0/24 to any port 10250 proto tcp comment 'kubelet API'
sudo ufw allow from 10.0.0.0/24 to any port 10256 proto tcp comment 'kube-proxy'
sudo ufw status numbered

Replace 10.0.0.0/24 with the network your nodes share. ufw status numbered lists the active rules with an index, so you can delete a wrong one with sudo ufw delete <number>. The ufw basics for a VPS covers the ordering rules that decide which of your entries actually applies.

One ufw setting breaks Kubernetes on its own. Pod traffic crossing the node is forwarded, not delivered locally, and ufw drops forwarded packets by default. Set DEFAULT_FORWARD_POLICY="ACCEPT" in /etc/default/ufw and run sudo ufw reload. Without it, port 10250 can be wide open and pod to pod traffic between nodes still fails.

Check your provider's firewall as well. Most VPS panels have a network level firewall that sits in front of the server and is invisible to ufw status. A rule you added on the node changes nothing if the packet never arrives.

When the port is reachable and the request still fails

Some 10250 failures come back instantly instead of hanging, which tells you the connection succeeded and the request was rejected. x509: certificate signed by unknown authority in the metrics-server log means the kubelet is serving a self signed certificate that the scraper does not trust. The usual answers are to enable kubelet serving certificate rotation so the cluster CA signs it and then approve the certificate signing request, or to accept the risk on a lab cluster and run metrics-server with --kubelet-insecure-tls.

A message containing Forbidden together with nodes/proxy or nodes/metrics is an RBAC (role based access control) failure. The caller reached the kubelet, the kubelet asked the API server whether that identity may use the subresource, and the answer was no. Fix the ClusterRole of the caller. No firewall change will help, because nothing was blocked.

If you only want one small cluster

If you are hitting these errors while standing up kubeadm on a single VPS for the first time, consider whether you need kubeadm at all. A single node k3s cluster on a VPS gives you a working Kubernetes API in one command, with the kubelet, kube-proxy and a CNI already wired together. Port 10250 still exists there and the same rules apply to it, but you no longer have to assemble the control plane yourself.

FAQ

What is port 10250 used for in Kubernetes?

It is the kubelet's authenticated HTTPS API on every node, control plane and worker alike. The API server connects to it for kubectl logs, kubectl exec, kubectl attach and kubectl port-forward, and metrics-server scrapes /metrics/resource on it to supply kubectl top. Node status does not use it, because the kubelet posts its heartbeat outbound to the API server on port 6443. That is why a blocked 10250 leaves nodes showing Ready while logs and exec fail.

How do I find what is listening on port 10250?

Run sudo ss -lntp | grep 10250 on the node. The users:((...)) field at the end of the line names the process and its PID. sudo matters, because without root the process column is blank. If the owner is the kubelet, sudo systemctl status kubelet --no-pager tells you whether it is a healthy kubelet or one restarting in a loop. If the owner is k3s, you have two Kubernetes distributions installed on one server and need to remove one.

Do I have to open port 10250 in my firewall?

Yes, between your nodes. The control plane must reach 10250 on every node, including on itself, or logs, exec, port-forward and metrics all fail. Restrict it by source to the network your nodes share, for example sudo ufw allow from 10.0.0.0/24 to any port 10250 proto tcp. Do not open it to the internet: anything that can authenticate to that port can run a process in any container on the node.

Why does kubectl logs fail for pods on one node only?

Because the block is per node, and the API server connects to the specific node that hosts the pod. Read the error text: it contains the node IP address it tried to reach. Then run nc -zv <node-ip> 10250 from a control plane node. A timeout points at the firewall on that node or at your provider's network firewall. connection refused points at a kubelet that is not running there, so check systemctl status kubelet on that node instead.