Metallb on Kubernetes
March 27, 2023•895 words
Metallb on Kubernetes
NodePorts limitations
My original problem was fairly simple: I wanted my kubernetes cluster to serve webpages in HTTPS. I started by running a simple Nginx server in a Pod. Kubernetes would allocate an IP address when creating the pod, but:
- The IP address was from a private network, which would make it unreachable from the internet.
- The IP address would change every time I re-created the Pod.
Kubernetes solve the second problem with the notion of Service. It is essentially an IP address that is allocated to Nginx, but will never change. When a request reaches this IP address, it is simply forwarded to one of the Servers endpoints. This solves the problem of the ever-changing IP address, but although the IP is allocated from another pool, it is still a private IP address.
┌─────────┐
│ │
┌───►│Server │
│ │ │
┌─────────┐ │ └─────────┘
│ ├───┘
│Service │
│ ├───┐ ┌─────────┐
└─────────┘ │ │ │
└───►│Server │
│ │
└─────────┘
To make the IP address reachable from the internet, one can create a special type of service called NodePort, which opens a given port on all the nodes of the Kubernetes cluster, and forwards requests to the underlying servers.
$ cat nginx/nginx-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: NodePort
selector:
name: nginx
ports:
- protocol: TCP
# The service listens to incoming requests on this port ...
port: 50000
# And forwards them to the container which listens on that port.
targetPort: 443
# On the host system, this port is bound.
nodePort: 30003
name: https
This works, but the caveat is that the port range is [30000, 32767], and nobody wants to type a url like http://foo.dodges.it:30000. We want port 443.
Another concern is that it opens a port on every node, including those who shouldn't be exposed to the internet. This is a security drawback.
Metallb
Enters Metallb, which aims to solve these problems. Installing it is fairly easy:
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.9/config/manifests/metallb-native.yaml
namespace/metallb-system created
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
secret/webhook-server-cert created
service/webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
Metallb works by sending BGP advertisements from the nodes to the local routers. But which routes to advertise must be first defined. Let's try with 192.168.5.0/24:
$ cat ipaddresspool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- 192.168.5.0/24
And to start advertising, we can simply create a BGP advertisement object. I'm not specifying any ipAddressPool object as the default is to advertise them all.
$ cat bgpadvertisement.yaml
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: example
namespace: metallb-system
With this new IP address pool deployed, let's modify my nginx service to use Metallb:
$ cat nginx/nginx-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
metallb.universe.tf/loadBalancerIPs: 192.168.5.1
spec:
type: LoadBalancer
selector:
name: nginx
ports:
- protocol: TCP
port: 443
targetPort: 443
name: https
- protocol: TCP
port: 80
targetPort: 80
name: http
After reloading, I unfortunately get an error message when connecting to Nginx:
$ curl -vv https://192.168.5.1
* Trying 192.168.5.1:443...
* connect to 192.168.5.1 port 443 failed: No route to host
* Failed to connect to 192.168.5.1 port 443 after 3033 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to 192.168.5.1 port 443 after 3033 ms: Couldn't connect to server
Let's check if I have a route for this address:
$ arp -v -e 192.168.5.1
Address HWtype HWaddress Flags Mask Iface
192.168.5.1 (incomplete) enp9s0
Entries: 6 Skipped: 5 Found: 1
Looks like nothing is advertising this IP address. swip.dodges.it should be, but running a tcpdump on it doesn't return anything:
$ sudo tcpdump -i enp9s0 "tcp port 179"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp9s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
From the ARP table above, it's clear that BGP announcements are not being made. But why? Let's try to use arping from another node in the same LAN to find who has 192.168.5.1.
When using the local area network:
$ sudo arping -I eth0 192.168.5.1
ARPING 192.168.5.1
Timeout
Timeout
So Metallb isn't advertising on the LAN. Which makes sense, since the cluster thinks that the underlying network where the nodes live are a Wireguard mesh. Let's try over Wireguard network:
> $ sudo arping -I wg15 192.168.5.1
arping: libnet_init(LIBNET_LINK, wg15): unknown physical layer type 0xfffe
Oh no… Wireguard doesn't transport ARP requests (well it's a layer 3 VPN so that seems working as intended). Fortunately he documentation seems to allow us to define the subnet over which the BGP advertisement are sent.
Let's look at the logs of speaker in the metallb-system namespace:
$ k logs -n metallb-system speaker-8lr76 | tail
{"caller":"native.go:90","error":"dial \"192.168.1.1:179\": getsockopt: connection refused","level":"error","localASN":64512,"msg":"failed to connect to peer","op":"connect","peer":"192.168.1.1:179","peerASN":64512,"ts":"2023-03-27T20:21:35Z"}
{"caller":"native.go:90","error":"dial \"192.168.1.1:179\": getsockopt: connection refused","level":"error","localASN":64512,"msg":"failed to connect to peer","op":"connect","peer":"192.168.1.1:179","peerASN":64512,"ts":"2023-03-27T20:21:39Z"}
It looks like BGP advertisement between the node and the router isn't working. A quick glance at my router's user guide shows that there is no support for BGP coming from the LAN.
So next step will be to reconfigure Metallb using L2Advertisement.