So TL;DR Haproxy has a builtin prometheus-exporter now: https://github.com/haproxy/haproxy/tree/master/contrib/prometheus-exporter
In on-prem Kubernetes clusters there is no IAAS loadbalancer options, so a common setup is to run haproxy and keepalived as static pods. This adds some complexity upfront but runs rock solid in production as the control plane VIP is always available and HaProxy automatically removes control plane nodes from rotation.
I’ve been maintaining a Kubernetes cluster at home for dev-work and one of the more recent things I did was spool up some monitoring IE Grafana and Prometheus. By the way the prometheus-operator ( https://github.com/coreos/prometheus-operator ) from the former coreos folks is works perfectly for this. So, I’ve been slowly going through different cluster components and adding scrapers and exporters. And, one of the recent targets has been my cluster control plane load-balancer and vip (haproxy+keepalived) so I’ve been looking into exporters. Coincidentally, with the release of haproxy 2.0.0 there is an embedded exporter. Anyways, here’s the container I’ve been working on: https://hub.docker.com/r/whisperos/haproxy
Below is an example static manifest that lives on master nodes:
---
apiVersion: v1
kind: Pod
metadata:
name: ha-lb
namespace: kube-system
labels:
k8s-app: ha-control-plane
spec:
hostNetwork: true
volumes:
- hostPath:
path: /etc/haproxy
name: etc-haproxy
- hostPath:
path: /etc/keepalived
name: etc-keepalived
- name: var-iptables
hostPath:
path: /var/lib/iptables
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
containers:
- name: haproxy
image: docker.io/whisperos/haproxy:2.0.0
command:
- /usr/sbin/haproxy
args:
- -db
- -f
- /etc/haproxy/haproxy.cfg
securityContext:
privileged: true
volumeMounts:
- mountPath: /etc/haproxy
name: etc-haproxy
readOnly: true
- name: keepalived
image: docker.io/whisperos/keepalived:2.0.16
command:
- /usr/sbin/keepalived
args:
- --no-syslog
- --log-console
- --dont-fork
- --use-file=/etc/keepalived/keepalived.conf
securityContext:
privileged: true
volumeMounts:
- mountPath: /etc/keepalived
name: etc-keepalived
readOnly: true
- mountPath: /var/lib/iptables
name: var-iptables
readOnly: false
- mountPath: /run/xtables.lock
name: xtables-lock
readOnly: false
Then my haproxy config looks like this:
global
maxconn 200000
nbthread 8
tune.ssl.default-dh-param 2048
defaults
log global
retries 3
maxconn 200000
timeout connect 5s
timeout client 50s
timeout server 50s
frontend exporter
mode http
bind *:8404
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /stats
stats refresh 10s
frontend kubernetes_lb
bind 10.12.4.3:6443
default_backend kubernetes_back
mode tcp
backend kubernetes_back
balance leastconn
mode tcp
server titan01.iag.d3fy.net 10.12.4.114:6443 check port 6443
server titan02.iag.d3fy.net 10.12.4.175:6443 check port 6443
server titan03.iag.d3fy.net 10.12.4.171:6443 check port 6443
frontend etcd_lb
bind 10.12.4.3:2379
default_backend etcd_back
mode tcp
backend etcd_back
balance leastconn
mode tcp
server titan01.iag.d3fy.net 10.12.4.114:2379 check port 2379
server titan02.iag.d3fy.net 10.12.4.175:2379 check port 2379
server titan03.iag.d3fy.net 10.12.4.171:2379 check port 2379
# vi:syntax=haproxy
Now I’ll configure the prometheus-operator:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: prom-haproxy-ha
release: mon
name: prom-haproxy-ha
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 15s
port: http-metrics
jobLabel: jobLabel
namespaceSelector:
matchNames:
- kube-system
selector:
matchLabels:
app: prom-ha-control-plane
And don’t forget the headless service:
apiVersion: v1
kind: Service
metadata:
labels:
app: prom-ha-control-plane
jobLabel: ha-control-plane
name: prom-ha-control-plane
namespace: kube-system
spec:
clusterIP: None
ports:
- name: http-metrics
port: 8404
protocol: TCP
targetPort: 8404
selector:
k8s-app: ha-control-plane
type: ClusterIP