K3s GitOps Cluster

A three-node Kubernetes cluster running on repurposed hardware, managed entirely through git. No kubectl apply in production — every change goes through a pull request, ArgoCD detects the diff, and the cluster converges.

Architecture

┌─────────────────────────────────────────────┐
│               K3s Cluster                           │
│                                                     │
│  control-plane (1 node)                             │
│  ├── kube-apiserver                                │
│  ├── ArgoCD          ← watches git repo            │
│  └── cert-manager    ← ACME / Let's Encrypt        │
│                                                     │
│  workers (2 nodes)                                  │
│  ├── app namespace   (services + ingress)           │
│  ├── monitoring      (Prometheus + Grafana)         │
│  └── traefik         (ingress controller)           │
└─────────────────────────────────────────────┘

GitOps workflow

The cluster state lives in a cluster/ directory in git. ArgoCD watches it and reconciles every 3 minutes — or immediately on a webhook push.

cluster/
├── apps/
│   ├── api/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ingress.yaml
│   └── frontend/
│       └── ...
├── monitoring/
│   └── kube-prometheus-stack/   (Helm chart values)
└── argocd/
    └── applications/            (ArgoCD Application CRs)

Deploying a new version is a one-line PR: bump the image tag in deployment.yaml, merge, and ArgoCD rolls it out with a zero-downtime RollingUpdate.

Key Kubernetes concepts used

Concept	Where
Deployments + ReplicaSets	all application workloads
Services (ClusterIP / NodePort)	internal routing
Ingress + IngressClass	Traefik routes traffic to services
ConfigMaps	non-secret app configuration
Secrets (sealed with kubeseal)	credentials, never plaintext in git
PersistentVolumeClaims	Prometheus TSDB, Grafana state
HorizontalPodAutoscaler	scales the API on CPU > 70 %
RBAC	ArgoCD has read/write, CI has read-only
Namespaces	hard isolation between app and monitoring
NetworkPolicies	monitoring can scrape app, app cannot reach monitoring

Sealed Secrets

Kubernetes Secrets are base64, not encrypted. I use kubeseal to encrypt them with the cluster's public key before committing:

kubectl create secret generic db-creds \
  --from-literal=password="$DB_PASS" \
  --dry-run=client -o yaml \
| kubeseal --format yaml > cluster/apps/api/db-creds-sealed.yaml

The sealed file is safe to commit. Only the cluster can decrypt it.

Observability

The kube-prometheus-stack Helm chart deploys the full monitoring suite in one values file: Prometheus, Grafana, Alertmanager, and exporters for nodes, etcd, and the kubelet. Custom alerts fire on:

pod restarts > 3 in 10 minutes
node memory > 85 %
ArgoCD sync failures

What I learned

Declarative beats imperative: once I stopped running kubectl commands and started editing YAML, the cluster became reproducible — I can rebuild it from git clone.
RBAC is not optional: giving ArgoCD cluster-admin was the fast path and also the wrong one. Scoping roles to namespaces surfaced two misconfigurations I didn't know existed.
Requests and limits matter: without resource constraints, one noisy pod starved the monitoring namespace during a load test. Setting them on every workload made scheduling predictable again.
Kubeseal solved the secrets problem: I was stuck on how to store secrets in git without exposing them. Sealed Secrets made the answer obvious and auditable.