Kubernetes for Public Services: SLOs, Accessibility, and the Compliance Drag

When I joined the team tasked with migrating a suite of public-facing government services to Kubernetes, I expected technical challenges. What I didn't fully anticipate was the sheer operational burden of compliance and the non-negotiable demand for true accessibility, which quickly became our most critical — and hardest to measure — Service Level Objective.

The Initial Promise: Agility and Scale

Our mandate was clear: reduce infrastructure costs, increase deployment velocity, and improve the resilience of citizens' access to essential services. Kubernetes, specifically an on-prem vanilla distribution (Kubespray 2.19.1 on RHEL 8), seemed like the natural fit. We started with a small cluster of 9 worker nodes, each with 32 vCPUs and 128GB RAM, handling about 15 core applications. The initial dev/test environments were spun up in about 4 weeks, a significant improvement over the traditional 6-month provisioning cycles we'd seen before.

The Accessibility SLO that Bit Us

Early in week three, after our first internal PoC, we got a harsh reality check. While performance SLOs (latency < 100ms, availability > 99.9%) were straightforward to monitor, the accessibility requirements for public services (WCAG 2.1 AA) proved far more complex. It wasn't just about Lighthouse scores; it was about screen reader compatibility, keyboard navigation, and cognitive load. We discovered that a seemingly minor UI change in one service's Deployment could inadvertently break accessibility for a segment of users, leading to formal complaints. This became a paramount SLO, driving significant changes in our CI/CD pipelines.

Establishing Blue/Green for Critical Services

For high-traffic citizen-facing applications, like our tax filing portal processing 50,000 requests per minute during peak season, downtime was a non-starter. We implemented blue/green deployments using NGINX Ingress and Kubernetes Services with label selectors. This allowed us to shift traffic gradually, rolling back instantly if pre-defined health checks failed. This reduced deployment risks significantly, dropping incident rates from 0.7 per deployment to 0.05.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: citizen-portal-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
  - host: citizen.gov.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: citizen-portal-service-blue
            port:
              number: 80

The Compliance and Audit Drag

This was the biggest elephant in the room. For every change, every new manifest, every cluster configuration, we faced rigorous audit requirements: NIST 800-53, FedRAMP, GDPR (for some data types). This meant extensive documentation, change management board approvals, and security scans at every stage. We used tools like Falco for runtime security, Open Policy Agent (OPA) Gatekeeper for admission control, and Anchore for image scanning. These added significant overhead, increasing our release cycle time by roughly 30% compared to a purely commercial environment. For instance, just getting an OPA policy approved and deployed could take up to 2 weeks due to required security reviews.

Optimizing for Multi-Tenancy and Cost Allocation

With multiple agencies and development teams sharing the clusters, effective multi-tenancy was crucial. We leveraged Kubernetes namespaces, ResourceQuotas, and LimitRanges extensively to ensure fair resource distribution. Each agency had its own namespace(s), and we integrated custom metrics with Prometheus and Grafana to track resource consumption per namespace. This data was then used for internal chargebacks, helping agencies understand and control their cloud-native spend.

What I'd Do Differently

If I were to start over, I would embed compliance and accessibility specialists into the team from day one, not bring them in for reviews post-development. Their early involvement would have saved us countless rework hours and prevented several near-misses during crucial public launches. I'd also push harder for standardized service meshes (like Istio 1.15) earlier to simplify traffic management.

Lingering Challenges

While we achieved significant gains, one persistent challenge remains: balancing developer autonomy with strict government security protocols. Developers want to deploy quickly; security needs exhaustive vetting. Automating security policy creation and validation, particularly for ephemeral environments, is still an active area of research for us.