Containers on Retail Edge: Bandwidth, Boot-Time, and Burnt Weekends

When my team got the directive to migrate all our retail store gateway workloads to containers, I initially saw it as a straightforward win. Reduced snowflake instances, better dependency management, faster deployments. What I hadn't fully appreciated was the brutal reality of intermittent network connectivity on 3,000 geographically dispersed sites, each with a consumer-grade 50Mbps uplink, and the profound impact of bloated container images. The biggest gotcha that bit us hard in week three was the 7GB base image download that regularly stalled and timed out, leaving 50 stores offline for hours.

The Initial Mandate: All-In on Containers

Our fleet of 3,000 edge gateways, deployed across various retail franchises, was a mix of custom Python services, some legacy Java apps, and a few C++ binaries. They handled inventory synchronization, point-of-sale data aggregation, and local AI inferencing for security cameras. The plan was to containerize every single service using Docker, manage them with a custom orchestrator running on each gateway, and deploy updates via a centralized CI/CD pipeline. Our chosen gateway hardware was a modest Intel NUC with 8GB RAM, 128GB SSD.

The Network Nightmare and Image Bloat

Our central artifact repository was hosted on AWS S3, fronted by CloudFront. We assumed this would be fast enough. It wasn't. The average store internet connection fluctuated wildly. Our initial Docker images, built naively, were massive. A python:3.8-slim-buster base image plus all our pip install dependencies for a single service ballooned to 800MB. Multiply that by 10 services per gateway, plus a core orchestration image, and we were looking at over 7GB of initial download. Downloads frequently failed, resumed, re-failed, and eventually timed out.

Systemd vs. Containerd: A War Every Tuesday

The debate raged for months: should systemd launch our docker run commands, or should we use containerd directly with crictl and manage the lifecycle with a thin wrapper? I argued for systemd for its battle-hardened process supervision, logging integration, and dependency management. Our platform lead pushed for containerd directly, citing lower resource overhead and future Kubernetes compatibility. Ultimately, we settled on systemd unit files launching docker run --restart=always commands, primarily because our field engineers were already familiar with journalctl and systemctl for debugging. This pragmatic choice saved us a ton of support calls.

[Unit]
Description=My Retail Edge Service
After=network-online.target docker.socket
Wants=network-online.target

[Service]
ExecStartPre=/usr/bin/docker pull myrepository/retail-service:latest
ExecStart=/usr/bin/docker run --rm --name retail-service -p 8080:8080 myrepository/retail-service:latest
ExecStop=/usr/bin/docker stop retail-service
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Image Layer Optimization and Alpine Linux

We spent two months brutally optimizing our images. Multi-stage builds became mandatory. We switched from python:3.8-slim-buster to python:3.8-alpine where possible, immediately cutting image sizes by 50-70%. We meticulously removed build-time dependencies from final images. For Python, pip install --no-cache-dir and cleaning up APT package caches aggressively became standard practice. The 800MB image for a single Python service shrank to 150MB.

Rollback Strategies and Edge Updates

One critical lesson was the need for robust rollback capabilities. Our initial latest tag strategy was disastrous. If a bad image was pushed, all 3,000 gateways would pull it, break, and need manual intervention. We quickly moved to immutable version tags (e.g., myrepository/retail-service:1.2.3). Our custom orchestrator included a simple rollback logic: if a container failed to start 3 times within 5 minutes, it would attempt to roll back to the previously deployed, known-good image version. This saved countless truck rolls.

What I'd Do Differently

If I were to start this project today, I would make image size auditing an early and explicit gate in our CI pipeline. We added it eventually with tools like dive and custom scripts that failed builds if image size exceeded arbitrary thresholds, but it was reactive. I'd also push harder for a localized, store-level image caching proxy from day one to minimize repeated downloads and mitigate slow connections.

Honest Limitations

Despite all our efforts, cold boot times on a completely new gateway still take about 45 minutes to pull all images and spin up services. This is acceptable for initial deployment but highlights the constant tension between feature richness and operational overhead. We also still struggle with inconsistent network performance in about 5% of our locations, requiring manual interventions or satellite uplinks in some extreme cases.