MLOps at the Edge: 300 IoT Devices and Our Over-The-Air Ordeal

When I joined a startup focused on industrial automation, part of my role as a DevOps engineer expanded rapidly into MLOps. We were tasked with deploying computer vision models to approximately 300 edge devices scattered across various manufacturing plants. The initial pitch was simple: deploy models, run inference locally, and reduce network bandwidth. What actually happened was a nine-month saga of unexpected challenges, with over-the-air (OTA) update failures becoming the bane of my existence for weeks on end.

The Edge Devices and Their Constraints

Our fleet consisted of custom-built embedded Linux boxes, each with 2GB RAM, a 16GB eMMC, and an ARM Cortex-A53 quad-core CPU. They ran a stripped-down Yocto Linux build. Connectivity was often via unstable 4G LTE or plant Wi-Fi, with latency sometimes hitting 300ms. The key constraint was storage: we couldn't just dump all model versions on the device. Each model binary for our object detection task ranged from 30MB to 70MB, depending on the quantization strategy.

Model Packaging: The First Optimization Dive

Initially, data scientists trained models in PyTorch. My first task was to figure out how to get these models onto our devices efficiently. We experimented with several formats. TensorFlow Lite (TFLite) proved to be the most viable for our ARM architecture, offering significant size reductions and inference speedups on the limited computational resources. We adopted a workflow to export PyTorch models to ONNX, then convert ONNX to TFLite.

import torch
import onnx

model = MyPyTorchModel()
model.load_state_dict(torch.load('model.pth'))
model.eval()

dummy_input = torch.randn(1, 3, 224, 224, requires_grad=True)
torch.onnx.export(model, dummy_input, "model.onnx",
                  opset_version=11,
                  do_constant_folding=True,
                  input_names = ['input'],
                  output_names = ['output'])

This conversion reduced a typical FP32 PyTorch model from 120MB to a quantized TFLite model of 35MB. We achieved about a 2.5x speedup in inference time on the device, dropping from 250ms to ~100ms per frame.

The OTA Update Fiasco

The architecture for model deployment involved a custom agent on each device continually polling our central deployment service for new model versions. When a new version was detected, the agent would download it over HTTP/S, verify its integrity (SHA256 checksum), and then activate it. This worked fine for the first 50 devices in a pilot. When we scaled to 300, OTA updates became a nightmare.

The problem was network instability. A 35MB download over a flaky 4G connection frequently timed out or corrupted. Our agent wasn't robust enough to handle partial downloads gracefully. Devices would get stuck in a download loop, consuming bandwidth and failing to update. I spent three weeks debugging these issues, often SSHing into individual devices to manually restart the agent or clear corrupted files.

Solving OTA: Resumable Downloads and A/B Updates

We eventually rewrote the agent's update logic to support resumable downloads using HTTP Range headers. This drastically improved reliability. We also implemented an A/B update scheme: new models were downloaded to a separate partition. If activation failed, the device would roll back to the previously working version. This reduced our update failure rate from 40% to less than 2% across the fleet.

Drift Detection in a Data-Scarce Environment

One of the biggest challenges was monitoring model performance and concept drift without backhauling large amounts of raw inference data. Shipping 1080p video frames back to AWS for every inference was a non-starter due to cost and bandwidth. Instead, we adopted a strategy of sending aggregated inference metadata: bounding box counts and classes, confidence score histograms, inference latency, and environmental metadata.

{
  "timestamp": "2023-10-27T10:30:00Z",
  "device_id": "iot-device-067",
  "model_version": "v1.2.1-quant",
  "metrics": {
    "object_counts": {"person": 5, "car": 2},
    "avg_confidence": {"person": 0.88, "car": 0.76},
    "inference_latency_ms": 105,
    "device_temp_c": 45.2
  }
}

This light-weight telemetry allowed us to build dashboards and trigger alerts for significant deviations, indicating potential data drift or model degradation on specific devices, without incurring massive data transfer costs.

What I'd Do Differently

If I were to start this project again, I would prioritize a robust OTA update mechanism from day one. Instead of building our own agent from scratch, I'd seriously evaluate an existing solution like Mender or RAUC for handling embedded Linux updates. Their battle-tested reliability and features like atomic A/B updates would have saved us months of development and debugging.

Honest Limitations

Despite our progress, true root cause analysis for model errors on the edge remains challenging. We still can't easily retrieve specific inference images from devices when an anomaly is detected without manual intervention. This means figuring out why a model started misclassifying requires a detective's intuition, and often, a site visit to collect sample data.