From Zabbix to Datadog: A Real Migration Playbook

In a large-scale financial services environment, I led the migration of a ~12-year-old Zabbix deployment monitoring roughly 1,400 hosts and 60+ business services onto Datadog. This is what the playbook actually looked like — including the alerting gap that bit us in week three.

Why we left Zabbix

Zabbix wasn't broken. It was, in many ways, doing exactly what it was deployed to do in 2013: poll SNMP, scrape agents, fire triggers, page humans. The reason we migrated wasn't that Zabbix was bad — it was that the operational surface around it had outgrown the tool. We were running three Zabbix proxies, a partitioned MySQL backend that needed a dedicated DBA every quarter, and a custom Grafana layer bolted on top because the native UI couldn't keep up with how product teams wanted to slice data. APM was missing entirely; we were correlating latency to infra by eye.

Leadership wanted one pane of glass for infra, APM, logs, and synthetics. Datadog won the bake-off, mostly because the agent's autodiscovery on Kubernetes meant we'd stop hand-rolling host templates for every new service. For deeper context on why tool consolidation matters beyond just licensing, see [INTERNAL LINK: The Hidden Cost of Observability Tool Sprawl in Enterprise DevOps].

[IMAGE: Before/after architecture diagram — left side shows Zabbix server + 3 proxies + MySQL + Grafana feeding PagerDuty; right side shows Datadog Agent on every host/pod + API integrations + native dashboards + monitors feeding PagerDuty.]

Thresholds don't map 1:1 — and that's where you get hurt

The biggest misconception going in was that we could spreadsheet the existing Zabbix triggers and "translate" them. We tried. It fell apart fast.

Zabbix trigger expressions are evaluated against the latest value (or an aggregate function like avg(), last(), min()) per item, and they fire when the boolean expression flips to true. A typical trigger looked like:

{host:system.cpu.util[,user].avg(5m)}>85 and {host:system.cpu.util[,user].min(5m)}>70

The naive Datadog equivalent — avg(last_5m):avg:system.cpu.user{host:foo} > 85 — looks identical but isn't. Datadog evaluates on a rolling window and applies its own no-data and recovery logic. Concretely: Zabbix flapped less because of that secondary min() > 70 guard; Datadog's default recovery threshold equals the alert threshold unless you set a separate recovery_threshold. We got hammered with flapping during week one until we explicitly set hysteresis on every CPU/memory monitor.

Multi-condition Zabbix triggers (the and/or chains we'd accumulated over a decade) also don't translate cleanly. Datadog wants you to express that as a composite monitor, and composite monitors have their own evaluation cadence — they can lag the underlying monitors by 30–60s. For high-frequency triggers that mattered.

The dashboards that took the longest

Anything plotting a single host metric: trivial. A dashboard rebuilt itself in an afternoon. The dashboards that consumed weeks were the ones built around business services — the "payments platform" view that aggregated 40+ hosts, 8 application tiers, and a handful of external dependencies into one health score. Zabbix did this via service trees with weighted SLAs. Datadog has Service Level Objectives, but the data model is different enough that we ended up redesigning the underlying service map rather than porting the tree. Plan for that. If your org has heavy use of Zabbix services or IT services, budget 2–3x what you think.

The gap that caused a missed incident

Three weeks in, during the dual-run period, a Postgres replica on one of our reporting clusters fell behind by 18 minutes before anyone noticed. Zabbix had a trigger on pgsql.replication.lag via a custom user parameter; the Datadog Postgres integration ships a postgresql.replication_delay metric out of the box. We'd ticked the box. The problem: the Datadog check was scoped to role:primary in the agent config we'd copy-pasted across the fleet. On the replica it was tagged role:replica, so the monitor's query (avg:postgresql.replication_delay{role:primary}) returned no data — and our no-data alerting was set to "notify after 10 minutes of no data" only on hosts, not on monitors with empty result sets.

What caught it: a manual reconciliation pass where we diff'd the active Zabbix trigger list against the active Datadog monitor list every Friday during the cutover. The diff flagged 11 triggers with no Datadog equivalent firing in the last 7 days despite being noisy in Zabbix. Replication lag was one of them.

The fix was a two-line change to the agent config plus enabling notify_no_data: true with no_data_timeframe: 15 on every infra check. Cheap fix. Could have been a missed regulatory reporting SLA.

[IMAGE: Side-by-side table — left column Zabbix trigger expressions including the CPU example and the replication lag user parameter; right column the equivalent Datadog monitor query plus the recovery threshold and notify_no_data settings that had to be added.]

Cost and timeline, honestly

End-to-end took ~14 weeks across roughly 1,400 hosts, with a 6-week dual-run period where both systems paged. Zabbix self-hosted cost was mostly hidden — DBA time, the Grafana stack, the two engineers who knew the templating language well enough to be on call for the monitoring system itself. Datadog's SaaS pricing is transparent but front-loaded; the per-host plus per-custom-metric plus APM-host model adds up faster than the sales quote suggests, and custom metrics in particular need a governance policy on day one or your bill drifts 20–30% in a quarter. The Datadog [EXTERNAL LINK: Datadog official pricing and billing docs] spell out the dimensions you'll be billed on — read them before signing, not after.

What I'd do differently

Three things. First, I'd skip the "translate triggers" phase entirely and rebuild the alert catalog from the SLOs backwards — most of those decade-old Zabbix triggers shouldn't have existed in the new system at all. Second, I'd run the Friday diff from day one, not week three. Third, I'd put a hard cap on custom metrics in the agent config from the first deploy; retrofitting governance after teams have already shipped 8,000 custom metrics is a political problem, not a technical one. For more on how monitoring blind spots compound into incidents, see [INTERNAL LINK: Building an Organizational "Failure Memory" for CI/CD].

— Ram