From Monolith to 18 Microservices: The One We Merged Back

When we decided to break up the core monolithic application at my previous company, I genuinely believed it would be a straightforward path to improved scalability and developer velocity. What I didn't foresee was the sheer operational complexity we'd introduce, particularly the contract testing nightmare that bit us hard in week three of our integration phase, and the service we eventually merged right back into its parent.

The Initial Monolith: A Single Ruby Beast

Our application, affectionately (or un-affectionately) named 'Guardian', was a 350,000-line Ruby on Rails monolith running on 12 EC2 instances (c5.xlarge). It handled everything from user authentication and order processing to invoice generation and reporting. Deployment times were averaging 25-30 minutes, and scaling any specific part of the system independently was impossible. New feature development often meant touching widely disparate parts of the codebase, leading to frequent merge conflicts and a general slowdown in release cycles.

The "Big Bang" Plan and the 18 Services

Our architectural committee, after several whiteboard sessions, decided on a "big bang" approach to split. The plan was to identify 18 distinct domains, each to become its own microservice. These included services for User Authentication, Product Catalog, Order Management, Payment Gateway Integration, Inventory, Notifications, and Reporting. We decided on Kubernetes 1.18 running on EKS for orchestration, with PostgreSQL 12 as our primary datastore for most services, and RabbitMQ for inter-service communication.

Contract Testing: Our First Major Roadblock

Within three weeks of teams starting to build out their services, our CI/CD pipelines became a sea of red. The problem wasn't individual service bugs, but rather breaking changes in API contracts between services. The Order Management Service would roll out a new payload structure for an order confirmation, and suddenly Notifications couldn't parse it, leading to silent failures. We initially tried Swagger/OpenAPI definitions, but these were often out of sync with reality. The solution, after a painful six weeks, was implementing consumer-driven contract testing using Pact.js for TypeScript services and Pact Consumer for Ruby services. It added 5-8 minutes to our CI pipelines but saved countless hours of debugging.

The Saga Pattern: Twice We Got It Wrong

For complex business processes spanning multiple services, like placing an order (checking inventory, reserving stock, processing payment, sending notification), we opted for the saga pattern. Our first attempt used direct orchestrator-based sagas. The Order Orchestrator Service would send commands to Inventory, Payment, Notifications and wait for responses, handling rollbacks. This became a behemoth service itself, tightly coupled to the internal logic of all other services. When Payment changed its API, the orchestrator broke.

Our second attempt was choreography-based sagas using RabbitMQ. Services would emit events, and other services would react. This decoupled services nicely but made tracing an individual transaction incredibly difficult. We initially had no dedicated event logger, so debugging a failed saga involved sifting through logs across 5-7 services, often hours after the incident. We eventually implemented a centralized event log via Kafka Connect and Splunk, which dramatically improved visibility.

The Billing Calculator Service: The One We Merged Back

Among the 18 services, 'Billing Calculator' was designed to take raw usage data and apply complex, client-specific pricing rules. It was a small service, about 2,000 lines of Ruby, highly CPU-intensive, and stateless. While it worked, the overhead of managing a separate CI/CD pipeline, monitoring, deployment, and operational runbook for such a small, tightly coupled domain felt disproportionate. Its API surface was minimal (one endpoint, /calculate), and it was only ever called by the Invoice Generation service in a synchronous, blocking fashion. After six months of fighting minor deployment issues and network latency concerns for this minuscule component, we made the pragmatic decision to merge it back into the Invoice Generation service. This reduced our service count to 17, and surprisingly, developer productivity for invoicing improved slightly due to reduced cross-service communication overhead.

What I'd Do Differently: Smaller, More Focused Splits

Looking back, I'd advocate for a more iterative, strangler-fig pattern approach. Instead of 18 services at once, I would have targeted 3-5 core, highly independent domains first (e.g., User Auth, Product Catalog). This would allow teams to learn operational patterns and tooling without the simultaneous chaos of 15+ new deployments. I'd also push for a dedicated API Gateway (like Kong or Ambassador) earlier for better routing, observability, and authentication/authorization at the edge.

Ongoing Challenges: Observability and Data Consistency

Even with contract testing and improved saga patterns, observability remained a constant challenge. Distributed tracing with Jaeger helped, but ensuring every service emitted consistent and useful span data was a continuous battle against developer fatigue. Furthermore, maintaining eventual data consistency across dozens of databases, especially for critical financial metrics before our eventual Kafka Connect implementation, was a daily source of anxiety. We still had cases where a refund processed by Payment Gateway didn't immediately reflect in Reporting for several minutes, sometimes causing customer support confusion.