fosdick.io

Building Service Frameworks

I spent time at Stripe building the internal Java and Ruby service frameworks that hundreds of Stripe's services were built on (500+ at the time of this writing). This post takes a look at the approach behind them, the decisions we made, and what worked (and what didn't).

Specifically, I worked on Stripe's Service Frameworks team. We owned the internal framework that teams used to build Java services. Over time, this grew to include the Ruby service framework as well. This unification helped to standardize how services get built at Stripe.

I'll keep this at a high level and spend most of the time on the parts that are easy to underestimate: migrations, developer experience, and the challenge of managing multiple service frameworks in different language ecosystems.

Background

Stripe runs one of the largest Ruby codebases in the world (roughly 15 million lines across roughly 150,000 files at the time). For most of the company's history, the product was a single monolithic Ruby application. The monolith was a big part of why Stripe could move as fast as it did for as long as it did. However, past a certain scale, a single shared codebase begins to have significant drawbacks.

Stripe made a few strategic bets to solve this. One of these was to make it straightforward to take well-defined pieces out of the monolith and run them as independent services. With that, the Java service framework was born with a set of conventions, tooling, and abstractions for creating, deploying, and operating a new service without reinventing the wheel every time. The team's original name, Horizon, reflected that forward-looking mission.

Services were extracted where boundaries were clear and the independence was worth it. That said, a large amount of important logic stayed in the monolith.

Design

A framework is really a set of opinions about how software should be built, written down as code. Having a set of common abstractions meant that when engineers switched teams, they could take what they knew of the framework and focus on business logic. We did not try to support every possible pattern, but instead provided sane, opinionated defaults. We tried to make the common case easy and the correct behavior the default.

Building a Service Framework

It was decided early on to build a framework in-house rather than adopt something like Spring. This mostly stemmed from the requirements to use gRPC, compile-time dependency injection, and Bazel as the primary build system. At the time, no existing framework supported all of these cleanly.

The framework was built as an integration layer over existing open-source libraries. This includes Dagger for dependency injection and Temporal for durable, long-running work.

This decision to build in-house gave a lot of control over the developer experience, but the cost was the ownership and maintenance burden that came with it.

Declarative Configuration / Code Generation

The framework took a configuration-first approach. Engineers would describe what they want (e.g., which pods to run, which services to expose, which databases to attach) and the tooling would generate the scaffolding to match. A simplified configuration for a URL shortener might look like this:

bundle_name: links
pods:
  - flavor: rpc-server
    services:
      - LinkService
  - flavor: worker
    workflows:
      - PruneExpiredLinks
databases:
  - type: mongo
    name: linkdb

This configuration file could then be used to generate the skeleton for an entire service. This included Kubernetes configuration, dependency-injection wiring, directory structure, and stub implementations. The configuration served as the source of truth with everything downstream being derived from it.

A build-time code generation pipeline: a declarative bundle config and source annotations feed a Bazel build, which produces hidden dependency-injection wiring, Kubernetes config, and gRPC stubs.

Figure 1. Build-time code generation pipeline. The declarative configuration and a handful of source annotations drive the build.

The initial version generated files that engineers were meant to check in and edit. That was a mistake because engineers would modify the generated files in ways we never expected. When we would make changes to a template, this would then collide with their edits.

So we moved to generating hidden files at build time using Bazel macros. Annotation processors were used when something needed to be registered at runtime (like a database model).

@Model(db = "linkdb", collection = "ShortLink")
abstract class ShortLink implements Model<ShortLinkPb> {
  @Id(prefix = "lnk", type = RANDOM)
  abstract String id();
}

Service Bundles

One key principle for the framework was data isolation. A service's data belongs to that service. Everything else talks to it through a well-defined interface (such as a gRPC endpoint or a Kafka event). Breaking this rule would result in a distributed monolith.

The "service bundle" was built on this principle. A service bundle is a cluster of pods that are allowed to share direct database access. We had different types of pods for each bundle, such as:

  • rpc-server pods that handle synchronous gRPC traffic
  • worker pods that handle asynchronous jobs

For the previous URL shortener example, the rpc-server pod answers the calls to shorten or resolve URLs, while the worker pod runs the job that prunes expired links in the background. Both touch the same linkdb, but the rest of the company only ever sees LinkService.

The service bundle: isolated externally, but shared internally.

Figure 2. The service bundle: isolated externally, but shared internally. Pods within a bundle may share a database directly, but anything outside the bundle must go through the public interface.

The service bundle enabled teams to split synchronous work from asynchronous work and scale the two separately while presenting a single clean boundary to external services.

Migrations

One of the most painful and time-consuming aspects of building and maintaining service frameworks is migrations. Every time the framework would evolve, we would need to make sure existing services got updated. When the framework exposes a large surface of generated, checked-in code, then even small template changes mean code modifications for every service that uses it. That doesn't even factor in the human coordination required for looping teams in and having them review changes.

As previously mentioned, this is the reason we moved away from having teams check in their generated code. When the code is regenerated at build time, a framework update becomes much simpler for the service owner. That said, migrations never fully go away. Anything that touches the APIs that bundles expose to one another needs careful versioning and compatibility.

It is important to treat every migration like a product, with users, adoption metrics, and a high quality bar. The tooling to move services safely does not demo well and is not generally considered to be exciting work, but it is the thing you most want to have already built before you actually need it.

Developer Experience

An important pattern is to abstract away sharp-edged libraries by providing safe, opinionated defaults out of the gate. Stripe's use of Temporal is a good public example, as the team wrapped the Temporal SDK and kept it out of product teams' hands. This exposed a safer interface with guardrails like mandatory retry policies and static analysis to catch code that could break workflow replay.

The other place we invested heavily in developer experience was observability. Debugging a distributed system is hard, so it's important to ensure instrumentation is effortless by default. Every gRPC call should be traced out of the box, and there should be an easy way to annotate a function with the metrics it should emit. When production blew up, tagged metrics for dashboards/alerting, queryable logs, and distributed traces across services were all available.

A single inbound gRPC request passing through server and client interceptors across two services, automatically emitting metrics, logs, and a distributed trace.

Figure 3. A single gRPC request, automatically instrumented end to end. The interceptors emit metrics, logs, and trace spans without the service author having to wire anything up.

Expanding to Ruby

When the Ruby service framework moved under our team, we aimed to unify the approach to building services at Stripe. Ruby and Java services had to interoperate, which provided a great opportunity for alignment. They called each other over gRPC, produced and consumed the same Kafka events, and shared the same observability infrastructure. One team owning both frameworks meant we could make coordinated decisions about how those shared layers behaved.

The challenge was that the Ruby and Java ecosystems are genuinely different. Patterns that feel natural in Java (like compile-time dependency injection with Dagger) do not always translate to Ruby. We were able to separate the abstractions that were truly language-agnostic (e.g., how a service is described, how it is deployed, how it is traced) from the ones that had to be adapted to each language. This helped to unify goals and let implementations diverge where required by the languages.

Key Takeaways

Building infrastructure that other engineers depend on is a different kind of work than building product features. Your customers are your colleagues, which makes the relationship more forgiving in some ways and much less forgiving in others. The cost of a bad API decision is not just a bad afternoon for one user, but can be many hundreds of hours of migration work spread across every dependent team.

A few things I would carry into any similar project:

  • Keep the surface area small. Every abstraction exposed is one that will potentially need to be migrated. Be aggressive about keeping generated code out of your users' repositories.
  • Invest in migrations early. The tooling to move hundreds of services safely is not glamorous, but its value compounds. You want it built well before the day you need it. Treat migrations as a product rather than a chore.
  • Unification pays off. Bringing the Ruby and Java frameworks under one team created real leverage, as work done once could benefit both ecosystems. Knowledge moved more cleanly across a boundary that used to be two separate silos.
  • Opinionated defaults are a feature. Saying "this is how we do it here" is a gift to the engineers who use your framework. Every default you set well is a decision that a hundred teams do not each have to make poorly.

While some of this is obvious when said out loud, it is expensive to learn on your own. In the end, I think that's a fair description of what a good framework is for in the first place.