Rebuilding the Payroll Engine That Moves Billions

Our partners entrust Check to power payroll for tens of thousands of employers across the US. Those employers rely on us to get their people paid on time. We take that responsibility seriously and sweat the details.

A year ago, we rebuilt the engine behind those payrolls. Today, payroll processing is reliable, easier to operate, and easier to change.

A workflow for every payroll

The old engine had carried us from our first payroll to billions of dollars a year, but it was showing strain. A payroll’s lifecycle was managed in multiple places. Scheduling and rescheduling logic had glommed on layers, each added for narrow reasons. Sandbox and production ran substantially different code paths. Everything moved slower and we held our breath on small changes.

Batch processing has long been a mainstay of payroll and the banking system behind it, and our early cron-based engine followed that model. But we realized that payroll processing isn't actually a batch job. It's a long-lived process unique to each payroll: funding—sometimes late, sometimes by wire, sometimes not at all—disbursement, reconciliation, failure, and recovery.

We built the new engine around that reality, giving each payroll a long-lived workflow responsible for its lifecycle. They recover automatically from database blips and worker restarts. Events like funding or company termination signal that workflow, which coordinates what happens next.

Draw the boundaries

The work started with rebuilding our mental model. We enumerated every processing call site, yanked code out of monolithic models into an intentionally designed interface, and mapped the conceptual flow over and over again until it stuck. We then drew our box around orchestration and scheduling.

Through the rigor, we discovered an elegant core underneath years of accretion. There’s a tight set of events that affect how payroll executes: funding settled, funding failed, expedite, and a couple company lifecycle moments. Only these events could enter the box.

Thinking in APIs and drawing that box made the rearchitecture possible.

Rewrite narrowly, migrate gradually

Big rewrites often fail through biting off too much, scope creep, or poor execution. We would’ve preferred to evolve incrementally, but switching from batch to per-payroll durable workflows meant replacing the orchestration model. We kept the rewrite narrowed to orchestration and scheduling, and migrated incrementally while we ran both old and new engines side by side.

Maintaining both systems was a feature, not a bug**.** We spent dedicated time every day reviewing any “stuck” payrolls in the old system. That ritual surfaced real requirements from the operational realities at our scale. There's always a gap between how you assume software will be used and how it's used under pressure.

Make testing a deliverable

Before this effort, manual QA of payroll processing was hard, so we did too little of it. Our automated tests left us with blindspots on how parts interacted across a payroll’s lifecycle.

To counter that, we built more substantial simulation tooling and consolidated code paths between sandbox and production. An internal panel provided capabilities for inspecting a payroll's full payment lifecycle, and a simulator for nearly any scenario. Want a next-day payroll with 100 workers at a company whose last funding attempt failed? A few clicks. Making QA easy paid off immediately. Everyone involved uncovered at least one issue that could have caused an incident while testing end-to-end scenarios and complex interactions our automated suite had missed.

These improvements reached our partners, too. Previously, sandbox supported largely happy-path testing of the payroll lifecycle. The granular simulation features we shipped let partners trigger multiple failures and recoveries, testing their integrations before these things happen in production.

Start small, not simple

Flipping a switch to 100% wasn’t a risk we were willing to take, so we rolled out incrementally over two months. We started with the hard cases at small scale.

• Crisp interfaces enabled the gradual rollout. First, opt in individual payrolls, then entire partners, one flag at a time.
• Daily reps. Automated money movement canary tests exercised the engine every day before it touched any end user payroll.
• We ran our own payroll first. There’s nothing like your own paycheck riding on your code.
• Risk-based sequencing. We began with limited volume that exercised our most operationally complex paths, wires and manual operations, then expanded to larger payrolls and broader volume as the engine proved itself.
• Daily operational review. We reviewed a rollout dashboard every morning for two months, staying curious about the wide state space of payroll processing.

The right to move faster

Our partners didn’t notice the engine changing. Payroll became more reliable, and we earned the right to move faster. As our partners grow, they need payroll to support more employers, larger employers, more complex needs, and new ways to pay people. We’ll still sweat the details, but we now have an engine we can evolve with them.

For the people getting paid, it will be just another payday.