Rebuilding a payment ledger that finally reconciles

The situation

A mid-stage fintech was processing millions of transactions a month through a system that had grown organically. Settlement was tracked in spreadsheets, reconciliation was a manual process owned by two people, and edge cases (chargebacks, partial refunds, currency reversals) caused weekly fire drills. Audits took months.

Leadership had two open jobs they couldn't fill: a senior payments engineer and a reliability lead. They needed both, fast.

What we did

• Designed a double-entry ledger. Wrote down the accounting model first — every money movement creates two entries, with a single source of truth for balances.
• Made every operation idempotent. Webhooks, processor callbacks, and internal events became safely-retryable with deduplication keys persisted in the ledger.
• Built reconciliation as a query, not a script. Daily reconciliation reports run against the ledger and the processor's settlement files — discrepancies surface as a count, not a hunt.
• Reduced PCI scope. Card data was tokenized at the edge and never crossed our application boundary. The compliance footprint shrank dramatically.
• Cutover with a shadow run. Ran the new ledger in shadow mode for six weeks against production traffic before flipping over. Zero customer-visible incidents at switch.

Outcomes

• Reconciliation finishes within T+1 for 99.99% of transactions.
• Ops time on payments dropped by roughly five business days per month.
• The first audit after migration completed in two weeks instead of two months.
• No duplicate-charge incidents since launch (previously a quarterly recurrence).

Stack

AWS, Postgres (with strict transaction discipline), Go, TypeScript / Node.js, Terraform, Datadog. Card processor and bank rail integrations handled with at-least-once delivery semantics.