Financial InfrastructureFeatured

Data Drift Diagnosis & Reconciliation Service

Most reconciliation tools stop at "something is wrong." This system goes further, answering what broke, when it broke, and why — combining confidence-weighted transaction matching with statistical drift detection on a rolling 30-day window, and surfacing rule-based root-cause hypotheses for every drift event it flags.

4 monthsSource

The Problem

Reconciliation tools typically surface mismatches without context. When match rates degrade, teams are left manually digging through transactions to figure out whether it's a data quality issue, a timing shift, or a genuine upstream break — with no historical baseline to tell normal noise from a real problem.

The Solution

Built a FastAPI service with a multi-factor weighted matching engine (amount, date, reference, description), a dead-letter quarantine so invalid rows are never silently dropped, and a drift analyzer that computes z-scores against a 30-day rolling baseline across match rate, confidence, and date delta — only activating once enough history exists to be statistically meaningful.

Architecture

Nginx (TLS) routing to FastAPI (ingestion, matching, drift analysis) and a Streamlit dashboard

PostgreSQL storing transactions, bank statements, reconciliation results, snapshots, drift events, and quarantined records

Redis for rate limiting and the APScheduler job store

APScheduler with a PostgreSQL-backed store so scheduled runs survive restarts

Idempotent ingestion via SHA-256 content hashing into deterministic batch IDs

Technologies Used

FastAPIPostgreSQLRedisStreamlitDockerAPScheduler

Challenges

1Designing a confidence score that weighs amount, date, reference, and description fairly across different data sources
2Setting drift thresholds (z-score bands) that catch real anomalies without flagging normal day-to-day variance
3Making ingestion truly idempotent so reruns and retries never duplicate or corrupt data

Engineering Trade-offs

1Chose SHA-256 content hashing for idempotent ingestion over a manual dedup step — reruns are safe by construction, at the cost of needing careful hash design to avoid false-positive collisions on near-duplicate files
2Chose a rolling 30-day z-score baseline over fixed thresholds — drift detection adapts to each source's own normal variance, at the cost of a 7-snapshot warm-up period before it activates at all
3Added a human-review band between auto-matched and unmatched instead of a single binary threshold — reduces false confidence in automated decisions, at the cost of an ongoing manual review queue
4Quarantined invalid rows instead of dropping them — nothing the system has seen is ever silently lost, at the cost of extra storage and a quarantine table that needs periodic triage

Lessons Learned

1A reconciliation tool is judged by what it does after it finds a mismatch, not by the matching itself
2Idempotency has to be the default behavior of every endpoint, not a special case for retries
3A human review band between auto-matched and unmatched reduces false confidence in fully automated decisions

Related Projects

View all

Featured

Financial Infrastructure

TxCore

Financial transaction infrastructure with distributed backend, double-entry accounting, and webhook processing.

View project

Featured

Financial Infrastructure

Sentinel

Open-source trust layer for financial systems where AI agents, services, and humans all act on shared infrastructure — built around an immutable audit ledger and real-time risk intelligence.

View project

Financial Infrastructure

LedgerLens Recon

A lightweight, auditable reconciliation CLI that matches Stripe payments against an internal ledger and produces a color-coded Excel report.

View project

Data Drift Diagnosis & Reconciliation Service