Loading...
Loading...
TxCore processes millions in monthly volume with a hard guarantee: no transaction is ever processed twice, lost, or left in an inconsistent state. Here's how we built that.
Field Note — TxCore
---
Financial transaction infrastructure has one non-negotiable property: correctness. Not performance, not elegance, not developer experience. Correctness. A payment system that is fast and wrong is worse than a payment system that is slow and right.
TxCore was built to handle high-volume payment processing with a guarantee that every transaction is recorded exactly once, every ledger is balanced, and every failure is recoverable without human intervention. This is the architectural story of how we approached that.
---
Most payment infrastructure falls into two categories: expensive enterprise platforms (Stripe, Adyen) that are reliable but costly at scale, and open-source or in-house solutions that are cheap but fragile.
For startups in emerging markets, neither category works well. Enterprise platforms charge fees that eat margins. In-house solutions often lack the reliability and audit capabilities required for financial compliance — and once you're processing real money, "we'll fix reconciliation later" is not a viable strategy.
TxCore sits in the middle: a self-hosted, distributed transaction processing system with built-in double-entry accounting, idempotent operations, and comprehensive webhook delivery guarantees.
---
TxCore is split into four core services:
1. Ingestion Service: Receives transaction requests via REST API, validates them, and emits events.
2. Ledger Service: Maintains the double-entry ledger, ensures balance invariants, and records every debit/credit pair.
3. Webhook Service: Manages delivery of real-time notifications to downstream systems with retry logic and dead-letter handling.
4. Reconciliation Service: Nightly jobs that verify ledger consistency, reconcile against external payment processors, and flag discrepancies.
Each service is independently deployable, independently scalable, and communicates via events. The event bus is RabbitMQ — chosen over Redis Streams because TxCore needs stronger delivery guarantees than Redis can provide. RabbitMQ supports publisher confirms, consumer acknowledgments, and dead-letter exchanges. For a financial system, "maybe the message was delivered" is not acceptable.
The ledger is the heart of TxCore. Every financial operation — payment, refund, transfer, fee — creates at least two ledger entries: one debit and one credit. The sum of all debits must equal the sum of all credits. This is enforced at the database level with a constraint trigger:
CREATE OR REPLACE FUNCTION enforce_ledger_balance()
RETURNS TRIGGER AS $$
BEGIN
IF (SELECT COALESCE(SUM(amount), 0) FROM ledger_entries WHERE transaction_id = NEW.transaction_id) != 0 THEN
RAISE EXCEPTION 'Ledger imbalance detected for transaction %', NEW.transaction_id;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;This trigger runs after every insert to ledger_entries. It is expensive — it forces a sequential scan of all entries for a transaction on every insert. We mitigated this by partitioning ledger_entries by transaction_date and indexing transaction_id. The performance cost is acceptable because the correctness guarantee is absolute.
Idempotency is not optional in payment processing. A network retry, a user double-click, a webhook redelivery — any of these can cause a transaction to be processed twice. TxCore requires an idempotency key on every mutating request:
@app.post("/transactions")
async def create_transaction(
request: TransactionRequest,
idempotency_key: str = Header(...),
db: AsyncSession = Depends(get_db),
):
existing = await db.execute(
select(Transaction).where(Transaction.idempotency_key == idempotency_key)
)
if existing.scalar_one_or_none():
return existing_result # Return cached response, don't reprocess
# ... process transactionIdempotency keys are stored with a TTL in Redis (24 hours) and permanently in PostgreSQL. The Redis layer handles the fast path — "have I seen this key recently?" The PostgreSQL layer handles the slow path — "did I see this key six months ago?" Both checks happen before any business logic runs.
Distributed transactions are hard. The classic solution is two-phase commit (2PC): prepare, vote, commit. But 2PC holds locks across services during the voting phase. If one service is slow, every other service waits. In a financial system, held locks mean held money.
We chose the saga pattern instead. A saga breaks a distributed transaction into a sequence of local transactions, each with a compensating action:
[Debit Payer Account] → [Credit Merchant Account] → [Record Fee] → [Notify Webhook]
↓ compensate: credit payer
↓ compensate: debit merchant
↓ compensate: reverse feeIf any step fails, the saga executor runs compensating actions in reverse order. The system is eventually consistent — there are brief windows where money has left the payer but not reached the merchant — but every state is valid and every failure is reversible.
The tradeoff is complexity. Compensating transactions have to be carefully designed. "Reverse a debit" is not always "credit the same amount" — currency conversion, fees, and promotional credits complicate the math. We spent three weeks designing the compensation logic for the fee step alone.
But the benefit is availability. A slow webhook service doesn't block ledger writes. A database maintenance window on the fee service doesn't halt all payments. Each saga step is independent, with its own retry policy and dead-letter handling.
Downstream systems depend on TxCore's webhooks to know when money moves. A missed webhook means a merchant doesn't ship a product, or a user doesn't see their balance update.
Our webhook service implements at-least-once delivery with exponential backoff:
async def deliver_webhook(endpoint: WebhookEndpoint, payload: dict, attempt: int = 1):
try:
response = await httpx.post(endpoint.url, json=payload, headers=signature_headers)
response.raise_for_status()
await record_delivery_success(endpoint, payload)
except Exception:
if attempt >= MAX_RETRIES:
await move_to_dead_letter(endpoint, payload, attempt)
else:
delay = min(BASE_DELAY * (2 ** attempt), MAX_DELAY)
await schedule_retry(endpoint, payload, attempt + 1, delay=delay)The backoff schedule is: 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s. After 10 attempts (~17 minutes total), the webhook moves to a dead-letter queue and an alert fires.
Every webhook payload is signed with HMAC-SHA256. The receiving system can verify the signature to confirm the payload genuinely came from TxCore and wasn't modified in transit. This is critical for financial webhooks — a forged "payment received" webhook could trigger a product shipment for a non-existent payment.
We chose event sourcing over storing only current state. In event sourcing, the ledger is not a table of balances — it's a table of events. The current balance is a projection, computed by replaying all events for an account.
class AccountEvent(BaseModel):
event_id: UUID
account_id: UUID
event_type: Literal["CREDIT", "DEBIT", "FEE", "REFUND"]
amount: Decimal
currency: str
metadata: dict
created_at: datetimeThe benefits are profound:
The tradeoff is storage volume and read complexity. Event stores grow linearly with transaction volume. Read models require projection logic that stays in sync with the event schema. We accept this because for financial data, the ability to reconstruct history is worth the storage cost.
---
"Exactly once" is the holy grail of distributed systems and it's theoretically impossible. What you can build is "effectively exactly once" — idempotent operations plus deduplication plus careful ordering.
Our approach combines three mechanisms:
1. Idempotency keys: Every request carries a client-generated key. Duplicate keys are rejected at the API gateway.
2. Deduplication window: Redis stores processed keys for 24 hours. PostgreSQL stores them forever. The Redis window catches retries; the PostgreSQL table catches replay attacks.
3. Deterministic event IDs: Every event generated by TxCore has a deterministic ID derived from its content and timestamp. If the same logical event is produced twice (e.g., by two different processors), the second insert conflicts on the unique index and is silently dropped.
This is not mathematically exactly-once. It is practically exactly-once — the probability of a duplicate transaction slipping through is lower than the probability of a cosmic ray flipping a bit.
PostgreSQL gives us ACID properties on a single node. But TxCore is distributed. A saga step might debit an account in the ledger service and then fail to credit the merchant in the merchant service. During the failure window, the ledger is inconsistent.
We handle this by making inconsistency explicit and temporary. Every saga has a status: PENDING, COMPLETED, COMPENSATING, FAILED. Dashboards and APIs expose this status. A merchant seeing a PENDING transfer knows the money is in flight. A COMPENSATING status means something went wrong and the system is rolling back.
The key insight is that users can handle temporary inconsistency if they can see it. What they can't handle is hidden inconsistency — a transfer that looks complete but actually failed, or a balance that looks correct but is missing a pending debit.
---
Financial systems require a fundamentally different approach to error handling. In a normal web app, you catch an exception, log it, and return a 500. In a financial system, an exception might mean money is in the wrong place. Every error path has to be designed as carefully as the success path.
Idempotency is not optional in payment processing. It is the difference between a system that works and a system that accidentally charges customers twice. Build it in from day one. Retrofitting idempotency onto a system that already has duplicate transactions is a nightmare.
Observability is critical for debugging distributed financial transactions. When a saga fails, you need to know exactly which step failed, why it failed, what the compensating action did, and what state every account is in. Distributed tracing (OpenTelemetry) and structured logging (JSON) are not luxuries — they are requirements.
---
TxCore is a private infrastructure project. Architecture details shared with permission.