Loading...
Loading...
At some point in building a platform, you hit a wall that has nothing to do with features. The code works. The tests pass. But can it actually be operated?
This is the ninth post in the Sentinel series. Previous: [Building Sentinel: Kafka, Multi-tenancy & Python SDK](#).
At some point in building a platform, you hit a wall that has nothing to do with features.
The code works. The tests pass. The architecture is sound. But the question of whether the system can actually be operated — deployed reliably, monitored effectively, recovered from when things go wrong — remains open.
This is the operational envelope problem. It's the gap between software that works in development and software that can be trusted in production.
Sentinel reached that wall after Phase 5. The core capabilities were complete: audit ledger, risk intelligence, AI actor tracking, dashboard, compliance reports, multi-tenancy, Kafka streaming, SDK. But answering "is this production-ready?" honestly required more than checking the feature list.
---
There are exactly three questions that determine whether a system is production-ready, and they're not about the code:
Can it be deployed reliably, repeatedly, by anyone on the team?
"Works when I deploy it" is not the same as "reliable deployment." Reliable deployment means a documented, automated process that produces the same result every time. It means zero-downtime rolling updates so a new deployment doesn't cause the very incidents Sentinel is designed to detect. It means image digest pinning — not image: backend:latest which silently changes under you — but image: backend@sha256:abc123 so you know exactly which code is running.
It means the deployment process is testable against staging before production sees it, with a manual gate that forces a human decision before the production environment is touched.
Can you tell, at any moment, whether the system is healthy?
Not just "is the server running" — that's the liveness probe, which is table stakes. Real health monitoring means knowing: is the event ingestion pipeline within SLA? Is the risk engine scoring in time to matter? Is the Kafka consumer keeping up, or is it falling 1000 events behind and providing stale risk scores? Are we using 85% of Redis memory and approaching the point where JWT blacklist entries and Celery task queues start competing for space?
These are specific, measurable thresholds. Prometheus can watch all of them and fire alerts when they cross. But only if someone has written the alerting rules, which means deciding what the thresholds should be and documenting why.
When something goes wrong, can someone fix it?
Not "can you fix it" — you know the system, you built it. Can someone fix it at 2am when you're offline? Can a new engineer who joined last month fix it?
This requires runbooks. Specific, step-by-step documentation that starts from the alert that fired and ends with the system restored. The commands to run. The things to check in order. The escalation path when the runbook doesn't resolve it.
Writing runbooks also forces a kind of discipline: you can only write a runbook for a failure mode you've thought through. The act of writing them surfaces gaps in the operational design before they become incidents.
---
Kubernetes is often presented as an infrastructure choice — a way to run containers at scale. It's more accurate to think of it as an operational contract.
When you define a Deployment with maxUnavailable: 0, you're making a commitment: deployments will never take the service down. When you define an HPA, you're making a commitment: the system will scale automatically when load increases, without human intervention. When you define a terminationGracePeriodSeconds: 120 on the worker, you're making a commitment: in-flight tasks will complete before the pod is forcibly terminated.
These are not performance optimizations. They're reliability guarantees expressed as infrastructure configuration.
The Kafka consumer deployment uses strategy: Recreate instead of RollingUpdate. This is a specific operational decision: during a rolling update, two consumer instances would be running simultaneously, both claiming partitions. Kafka's rebalancing protocol would re-assign partitions between them mid-deploy, which can cause duplicate processing or missed offset commits depending on timing. Recreate terminates the old pod before starting the new one. Deployment takes slightly longer, but the partition assignment is clean.
Every one of these decisions represents a trade-off that's documented in the deployment manifests. The manifests are the documentation.
---
Sentinel has its own alert system: the AlertRule model that evaluates conditions against audit events and fires application-level alerts when risk thresholds are crossed.
But those alerts are about the data Sentinel is watching. They don't tell you anything about whether Sentinel itself is healthy.
That's what Prometheus alerting rules are for. Two separate alert paths, serving different purposes:
The application alerts ask: "Is something suspicious happening in our financial systems?" They fire when risk_score > 75 or when an AI agent accesses a new resource type or when an admin action happens at 3am. They're stored in PostgreSQL and delivered via Slack and email.
The infrastructure alerts ask: "Is Sentinel itself working correctly?" They fire when the event ingest p99 latency exceeds 500ms, or when the Kafka consumer falls 1000 events behind real time, or when the API error rate exceeds 1% for five minutes. They go to Alertmanager, then PagerDuty, then someone's phone.
Conflating these two would be a mistake that surfaces in exactly the wrong moment: when Sentinel is experiencing issues, you want the infrastructure alerts to fire independently of whether the application alert system is working.
---
After Phase 6, Sentinel has a complete operational story alongside its technical one:
The next post covers the technical implementation — the Kustomize base/overlay pattern, the CD pipeline's image digest pinning approach, and the alerting rule design that watches Sentinel's own health metrics.
---
Sentinel is open source: [github.com/Gwerdonatus/Sentinel](https://github.com/Gwerdonatus/Sentinel)