RelA — Reliability Autohealer is a small prototype built to observe how reliability decisions behave under load.
Not because mechanisms are missing — most systems already have retries, timeouts, hedging, and load shedding — but because the structure and timing of decisions begin to matter more as systems approach saturation.
In an earlier post, I described the motivation behind RelA and the problem space it explores. This post focuses on what became visible once reliability decisions were made explicit, observable, and separable from request execution.
At its core, RelA asks -
What changes when reliability decisions move out of the request path and into a control plane?
RelA explores this by modeling a complete control loop on a single machine: metrics → policy evaluation → action → verification, with an explicit audit trail. The goal is not to optimize latency, but to understand how reliability decisions interact, conflict, and fail under real load and saturation. [Source code: github]
Why this problem appears even when systems look 99% healthy
In distributed systems, average latency rarely tells the full story. Reliability is shaped by tail behavior, and tails are what tend to break user journeys.
In a fan-out architecture, if a single request fans out to 20 downstream calls and each succeeds 99% of the time, the end-to-end experience succeeds only about 82% of the time:
Closing this gap requires mitigations such as hedging, load shedding, and timeouts. These mechanisms are effective under normal conditions. Under saturation, they begin to interact in less predictable ways.
RelA was built to observe that interaction directly and explicitly: create congestion, observe tails, apply mitigation, and observe how the system responds over time (second-order effects).
Architecture: policy outside the hot path
Most production systems embed reliability logic directly inside application SDKs. This works until the service is under stress. At that point, the application is doing two jobs:
- serving traffic
- deciding how to survive the traffic
As saturation increases, these responsibilities begin to compete. Decisions about what to drop and what to preserve are made locally, with limited visibility into global conditions or downstream impact.
This coupling leads to a classic failure mode: Priority Inversion - a busy system spends its last compute cycles deciding how to save itself.
RelA moves reliability decisions out of the hot path and into a control plane, inspired by the control-plane/data-plane split used in Software-Defined Networking (SDN). The separation changes three things: failure isolation, reasoning clarity, and blast radius.
-
Data Plane (
main.py) A FastAPI service that processes/workrequests. It is stateless and acts only on externally supplied flags -hedge_enabled,degraded_mode,shed_bronze. -
Control Plane (
agent.py) A Python loop that polls Prometheus metrics, evaluates policy, and asserts decisions back to the application. -
Source of truth (
config_server.py) A minimal config service holding global state. The application polls it like a sidecar.
This separates the cycle of deployment from the cycle of control (changing decision logic). Decision logic can change without redeploying application code, and the behavior of the control loop can be inspected independently of request handling..
Reliability decisions are evaluated outside the service and asserted back as control flags.
Design constraints that surfaced
1. A controller acts on sampled reality
Control loops never react to reality directly. They react to a measured projection of it. That projection is shaped by scrape cadence, aggregation windows, evaluation intervals, and configuration propagation delays.
In RelA, the decision loop is bounded by 4 independent clocks:
- Prometheus scrape cadence
- query windowing semantics
- controller evaluation interval (2s)
- application-side config polling interval (2s)
Those clocks exist in every production system. Together, they define the earliest possible moment at which a change can be detected, reasoned about, and acted upon.
No amount of fast logic can overcome this bound.
This is where controllers become correct but late.
Spikes that begin and end inside a window may be invisible. Spikes that have already subsided may continue influencing decisions. This is not a bug — it is the cost of sampling.
Production systems account for this through intentional inertia: consecutive breaches to enter a mode, consecutive clears to exit, and minimum dwell times to prevent flapping. RelA makes these constraints explicit and centralized, rather than scattering them across SDKs and request paths.
Controller transition modes only after sampled signals converge, not at the moment conditions change.
2. Tail mitigation can self-amplify under saturation
When tail latency rises, the instinctive response is to enable hedging. In RelA, hedging improves tail behavior by spending additional work on slow requests — until the system is already resource-bound.
Under saturation, spending more work to reduce latency can be the wrong move.
RelA exposes this failure mode by forcing explicit policy arbitration. Policies are evaluated in priority order and exit on the first match:
for actionin policy.get("actions", []):
if triggers.intersection(breaches):
target_action = action["name"]
break #selects the first applicable action and exits
This single break matters.
It enforces mutual exclusion between:
- survival mode (capacity protection via shedding)
- performance mode (tail optimization via hedging)
The system is never allowed to optimize latency while it is failing capacity. That trade-off is resolved structurally, not heuristically.
Under saturation, the controller suppresses tail optimizations and enters a single, capacity-preserving mode
3. QoS as a product decision
Random shedding treats all requests equally. In practice, systems often care about which requests succeed.
RelA allows business context (e.g., user tier) to participate in control decisions,reframing reliability from a fairness problem to a value-preservation problem.
- Gold tier - bypass shedding
- Bronze tier - dropped first during surviaval mode (fast-fail with 503)
From a CAP perspective, this accepts lower consistency to preserve availability for the partition that matters most.
Reliability outcomes are asymmetric by design
Decision and outcome alignment
RelA treats policy-driven drops as first-class outcomes rather than errors. Errors and drops are intentionally tracked separately to preserve decision semantics.
Agent logs below show deterministic mode selection aligned with observed signals and policy priority. Application behavior reflects control decisions directly, making trade-offs visible at the boundary.
The specific numbers matter less than the alignment between signal, decision, and outcome.
What would need hardening in production
RelA is intentionally minimal. Hardening would focus less on adding features and more on controlling failure modes:
1. Control-plane availability and scope: The controller can’t be a single point of failure. In production this evolves into:
- HA - per-pod, per-zone agents
- Leader election - for shared policy ownership
- Sharded controllers - decisions apply to a shard, not the entire fleet
In a nutshell, the goal is not global coordination, but local survivability when the parts of the system already degraded.
2. Guardrails on autonomy: Autonomy needs limits. Without guardrails, automation quietly becomes new source of instability.
- Cooldowns - minimum dwell time per mode
- Max durations - upper bounds on how long a mitigation can remain active
- Exit conditions - explicite exit criteria, not just threshold recovery.
3. Auditability: Persistent logs should be tied to metrics and timelines.
At scale this means:
- persistent decision logs (metric → threshold → action)
- correlation with request traces or incident timelines
- human-readable explanations for why a mode was entered or exited
4. Progressive rollout of policy changes: progressive rollout of policy changes, with override paths to limit blast radius by service, tier or region.
Retrospective
2 patterns stood out once the control loop became observable:
1. Remediation speed is bounded by signal resolution - You cannot correct in seconds what you measure in minutes. Scrape intervals, query windows, evaluation cadence, and config propagation together define the minimum achievable recovery time.
If the signal arrives late, the decision will be correct — and still wrong for the moment.
This constraint exists in every production system, whether acknowledged or not.
2. Control must be level-triggered, not edge-triggered - Desired state must be continuously asserted to prevent drift.
RelA continuously asserts desired state based on current conditions:
- mitigations stay active while breaches persist
- mitigations exit only when conditions are demonstrably stable
This prevents configuration drift and avoids half-healed systems where automation fired once and never fully recovered.
Together, these observations reinforce a simple idea: Many reliability failures are decision-structure problems rather than mechanism gaps.
Final note on scope
RelA is not a production system. Its value lies in surfacing behaviors that remain consistent across scale — only with higher cost.
Latency numbers come from local runs and will vary. What remains invariant is the shape of behavior and the constraints behind it.