ECS for Control: Observations – Concepts

Watching a control system over time: from per-event failure detection to observations – events counted against operations, distilled into evidence, deciding adaptation.

From Failures

A control program meant to run unattended has to detect that something has gone wrong before it can do anything about it – detection is the basis of every recovery measure, from correcting a parameter to restarting the world. The familiar form of detection is per-event: a run-time check fires, an ASSERT fails, a fault is raised. One event, one verdict, immediate consequence. Per-event detection is good at exactly what it does: point defects, violated invariants, conditions where a single occurrence already proves the program wrong and continuing is not an option.

This set of documents is about the conditions per-event detection cannot express – and about the mechanism that can. The mechanism grew out of failure detection, and its naming walked away from it in two steps: what began as counting failures became recording disturbances, and settled as gathering observations – each step widening the reach of the same machinery, from things going wrong, to things being out of the acceptable, to facts worth watching regardless of direction. The result is a small set of concepts: operations and events, observations, an instrument that accounts and an observer that decides.

What Per-Event Detection Cannot See

Consider a print buffer that occasionally runs out of space, so that part of an output line is dropped – clipped. Is that a failure?

A single clip is not. For a diagnostic text channel, losing a line fragment now and then may be entirely acceptable – the alternative would be to size the buffer for the worst burst that can ever occur, and pay for that memory permanently. A sustained stream of clips is another matter: output is degrading faster than anyone accepted, and something ought to react. The difference between the two cases is not visible in any single event. It is a rate – so many occurrences per so much work – and a rate cannot be checked by a predicate on one occurrence. Per-event detection offers exactly two responses, stop or ignore, and for the clip both are wrong: stopping over one clip is absurd, ignoring all clips is blindness.

Three properties separate this class of condition from the per-event class:

tolerance is part of the specification. Some amount of the condition is declared acceptable – the acceptable amount is an engineering commitment, not an apology. Detection must fire when the commitment is exceeded, and stay quiet below it.
the condition exists only over time. A rate, a trend, a persistent deviation: each is a statement about many occurrences against much work. Watching a single instant reveals nothing.
the interesting direction is not always "worse". The absence of trouble is itself information – sustained quiet is the evidence on which a supervisor may relax a corrective measure, dial a rate back down, or return to normal operation. No event carries that; only the record of work passing without events does.

The print buffer serves as the example throughout the Observations set for two reasons: it is easy to understand, and it can be implemented in test code on a bare evaluation board, eg. a Pico 2, without any external hardware. But it stands for any comparable peripheral device operation, and the concepts hold unchanged. Consider an RCS thruster: intermittent misfires are permitted as long as attitude control can correct for them – but with too frequent misfires the regulation boundaries of the attitude control may be exceeded, and corrective measures are required.

Observations

An observation is a recorded fact gathered by watching over time, carrying its reference with it: "drifted from the acceptable clip rate", not "17 clips". Two properties define it:

time-integrated – an observation condenses many occurrences and much elapsed work into one current reading; it is the result of watching, not a single measurement.
direction-neutral – trouble evidence and slack evidence alike, each a recorded fact in its own right. For the print buffer: a clip is trouble evidence – demand exceeded the buffer's capacity. A quiet window – one in which the pending output stayed within what a single drain run can clear – is slack evidence: the drain had capability to spare. The second is not the mere absence of the first. It is its own observation, gathered against its own reference (the drain's capability, not the acceptable clip rate), and it is what a supervisor needs to dial the drain back down: a mechanism that only recorded trouble could tighten a response but never justify relaxing it.

The direction-neutrality is why the mechanism is named for observations and not for failures or disturbances. One word keeps its old scope: a disturbance is an observation that has crossed the acceptable – the trouble reading of an observation, never the mechanism itself.

Events and Operations

Observations are built from two counted streams, and the pair of terms carries the whole counting vocabulary of the Observations set:

an operation is the unit of usage work – one use of the watched mechanism. For the print buffer: one string put.
an event is an occurrence of the watched condition during an operation. For the print buffer: the put clipped.

Every operation is counted; an event is counted only when the condition occurs. The two streams are deliberately asymmetric – operations measure exposure, events measure occurrences – and every rate in these documents is a pair of them:

acceptable rate:   1 event per 8 operations   (the commitment)
triggering rate:   1 event per 6 operations   (the declared excess)

The word trigger belongs to the operation level: an operation that produces its event – or events, for compound conditions – triggers them; an operation that does is a triggering operation, and the declared excess is named the triggering rate after the events it counts.

The vocabulary is layered, on purpose. The observation mechanism is neutral: it counts events without knowing what they mean. The source – the mechanism being watched – knows exactly what they mean, and names them in its own terms: at the print buffer, a clip is a failure of the put, and the buffer's code says so. Neutrality binds the observation vocabulary, not its users; meaning belongs to the owner of the watched condition and crosses into the observation machinery as a count, nothing more.

From the two streams, the mechanism maintains a level: a single integer that rises with events and drains with operations, weighted so that a source running exactly at the acceptable rate holds the level steady. Below the acceptable rate the level drains to zero; above it the level climbs – and crossing a threshold is the detection: the level trips, in the circuit breaker's sense of the word, the sense used throughout these documents. An operation triggers, the level trips – the two verbs never trade places. The level is the observation in its condensed form: one current reading, carrying its reference (the acceptable rate is built into the weighting). One level watches one condition against one reference; direction-neutrality lives in the mechanism, not in any single level – slack is a condition of its own, watched the same way against its own reference (§ Observations). How the weights and thresholds are derived from the declared rates is the calibration document's subject.

The Instrument and the Observer

Two roles divide the work of observing, and the line between them is mechanism versus policy:

the distiller is the instrument: it keeps the accounts – reads the counters, applies the weights, maintains the level, publishes the current reading. Like any instrument it never concludes; a meter is not a judgement.
the supervisor is the observer: it samples the published reading, holds the reference, finds the condition – "acceptable", "excessive", "quiet long enough" – and decides what to do about it. The term follows the established usage of supervisory control: a supervisor observes and directs, but does not do the work.

Detection itself – noticing an event and counting it – is a duty of neither role. It belongs to whoever can observe the condition, and that is usually not the party suffering it: the code that calls the print buffer cannot see that its output was clipped, but the buffer knows – only the mechanism can translate "no space" into an event. The most productive detection sites are exactly these contract boundaries, where a mechanism is the sole witness of its own limits; the mechanism counts the event during the caller's own operation, and the accounting picks it up from there.

Note what this leaves on the usage side: nothing. The operation completes the same way whether it triggered an event or not – no error return to test, no handling path, no awareness. Absorption is the mechanism's, accounting the distiller's, decision the supervisor's; the operational code carries none of it. That is a distinguishing property between the two detection mechanisms: a per-event failure is by nature immediate – it surfaces at the point of occurrence and must be dealt with there and then, whatever the handling is: in the calling code, or through the error handling path, up to partial or full recovery. An observed event surfaces nowhere on the operational path; its consequence, if any, arrives later and elsewhere, as a considered decision. For a control program this is the concrete payoff: the control laws stay free of error handling for every tolerated condition.

In the ECS role vocabulary, the distiller and the supervisor are Systems like any others – one whose work is accounting, one whose work is deciding. The operational Systems do the program's actual work and are exempt from nothing: any System that can observe a condition counts it. What keeps the arrangement clean is that the three concerns never share a home: operational code works and counts, the distiller accounts, the supervisor decides.

The Escalation Ladder

Observation-based detection feeds a ladder of responses, ordered by cost and reach – each step up the ladder is a rung. The control system closes three kinds of loop, one per rung, each with a different plant:

rung	loop	closed by	acts on	timescale
0	regulation	operational Systems	the controlled system	every tick
1	adaptation	supervisor	the control system's own parameters	slow, evidence-driven
2	lifecycle	supervisor's last resort	the world's existence	rare, one-shot

rung 0 includes absorption: an event is handled gracefully where it occurs, and accounted – it never becomes an immediate failure. The clip mechanism is the absorber: a put that finds insufficient space delivers what fits, counts the loss, and returns; the program runs on undisturbed. Up to the acceptable rate this is the whole response – absorbed and accounted, no action taken. Capacity sits below the ladder entirely: a burst that fits the buffer is no event at all, nothing to absorb. And absorption does not replace detection – it handles each event singly; only the accounting sees their rate.
the adaptation loop is the supervisor acting on observations by commanding operational parameters – a rate, a period, a batch size – inside an authored envelope. It must run slower than the loops it adapts, or the two fight; the level's own inertia and the thresholds' hysteresis provide that separation. Adaptation is also where slack evidence pays: the same loop that escalated on trouble de-escalates on quiet.
the lifecycle loop is the last rung: adaptation exhausted, the condition persists – revoke the world and re-create it through the recovery path. Re-creation need not mean the whole world: it can be a partial world – one self-contained part of the program, such as a control system partition – re-created while the rest keeps running. This is where the opening motivation returns: observations are one of the two feeders of recovery, alongside per-event faults.

The ladder is also a rule about where loops close: at the innermost level that can close them. An absorbed event never becomes a failure; a condition adaptation can hold never reaches revocation.

Positioning

None of these shapes is novel, and that is their recommendation. The concepts presented here correspond, piece for piece, to long-established practice:

the regulatory/supervisory split of process control, and supervisory control theory (Ramadge–Wonham): a supervisor observes and directs, but does not do the work – the instrument/observer line drawn here.
the leaky bucket of telecom rate policing: a counter that fills with occurrences and drains at a declared rate – the level mechanism's family.
spacecraft FDIR (fault detection, isolation, recovery): detection embedded in the operational units, attribution in a dedicated tier, response in a supervisor – the same three-way division of labour.
alarm management (ISA-18.2): domain experts declare conditions and severities as data; a separate mechanism annunciates and manages response – expertise crossing a boundary as data, not code.
automotive diagnostics (AUTOSAR's diagnostic event manager): software components report events; a central mechanism debounces, weighs, and records them – an event/accounting split of the same shape.
avionics built-in test: continuous background checks feeding maintenance and reconfiguration decisions – observation as a standing activity, not an exception path.

The Observations set makes no claim to these systems' scale or ceremony – an Astrobe control program is not meant to be deployed as commercial flight software, with the certification and infrastructure that entails. The claim is that the shape is the proven one, applied at a scale where it stays fully inspectable: a few counters, one level per watched condition, one supervisor loop, all of it ordinary Systems over the ordinary store.

The calibration document takes the next step: from the declared rates to every derived quantity – weights, thresholds, detection deadlines, recovery horizons – and who authors which.