ECS for Control: Observations – Calibration

From the declared rates to every derived quantity: the observation contract, who authors which number, and the relations that bind them.

The Declaration

Everything the observation machinery does is derived from a small authored declaration – no quantity below is picked, tuned, or adjusted after the fact. This document states the declaration and the derivations as relations; concrete values for every quantity – one calibration, used by the whole Observations set – are collected in § The Calibration Row. The declaration has six members:

acceptable rate:  eventsAccept events per opsAccept operations
triggering rate:  eventsTrig events per opsTrig operations
eventsPerOp:      events one triggering operation produces
opsBurst:         the burst that just reaches the set threshold
clearPct:         clear threshold, as a percentage of the set threshold
capPct:           level cap, as a percentage of the set threshold

the acceptable rate is the commitment: up to this many events per this many operations is declared service quality, not trouble. A rate is always a pair of counts – events per operations – never a fraction.
the triggering rate is the declared excess: at this rate, sustained, the service is failing its purpose, and detection must fire within a bounded number of operations. The gap between the two rates is the band; inside it the machinery deliberately promises nothing (§ The Detection Curve).
eventsPerOp covers mechanisms whose watched condition produces more than one event in a single operation – compound occurrences count heavier, which is the correct arithmetic for "worse". For most conditions – the print buffer clip of EcsControlWatch among them – it is 1.
opsBurst is the separate clustering judgement. The rates cannot supply it: a source honouring the acceptable rate over every long stretch can still cluster its tolerated events into a short run, and at the acceptable rate the level is exactly neutral (§ The Counting Regime) – so how much clustering the detector absorbs is not implied by any rate. Rate and burst together are the commitment; neither is derivable from the other. The declared burst is the run of consecutive triggering operations that just reaches the set threshold – the trip lands exactly on its last member. It is counted in operations; the events it carries follow through eventsPerOp.
clearPct and capPct place the two remaining thresholds relative to the set threshold: the level must fall to clearPct of it before the condition counts as cleared (the hysteresis), and the level is capped at capPct of it (bounding how much accumulated history a recovery must work off).

The declaration is authored by the owner of the watched condition – for a mechanism and its serving loop, the provider. Capability bounds the declaration but does not source it: capacity determines what the mechanism can hold, the declaration is what its owner commits to, and commitment must not exceed capability. Absorption – the graceful per-event handling, rung 0 of the concepts document's ladder – is likewise capability, not declaration.

Ownership

The contract has two sides, and every quantity in this document belongs to exactly one of them:

the provider owns the watched mechanism and declares its service: the six members above. From them, everything in the following sections is derived – weights, thresholds, the detection curve. The provider publishes the declaration and its curve, and learns nothing about the callers.
the usage engineer owns the operations: how often the mechanism is used, in what bursts, to what purpose. The usage side neither authors nor receives any monitoring constant. Its part of the contract is selection and acceptability: read the published curve, convert it with its own call rate into its own time scale, and judge whether the promised detection serves the purpose. No usage-side number crosses the contract.

This split is why the whole calibration is denominated in operations, not in time. The machinery can promise "detect within so many operations of sustained excess" – it cannot promise any wall-clock time, because without operations there is no evidence: an idle mechanism trips nothing, ever. Time enters only at the two ends – in the plant requirements the usage engineer starts from, and in the supervisor's judgement of a detected condition – never in the middle (§ The Temporal Boundary).

The declared-burst form above is the provider-owned case, and the one EcsControlWatch uses for the print buffer clip. The mirrored case exists: where the monitored loop belongs to the usage side, the usage engineer authors required detection points – rate and deadline pairs – and the set threshold is derived as the tightest one; the burst the choice implies is then reported back as a behavioural fact. Either way exactly one side authors, and the derivation below is the same.

The Counting Regime

The mechanism's side of the machinery is two counters at the event source, maintained during the operation itself:

every operation counts, exactly once – the operation counter is the exposure record.
a triggering operation counts its events in the same act – the event counter and the operation counter advance together, never separately.

The distiller reads both counters and applies the change since its last pass as one net quantity. This makes the level a function of the counts alone – not of when the distiller looked. Two consequences carry the whole discipline:

batching invariance: whether the distiller sees ten operations in one pass or one operation in ten passes, the level lands in the same place. Distiller scheduling cannot distort evidence.
exactness: every relation in this document is exact, not approximate. There is no sampling loss to allow for, and no safety margin to add – a margin written "to be safe" would be an undeclared parameter.

The weighting is fixed by one identity. Events add weight to the level, operations drain decay from it, and the two are chosen so that a source running exactly at the acceptable rate holds the level steady:

neutrality:  eventsAccept * weight - opsAccept * decay = 0

simplest choice, used throughout this document:
weight = opsAccept
decay  = eventsAccept

The identity fixes only the ratio of the two. Any common scaling of weight and decay – and with them of every level-denominated threshold – preserves every relation in this document, and leaves every operation-denominated quantity (the detection curve, the recovery horizons) unchanged. The implementation uses that freedom for integer resolution; here the unscaled form stands.

Below the acceptable rate the level drains to zero; above it, it climbs. The acceptable rate is thereby built into the level itself – the reference the observation carries.

The Level Machinery and Its Thresholds

Each distiller pass applies the counter deltas and clamps:

level' = level + deltaEvents * weight - deltaOps * decay
level' clamped to 0 .. Cap
trip:    level' >= Tset     (tested on the post-application level)

The three thresholds all derive from the declaration:

Tset   = opsBurst * (eventsPerOp * opsAccept - eventsAccept)
Tclear = clearPct * Tset / 100
Cap    = capPct * Tset / 100

One triggering operation contributes eventsPerOp * weight - decay to the level – its events' weight less its own operation's decay. The set threshold is that contribution taken opsBurst times: a run of consecutive triggering operations reaches Tset exactly on the declared burst's last member, and trips there. Two properties follow:

a single triggering operation never trips – with the burst validity below, Tset sits strictly above one operation's contribution. One occurrence is never detection; that is what separates this machinery from the per-event path.
the burst is exact: the trip lands on the burst's last member, not near it. There is no margin in the threshold, by the counting regime's rule.

Validity of a declaration is checked before anything is derived:

eventsAccept >= 1                                       (zero tolerance has no rate to discriminate)
eventsTrig * opsAccept - opsTrig * eventsAccept > 0     (the band is non-empty)
opsBurst * eventsPerOp > eventsAccept                   (the burst's events exceed the tolerated clustering)
clearPct < 100
capPct >= 100

The first check draws the line back to the concepts document's opening: a condition with zero tolerance – any single occurrence intolerable – is not an observation problem. Its correct detector is the per-event path: a direct trip on the occurrence, no counting, no calibration. The observation machinery begins where a tolerance exists.

The third check is the clustering necessity: a source honouring the acceptable rate can legitimately present up to eventsAccept events in a cluster, so a burst carrying no more than that would trip on behaviour the rate face tolerates.

The Detection Curve

From the thresholds follows the machinery's central promise: how many operations of sustained excess until the trip. At a sustained rate of e events per o operations:

opsDetect(e, o) = Tset * o / (e * opsAccept - o * eventsAccept)

valid above the acceptable rate:  e * opsAccept - o * eventsAccept > 0

This is the detection curve – an upper bound, since front-loaded arrangements of the same events trip earlier. The provider publishes it alongside the declaration; each usage engineer converts it with their own call rate and judges acceptability in their own time scale. Three of its properties are the contract's fine print:

at the triggering rate, the curve gives the deadline the declaration promises.
at full excess – every operation triggering – the curve returns the burst: detection in exactly opsBurst consecutive triggering operations. The floor of any detection deadline is burst-determined, not rate-determined.
towards the acceptable rate the curve diverges, and must: bounded detection arbitrarily close to the tolerance would force false trips at the tolerance, since a stream within the tolerance can look marginally offending over any finite window. Detection time growing without bound as the rate approaches the acceptable one is the continuity between "never trips within the commitment" and "trips within deadline at the declared excess". Inside the band the level drifts slowly, deliberately unspecified – that is what tolerance means on the operations axis. The width of the band is thereby priced on the curve, where the usage engineer can read it: a narrow band buys sharp discrimination at the cost of a long deadline.

Fine discrimination near the boundary, if a purpose ever needs it, is a second, slower observation on the same source – a longer window with a tighter band – never a tighter threshold on this one.

Recovery Horizons

After a trip and a corrective action, the level must work its way down – and the decline is evidence in its own right. Clean operations drain decay each, so from any level the count of clean operations to reach the clear threshold is fixed:

opsRecover(L) = (L - Tclear) / decay

recOps    = opsRecover(Tset)     (from the set threshold)
capRecOps = opsRecover(Cap)      (from the cap: the worst accumulated history)

These two horizons are the calibration's gift to the supervisor:

recOps is the dwell: after commanding a correction, the supervisor holds further decisions until that many operations of fresh evidence have passed – one decision per correction's measurable effect. Anything faster reacts to evidence the previous decision has not yet had time to influence.
the horizon is denominated in operations, never wall time. Recovery is absence of events over exposure: an idle stretch proves nothing, however long – quiet has to be earned by clean operations. The same rule guards the reversal side: relaxing a correction on "no trouble for a while" requires that the while contained work.
capRecOps bounds the worst case: however long the excess lasted, the cap limits the accumulated history, and with it the longest recovery the machinery can demand.

The clear threshold below the set threshold is the hysteresis that prevents decision flapping: tripped and cleared are separated by a band of level the evidence must actually cross.

Period Rules

The observation machinery runs on three periods – the usage Systems' run period, the distiller's, the supervisor's – and the counting regime settles how they relate:

a distiller period shorter than the usage period gains nothing. New evidence arises only when operations run; a distiller pass between two usage runs finds no new counts and is a no-op (batching invariance). The shortest useful distiller period is therefore the usage period itself, with the distiller scheduled to run after the usage Systems within the tick: evidence is distilled in the very tick it arises. A longer distiller period is legitimate – it adds up to one distiller period of observation latency, budgeted against the detection deadline.
a supervisor period shorter than the distiller period gains nothing. The published reading changes only when the distiller runs; between two distiller passes the supervisor would re-read an unchanged value. The shortest useful supervisor period is the distiller period, with the supervisor scheduled to run after the distiller within the tick: detection and decision fall in the same tick. A longer supervisor period adds correction latency, budgeted the same way.
with the full chain ordered within one tick – usage, then distiller, then supervisor – the observation machinery adds zero latency of its own.

The one thing distiller timing does affect is recency: the published reading's timestamp is the distillation tick, late by at most one distiller period against the operations it condenses. A supervisor judging staleness authors its bound against that granularity.

The Temporal Boundary

Everything above is denominated in operations, and the calibration stays inside that line. Where an external requirement is temporal – "corrective action active within so many milliseconds" – the conversion is a composite:

temporal deadline  =  detection deadline (operations)  /  operation-rate floor (liveness)

The operations half is this document's; the liveness half – the guarantee that operations keep coming at a known floor rate – belongs to the schedule, which in this architecture is authored and enforced elsewhere. The conversion itself walks the chain backwards: from the temporal deadline, subtract the supervisor's and the distiller's latency budgets, then convert ticks to operations with the usage System's run period and calls per run. It is an authoring-time calculation – performed when the world is designed, never by the running machinery, which receives only the resulting constants.

This boundary is drawn deliberately. Producer rates, millisecond deadlines, and whole-program temporal composition are their own subject, and this set of documents stops at its edge: the observation machinery promises operations, and keeps that promise exactly.

The Calibration Row

The four documents of this set use one calibration throughout – the row EcsControlWatch declares for the print buffer clip, and the verification document exercises:

declared:
  acceptable rate:  1 event per 8 operations
  triggering rate:  1 event per 6 operations
  eventsPerOp:      1
  opsBurst:         6
  clearPct:         34        capPct:  200

derived:
  weight  =  8      decay  =  1
  Tset    = 42      Tclear = 14      Cap = 84
  opsDetect(1 per 6) = 126           (the promised deadline)
  opsDetect(1 per 1) =  6            (full excess: the burst)
  recOps  = 28      capRecOps = 70

A deliberately narrow band – 1/6 declared against 1/8 committed – whose price is the long deadline, read off the curve exactly as § The Detection Curve describes. The implementation document shows these numbers computed, checked, and locked at world creation; the verification document shows every one of them confirmed on target, character-exact.