ECS for Control: extract-ecs

Manual page for extract-ecs, the extractor: the ECS file and the ECS map file written from the sources of a built program, compared against a previous pair.

Name

extract-ecs – extract an ECS program's definition from its built sources: the ECS file and the ECS map file, the pair, compared against a previous pair

Synopsis

extract-ecs program-dir [-o stem] [-b stem] [--allow-lists file]

Description

extract-ecs reads a program written to the architecture of ECS for Control: Concepts and writes its definition: which modules exist, of which kind, under which names they play their kind's roles, where they belong, and what the sources establish for each of them. It writes two files beside the sources, under the stem -o names, with two extensions:

<stem>.ecs, the ECS file: the head line, the ALLOWLISTS line, ROLENAMES, MODULES and DECLARATIONS, in the grammar of ECS for Control: The ECS File;
<stem>.ecsm, the ECS map file: the same definition in the forms a reviewer reads, SCHEDULES, ADAPTERS, TOKENMAP and MANIFESTS, with the evaluated values where the ECS file states an expression as written.

The two files are written by one run and form a pair. Nothing in them is inferred and nothing is authored: every entry is a declaration in a module's text, every fact a construct of the sources or a constant expression evaluated. The program's label, the name of its PROGRAM group, is the program directory's name.

The sources are read in three stages, each on the result of the one before:

stage 0 reads the program as a built program: the linker's map file in the program directory is the module list, and the build is verified by its own products (The Built Program, below);
stage 1 reads the kind declaration of every linked module, the ECSKIND comment before its MODULE line;
stage 2 reads every module as its kind and extracts what the kind establishes (What Is Extracted, below).

With -o, the pair is written; with -b, the fresh extraction is compared with a previous pair; with neither, the program is read and reported and nothing is written. Every run prints its findings first and its result last (The Report, below).

Options

-o stem / --output stem: Write the pair under stem, <stem>.ecs and <stem>.ecsm, beside the sources. An extension given is stripped and said. Without -o nothing is written. The pair is withheld when any violation exists (The Report, below).
-b stem / --basis stem: Compare the fresh extraction against the pair stem names in the program directory, a previous pair or an authored ECS file, and print the differences (The Basis, below). With -o naming the same stem, the regeneration check: the existing pair is left untouched when identical and rewritten when not. Without -o, nothing is written. A stem that names no file is an error; the basis is never guessed from the directory.
--allow-lists file: Read the library imports against file, a project's own copy of the allow-lists file, instead of the library's file in ref/ beside the programs. The file's head line is recorded in the ECS file (The Allow-Lists File, below).

The Built Program

The program is read as a built program: one that compiles and links as it stands. extract-ecs never invokes the compiler and never reads a module the linker did not link. The linker map file, written beside the main program module, is the module list, and its file table says where each module's code file lies; a module in the tree without a line in that table is not in the program, and a library transport an adapter names is found in that table by name, never along a search path.

The build is verified by its products and their dates: every source older than its code file, every importer compiled after the interface it imports, the linker map file newer than every code file, the binary newer than every code file, and exactly one linker map file in the program directory. A program that fails this is refused, with every failure listed, and the rebuild is the check: nothing of the architecture is read from a tree whose build is stale, and no finding of extract-ecs is ever a finding about the Oberon.

The program directory is the directory that holds the main program module and the linker map file. A module's place is the first directory below it in the module's linked path: the program directory itself holds the program-level modules, a directory holding exactly one PARTITION module is that partition's directory, and a directory holding none is a helper directory whose modules are read at the program level. Anything deeper is the programmer's own organisation and reads the same.

The Kind Declaration

Every module of the program declares its kind in its own text, in a comment before the MODULE line, with one parameter per role of the kind:

(* ECSKIND: SYSTEM Manifest Name Run *)
MODULE BlinkSystem;

The kinds, their roles and the parameters are the ones of the ECS file reference. Stage 1 reads every declaration and refuses the program when any cannot be read: a declaration missing or doubled, malformed, naming an unknown kind, carrying the wrong number of parameters, or a parameter that is no identifier. Every such finding is listed before the refusal, and nothing beyond stage 1 is read.

What Is Extracted

Stage 2 reads each module as its declared kind. The roles come first: each role parameter must name an exported item of the module, of the sort the kind table says, a variable or a procedure; schedule 0's gate procedure is local and is the exception. Then the kind's constructs:

SCHEDULE: the gate procedure's statement list, the tick period passed to the kernel, the partitions created and registered in their order, the calls into the program's modules; the deployment, the call that passes another SCHEDULE module's exported gate procedure as a procedure actual, recorded as the source writes it; schedule 0 is the main program module, numbered, never deployed.
PARTITION: the registrations, each read against the registered SYSTEM module's declaration, role by role, with the period as registered.
SYSTEM: the name, a string literal or a local string constant, else the module's name; the manifest, each set's expression as written, module-qualified; the VALUES constants the module names; the module body read for the name and manifest assignments and nothing else.
STATE and LOCKED: the store's pointer type and record, the leaves; the locked record's leaves and the interface record's.
TOKENS: the tokens record's leaves, each token's assignment evaluated and the TOKENBASES constant it reaches, so that the file records the ordinal and the map the program-wide value; the surfaces as braced sets of the module's tokens; the names; the back-assert evaluated.
ADAPTER: each exported procedure's direction, read from its parameter modes, and the store tokens it reaches; the transport named by the declaration, located in the linked program or in the library.
BOUNDARY: the crossings, each index evaluated against the program's schedule numbers, each entry a surface expression as written.
PERIODS, RESOURCES, TOKENBASES, WIRING, VALUES: the exported constants as written, module aliases resolved to module names.
every module: the library imports, the imports beyond the program's own modules and the framework modules the allow-lists file names.

The pair checks between kinds are read here too: a token that mirrors no store variable and a store variable that no token mirrors, a manifest naming a token or surface that does not exist, a crossing naming a token, a registration of another directory's SYSTEM, a partition created by no schedule or by two, a SYSTEM registered nowhere or twice, a doubled program-level module and a partition directory without its STATE or TOKENS module. Where a construct cannot be read, say a tokens record that is no record, the modules that depend on it are reported as unresolved by name, never as a cascade of findings.

The facts the file records and their grammar are the subject of the ECS file reference; the ECS map file's four sections are described there under The ECS Map File.

The Report

The report goes to standard output. Each finding is one line: the identifier of the check-point and the reading it is read under, as the review guide numbers them, then the file and the line it concerns, then the finding:

PARTITION.2-a  BlinkPartition/BlinkPartition.mod:85: registers BlinkSystem.Manifst as its manifestvar; BlinkSystem declares manifestvar = Manifest

A refusal for a program that is not built as it stands lists what is stale under one line, each entry with its check-point:

extract-ecs: EcsControlSchedule-v29: not built as it stands, build first
  MODULE.3-a  BlinkSystem: compiled before its source changed

The findings are of three classes, and the class decides the run:

a refusal: the program is not built as it stands, or a kind declaration cannot be read. Every refusal is listed, the program is not read further, and the exit code is 2;
a violation: a structure of the architecture that the sources do not establish. Every violation is listed, nothing is written, and the exit code is 1;
an observation: a reading the reviewer should see that withholds nothing. It is marked (observation) at the end of its line, the pair is written, and the exit code is 0. A SYSTEM registered nowhere, a partition directory named otherwise than its PARTITION module, and a source in the tree the linker did not link are observations.

A finding about a module the extractor could not read is marked -- unresolved and names that module. The report ends with the result: when the pair is written, the module, schedule and partition counts with the count of observations, then what was written and compared; when it is not, the finding counts. The check-points of ECS for Control: Review Guide say which readings extract-ecs holds, which check-ecs holds and which the reviewer reads.

The Basis

The basis, -b stem, is a previous pair in the program directory, or an authored ECS file; it is never guessed from the directory. With a basis, the fresh extraction is compared against it, after the pair is written when -o is given, and the differences are printed:

the ECS file by its meaning: ROLENAMES factored out, an entry or fact added +, removed -, changed ~, each in the file's own terms and named by its group and module;
the ECS map file as text below its head line, section by section, the differing sections named.

The head lines take no part in the comparison. A basis without DECLARATIONS, an authored file, compares its entries only and says so. A basis with an ECS map file beside it compares both; one without compares the ECS file alone.

-o and -b naming the same stem is the regeneration check: the existing pair is compared with the fresh extraction, left untouched when identical, rewritten when not, and the report says which. -b without -o compares without writing anything, the fresh extraction against the named basis; it is the form for a question, not a record. A module the basis does not list is annotated as a new module in its findings.

The Allow-Lists File

The library imports are read against the allow-lists file, the file of the import allow-lists the library ships with the reference implementations, ref/imports.ecs beside the example programs. --allow-lists names another file, a project's own copy. The file's head line is recorded in the ECS file as its ALLOWLISTS line, so that the file a definition was extracted under is known; the modules its FRAMEWORK group names are left out of every library fact. A file that cannot be loaded, missing or mis-defined, is an error listed before anything is read, with every error of the file at once.

What It Does Not Do

It does not compile, link or build: the program is read as built, and a stale build is refused, never repaired.
It does not check the program beyond what extraction establishes: the check-points of the review guide belong to check-ecs and to the reviewer, and the station sequence of a schedule's gate procedure is read by hand.
It does not author: no entry, fact or name is ever inferred, defaulted or proposed. A program without kind declarations has no definition to extract.
It does not read an ECS file to extract from it. An ECS file is read only as a basis, for the comparison.
It does not edit a source, a build product or a basis. The pair is the only thing it writes.
It does not prescribe a process: when to extract, which stem to use and which basis to compare against are the programmer's choices.

Examples

The examples are run in the program directory and write the tool by its name; how the name is run, python -m extract-ecs with PYTHONPATH set or the path to the script, is the subject of Running the Python Tools. The ECS tools live beside the example programs, in examples/v3.2/rpi/pico2/EcsControlSchedule/ecs-tools/, not in the repository's tools/ directory; a PYTHONPATH that names tools/ is extended by that directory for the module form.

Read the program and report, writing nothing:

extract-ecs .

Write the first pair of a version:

extract-ecs . -o v29-01

Write a second pair and compare it against the first:

extract-ecs . -o v29-02 -b v29-01

The regeneration check: compare the existing pair with a fresh extraction, rewrite it when the sources changed:

extract-ecs . -o v29-01 -b v29-01

Ask whether the sources still match a pair, writing nothing:

extract-ecs . -b v29-01

Extract under a project's own allow-lists file:

extract-ecs . -o v29-02 -b v29-01 --allow-lists project-imports.ecs