Library Customisation

Introduction

Oberon RTK or the Astrobe library, or any library, should be customisable by its user. Since Astrobe for RP2040, or any other of the supported MCUs, builds a single program that exclusively runs on the target processor, all customisation needs to be part of the build process. That is, contained in the linked modules that make up the program. As opposed to, for example, the Oberon system, where configuration data could be read from a configuration file without changing any module for a different set-up. With our embedded programs, we have only code.¹

Such customisation could be simple parametrisations, such as buffer sizes, number of possible threads, serial baud rates, memory layout, pin assignments, and so on. There are quite a few of corresponding design decisions baked right into the library code, both in Oberon RTK and in the Astrobe library. They may or may not fit your requirements and system design.

Or maybe Oberon RTK’s memory allocator does not suit your program’s (project’s, system’s) specific needs, so you need to replace module Memory with a customised version, while still using the rest of the framework as-is, out of the box.

In any case, customisation boils down to replacing a library module with a customised one, as program code is all we have for this purpose. Since the rest of the library is to be used as-is, the customised version needs to have the same module and file name as the replaced one, and provide the same, or a superset of, the public interface, else the corresponding import relationships break.

Module Hierarchy

We do have the library source code,² so why not simply make the changes there, and be done with it? Of course we can do that, but in my experience this does not work well for more complex systems that usually need to be maintained over years. I believe we need a clean separation of library code and program code. The modules in the library are not to be altered, otherwise updating the system to a new release of the library – for example to profit of fixed defects, or more efficient implementations under the hood – becomes a labour-intensive and error-prone endeavour. Note that this can also include your own in-house library, to be used across different projects.

From Oberon RTK’s perspective, we have this hierarchy of modules:

1. modules specific for a program or system
2. your own library
3. the Oberon RTK library
4. the Astrobe library

The build process should pick the modules from top to bottom, from most specific to generic. For the purpose of customisation, customised modules that replace a library module with the same name higher up in this search hierarchy should be picked in lieu of the library module. To underline this again, of course the customised module must provide the same, or a superset of, the public interface of original library module.

Astrobe features the hierarchical library search path to find the correct modules, right? Let’s assess why this generally powerful functionality – which I like a lot – is not a solution for our problem at hand here.

The Library Search Path

From the Astrobe documentation:

The editor, compiler, linker and builder first search the current folder when trying to locate imported source code, symbol and object files. They then search each of the folders listed in the current configuration’s Library Pathnames textbox. The search continues until the file is found or the last folder in the list has been searched.

The first sentence holds the clue: compiling and linking always first searches the directory of the target module. That is, in general we cannot put our customised module “higher up” in the search hierarchy so it will be picked in lieu of the library module. It does work for framework modules that are not imported by any other framework module, such as Main. A few of the example programs make use of this already.

But other than this special case we’ll run into version consistency problems while linking, because different modules may have been imported during compilation due to the “current folder” rule. Therefore it’s deemed “unwise”³ to have more than one module with the same name in the search path hierarchy. But – here we actually want to replace a module with our version with the same name, as outlined above.

If you check out the Astrobe config files in the repository you see a representation of the above search hierarchy, sans your in-house library, of course: both Oberon RTK and the Astrobe library are on the search path. Oberon RTK includes a few modules with the same name as the Astrobe library. Main is the most obvious one, and its name cannot be freely chosen, as the linker depends on this exact name.

Programs using these config files build just fine. That is, what counts is not the static availability of modules with the same name along the search path, but which modules are actually picked by the build system. From Oberon RTK’s perspective, the modules imported from the Astrobe library must be carefully chosen to avoid the aforementioned module version conflicts. And that’s why MCU2 is not named MCU.

Back to our challenge at hand here, ie. customised versions of any library module in general.

We could work around the “current folder” rule to an extent by segregating modules that may need customisation into separate directories in the library, wherein we have to make sure there are no imports between those modules. This way they are never in the “current folder” with their importers, and thus will never be picked other than via the search path.

However, the problem for the library programmer is which modules to select for a potential customisation – upfront, that is: the library structure would select and restrict the customisable modules. Not flexible, prone to structural changes, restrictive for the library user: I cannot know, in general, which modules you want to customise.⁴ Also, given that the modules in these separate segregation directories must not import others in the same directory, we may need several such customisation directories, or one for each potentially customisable module. And we also may need specific segregation directories for the same module for the different MCUs supported.

Bottom line: the module search path does not help our customisation problem⁵ – unless we change the context to better suit our needs.

Solution Approach

After mulling this over for quite some time now, I have decided to solve this problem by not solving it at the framework level. No segregated modules. Maybe I’ll relent at some point :) and implement a hybrid solution, with segregated modules for simple parametrisations.

The proposal is to create and maintain a program-local framework library. Yes, it’s a bit “duh!” and obvious, but here we are. The basic idea is to selectively copy the required module files from the installation directory to a project-specific library directory, and keep the customised versions outside this directory.

I have created a tool makelib that automates the process for easy updating and maintaining the project-local copy. Obviously this can be done manually as well, or using your own script or program. A program or script has the benefit of systematic repeatability.

The solution is simple, and I don’t claim this to be very original:

• The program (project, system) directory contains a local framework library directory, such as rtk-lib, or rtk-lib-rp2040.
• All required framework library modules, excluding all customised ones, are copied into this rtk-lib directory, flat, ie. without any sub-directories.
• All customised library modules are located in the program directory, or a sub-directory thereof, such as rtk-custom.
• The module search path includes rtk-lib and rtk-custom at the right hierarchy levels, see below.

The modules in the local library rtk-lib (or your choice of name) are not to be changed. Or with other words, the principle is that they can be replaced from the library’s installation directory at any time (with some room for experimentation, see below).

As already noted above, for framework library modules that are not imported by other library modules, such as Main, or Recovery, we can provide customised variants directly without creating a local framework library.

Set-up

Overview

makelib strictly uses the library search path to determine which framework files to copy into the program-local library.⁶ Any library module found with the same name as a module in the project directory (or its sub-directories) is not copied.

See Module Search Path for a detailed discussion of the library search path in general.

Absolute Library Search Path

Let’s assume – as starting point – that without customisation we have this search path for one of the example projects. Note the use of absolute directory paths, and that both Astrobe and Oberon RTK library directories are on the search path.⁷

%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/rp2040
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any/kernel-v1
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/pico
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/any
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/PiPico
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/General

Preparing the search path for the use of makelib, we insert a directory entry for the intended project-local framework library, here using rtk-lib:

%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-lib <= here
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/rp2040
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any/kernel-v1
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/pico
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/any
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/PiPico
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/General

Note the location in the hierarchy right above the framework library directories (the lib ones).

If we want to put all customised library modules into a specific directory, we’d also insert a corresponding entry, here using rtk-custom:

%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-custom <= here
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-lib
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/rp2040
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any/kernel-v1
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/pico
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/any
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/PiPico
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/General

Note the position of rtk-custom above rtk-lib.

Running makelib, or your equivalent own script or even manual copy operation, does the following:

• Determine which framework modules to copy, either by inspecting the program module and its direct and indirect imports, or all:
  • Recursively collect all imports of all modules in a program, here Recovery (if not using the “all” option).⁸ makelib inspects the module source texts.
  • Find and mark the modules below rtk-lib for copying, following the search path. All customised modules, which are above rtk-lib, are omitted.
• Copy the marked library modules into rtk-lib without any sub-directory structure.

After creating the local framework library in rtk-lib, we would only need this minimised search path:

%AstrobeRP2350%/oberon-rtk/examples/v2.1/rpi/any/Recovery
%AstrobeRP2350%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-custom
%AstrobeRP2350%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-lib

But since all not customised modules are now in the local framework library rtk-lib, they will be picked in lieu of any modules in the framework installation directories (oberon-rtk/lib, astrobe-mx/Lib), so there’s no need to create and use a different config file, which makes it easy to run makelib again to update the local framework library at any point. Also, makelib provides an option to selectively pick a subset for copying only, so not copied modules will still be found a their original location along the search path. For this, we need the full search path intact.

Any project-specific sub-directories, such as for sub-systems of your program, are referred to by directory paths above rtk-lib, just as we structure project-specific search path anyway, nothing new here.

For example:

%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/sub-system1 <= here
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/sub-system0 <= here
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-custom
%AstrobeRP2040%/oberon-rtk/examples/v2.1/rpi/any/Recovery/rtk-lib <= "dividing line"
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/rp2040
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any/kernel-v1
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/pico
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/any
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/PiPico
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/General

rtk-lib (again: or your choice of name) is the “dividing line” between framework library and program (project, system) modules. The modules in rtk-lib are not meant to be edited. makelib will overwrite any changes there upon updating. But we can experiment with the local framework library modules, eg. to quickly evaluate a possible customisation before making it permanent in the form of a custom module. Then run makelib to systematically discard those experimental changes, and the framework modules and the project modules are again strictly kept separate, which they should be for the benefit of maintenance.

Relative Library Search Path

Please refer to Module Search Path, and the recommendation at the bottom there: for any other than very simple project structures I would not use relative directory paths to specify the module search path.

However, a nice property of Astrobe is that we can often use generic config files for different projects and programs, in particular if all project module files are located directly in the project’s directory. Generic meaning that we don’t need to specify a project directory in the search path.

This search path does the trick:

..
rtk-lib
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/rp2040
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any/kernel-v1
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/mcu/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/pico
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/board/rpi/any
%AstrobeRP2040%/oberon-rtk/lib/v2.1-dev/any
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/PiPico
%AstrobeRP2040%/astrobe-mx/rp2040/Lib/General

makelib does work with relative directory entries, but for the purpose of the collection of the imported framework modules, they are only resolved relative to the project directory, not for each module absolute directory path in the search path, as they are for compilation. The relative directory paths .. and rtk-lib will resolve to directories that are probably not meant to be in the search path during compilation and linking, but with only the two relative directory specs these cases are limited and manageable.

Bottom Line

Any misunderstanding on my part regarding Astrobe’s compilation, linking, and building processes, and defects in makelib notwithstanding, this solution approach works well in my tests and use: any framework module can be replaced by a customised one (see section Example Programs, below).

The basic idea is simple: work around the “current folder” rule by collecting all required library modules into a project-local library directory, apart from the customised module (or modules), and provide the custom version as part of the project directories, from where it will be imported by the modules in the project library directory. This also cleanly separates library from project code, easing maintenance.

There’s an additional benefit: programs are now self-contained in one directory, including all framework modules. You can always go back and examine exactly how a program was built, ie. using which versions of the framework modules, including Astrobe’s and your own, if you so choose. Updates to the framework modules can be inspected and evaluated in their installation directories after pulling from the repository, before applying them to the project using makelib.

As said before, this whole set-up is pretty obvious and straight-forward, and not very original. You may use it already. The tool support is a useful bonus.

Example Programs

To try, you can apply this solution and makelib to any example program of Oberon RTK.

• Create a config file with the generic search path outlined above (there’s an example in the repository, look for a config file name including plib).
• Build the example program, to be sure we have a working baseline.
• Run makelib on the program main module (see makelib for parameters and options).
• Build the program. By checking the build system’s output, observe how the library modules are now found in the project-local rtk-lib, not the library installation locations.
• Copy any library module into the program directory, and run makelib to update. The tool will report the removal of the “customised” module (use option -v, verbose) from the local library.
• Build the program, again checking where the build system finds the modules.

Copyright, Licenses

Note that this approach and makelib potentially put copyrighted files into the program-local framework library, for instance from the Astrobe library. Or a library module is used as direct basis for a customised one. Which is OK for all development purposes, but in case you distribute your program in source form, or use a public repository, you need to make sure to comply with the corresponding licences.