Developer guide

OpenServerless is a complete serverless development environment.

It allows you to build full stack, scalable, cloud-native applications with backend and frontend, using services like PostgreSQL (SQL), FerretDB (NoSQL) and Milvus (vector) databases, Redis cache and S3-compatible object storage, with minimum effort. Check the tutorial on https://openserverless.apache.org for a complete guide on how to use it.

This guide focuses on how to develop the system itself, not on how to use it.

TL;DR

The short version, for the impatient. Everything below this section explains why each step is what it is; this is the sequence that gets you to a running local build.

1. Clone with submodules. Most of the code is in submodules, so this is not optional:

git clone --recurse-submodules https://github.com/apache/openserverless
cd openserverless

2. Build the CLI first. Nothing else works until ./ops exists:

task cli        # compiles cli/ into ./ops

The task files it runs are the ones in oplugins/ — plain YAML, editable in place.

[!WARNING] ./ops behaves differently depending on where you are. Run it from this directory and it uses your local oplugins/. Run it from anywhere else and it downloads the task files from GitHub and uses those instead — your local edits are silently ignored. Always check with ./ops -info that OPS_ROOT points inside your working copy. See How ops finds its tasks.

3. Know where the images come from. oplugins/opsroot.json holds, under .config.images, the pointer to every image that gets deployed. Read it before you wonder why the cluster is running something other than what you built — see opsroot.json.

4. Build only what you need, then load it into kind. Each component has its own task, and each one builds the image and saves it so kind can pick it up:

task operator     # or: task openwhisk, task admin-api, task streamer, task runtimes
task opsroot      # rewrite opsroot.json to point at what you just built
ops setup docker  # create the kind cluster and load your local images into it

task build does all of them at once — useful the first time, rarely what you want after. Nothing is pulled from a registry: the images are saved to disk and imported straight into the kind node. See The development loop, end to end.

5. Pick the submodules you actually work on. You generally do not need to build everything. Work inside a single subrepo, rebuild just that component, redeploy — see Iterating on a single component.

6. To publish images, fork and push a tag. CI builds are triggered by tags, and they run in whichever repository received the tag — so the tag must land in your fork:

# fork the repo on GitHub, then, in the component's directory:
git remote set-url origin https://github.com/<your-org>/<the-fork>
task tag          # create the timestamp tag (this deletes existing local tags)
task ci           # push the tag → GitHub Actions builds and pushes the image

See Publishing images.

Components

The key components, each one a separate git repository checked out as a subdirectory of this one:

DirectoryRepositoryRole
cliapache/openserverless-clithe ops CLI: controls the operator and implements the development environment
oplugins-opapache/openserverless-operatorthe operator, installing all the components including the integrated services
admin-apiapache/openserverless-admin-apithe admin API, complementing the operator
streamerapache/openserverless-streamerstreaming support for AI applications
runtimesapache/openserverless-runtimesthe runtimes to run applications in Python/Node.js/Go/Java using the integrated services
buildapache/openserverless-buildthe OpenWhisk fork: controller and invoker images
devcontainerapache/openserverless-devcontainerthe development container image
opluginsapache/openserverless-taskthe task files (the “plugins”) that ops executes
testingthe end-to-end test suite

Start here: build the CLI

Everything else in this guide goes through ops, so build it first. From the root of this repository:

task cli

That compiles the Go sources in cli/ and leaves the binary in the repository root as ./ops. Nothing else is needed to start — no prerequisites to install, no configuration to write.

Now ask it where it is:

./ops -info

This prints the environment the CLI resolved for itself — which task files it will run, where it caches things, which version it is. It is the first thing to check whenever something behaves unexpectedly, and it is explained field by field in How ops finds its tasks.

Then ask it what it can do:

./ops -t

The first run does real work. ops ships with no task files and no external tools, so before it can list anything it:

  1. downloads the tasks — a git clone of $OPS_REPO (default apache/openserverless-task) into ~/.ops/$OPS_BRANCH/oplugins, unless it finds them locally first. You will see Cloning tasks... / Tasks downloaded successfully. In this repository the checked-out oplugins/ is used instead, so nothing is downloaded.
  2. downloads the prerequisites — the external binaries the tasks declare in prereq.yml, cached under ~/.ops/<os>-<arch>/bin: kubectl, helm, kind, k3sup, yq, rg, uv, bun, 7zz, coreutils. They are fetched once, on demand, and pinned by version.

Expect the first ops -t to pause while that happens; later runs are immediate. When it finishes you get the banner and the list of top-level tasks:

$ ./ops -t
Welcome to ops, the all-mighty, extensibile apache OPenServerless CLI Tool.
...
*****
Tasks
*****

Type ops <task> to see usage and subtasks.

-----------------------------------
OpenServerless Administration Tasks
-----------------------------------

 admin       Manage additional users in OpenServerless
 config      Manage the Apache OpenServerless configuration
 setup       Setup the Apache OpenServerless platform on multiple environments
 debug       Debug utilities for the Apache OpenServerless platform
 cloud       OpenServerless setup utilities for supported Deployment models on Cloud Providers
 util        Utilities
 ...

ops <task> with no arguments shows that task's own usage and subtasks, so you can explore downwards from here.

If a download goes wrong, ops -update re-fetches the tasks and prerequisites, and ops -reset clears ~/.ops entirely so the next run starts from scratch.

The CLI

The system is controlled by ops, a single self-contained Go binary. It is essentially a task executor: it is based on Task but extends it in ways that turn it into a complete portable scripting environment.

It is Task, extended

  1. It downloads prerequisite binaries. Task files declare what they need in a prereq.yml; ops fetches and caches the missing binaries on first use (see cli/prereq.go). You do not have to install them yourself.

  2. It embeds tools. wsk, wskdeploy, jq, sh, awk, envsubst, task and a set of small purpose-built helpers are compiled into the binary itself (see cli/_patches/ and cli/tools/). They are invoked with a - prefix and need no download:

    ops -wsk action list
    ops -jq . file.json
    ops -h              # list all the embedded tools
    

The result is that a single binary gives you a complete, portable scripting environment — the same on Linux, macOS and Windows.

General syntax

ops -<tool>  <args>...     # run an embedded tool
ops <task>   <args>...     # run a task from the task files

Essential flags:

-h | -help      list embedded tools
-v | -version   current version (always mention it when asking for help)
-t | -tasks     list top level tasks (downloads them if needed)
-i | -info      show the CLI environment
-u | -update    download the latest tasks and prerequisites
-c | -config    manage the server configuration
-l | -login     access the system
-reset          clean the downloads (if nothing works, try this)

How ops finds its tasks

When you run a task, ops locates its root — the directory holding opsfile.yml and opsroot.json — in this order (see locateOpsRoot in cli/prepare.go:171):

  1. $OPS_ROOT, if set.
  2. Searching upwards from the current directory for a folder containing both opsfile.yml and opsroot.json.
  3. A oplugins (or olaris) subfolder of the current directory. This is what happens when you work in this repository: the checked-out oplugins/ is used directly.
  4. ~/.ops/$OPS_BRANCH/oplugins. If it is not there, ops downloads it with a git clone from $OPS_REPO (default http://github.com/apache/openserverless-task) at the current branch.

So in normal use ops self-provisions its task files; in development it picks up your local oplugins instead. Use ops -info to know which values are actually in effect:

$ ops -info
OPS & OPS_CMD: /Users/me/Projects/openserverless/ops
OPS_VERSION: 0.9.0-2609031223.SNAPSHOT
OPS_BRANCH:
OPS_BIN: /Users/me/.ops/darwin-arm64/bin
OPS_TMP: /Users/me/.ops/tmp
OPS_HOME: /Users/me/.ops
OPS_ROOT: /Users/me/Projects/openserverless/oplugins
OPS_REPO: http://github.com/apache/openserverless-task
OPS_PWD: /Users/me/Projects/openserverless
OPS_TASKS: 7e523adb9af3f17ae221220e9c17bce31fd92090
OPS_ROOT_PLUGIN: /Users/me/Projects/openserverless

OPS_ROOT pointing inside your working copy is the sign that you are running against your local task files.

The configuration files

opsroot.json

oplugins/opsroot.json is the glue of the whole system. It declares which images to deploy and other information; .config.images is the part that matters most during development:

{
  "config": {
    "images": {
      "operator":    "docker.io/apache/openserverless-operator:0.9.0-incubating.26i08j16-snapshot",
      "controller":  "docker.io/apache/openserverless-wsk-controller:0.9.0-incubating.26i07r51-snapshot",
      "invoker":     "docker.io/apache/openserverless-wsk-invoker:0.9.0-incubating.26i07r51-snapshot",
      "streamer":    "docker.io/apache/openserverless-streamer:...",
      "systemapi":   "docker.io/apache/openserverless-admin-api:...",
      "devcontainer":"docker.io/apache/openserverless-devcontainer:...",
      "couchdb":     "docker.io/apache/couchdb:2.3",
      "redis":       "docker.io/bitnamilegacy/redis:8.2.1",
      "...":         "... third-party service images ..."
    },
    "ops": { }
  },
  "version": "..."
}

.config is a source of environment variables

The config object is not read by the tasks as JSON. It is flattened into environment variables that are then passed to every task. The rule (see flatten in cli/config/config_map.go:156) is: walk the tree, join the keys with _, uppercase the result. So

.config.images.operator   →   IMAGES_OPERATOR

and a task can simply use $IMAGES_OPERATOR. ops -config -d shows the whole set — see Inspecting and editing the configuration below.

This is why opsroot.json is the glue: changing an image there changes what every task deploys, with no code change anywhere.

The flattening merges three sources, in increasing order of precedence (cli/main.go:460):

  1. opsroot.json from the task root — the defaults shipped with the release;
  2. config.json — the user's own configuration, written by ops -config;
  3. the opsroot.json of any installed plugin, under its own key (a plugin whose name collides with an existing key is ignored, with a warning).

Inspecting and editing the configuration: ops -config

ops -config is how you read and change that merged configuration without editing JSON by hand (cli/config/config_tool.go).

Dump every variable-d (or --dump) prints the whole flattened set, exactly as the tasks will see it. This is the quickest way to answer “what is this task actually going to use?”:

$ ops -config -d
IMAGES_OPERATOR=docker.io/apache/openserverless-operator:0.9.0-incubating.26i08j16-snapshot
IMAGES_CONTROLLER=docker.io/apache/openserverless-wsk-controller:0.9.0-incubating.26i07r51-snapshot
OPERATOR_CONFIG_APIHOST=miniops.me
OPERATOR_COMPONENT_POSTGRES=true
POSTGRES_CONFIG_REPLICAS=1
...

The dump includes the generated passwords and API keys (SECRET_*, REGISTRY_CONFIG_SECRET_*). Redact it before pasting into a bug report.

Read one value — pass a bare key, with no =. Several keys print several values:

$ ops -config IMAGES_OPERATOR
docker.io/apache/openserverless-operator:0.9.0-incubating.26i08j16-snapshot

Set a value — pass KEY=VALUE. Several pairs at once are fine:

ops -config OPERATOR_CONFIG_APIHOST=my.host.example
ops -config IMAGES_STREAMER=ghcr.io/me/openserverless-streamer:mytag POSTGRES_CONFIG_REPLICAS=3

Remove a value-r (or --remove) with the keys to drop:

ops -config -r POSTGRES_CONFIG_REPLICAS

All writes go to config.json in $OPS_HOME (~/.ops/config.json), which is layer 2 of the merge above — so a value you set there overrides opsroot.json and survives the task files being re-downloaded.

That layering has two consequences worth knowing:

  • -r removes a key from config.json only. It cannot delete anything that came from opsroot.json; removing your override just lets the underlying default show through again. Removing a key that only exists in opsroot.json fails with invalid key: '<KEY>' - key does not exist in config.json.
  • You cannot blank a value. ops -config -h claims that KEY="" disables an opsroot.json value, but the parser rejects an empty right-hand side (cli/config/config_tool.go:161) and the command fails with invalid key-value pair: "KEY=". To neutralise an inherited value today, set it to something the task treats as off (false, 0) — or edit oplugins/opsroot.json directly, as described under Overriding a single image.

.version is the required CLI version

The top-level version field is the version of ops that these task files require. On every run the CLI compares its own version against it as semver (cli/prepare.go:129). If ops is older, it says so and updates itself automatically:

Your ops version (0.9.0) is older than the required version (0.9.1).

In development your CLI version is usually not valid semver (e.g. 0.9.0-2609031223.SNAPSHOT), so the check is skipped with a warning and you are never force-updated out of your own build.

runtimes.json

oplugins/runtimes.json points to the runtime images to use. It is a standard OpenWhisk runtimes manifest — the same format OpenWhisk itself consumes, with runtimes, blackboxes and description at the top level:

{
  "runtimes": {
    "nodejs": [
      {
        "kind": "nodejs:24",
        "default": false,
        "image": {
          "prefix": "docker.io/apache",
          "name": "openserverless-runtime-nodejs",
          "tag": "v24-26i07r17-snapshot"
        },
        "deprecated": false,
        "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" }
      }
    ],
    "python": [ ], "go": [ ], "java": [ ]
  },
  "blackboxes": [ ],
  "description": "..."
}

It is generated in the runtimes/ repository and copied over by task sync-runtimes. Because it is the standard format, the CLI also hands it to the embedded wsk through WSK_RUNTIMES_JSON (cli/main.go:291).

How to build

The system runs on Kubernetes and can be deployed in many environments — most notably k3s, microk8s, and cloud offerings like Amazon EKS, Azure AKS and Google GKE. For development you can use Docker, and OpenServerless will install kind for you.

Build everything locally with:

task build

This builds all the required images. It runs, in order (see Taskfile.yml:231):

StepWhat it does
gitinitinitializes git in the subrepos if missing (tags are the build's currency, so a repo without git cannot be built)
opslinkbuilds ops, symlinks it into ~/.local/bin, links oplugins into ~/.ops/$OPS_BRANCH
openwhiskbuilds the controller and invoker images
operatorbuilds the operator image
runtimesbuilds runtimes.json
streamerbuilds the streamer image
admin-apibuilds the admin-api image
opsrootrewrites oplugins/opsroot.json with the tags just built

You compile ops first, then ops builds the system.

Everything is driven by tags

All the compilations are driven by the git tag of each component repository. A build does not take a version from a variable — it reads the tag out of git, tags the image with it, and writes that same tag into opsroot.json. There are two kinds of tag, for two different purposes.

There is no way around this, not even the manual runs. Starting a workflow by hand from the GitHub Actions UI (workflow_dispatch) looks like it bypasses tagging, but under the hood it does exactly what you would have done locally: the job runs task tag and task ci on the runner, and the following step reads the tag back with git describe --tags --abbrev=0 to name the image (oplugins-op/.github/workflows/image.yml:54). So every image that exists corresponds to a tag — there is no such thing as an untagged build, and given an image tag you can always find the commit it was built from.

Development tags: the operator ref

The main build derives its tag from the git ref of the operator (Taskfile.yml:25):

RELEASE:
  sh: git -C oplugins-op rev-parse --short HEAD
OPERATOR_TAG: "{{.RELEASE}}"

Every other component inherits it — STREAMER_TAG, ADMIN_API_TAG, OPENWHISK_TAG, DEVCONTAINER_TAG and OPS_BRANCH are all {{.OPERATOR_TAG}}. One short commit hash labels the whole build, so a local build is always internally consistent.

These tags are meant for private development. They never leave your machine: the images are saved locally (under ~/.ops/<os>-<arch>/images/kind/) and loaded into kind.

Publication tags: the timestamp

To publish images you define a tag explicitly with task tag, in the component's own directory. These are the tags meant to publish images, and they are covered in full in Publishing images below.

Keeping opsroot.json in sync

Two tasks write opsroot.json, and it is worth knowing which one you are running:

  • task opsroot (part of task build) writes the tags of the images you just built locally — the operator-ref tags.

  • task sync-opsroot writes the latest published tag of each component, read from git:

    OPENWHISK_TAG:
      sh: git -C build for-each-ref --sort=-creatordate --count=1 --format='%(refname:short)' refs/tags
    

Both keep a pristine copy in opsroot.orig on first run and regenerate from it, so repeated runs are idempotent. task sync-images runs the runtimes and opsroot syncs together.

Gotcha. sync-opsroot reads whatever tags are local. A freshly cloned subrepo has no tags, for-each-ref returns the empty string, and you silently get a tagless image reference such as docker.io/apache/openserverless-wsk-controller:. If images fail to pull with an empty tag, run git fetch --tags in the offending subdirectory and re-run the task.

Developing against a local build

In development you set OPS_ROOT temporarily to your local task files, so that ops uses the oplugins in your working copy rather than the downloaded ones. This is what Taskfile.yml does when running the tests:

export PATH="$PWD:$PATH"
export OPS_ROOT="$PWD/oplugins"
export OPS_BRANCH="<the branch>"
which ops
ops -info

Always confirm with ops -info that OPS_ROOT and OPS_BIN are what you expect before concluding that a change did not work.

task opslink makes this semi-permanent: it symlinks ./ops into ~/.local/bin and your oplugins into ~/.ops/$OPS_BRANCH/oplugins, and appends OPS_BRANCH and the PATH entry to ~/.profile.

The development loop, end to end

Building for development means producing images that exist only on your machine, tagged with your operator ref, and pointing a local kind cluster at them. The full cycle:

# 1. build everything: ops, the images, and a matching opsroot.json
task build

# 2. make ops and your local task files the ones in use
export PATH="$PWD:$PATH"
export OPS_ROOT="$PWD/oplugins"

# 3. verify you are running against your own build
ops -info          # OPS_ROOT must point inside your working copy
ops -t             # list the tasks that will run

# 4. create the cluster and deploy
ops setup docker

The step that ties it together is invisible unless you go looking for it. Because these images are never pushed to a registry, task build saves each one to disk as it is built — task image-save writes it to ~/.ops/<os>-<arch>/images/kind/<base64-of-the-image-name> (Taskfile.yml:75). Later, ops setup docker calls ops util freeze kind-load, which walks that directory, decodes each filename back into an image name, and imports the layers straight into the kind node's containerd (oplugins/util/freeze/opsfile.yml:176).

So the chain from a code change to a running pod is:

your commit  →  operator ref = the tag  →  images built with that tag
             →  saved under ~/.ops/.../images/kind/
             →  opsroot.json rewritten with that tag   (task opsroot)
             →  IMAGES_* env vars                       (the CLI, on every run)
             →  kind-load imports them into the cluster (ops setup docker)
             →  the manifests reference exactly those images

Nothing is pulled from Docker Hub, because the tag in opsroot.json matches an image already present in the node.

Iterating on a single component

A full task build is rarely what you want after the first one. Each component has its own task, so rebuild just the one you touched and redeploy:

task streamer      # or: operator, admin-api, openwhisk, cli, runtimes
task opsroot       # rewrite opsroot.json with the new tags
ops setup docker   # reload the images and roll out

Two things to keep in mind:

  • task cli is a no-op once ./ops exists — it carries status: ! test -e ../ops (Taskfile.yml:95), and task opslink and task build both depend on it, so neither will rebuild the binary either. After changing the CLI, rm ./ops (or task clean) before rebuilding, or you will keep running the old one. ops -info showing a stale OPS_VERSION is the tell.
  • The tag only changes when the operator ref changes. RELEASE is git -C oplugins-op rev-parse --short HEAD, so rebuilding the streamer without committing in oplugins-op produces an image with the same tag as before. That is usually what you want — but it means kind may keep the previously imported layers, so delete the cluster (ops setup docker delete) if a change stubbornly refuses to appear.

Overriding a single image

You do not have to rebuild anything to try a different image. Because .config is just a source of env vars, editing oplugins/opsroot.json is enough:

jq '.config.images.streamer = "docker.io/apache/openserverless-streamer:some-other-tag"' \
   oplugins/opsroot.json > tmp && mv tmp oplugins/opsroot.json
ops setup docker

task opsroot and task sync-opsroot will overwrite this on their next run — they regenerate from opsroot.orig — so treat it as a temporary probe, not a durable change.

Starting from a clean slate

When the state gets confusing, the reset ladder, from cheapest to most drastic:

ops setup docker delete   # destroy the kind cluster
task clean                # remove ./ops and the cached kind images
ops -reset                # remove ~/.ops entirely (asks for confirmation)
ops -update               # re-download tasks and prerequisites

Publishing images

Everything above builds on your machine and stays there. Publishing is a different flow, and it is governed by one rule:

All CI builds are triggered by pushing a git tag. No branch push ever builds an image. Every component workflow is declared on: push: tags: with branches-ignore: '*' — the tag is not metadata attached to a build, the tag is the trigger, and its text is the image tag.

The three tasks

Every component — the operator, admin-api, streamer, devcontainer, the OpenWhisk build, the runtimes and the CLI — exposes the same trio, with the same meaning. You use them in this order:

TaskRuns whereEffectPushes?
task taglocalcreates (force-replacing) the git tag that names this buildno
task buildlocalbuilds using the current tag, into the local Docker daemonno
task cilocal, triggers remotepushes the tag, which starts the GitHub Actions buildyes, via CI

Run them in the component's own directory, not in the root — each subrepo has its own Taskfile.yml and its own workflow.

Two things newcomers consistently get wrong, so they are worth stating outright:

  • task tag deletes the existing local tags before creating the new one. This is deliberate — snapshot tags are disposable — but it is destructive, and it is not what git tag normally does.
  • task ci does not build anything. It runs git push --tags. The build happens in GitHub Actions on the repository that received the tag. This is the whole reason forking matters: the tag lands in your fork, so your Actions run, with your credentials, pushing to your registry.

There is also a manual path: every workflow declares workflow_dispatch, so it can be started from the GitHub Actions UI, where it creates the tag itself. Use it when you cannot or do not want to push tags from the command line.

The CLI is the one component with a wrinkle: task ci (alias task trigger) refuses to run on a dirty tree (git diff --exit-code), and its task tag creates a commit — it writes version.txt and branch.txt and commits them, because the CLI embeds its own version string. Every other component only moves a tag.

The tag format

Tags are timestamps in a compacted, sortable form. The encoding is uniform across all components, and it is worth learning because the tags are otherwise unreadable:

0.9.0-incubating.26i07r51-snapshot
└────┬─────────┘ ││││││└┬┘└───┬───┘
   BASETAG       ││││││ │     └───── SUFFIX: -snapshot marks a non-release build
                 ││││││ └─────────── minute
                 │││││└───────────── hour,  a letter: a=00 … x=23
                 │││└┴────────────── day of month
                 ││└──────────────── month, a letter: a=January … l=December
                 └┴───────────────── year, two digits

So 26i07r51 is 2026, September (i = 9th letter), day 07, hour 17 (r = 18th letter, zero-based → 17), minute 51.

Because the encoding is lexicographically sortable and the tags are created in order, “the newest tag” and “the latest build” are the same thing — which is exactly what task sync-opsroot relies on when it picks a tag with git for-each-ref --sort=-creatordate --count=1.

The runtimes are the exception: their tag carries a prefix naming the family to build, and CI parses it to decide what work to do (see below).

Where the images go

The destination registry and the credentials are derived automatically from the repository the tag was pushed to. In the normal case there is nothing to configure:

Tag pushed toRegistryNamespaceCredentials
the Apache repositorydocker.ioapachesecrets DOCKERHUB_USER / DOCKERHUB_TOKEN
any forkghcr.iothe fork owner's GitHub userthe current GitHub actor + the built-in GITHUB_TOKEN

So a contributor who forks and pushes a tag gets images at ghcr.io/<their-user>/… with no secrets to create and no .env to write — the workflow authenticates as the user running it, with the token GitHub hands to every Actions run.

To send images somewhere else, set the DOCKERHUB_REGISTRY secret together with DOCKERHUB_USER and DOCKERHUB_TOKEN. That is an override for people who want a specific destination, not a setup step.

The exception: repositories that publish other repositories' images

Two repositories publish images whose names do not match the repository itself:

RepositoryImages published
openserverless-buildopenwhisk2/controller, openwhisk2/invoker, openwhisk2/scheduler, openwhisk2/standalone, openwhisk2/scala
openserverless-runtimesopenserverless-runtime-<rt> — one per language version

The built-in GITHUB_TOKEN is scoped to packages owned by the repository running the workflow: GHCR links a package to its creating repository on first push and grants write access only to that repository. Since these two publish under names that do not match, GITHUB_TOKEN carries no permission and the push is refused.

For these two repositories only, when pushing to ghcr.io, a GHCR_TOKEN secret must be set on the fork — a classic Personal Access Token with the write:packages scope:

  1. https://github.com/settings/tokensGenerate new token (classic).
  2. Select write:packages (this implies read:packages and repo).
  3. In your fork: Settings → Secrets and variables → Actions → New repository secret, named exactly GHCR_TOKEN.

When pushing to Docker Hub instead, the equivalent is a DOCKERHUB_TOKEN with write permission on the target images.

The symptom when this is missing is denied: permission_denied: write_package at push time, after a long and otherwise successful build. Both workflows now fail fast with these instructions in the job summary rather than building first and failing at the end.

Naming your local build

Locally no registry is involved: each component builds under the same default image name it would use in CI, and the result exists only in your Docker daemon. To name it for your own registry instead, export MY_<COMPONENT>_IMAGE — the full image name without the tag; the tag is still computed as above. Each component ships a .env.dist to copy to .env (which stays untracked):

ComponentVariable
operatorMY_OPERATOR_IMAGE
streamerMY_STREAMER_IMAGE
admin-apiMY_ADMINAPI_IMAGE
devcontainerMY_DEVCONTAINER_IMAGE
OpenWhisk buildMY_CONTROLLER_IMAGE, MY_INVOKER_IMAGE, MY_SCHEDULER_IMAGE, MY_STANDALONE_IMAGE

What each component publishes

The trio above is identical everywhere; what differs is what one tag produces.

One tag → one imageoperator, admin-api, streamer, devcontainer. The plain case:

cd streamer
task tag        # 0.9.0-incubating.26i07r51-snapshot
task build      # local image, nothing pushed
task ci         # push the tag → CI builds linux/amd64 + linux/arm64 and pushes

CI also runs the license check (skywalking-eyes) and the unit tests before building the image, so a red build is not necessarily an image problem.

One tag → a family of imagesruntimes. The tag selects the work:

cd runtimes
task tag RT=python     # python_26i07r51-snapshot
task tag               # RT defaults to "all"

CI parses the prefix (all, common, experimental, or a single language) and builds only that family, producing one image per language version:

<registry>/<namespace>/openserverless-runtime-<rt>:<ver>-<tag>

Every runtime image is FROM the common base, so the common job runs first and the language matrix depends on it; building a language family against a missing or mismatched base is a failure mode, not a bug. This repository needs GHCR_TOKEN (see above). The resulting runtimes.json is what reaches an installation, via task sync-runtimes.

One tag → binaries, no imagecli. task build compiles ops into the repository root and is all you need locally; tagging matters only when the embedded version string does. task ci tags, commits, and pushes, and the release workflow publishes binaries for every supported platform.

The OpenWhisk imagesbuild. This one does not follow the pattern of the others and deserves its own note:

  • It vendors an OpenWhisk source tree and builds through a devcontainer rather than on the host, because the OpenWhisk Gradle build requires Java 11. On a host with a newer JDK it fails with Unsupported class file major version. Run it inside ops ide devcontainer.
  • Its task build does the whole local round trip: build the images, update opsroot.json, load into kind, roll out the controller.
  • The root Taskfile.yml and build/Taskfile.yml are separate build systems. This is the single most common wrong assumption. The root task build does invoke task openwhisk, but the two Taskfiles maintain their own tags and their own opsroot.json edits — when in doubt about which controller image is deployed, read .config.images.controller rather than inferring it.

Consuming a build you published

Once CI has pushed your images, point an installation at them by editing the image references in opsroot.json — either your working copy's oplugins/opsroot.json, or ~/.ops/$OPS_BRANCH/oplugins/opsroot.json for an installed CLI:

jq '.config.images.streamer = "ghcr.io/<your-user>/openserverless-streamer:<your-tag>"' \
   oplugins/opsroot.json > tmp && mv tmp oplugins/opsroot.json
ops setup docker

Runtimes are overridden the same way through runtimes.json.

Two failure modes worth recognising:

  • Locally built images are not automatically visible to kind. If you skip the kind-load step described earlier, the pod silently pulls the old upstream image instead. The symptom is a deployment that succeeds while running the wrong code.

  • A change visible in the custom resource is not proof it reached the workload. Confirm at the pod level:

    kubectl -n openserverless get pods -o jsonpath='{.items[*].spec.containers[*].image}'
    

Testing

Run the whole suite (defaults to kind):

task test

Run a single test — with an empty or unknown TEST it lists the available ones:

task test-one TEST=<name>.sh
task test-one TEST=

Both live in testing/tests/ and export OPS_ROOT to the local oplugins, so they always exercise your working copy.

Other useful tasks

TaskPurpose
tasklist all the tasks and print the current release
task cleanremove ./ops and the cached kind images
task licensecheck the license headers with license-eye (CMD=fix to fix them)
task sync-imagesrefresh runtimes.json and opsroot.json from the latest published tags
task release VER=<v> KEY=<gpg-key>build the signed source release tarball; the source must be in a version folder and you need a GPG key to sign

See also VERIFY.md for verifying a release, and CONTRIBUTING.md.