blob: 78d5a0b83807f1ccf9389e5cf180cb52ca0ddce5 [file] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Developing a storage plugin
A storage plugin implements the Registry's JDK-only SPI. It can live in a separate
repository and use a backend SDK without changing Core or the HTTP runtime.
The [standalone example](../examples/storage-plugin/README.md) is a complete,
buildable project; the [storage contract](storage-spi.md) defines the record and
failure semantics in detail.
## Project layout and dependencies
The example separates the plugin JAR from the executable application:
```text
storage-plugin/
pom.xml Independent Maven parent
plugin/ Provider, storage implementation, registration, TCK test
application/ Runtime dependencies, configuration, packaged-app test
```
| Module | Production dependencies | Test dependencies |
|---|---|---|
| Plugin | `sourcelume-registry-spi` and your backend's client libraries | `sourcelume-registry-spi-tck` |
| Application | `sourcelume-registry-runtime-quarkus` and your plugin | Quarkus integration-test support and REST Assured |
The plugin does not need the Registry parent POM, Core, Atlas, or Quarkus. Its
runtime dependency on Registry is just the SPI artifact. The TCK belongs in test
scope and supplies the JUnit contract suite. The application owns its Quarkus BOM
and build plugin; use the Quarkus version pinned by the Registry version you consume.
Keep the SPI, runtime and TCK on the same Registry version.
For example, the plugin's dependencies are:
```xml
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-spi</artifactId>
<version>${sourcelume.version}</version>
</dependency>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-spi-tck</artifactId>
<version>${sourcelume.version}</version>
<scope>test</scope>
</dependency>
```
The current snapshot artifacts must be installed locally. Follow the example's
[build commands](../examples/storage-plugin/README.md#build-and-run); they install
only the runtime, SPI, TCK and their dependencies, so developing this example does
not require an Atlas or PostgreSQL server.
## Implement the provider and plugin
The entry point is a public `RegistryPluginProvider` implementation with a public
no-argument constructor. `id()` supplies a unique provider ID; the example uses
`example`. Its constructor must not connect to the backend: ServiceLoader can
instantiate providers while the runtime is selecting one.
`open(PluginContext)` reads configuration and creates one `RegistryPlugin` instance.
The plugin owns its clients, storage and search implementations. Backend-specific
schema creation belongs in `initialize()`, which the runtime calls during startup.
| Interface and methods | Responsibility |
|---|---|
| `RegistryPluginProvider.id()` / `open(context)` | Provider identity, configuration and plugin creation |
| `RegistryPlugin.recordStore()` / `recordSearch()` | Return implementations operating on the same backend |
| `RegistryPlugin.initialize()` | Idempotently prepare backend resources and schema |
| `RegistryPlugin.isReady()` | Report whether the backend can serve requests |
| `RegistryPlugin.close()` | Release owned resources; repeated calls must be safe |
| `RecordStore.limits()` / `writeGuarantee()` | Declare enforced payload limits and conditional-write guarantees |
| `RecordStore.get(id)` / `getAll(ids)` | Authoritative reads returning complete snapshots |
| `RecordStore.write(request)` | Check the write condition and persist the complete replacement |
| `RecordSearch.search(query)` | Apply type, status and exact facet filters with the requested result limit |
The example uses two classes: `ExamplePluginProvider` and `InMemoryPlugin`.
The latter implements storage and search over one map. A backend with a client,
schema manager or substantial query logic can keep those as internal classes;
none of their types belongs in the public SPI.
The lifecycle and request paths are:
```mermaid
sequenceDiagram
participant Runtime as Shared runtime
participant Provider as RegistryPluginProvider
participant Plugin as RegistryPlugin
participant Store as RecordStore / RecordSearch
Runtime->>Provider: ServiceLoader discovery, then open(context)
Provider-->>Runtime: Plugin instance
Runtime->>Plugin: initialize()
Runtime->>Plugin: recordStore() / recordSearch()
Plugin-->>Runtime: Store and search implementations
Runtime->>Store: Record operations
Runtime->>Plugin: isReady() during health checks
Runtime->>Plugin: close() at shutdown
```
The runtime opens only the selected provider. Initialization failures are retried
according to the runtime configuration and keep readiness down if retries are
exhausted. Health checks combine initialization state with `isReady()`. The runtime
closes the plugin at shutdown; plugins must reject further storage operations.
Keep backend SDK exceptions inside the plugin and translate them into SPI failures.
## Register the provider
Create this file inside the plugin JAR:
```text
src/main/resources/META-INF/services/org.apache.sourcelume.registry.spi.RegistryPluginProvider
```
Its content is the provider's fully qualified class name:
```text
org.example.sourcelume.ExamplePluginProvider
```
No CDI annotation, custom launcher or Core registration change is needed. Select
the provider by ID in the application's `application.properties`:
```properties
sourcelume.storage.provider=example
sourcelume.storage.require-atomic-writes=true
sourcelume.example.max-document-bytes=${EXAMPLE_MAX_DOCUMENT_BYTES:4096}
```
The provider JAR and its backend dependencies must be included when building the
application. The runtime discovers providers on the application classpath at
startup. Use the ordinary plugin JAR so its service descriptor is preserved.
## Pass configuration to the plugin
`StorageProducer` collects `sourcelume.<provider-id>.*` settings and strips that
prefix before creating `PluginContext`. In the example,
`sourcelume.example.max-document-bytes` becomes `max-document-bytes`:
```java
int maxDocumentBytes = Integer.parseInt(
context.properties().getOrDefault("max-document-bytes", "4194304"));
StoreLimits limits = StoreLimits.builder()
.maxDocumentBytes(maxDocumentBytes)
.build();
```
Declare provider keys and environment-variable mappings in the application's
configuration, as above. This gives configuration enumeration an unambiguous
property name, including keys containing hyphens. Validate required values in
`open()` and fail clearly if they are missing or invalid. Do not log the properties
map: it may contain credentials. The provider should consume `PluginContext`
rather than reading framework configuration or environment variables itself.
## Preserve the storage contract
Core supplies a `RecordEnvelope` containing the exact document bytes, display
attributes, validation metadata, extracted facets and opaque attachments. Store
that snapshot without reserializing the document or dropping fields. An update
replaces the whole snapshot, including clearing removed facets and attachments.
An attachment is bound to the document digest; its signature format is outside the
storage plugin's responsibility.
Writes carry either `WriteCondition.absent()` or `WriteCondition.atRevision(...)`.
The plugin enforces the condition, domain-type immutability and the immutability of
validated records. Return the confirmed stored snapshot and its opaque revision.
A rejected condition throws `RecordConflictException` with the current snapshot
when one exists. See the [condition table](storage-spi.md#conditions-and-failures)
for the required outcomes.
Declare `ATOMIC_CONDITIONAL` only when the condition check and complete write are
atomic for all writers sharing that store. The example synchronizes access to a
single in-process map. That lock is not a concurrency mechanism for a shared remote
backend; use that backend's transaction or conditional-write support. If the
backend provides only `CHECK_THEN_WRITE`, document its writer restrictions. The
runtime rejects it when `sourcelume.storage.require-atomic-writes=true`.
`get()` returns `Optional.empty()` only when the record is missing. Backend failures
must throw. The default `getAll()` performs point reads; a plugin may override it
with an authoritative batch lookup. Missing IDs are omitted, duplicates are read
once, and a failure rejects the entire call. Each record must be a complete
snapshot; a search index is not an authoritative batch-read source.
Search can lag point reads. It must honor exact facet matching, OR within a facet,
AND across facets, and the result limit. Core extracts the facet values; plugins
persist and index them. An unsupported operation or failed query must not look
like a successful empty result. Storage and facet search are required by the current
SPI; graph support is a separate future contract.
Use `StorageException` for backend failures. `UNKNOWN_WRITE` means a mutation may
have committed but its result cannot be established. Do not turn a timeout into a
successful write or a definitive conflict without sufficient backend evidence.
Enforce `StoreLimits` before mutation and before consuming large backend resources.
The [PostgreSQL](../sourcelume-registry-postgres/src/main/java/org/apache/sourcelume/registry/postgres/)
and [Atlas](../sourcelume-registry-atlas-adapter/src/main/java/org/apache/sourcelume/registry/atlas/adapter/)
implementations show transaction-based and single-writer approaches respectively.
## Run the contract tests and package the application
Extend `RecordStoreContract` in your plugin's test sources. `createPlugin()` must
return an initialized plugin backed by an empty, isolated store for each test:
```java
class ExamplePluginContractTest extends RecordStoreContract {
@Override
protected RegistryPlugin createPlugin() {
RegistryPlugin plugin = new ExamplePluginProvider()
.open(PluginContext.builder().build());
plugin.initialize();
return plugin;
}
}
```
The base test closes the plugin after each test. For a persistent backend, create a
fresh test namespace or reset a dedicated test database; never use a live registry.
The TCK covers byte preservation, complete replacement, revisions, conflicts,
concurrent writes, batch reads, payload limits, search filters and limits, repeated
initialization, and access after close. Limit tests accept the declared boundary
and reject oversized creates and corrections without changing the stored snapshot.
Search assertions allow index lag and do not require a particular result order.
Use modest limits in the test instance: to bound test allocations, the TCK skips
a payload boundary test above 16 MiB or an attachment count test above 1024.
The combined attachment byte test also skips stores that allow no attachments.
Check the test reports for skips; a skipped dimension has not been verified.
Backend-specific tests must also cover failures and guarantees that a shared
contract test cannot induce, such as a lost commit acknowledgement or a partial
schema migration.
The application module depends on the runtime JAR and your plugin JAR, imports the
Quarkus BOM, and binds `quarkus-maven-plugin:build`. It needs no HTTP resource
classes: the runtime already provides the record API, validators, health checks
and shutdown handling. The example's `PluginApplicationIT` runs the resulting
fast-jar and verifies that the registered provider serves requests and receives
its configuration. It also checks that TCK, JUnit, Atlas and PostgreSQL libraries
are absent from this application's production package.
When packaging another backend, replace the example's implementation and provider
ID, add its SDK dependency to the plugin module, and declare its configuration in
the application. Run both the TCK and the packaged application test before shipping.
## Troubleshooting
| Symptom | Check |
|---|---|
| `No storage provider named ...` | The configured ID, plugin dependency, provider class name and service descriptor in the built JAR |
| `Multiple storage providers named ...` | Two provider implementations on the classpath use the same ID |
| An environment setting has no effect | The application declares the matching `sourcelume.<id>.*` property and the provider reads the stripped key |
| Startup rejects atomic-write requirements | The selected store's `writeGuarantee()` and the distribution's atomic-write policy |
| Readiness stays down | Initialization logs, backend connectivity and the plugin's `isReady()` implementation |
| Tests run but HTTP startup fails | The application packages the plugin and imports the Quarkus BOM matching the Registry runtime |