docs: add documentation section for migrations (#128)
* docs: add documentation section for migrations
* minor improvements
diff --git a/docs/06_extend-sdk-migrations.md b/docs/06_extend-sdk-migrations.md
new file mode 100644
index 0000000..bf822c1
--- /dev/null
+++ b/docs/06_extend-sdk-migrations.md
@@ -0,0 +1,179 @@
+---
+id: extend-sdk-migration
+title: "SDK Guide: Pipeline Element Migration"
+sidebar_label: "SDK: PE Migration"
+---
+
+Pipeline element migrations allow you to automatically update and migrate both existing and pipeline elements when a new
+version of StreamPipes is installed. This means that whenever you upgrade StreamPipes, all existing and future
+pipeline elements will be directly compatible with the new version without any manual interaction. Pipeline elements
+include adapters, data processors, and data sinks.
+
+:::info
+Migrations will make their debut in StreamPipes version `0.93.0` and will be an integral part of the system going
+forward.
+However, it's important to note that this feature is not available in any of the previous versions of StreamPipes. To
+take full advantage of migrations and their benefits, it is recommended to upgrade to version `0.93.0` or later. This
+will
+ensure that you have access to the latest enhancements and maintain compatibility with the evolving StreamPipes
+platform.
+:::
+
+## Define Migrations
+
+Whenever a pipeline element, be it an adapter, data processor, or data sink, undergoes changes that result in
+modifications to its configuration options, developers must additionally create a migration procedure. This migration
+process should be capable of smoothly transitioning all affected instances from the previous version to the new one.
+The migration itself is automatically managed and executed by StreamPipes. Developers are only responsible for two key
+aspects:
+
+* **Implementing the concrete migration**: Developers need to craft the specific migration logic that facilitates the
+ seamless transition of configuration options.
+* **Registering the migration**: Developers should register their migration procedures at the extensions service,
+ allowing StreamPipes to identify and apply the necessary updates to affected instances.
+
+By adhering to these two essential tasks, developers can ensure a hassle-free evolution of pipeline elements while
+StreamPipes handles the orchestration of the migration process.
+
+The following gives a concrete example of creating a migration for
+the [S7 adapter](./pe/org.apache.streampipes.connect.iiot.adapters.plc4x.s7.md).
+Thereby, we assume this adapter has received a new input element which determines whether the connection should be made
+authenticated or not.
+This is represented by a simple boolean that is visualized as a toggle button in the UI.
+
+### Implementing a Concrete Migration
+
+StreamPipes offers three distinct migration mechanisms tailored to specific types of pipeline
+elements: `IAdapterMigrator`, `IDataProcessorMigrator`, and `IDataSinkMigrator`.
+These migration mechanisms are presented as interfaces and require the implementation of two fundamental methods:
+
+* `config()`: This method defines the configuration for the migration, encompassing all essential metadata related to
+ the migration process.
+* `migrate()`: Within this method, the actual migration logic is to be implemented. It serves as the operational core
+ for facilitating the migration for the respective pipeline element.
+
+In accordance with the example described above, we will implement the `Plc4xS7AdapterMigrationV1` in the following.
+
+:::note
+Before we begin, it's important to familiarize ourselves with two key conventions that guide our approach to migrations:
+
+* To maintain clarity and organization, all migration classes associated with a specific pipeline element are located
+ within a dedicated sub-package named `migration`. This sub-package is nested within the package of the respective
+ pipeline element.
+* Migration classes are named according to a specific schema: `<elementClassName>MigrationV<targetVersion>`. For
+ example, if you are working on a migration for the PLC4x S7 adapter targeting version 1, the migration class would be
+ named `Plc4xS7AdapterMigrationV1`.
+:::
+
+Let's begin with providing the migration's configuration:
+
+```java
+@Override
+public ModelMigratorConfig config() {
+ return new ModelMigratorConfig(
+ "org.apache.streampipes.connect.iiot.adapters.plc4x.s7",
+ SpServiceTagPrefix.ADAPTER,
+ 0,
+ 1
+ );
+}
+```
+
+The migration config consists of the following four parts:
+
+* `targetAppId`: this needs to equal the app id of the targeted element
+* `modelType`: the type of the element to be migrated, this can be one
+ of: `SpServiceTagPrefix.ADAPTER`, `SpServiceTagPrefix.DATA_PROCESSOR`, `SpServiceTagPrefix.DATA_SINK`.
+* `fromVersion`: the version of the element that the migration expects as input
+* `toVersion`: the version the element has after the migration (needs to be at least `fromVersion + 1`)
+
+The second step is to implement the actual migration logic.
+In our example, we need to extend the existing static properties by an additional boolean property.
+
+```java
+@Override
+public MigrationResult<AdapterDescription> migrate(AdapterDescription element, IStaticPropertyExtractor extractor) throws RuntimeException {
+
+ var config = element.getConfig();
+
+ var slideToggle = new SlideToggleStaticProperty();
+ slideToggle.setDefaultValue(false);
+ slideToggle.setLabel("Authentication required?");
+ config.add(slideToggle);
+
+ element.setConfig(config);
+ return MigrationResult.success(element);
+}
+```
+
+We've completed all the necessary steps for our migration. The final task remaining is to register the migration within
+the service definition.
+
+### Registering the Migration
+
+Only when the migration is registered at the service definition, the migration is sent to the StreamPipes core service.
+Therefore, we need to add the migration to the same service definition as the element to migrate.
+In our example this is defined in `ConnectAdapterIiotInit`:
+
+```java jsx {22-24} showLineNumbers
+@Override
+public SpServiceDefinition provideServiceDefinition() {
+ return SpServiceDefinitionBuilder.create("connect-adapter-iiot",
+ "StreamPipes connect worker containing adapters relevant for the IIoT",
+ "",
+ 8001)
+ .registerAdapter(new MachineDataSimulatorAdapter())
+ .registerAdapter(new FileReplayAdapter())
+ .registerAdapter(new IfmAlMqttAdapter())
+ .registerAdapter(new RosBridgeAdapter())
+ .registerAdapter(new OpcUaAdapter())
+ .registerAdapter(new Plc4xS7Adapter())
+ .registerAdapter(new Plc4xModbusAdapter())
+ .registerAdapter(new KafkaProtocol())
+ .registerAdapter(new MqttProtocol())
+ .registerAdapter(new NatsProtocol())
+ .registerAdapter(new HttpStreamProtocol())
+ .registerAdapter(new PulsarProtocol())
+ .registerAdapter(new RocketMQProtocol())
+ .registerAdapter(new HttpServerProtocol())
+ .registerAdapter(new TubeMQProtocol())
+ .registerMigrators(
+ new Plc4xS7AdapterMigrationV1()
+ )
+ .build();
+```
+
+<br />
+
+## How Migrations are Handled Internally
+
+Migrations are handled by an interplay between the Extension Service, which provides the migrations,
+and the StreamPipes Core Service, which manages the migrations, as shown in the figure below:
+<img src="/img/06_sdk_migrations/migration-flow.png" alt="Interplay of extensions service and core to handle migrations"/>
+
+When an extensions service is initiated and has successfully registered itself with the core, it proceeds to send a
+request to the core. This request includes a comprehensive list of all available migrations that have been registered
+for it. Since this collection of migrations may encompass multiple migrations that affect the same pipeline element,
+the migrations are first de-duplicated and then sorted based on their version range before being transmitted.
+
+Upon receiving these migrations, the core's actions can be categorized into two distinct parts:
+
+* Update descriptions for new elements
+* Update descriptions for existing elements
+
+### Update Descriptions for New Elements
+
+Each migration transmitted from the extensions service to the core triggers the core to update the description of the
+corresponding element stored in CouchDB. This is achieved by requesting the current configuration from the extensions
+service and subsequently overwriting the existing configuration in the storage.
+
+### Update Descriptions for Existing Elements
+
+For each migration sent from the extensions service to the core, the core conducts a thorough check to determine if any
+existing elements are affected by this migration. If such elements are identified, the extensions service is tasked with
+requesting and subsequently executing the migration on behalf of the core.
+
+In scenarios where multiple applicable migrations exist for a single pipeline element, they are sequentially applied.
+Success in this process allows the core to seamlessly update the configuration. However, if any issues arise, the
+corresponding pipeline element is halted. In the case of processors and sinks, the associated pipeline is even marked
+with a `needs attention` label, which comes apparent in the UI.
diff --git a/website-v2/sidebars.json b/website-v2/sidebars.json
index 21ab0e4..a5d0c24 100644
--- a/website-v2/sidebars.json
+++ b/website-v2/sidebars.json
@@ -192,6 +192,7 @@
"extend-client",
"extend-sdk-functions",
"extend-sdk-event-model",
+ "extend-sdk-migration",
"extend-sdk-stream-requirements",
"extend-sdk-static-properties",
"extend-sdk-output-strategies",
diff --git a/website-v2/src/css/customTheme.css b/website-v2/src/css/customTheme.css
index 7240397..5b29da7 100644
--- a/website-v2/src/css/customTheme.css
+++ b/website-v2/src/css/customTheme.css
@@ -6,6 +6,9 @@
--ifm-color-primary-dark: #18125A;
--ifm-color-primary-darker: #171155;
--ifm-color-primary-darkest: #130E46;
+ --ifm-code-background: var(--color-primary-light);
+ --ifm-code-padding-horizontal: 0.2rem;
+ --ifm-code-padding-vertical: 0.12rem;
--color-primary: rgb(27, 20, 100);
--color-accent: rgb(57, 181, 74);
@@ -17,6 +20,10 @@
--ifm-heading-font-family: 'Roboto';
}
+code {
+ color: var(--color-accent);
+}
+
.navbar__logo {
height: 3rem;
margin-left: 0.25rem;
diff --git a/website-v2/static/img/06_sdk_migrations/migration-flow.png b/website-v2/static/img/06_sdk_migrations/migration-flow.png
new file mode 100644
index 0000000..9b9ac65
--- /dev/null
+++ b/website-v2/static/img/06_sdk_migrations/migration-flow.png
Binary files differ
