manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc - unomi - Git at Google

 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
 // You may obtain a copy of the License at
 //
 //      http://www.apache.org/licenses/LICENSE-2.0
 //
 // Unless required by applicable law or agreed to in writing, software
 // distributed under the License is distributed on an "AS IS" BASIS,
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
 //

 === Migration Overview

 Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0.

 There are two main steps in preparing your migration to Apache Unomi 2.0:
 - Updating applications consuming Unomi
 - Migrating your existing data

 === Updating applications consuming Unomi

 Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi.

 As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>.
 Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues.

 ==== Data Model changes

 There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual.

 ==== Create JSON schemas

 Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas.

 Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema.
 Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event you may be using.

 When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console),
 will point to errors in your schemas or will help you diagnose why the events are not being accepted.

 Note that it is currently not possible to modify or surcharge an existing system-deployed JSON schema via the REST API. It is however possible to deploy new schemas and manage them through the REST API on the `/cxs/jsonSchema` endpoint.
 If you are currently using custom properties on an Apache Unomi-provided event type,
 you will need to either change to use a new custom eventType and create the corresponding schema or to create a Unomi schema extension. You can find more details in the <<JSON schemas,JSON Schema>> section of this documentation.

 You can use, as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at:
  https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas].

 Finally, and although it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. This requires making sure that you don't allow open properties by using JSON schema keywords such as https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties[unevaluated properties]

 === Migrating your existing data

 ==== Elasticsearch version and capacity

 While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5.
 Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website].

 Your Elasticsearch cluster must have enough capacity to handle the migration.
 At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index.

 ==== Migrate custom data

 Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment.

 It relies on a set of groovy scripts to perform its data migration,
 located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration],
 these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own.

 In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes.

 Depending of the volume of data, migration can be lengthy. By paying attention to when re-indexation is happening (triggered in the groovy scripts by `MigrationUtils.reIndex()`),
 you can find the most appropriate time for your scritps to be executed and avoid re-indexing the same indices multiple times.

 For example if you wanted to update profiles with custom data (currently migrated by `migrate-2.0.0-10-profileReindex.groovy`), you could create a script in position "09" that would only contain painless scripts without a reindexing step.
 The script in position "10" will introduce its own painless script, then trigger the re-indexation. This way you don't have to re-index the same indices twice.

 You can find existing painless scripts in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/requestBody/2.0.0[tools/shell-commands/src/main/resources/requestBody/2.0.0]

 At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially.

 ==== Perform the migration

 ===== Checklist

 Before starting the migration, please ensure that:

  - You do have a backup of your data
  - You did practice the migration in a staging environment, NEVER migrate a production environment without prior validation
  - You verified your applications were operational with Apache Unomi 2.0 (JSON schemas created, client applications updated, ...)
  - You are running Elasticsearch 7.17.5 (or a later 7.x version)
  - Your Elasticsearch cluster has enough capacity to handle the migration
  - You are currently running Apache Unomi 1.6 (or a later 1.x version)
  - You will be using the same Apache Unomi instance for the entire migration progress. Do not start the migration on one node, and resume an interrupted migration on another node.

 ===== Migration process overview

 The migration is performed by means of a dedicated Apache Unomi 2.0 node started in a particular migration mode.

 In a nutshell, the migration process will consist in the following steps:

 - Shutdown your Apache Unomi 1.6 cluster
 - Start an Apache Unomi 2.0 migration node
 - Wait for data migration to complete
 - Start you Apache Unomi 2.0 cluster
 - (optional) Import additional JSON Schemas

 Each migration step maintains its execution state, meaning if a step fails you can fix the issue, and resume the migration from the failed step.

 ===== Configuration

 The following environment variables are used for the migration:

 |===
 |Environment Variable|Unomi Setting|Default

 |UNOMI_ELASTICSEARCH_ADDRESSES
 |org.apache.unomi.elasticsearch.addresses
 |localhost:9200

 |UNOMI_ELASTICSEARCH_SSL_ENABLE
 |org.apache.unomi.elasticsearch.sslEnable
 |false

 |UNOMI_ELASTICSEARCH_USERNAME
 |org.apache.unomi.elasticsearch.username
 |

 |UNOMI_ELASTICSEARCH_PASSWORD
 |org.apache.unomi.elasticsearch.password
 |

 |UNOMI_ELASTICSEARCH_SSL_TRUST_ALL_CERTIFICATES
 |org.apache.unomi.elasticsearch.sslTrustAllCertificates
 |false

 |UNOMI_ELASTICSEARCH_INDEXPREFIX
 |org.apache.unomi.elasticsearch.index.prefix
 |context

 |UNOMI_MIGRATION_RECOVER_FROM_HISTORY
 |org.apache.unomi.migration.recoverFromHistory
 |true

 |===

 If there is a need for advanced configuratiion, the configuration file used by Apache Unomi 2.0 is located in: `etc/org.apache.unomi.migration.cfg`

 ===== Migrate manually

 You can migrate manually using the Karaf console.

 After having started Apache Unomi 2.0 with the `./karaf` command, you will be presented with the Karaf shell.

 From there you have two options:

  - The necessary configuration variables (see above) have already been set, you can start the migration using the command: `unomi:migrate 1.6.0`
  - Or, you want to provide the configuration settings interactively via the terminal, in that case you can start the migration in interactive mode using: `unomi:migrate 1.6.0`

 The parameter of the migrate command (1.6.0 in the example above) corresponds to the version you're migrating from.

 At the end of the migration, you can start Unomi 2.0 as usual using: `unomi:start`.

 ===== Migrate with Docker

 The migration can also be performed using Docker images, the migration itself can be started by passing a specific value to the `KARAF_OPTS` environment variable.

 In the context of this migration guide, we will asssume that:

  - Custom migration scripts are located in `/home/unomi/migration/scripts/`
  - Painless scripts, or more generally any migration assets are located in `/home/unomi/migration/assets/`, these scripts will be mounted under `/tmp/assets/` inside the Docker container.

 [source]
 ----
 docker run \
     -e UNOMI_ELASTICSEARCH_ADDRESSES=localhost:9200 \
     -e KARAF_OPTS="-Dunomi.autoMigrate=1.6.0" \
     --v /home/unomi/migration/scripts/:/opt/apache-unomi/data/migration/scripts \
     --v /home/unomi/migration/assets/:/tmp/assets/ \
     apache/unomi:2.0.0-SNAPSHOT
 ----

 You might need to provide additional variables (see table above) depending of your environment.

 If the migration fails, you can simply restart this command.

 Using the above command, Unomi 2.0 will not start automatically at the end of the migration. You can start Unomi automatically at the end of the migration by passing: `-e KARAF_OPTS="-Dunomi.autoMigrate=1.6.0 -Dunomi.autoStart=true"`

 ===== Step by step migration with Docker

 Once your cluster is shutdown, performing the migration will be as simple as starting a dedicated docker container.

 ===== Post Migration

 Once the migration has been executed, you will be able to start Apache Unomi 2.0

 Remember you still need to submit JSON schemas corresponding to your events, you can do so using the API.
	//
	// Licensed under the Apache License, Version 2.0 (the "License");
	// you may not use this file except in compliance with the License.
	// You may obtain a copy of the License at
	//
	// http://www.apache.org/licenses/LICENSE-2.0
	//
	// Unless required by applicable law or agreed to in writing, software
	// distributed under the License is distributed on an "AS IS" BASIS,
	// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	// See the License for the specific language governing permissions and
	// limitations under the License.
	//

	=== Migration Overview

	Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0.

	There are two main steps in preparing your migration to Apache Unomi 2.0:
	- Updating applications consuming Unomi
	- Migrating your existing data

	=== Updating applications consuming Unomi

	Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi.

	As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>.
	Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues.

	==== Data Model changes

	There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual.

	==== Create JSON schemas

	Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas.

	Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema.
	Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event you may be using.

	When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console),
	will point to errors in your schemas or will help you diagnose why the events are not being accepted.

	Note that it is currently not possible to modify or surcharge an existing system-deployed JSON schema via the REST API. It is however possible to deploy new schemas and manage them through the REST API on the `/cxs/jsonSchema` endpoint.
	If you are currently using custom properties on an Apache Unomi-provided event type,
	you will need to either change to use a new custom eventType and create the corresponding schema or to create a Unomi schema extension. You can find more details in the <<JSON schemas,JSON Schema>> section of this documentation.

	You can use, as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at:
	https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas].

	Finally, and although it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. This requires making sure that you don't allow open properties by using JSON schema keywords such as https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties[unevaluated properties]

	=== Migrating your existing data

	==== Elasticsearch version and capacity

	While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5.
	Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website].

	Your Elasticsearch cluster must have enough capacity to handle the migration.
	At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index.

	==== Migrate custom data

	Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment.

	It relies on a set of groovy scripts to perform its data migration,
	located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration],
	these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own.

	In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes.

	Depending of the volume of data, migration can be lengthy. By paying attention to when re-indexation is happening (triggered in the groovy scripts by `MigrationUtils.reIndex()`),
	you can find the most appropriate time for your scritps to be executed and avoid re-indexing the same indices multiple times.

	For example if you wanted to update profiles with custom data (currently migrated by `migrate-2.0.0-10-profileReindex.groovy`), you could create a script in position "09" that would only contain painless scripts without a reindexing step.
	The script in position "10" will introduce its own painless script, then trigger the re-indexation. This way you don't have to re-index the same indices twice.

	You can find existing painless scripts in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/requestBody/2.0.0[tools/shell-commands/src/main/resources/requestBody/2.0.0]

	At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially.

	==== Perform the migration

	===== Checklist

	Before starting the migration, please ensure that:

	- You do have a backup of your data
	- You did practice the migration in a staging environment, NEVER migrate a production environment without prior validation
	- You verified your applications were operational with Apache Unomi 2.0 (JSON schemas created, client applications updated, ...)
	- You are running Elasticsearch 7.17.5 (or a later 7.x version)
	- Your Elasticsearch cluster has enough capacity to handle the migration
	- You are currently running Apache Unomi 1.6 (or a later 1.x version)
	- You will be using the same Apache Unomi instance for the entire migration progress. Do not start the migration on one node, and resume an interrupted migration on another node.

	===== Migration process overview

	The migration is performed by means of a dedicated Apache Unomi 2.0 node started in a particular migration mode.

	In a nutshell, the migration process will consist in the following steps:

	- Shutdown your Apache Unomi 1.6 cluster
	- Start an Apache Unomi 2.0 migration node
	- Wait for data migration to complete
	- Start you Apache Unomi 2.0 cluster
	- (optional) Import additional JSON Schemas

	Each migration step maintains its execution state, meaning if a step fails you can fix the issue, and resume the migration from the failed step.

	===== Configuration

	The following environment variables are used for the migration:

	\|===
	\|Environment Variable\|Unomi Setting\|Default

	\|UNOMI_ELASTICSEARCH_ADDRESSES
	\|org.apache.unomi.elasticsearch.addresses
	\|localhost:9200

	\|UNOMI_ELASTICSEARCH_SSL_ENABLE
	\|org.apache.unomi.elasticsearch.sslEnable
	\|false

	\|UNOMI_ELASTICSEARCH_USERNAME
	\|org.apache.unomi.elasticsearch.username
	\|

	\|UNOMI_ELASTICSEARCH_PASSWORD
	\|org.apache.unomi.elasticsearch.password
	\|

	\|UNOMI_ELASTICSEARCH_SSL_TRUST_ALL_CERTIFICATES
	\|org.apache.unomi.elasticsearch.sslTrustAllCertificates
	\|false

	\|UNOMI_ELASTICSEARCH_INDEXPREFIX
	\|org.apache.unomi.elasticsearch.index.prefix
	\|context

	\|UNOMI_MIGRATION_RECOVER_FROM_HISTORY
	\|org.apache.unomi.migration.recoverFromHistory
	\|true

	\|===

	If there is a need for advanced configuratiion, the configuration file used by Apache Unomi 2.0 is located in: `etc/org.apache.unomi.migration.cfg`

	===== Migrate manually

	You can migrate manually using the Karaf console.

	After having started Apache Unomi 2.0 with the `./karaf` command, you will be presented with the Karaf shell.

	From there you have two options:

	- The necessary configuration variables (see above) have already been set, you can start the migration using the command: `unomi:migrate 1.6.0`
	- Or, you want to provide the configuration settings interactively via the terminal, in that case you can start the migration in interactive mode using: `unomi:migrate 1.6.0`

	The parameter of the migrate command (1.6.0 in the example above) corresponds to the version you're migrating from.

	At the end of the migration, you can start Unomi 2.0 as usual using: `unomi:start`.

	===== Migrate with Docker

	The migration can also be performed using Docker images, the migration itself can be started by passing a specific value to the `KARAF_OPTS` environment variable.

	In the context of this migration guide, we will asssume that:

	- Custom migration scripts are located in `/home/unomi/migration/scripts/`
	- Painless scripts, or more generally any migration assets are located in `/home/unomi/migration/assets/`, these scripts will be mounted under `/tmp/assets/` inside the Docker container.

	[source]
	----
	docker run \
	-e UNOMI_ELASTICSEARCH_ADDRESSES=localhost:9200 \
	-e KARAF_OPTS="-Dunomi.autoMigrate=1.6.0" \
	--v /home/unomi/migration/scripts/:/opt/apache-unomi/data/migration/scripts \
	--v /home/unomi/migration/assets/:/tmp/assets/ \
	apache/unomi:2.0.0-SNAPSHOT
	----

	You might need to provide additional variables (see table above) depending of your environment.

	If the migration fails, you can simply restart this command.

	Using the above command, Unomi 2.0 will not start automatically at the end of the migration. You can start Unomi automatically at the end of the migration by passing: `-e KARAF_OPTS="-Dunomi.autoMigrate=1.6.0 -Dunomi.autoStart=true"`

	===== Step by step migration with Docker

	Once your cluster is shutdown, performing the migration will be as simple as starting a dedicated docker container.

	===== Post Migration

	Once the migration has been executed, you will be able to start Apache Unomi 2.0

	Remember you still need to submit JSON schemas corresponding to your events, you can do so using the API.