manual/src/main/asciidoc/new-data-model.adoc

//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//

=== Data Model changes for Apache Unomi 1.5.0

==== Data model and ElasticSearch 7

Since Apache Unomi version 1.5.0 we decided to upgrade the supported ElasticSearch version to the latest 7.4.2.

To be able to do so, we had to rework the way the data was stored inside ElasticSearch.

Previously every items was stored inside the same ElasticSearch index but this is not allowed anymore in recent ElasticSearch versions.

Since Apache Unomi version 1.5.0 every type of items (see section: link:#_items_and_types[Items and types]) is now stored in a dedicated separated index.


==== API changes

To be able to handle the multiple indices the Persistence API implementation
(https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java[ElasticSearchPersistenceServiceImpl])
have been adapted and simplified.

The good news is that there is no API changes, the persistence API interface didn't changed.

Any custom Apache Unomi plugins or extensions should continue to work on Apache Unomi 1.5.0.

The only notable changes are located at the
https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java[ElasticSearchPersistenceServiceImpl Java class].
This class should not be use directly, instead you should use OSGI service dependency injection using the interface https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-spi/src/main/java/org/apache/unomi/persistence/spi/PersistenceService.java[PersistenceService].

But if you are interested in the implementation changes:

. The property `index.name` have been renamed to `index.prefix`.
Previously used for the single one index name, now every index is prefixed using this property. (`context-` by default)
. We removed the property `index.names` originally used to create additional indices (used by the geonames DB for exemple).
This property is not needed anymore because the index is automatically created by the peristence service when the mapping configuration is loaded.
Example of mapping configuration file: (https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/extensions/geonames/services/src/main/resources/META-INF/cxs/mappings/geonameEntry.json[geoname index mapping])

Because of this changes the geonames DB index name is now respecting the index naming with prefix like any other item type.
Previously named: `geonames` is now using the index name `context-geonameentry`
(see: link:#_installing_geonames_database[Documentation about geonames extension]).

===== Migration

In order to migrate the data from ElasticSearch 5 to 7, Unomi provides a migration tool that is directly integrated.

It is HIGHLY RECOMMENDED to perform a full cluster backup/snapshot of the `source` cluster, and ideally to perform the
migration on a restore of the `source` cluster. For more information on ElasticSearch 5 snapshots and restore you can
find it here: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-snapshots.html

Note that it is possible to do the migration procedure on a single machine, but you will need to change the ports on one
of the ElasticSearch cluster. In the following example we have changed the ports on the `source` cluster, which requires
a cluster restart to take effect. It is also possible to change the ports on the `target` cluster but those would have
to be then changed again to your final setting.

So in the `source` 's ElasticSearch 5 `config/elasticsearch.yml` file we have modified the default ports to:

    transport.tcp.port: 9310
    http.port: 9210

Make SURE you change the ports out of the default 9200-9205 and 9300-9305 range (or whatever your cluster uses) otherwise
both clusters will attempt to merge!

On the `target` ElasticSearch 7 cluster configuration you will need to add the following setting in the `config/elasticsearch.yml`:

    reindex.remote.whitelist: "localhost:9210"

Replace "localhost:9210" which whatever location your `source` cluster is available at. Restart or start your
`target` ElasticSearch 7 cluster.

Important: Make sure you haven't started Apache Unomi before (using the `unomi:start` command or the autostart command
line parameter) otherwise you will need to restart your Apache Unomi installation from scratch. The best way to be sure
of that is to start a new Unomi install by uncompressing the archive and not launching it.

You can then start both instances of ElasticSearch 5 and ElasticSearch 7 and finally start Apache Unomi using:

    ./karaf

Once in the console launch the migration using the following command:

    migrate 1.4.0

Follow the instructions and answer the prompts. If you used the above configuration as an example you can simply use the
default values.

Be careful because the first address that the tool will ask for is the `target` (ElasticSearch 7) cluster, not the
ES 5 one.

Note that it is also possible to change the index prefix to be different from the default `context` value
so that you could host multiple Apache Unomi instances on the same ElasticSearch cluster.