manual/src/main/asciidoc/new-data-model.adoc

//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//

=== Data Model changes for Apache Unomi 1.5.0

==== Data model and ElasticSearch 7

Since Apache Unomi version 1.5.0 we decided to upgrade the supported ElasticSearch version to the latest 7.4.2.

To be able to do so, we had to rework the way the data was stored inside ElasticSearch.

Previously every items was stored inside the same ElasticSearch index but this is not allowed anymore in recent ElasticSearch versions.

Since Apache Unomi version 1.5.0 every type of items (see section: link:#_items_and_types[Items and types]) is now stored in a dedicated separated index.


==== API changes

To be able to handle the multiple indices the Persistence API implementation
(https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java[ElasticSearchPersistenceServiceImpl])
have been adapted and simplified.

The good news is that there is no API changes, the persistence API interface didn't changed.

Any custom Apache Unomi plugins or extensions should continue to work on Apache Unomi 1.5.0.

The only notable changes are located at the
https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java[ElasticSearchPersistenceServiceImpl Java class].
This class should not be use directly, instead you should use OSGI service dependency injection using the interface https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/persistence-spi/src/main/java/org/apache/unomi/persistence/spi/PersistenceService.java[PersistenceService].

But if you are interested in the implementation changes:

. The property `index.name` have been renamed to `index.prefix`.
Previously used for the single one index name, now every index is prefixed using this property. (`context-` by default)
. We removed the property `index.names` originally used to create additional indices (used by the geonames DB for exemple).
This property is not needed anymore because the index is automatically created by the peristence service when the mapping configuration is loaded.
Example of mapping configuration file: (https://github.com/apache/unomi/blob/9f1bab437fd93826dc54d318ed00d3b2e3161437/extensions/geonames/services/src/main/resources/META-INF/cxs/mappings/geonameEntry.json[geoname index mapping])

Because of this changes the geonames DB index name is now respecting the index naming with prefix like any other item type.
Previously named: `geonames` is now using the index name `context-geonameentry`
(see: link:#_installing_geonames_database[Documentation about geonames extension]).

===== Migration

In order to migrate the data from ElasticSearch 5 to 7, Unomi provides a migration tool that is directly integrated.

Note that it is possible to do the migration procedure on a single machine, but you will need to change the ports on one
of the ElasticSearch cluster. In the following example we have changed the ports on the `source` cluster.

So in the `config/elasticsearch.yml` file we have modified the default ports to:

    transport.tcp.port: 9310
    http.port: 9210

Make SURE you change the ports out of the default 9200-9205 and 9300-9305 range (or whatever your cluster uses) otherwise
both clusters will attempt to merge!

On the destination ElasticSearch cluster configuration you will need to add the following setting in the `config/elasticsearch.yml`:

    reindex.remote.whitelist: "localhost:9210"

Replace "localhost:9210" which whatever location your `source` cluster is available at.

Important: Make sure you haven't started Apache Unomi before (using the `unomi:start` command) otherwise you will need to
restart your Apache Unomi installation from scratch.

You can then start both instances of ElasticSearch 5 and ElasticSearch 7 and finally start Apache Unomi using:

    ./karaf

Once in the console launch the migration using the following command:

    migrate 1.4.0

Follow the instructions and answer the prompts. If you used the above configuration as an example you can simply use the
default values. Note that it is also possible to change the index prefix to be different from the default `context` value
so that you could host multiple Apache Unomi instances on the same ElasticSearch cluster.