manual/src/main/asciidoc/whats-new.adoc - unomi - Git at Google

 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
 // You may obtain a copy of the License at
 //
 //      http://www.apache.org/licenses/LICENSE-2.0
 //
 // Unless required by applicable law or agreed to in writing, software
 // distributed under the License is distributed on an "AS IS" BASIS,
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
 //
 === What's new in Apache Unomi 2.0

 Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product.
 The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which presented an opportunity for noticeable improvements in the overall performance.

 This new release also introduces a first (beta) version of the Unomi GraphQL API.

 ==== Scopes declarations are now required

 Scopes declarations are now required in Unomi 2. When submitting an event and specifying a scope,
 that scope must already be declared on the platform.

 Scopes can be easily created via the corresponding REST API (`cxs/scopes`)

 For example, an "apache" scope can be created using the following API call.
 [source]
 ----
 curl --location --request POST 'http://localhost:8181/cxs/scopes' \
 -u 'karaf:karaf' \
 --header 'Content-Type: application/json' \
 --data-raw '{
 "itemId": "apache",
 "itemType": "scope"
 }'
 ----

 ==== JSON Schemas

 Apache Unomi 2 introduces support for https://json-schema.org/specification.html[JSON Schema] for all of its publicly exposed endpoints.
 Data received by Apache Unomi 2 will first be validated against a known schema to make sure it complies with an expected payload.
 If the received payload does not match a known schema, it will be rejected by Apache Unomi 2.

 Apache Unomi 2 also introduces a set of administrative endpoints allowing new schemas and/or schemas extensions to be registered.

 More details about JSON Schemas implementation are available in the <<JSON schemas,corresponding section>> of the documentation.

 ==== Updated data model

 The introduction of JSON schema required us to modify Apache Unomi data model, One of the key differences is the removal of open maps.

 The properties field in the events objects provided by unomi are now restricted by JSON schema.
 This means object properties must be declared in a JSON schema for an event to be accepted.

 A new property, flattenedProperties has been introduced to the event object, this property has been added to store the properties as
 flattened in Elasticsearch and should avoid mapping explosion for dynamic properties.

 If there is dynamic properties that you want to send with your event, you should use the flattenedProperties field of the event.

 It's also necessary to specify the format of the values which are added to flattenedProperties by JSON schema but these value will be
 stored as flattened and will not create dynamic mapping contrary to the properties field of the events.

 Here is an example for objects that used dynamic properties for URL parameters:

 The following event example in Apache Unomi 1.x:
 [source]
 ----
 {
     "eventType":"view",
     "scope":"digitall",
     "properties":{
         "URLParameters":{
            "utm_source":"source"
         }
     },
     "target":{
         "scope":"digitall",
         "itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
         "itemType":"page",
         "properties":{
            "pageInfo":{
               "pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
               "nodeType":"jnt:page",
               "pageName":"Home",
               ...
            },
            "attributes":{},
            "consentTypes":[]
         }
     },
     "source":{
         "scope":"digitall",
         "itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
         "itemType":"site"
     }
 }
 ----

 Is replaced by the following in Apache Unomi 2.x:
 [source]
 ----
 {
     "eventType":"view",
     "scope":"digitall",
     "flattenedProperties":{
         "URLParameters":{
            "utm_source":"source"
         }
     },
     "target":{
         "scope":"digitall",
         "itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
         "itemType":"page",
         "properties":{
            "pageInfo":{
               "pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
               "nodeType":"jnt:page",
               "pageName":"Home",
               ...
            },
            "attributes":{},
            "consentTypes":[]
         }
     },
     "source":{
         "scope":"digitall",
         "itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
         "itemType":"site"
     }
 }
 ----

 If using the default Apache 1.x data model, our Unomi 2 migration process will handle the data model changes for you.
 If you are using custom events/objects, please refer to the detailed migration guide for more details.

 ==== Removal of the Web Tracker

 [TODO: Add more details about the web tracker]

 Apache Unomi 2.0 Web Tracker, previously located in `extensions/web-tracker/` has been removed.
 We considered it as outdated and instead recommend implementing your own tracker logic based on your project
 use case.

 ==== GraphQL API - beta

 Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API.
 Available behind a feature flag (the API disabled by default), the GraphQL API is available for you to play with.

 More details about how to enable/disable the GraphQL API are available in the <<GraphQL API,corresponding section>> of the documentation.

 We welcome tickets/PRs to improve its robustness and progressively make it ready for prime time.

 ==== Migrate from Unomi 1.x

 To facilitate migration we prepared a set of scripts that will automatically handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.

 It is worth keeping in mind that for Apache Unomi 2.0 we do not support “hot” migration,
 the migration process will require a shutdown of your cluster to guarantee that no new events will be collected while data migration is in progress.

 Special caution must be taken if you declared custom events as our migration scripts can only handle objects we know of.
 More details about migration (incl. of custom events) is available in the corresponding section <<Migrations,corresponding section>> of the documentation.

 ==== Elasticsearch compatibility

 We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0,
 this ensure you are on a recent version that is not impacted by the log4j vulnerabilities (fixed in Elasticsearch 7.16.3).

 This version increase is releated to Apache Unomi 2.0 makeing use of a new Elasticsearch field type
 called https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html[Flattened],
 and although it was available in prior versions of Elasticsearch, we do not recommend using those
 due to the above-mentioned log4j vulnerabilities.
	//
	// Licensed under the Apache License, Version 2.0 (the "License");
	// you may not use this file except in compliance with the License.
	// You may obtain a copy of the License at
	//
	// http://www.apache.org/licenses/LICENSE-2.0
	//
	// Unless required by applicable law or agreed to in writing, software
	// distributed under the License is distributed on an "AS IS" BASIS,
	// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	// See the License for the specific language governing permissions and
	// limitations under the License.
	//
	=== What's new in Apache Unomi 2.0

	Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product.
	The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which presented an opportunity for noticeable improvements in the overall performance.

	This new release also introduces a first (beta) version of the Unomi GraphQL API.

	==== Scopes declarations are now required

	Scopes declarations are now required in Unomi 2. When submitting an event and specifying a scope,
	that scope must already be declared on the platform.

	Scopes can be easily created via the corresponding REST API (`cxs/scopes`)

	For example, an "apache" scope can be created using the following API call.
	[source]
	----
	curl --location --request POST 'http://localhost:8181/cxs/scopes' \
	-u 'karaf:karaf' \
	--header 'Content-Type: application/json' \
	--data-raw '{
	"itemId": "apache",
	"itemType": "scope"
	}'
	----

	==== JSON Schemas

	Apache Unomi 2 introduces support for https://json-schema.org/specification.html[JSON Schema] for all of its publicly exposed endpoints.
	Data received by Apache Unomi 2 will first be validated against a known schema to make sure it complies with an expected payload.
	If the received payload does not match a known schema, it will be rejected by Apache Unomi 2.

	Apache Unomi 2 also introduces a set of administrative endpoints allowing new schemas and/or schemas extensions to be registered.

	More details about JSON Schemas implementation are available in the <<JSON schemas,corresponding section>> of the documentation.

	==== Updated data model

	The introduction of JSON schema required us to modify Apache Unomi data model, One of the key differences is the removal of open maps.

	The properties field in the events objects provided by unomi are now restricted by JSON schema.
	This means object properties must be declared in a JSON schema for an event to be accepted.

	A new property, flattenedProperties has been introduced to the event object, this property has been added to store the properties as
	flattened in Elasticsearch and should avoid mapping explosion for dynamic properties.

	If there is dynamic properties that you want to send with your event, you should use the flattenedProperties field of the event.

	It's also necessary to specify the format of the values which are added to flattenedProperties by JSON schema but these value will be
	stored as flattened and will not create dynamic mapping contrary to the properties field of the events.

	Here is an example for objects that used dynamic properties for URL parameters:

	The following event example in Apache Unomi 1.x:
	[source]
	----
	{
	"eventType":"view",
	"scope":"digitall",
	"properties":{
	"URLParameters":{
	"utm_source":"source"
	}
	},
	"target":{
	"scope":"digitall",
	"itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
	"itemType":"page",
	"properties":{
	"pageInfo":{
	"pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
	"nodeType":"jnt:page",
	"pageName":"Home",
	...
	},
	"attributes":{},
	"consentTypes":[]
	}
	},
	"source":{
	"scope":"digitall",
	"itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
	"itemType":"site"
	}
	}
	----

	Is replaced by the following in Apache Unomi 2.x:
	[source]
	----
	{
	"eventType":"view",
	"scope":"digitall",
	"flattenedProperties":{
	"URLParameters":{
	"utm_source":"source"
	}
	},
	"target":{
	"scope":"digitall",
	"itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
	"itemType":"page",
	"properties":{
	"pageInfo":{
	"pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
	"nodeType":"jnt:page",
	"pageName":"Home",
	...
	},
	"attributes":{},
	"consentTypes":[]
	}
	},
	"source":{
	"scope":"digitall",
	"itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
	"itemType":"site"
	}
	}
	----

	If using the default Apache 1.x data model, our Unomi 2 migration process will handle the data model changes for you.
	If you are using custom events/objects, please refer to the detailed migration guide for more details.

	==== Removal of the Web Tracker

	[TODO: Add more details about the web tracker]

	Apache Unomi 2.0 Web Tracker, previously located in `extensions/web-tracker/` has been removed.
	We considered it as outdated and instead recommend implementing your own tracker logic based on your project
	use case.

	==== GraphQL API - beta

	Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API.
	Available behind a feature flag (the API disabled by default), the GraphQL API is available for you to play with.

	More details about how to enable/disable the GraphQL API are available in the <<GraphQL API,corresponding section>> of the documentation.

	We welcome tickets/PRs to improve its robustness and progressively make it ready for prime time.

	==== Migrate from Unomi 1.x

	To facilitate migration we prepared a set of scripts that will automatically handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.

	It is worth keeping in mind that for Apache Unomi 2.0 we do not support “hot” migration,
	the migration process will require a shutdown of your cluster to guarantee that no new events will be collected while data migration is in progress.

	Special caution must be taken if you declared custom events as our migration scripts can only handle objects we know of.
	More details about migration (incl. of custom events) is available in the corresponding section <<Migrations,corresponding section>> of the documentation.

	==== Elasticsearch compatibility

	We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0,
	this ensure you are on a recent version that is not impacted by the log4j vulnerabilities (fixed in Elasticsearch 7.16.3).

	This version increase is releated to Apache Unomi 2.0 makeing use of a new Elasticsearch field type
	called https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html[Flattened],
	and although it was available in prior versions of Elasticsearch, we do not recommend using those
	due to the above-mentioned log4j vulnerabilities.