UNOMI-646 : update documentation (#494)
* UNOMI-646 : update documentation
diff --git a/manual/src/main/asciidoc/datamodel.adoc b/manual/src/main/asciidoc/datamodel.adoc
index 8f04857..c4ac43a 100755
--- a/manual/src/main/asciidoc/datamodel.adoc
+++ b/manual/src/main/asciidoc/datamodel.adoc
@@ -21,15 +21,17 @@
The following data model only contains the classes and properties directly related to the most important objects of Apache Unomi.
There are other classes that are less central to the functionality but all the major ones are represented in the diagram below:
-image::data-model-overview.png[]
+image::data-model.png[]
We will detail many of these classes in the document below.
=== Scope
-Scopes are not actually an object in the system, but simply unique strings that are used to "classify" objects.
+Scopes are objects which simply contains unique strings that are used to "classify" objects.
For example, when using scopes with a web content management system, a scope could be associated with a site identifier or even a host name.
+In events, scopes are used to validate event. Events with scope which are unknown by the system will be considered as invalid
+
____
Unomi defines a built-in scope (called `systemscope`) that clients can use to share data across scopes.
@@ -204,12 +206,16 @@
| scope | String | (Optional, event type specific) An identifier for a scope
+| persistent | Boolean | Defines if the event should be persisted or not (default: true)
+
| source | <<Item>> | An Item that is the source of the event. For example a web site, an application name, a web page
| target | <<Item>> | An Item that is the target of the event. For example a button, a link, a file or a page
| properties | Map<String,Object> | Properties for the event. These will change depending on the event type.
+| flattenedProperties | Map<String,Object> | Properties that will be persisted as flattened. These will change depending on the event type.
+
|===
==== Event types
@@ -253,7 +259,7 @@
| scores | Map<String,Integer> | A map of scores with the score identifier as the key and the score total value as the value.
-| mergedWith | String | If merged with another profile, the profile identifier to the master profile is stored here
+| @Deprecated mergedWith | String | If merged with another profile, the profile identifier to the master profile is stored here
| consents | Map<String,<<Consent>>> | The consents for the profile, as a map with the consent identifier as a key and the Consent object type as a value.
@@ -314,13 +320,13 @@
"lists": [
"userListId"
],
-"goals": {
- "viewLanguagePageGoalTargetReached": "2020-02-10T19:30:31Z",
- "downloadGoalExampleTargetReached": "2020-02-10T15:22:41Z",
- "viewLandingPageGoalStartReached": "2020-02-10T19:30:27Z",
- "downloadGoalExampleStartReached": "2020-02-10T19:30:27Z",
- "optimizationTestGoalStartReached": "2020-02-10T19:30:27Z"
-}
+ "goals": {
+ "viewLanguagePageGoalTargetReached": "2020-02-10T19:30:31Z",
+ "downloadGoalExampleTargetReached": "2020-02-10T15:22:41Z",
+ "viewLandingPageGoalStartReached": "2020-02-10T19:30:27Z",
+ "downloadGoalExampleStartReached": "2020-02-10T19:30:27Z",
+ "optimizationTestGoalStartReached": "2020-02-10T19:30:27Z"
+ }
},
"segments": [
"leads",
@@ -394,7 +400,6 @@
"systemProperties": {},
"segments": [],
"scores": null,
- "mergedWith": null,
"consents": {}
}
----
@@ -786,7 +791,7 @@
==== Example
-In this exmample we should the default `updateProperties` built-in rule that matches the `updateProperties` event and
+In this example we can see the default `updateProperties` built-in rule that matches the `updateProperties` event and
executes the built-in `updatePropertiesAction`
[source,json]
@@ -1001,7 +1006,7 @@
In the following example a campaign that starts January 1st 31, 2020 at 8:38am and finished on February 29th, 2020 at the same time has the following entry condition: the session duration must be less or equal to 3000 milliseconds (3 seconds) and the profile has viewed the “about” page on the “digitall” website.
The cost of the campaign is USD 1’000’000 and the timezone is Europe/Zurich.
-The primary goal for the campaign is the goal we should as an example in the Goal section.
+The primary goal for the campaign is the goal we should have as an example in the Goal section.
[source,json]
----
diff --git a/manual/src/main/asciidoc/images/data-model-overview.png b/manual/src/main/asciidoc/images/data-model-overview.png
deleted file mode 100755
index 8243b9e..0000000
--- a/manual/src/main/asciidoc/images/data-model-overview.png
+++ /dev/null
Binary files differ
diff --git a/manual/src/main/asciidoc/images/data-model.png b/manual/src/main/asciidoc/images/data-model.png
new file mode 100644
index 0000000..e63d771
--- /dev/null
+++ b/manual/src/main/asciidoc/images/data-model.png
Binary files differ
diff --git a/manual/src/main/asciidoc/whats-new.adoc b/manual/src/main/asciidoc/whats-new.adoc
index b042968..97e5a16 100644
--- a/manual/src/main/asciidoc/whats-new.adoc
+++ b/manual/src/main/asciidoc/whats-new.adoc
@@ -13,14 +13,14 @@
//
=== What's new in Apache Unomi 2.0
-Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product.
+Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product.
The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which presented an opportunity for noticeable improvements in the overall performance.
This new release also introduces a first (beta) version of the Unomi GraphQL API.
==== Scopes declarations are now required
-Scopes declarations are now required in Unomi 2. When submitting an event and specifying a scope,
+Scopes declarations are now required in Unomi 2. When submitting an event and specifying a scope,
that scope must already be declared on the platform.
Scopes can be easily created via the corresponding REST API (`cxs/scopes`)
@@ -39,8 +39,8 @@
==== JSON Schemas
-Apache Unomi 2 introduces support for https://json-schema.org/specification.html[JSON Schema] for all of its publicly exposed endpoints.
-Data received by Apache Unomi 2 will first be validated against a known schema to make sure it complies with an expected payload.
+Apache Unomi 2 introduces support for https://json-schema.org/specification.html[JSON Schema] for all of its publicly exposed endpoints.
+Data received by Apache Unomi 2 will first be validated against a known schema to make sure it complies with an expected payload.
If the received payload does not match a known schema, it will be rejected by Apache Unomi 2.
Apache Unomi 2 also introduces a set of administrative endpoints allowing new schemas and/or schemas extensions to be registered.
@@ -51,11 +51,50 @@
The introduction of JSON schema required us to modify Apache Unomi data model, One of the key differences is the removal of open maps.
-For example, the following event model in Apache Unomi 1.x:
+The properties field in the events objects provided by unomi are now restricted by JSON schema.
+This means object properties must be declared in a JSON schema for an event to be accepted.
+
+A new property, flattenedProperties has been introduced to the event object, this property has been added to store the properties as
+flattened in Elasticsearch and should avoid mapping explosion for dynamic properties.
+
+If there is dynamic properties that you want to send with your event, you should use the flattenedProperties field of the event.
+
+It's also necessary to specify the format of the values which are added to flattenedProperties by JSON schema but these value will be
+stored as flattened and will not create dynamic mapping contrary to the properties field of the events.
+
+Here is an example for objects that used dynamic properties for URL parameters:
+
+The following event example in Apache Unomi 1.x:
[source]
----
{
- "TODO": "ADD JSON"
+ "eventType":"view",
+ "scope":"digitall",
+ "properties":{
+ "URLParameters":{
+ "utm_source":"source"
+ }
+ },
+ "target":{
+ "scope":"digitall",
+ "itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "itemType":"page",
+ "properties":{
+ "pageInfo":{
+ "pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "nodeType":"jnt:page",
+ "pageName":"Home",
+ ...
+ },
+ "attributes":{},
+ "consentTypes":[]
+ }
+ },
+ "source":{
+ "scope":"digitall",
+ "itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
+ "itemType":"site"
+ }
}
----
@@ -63,14 +102,38 @@
[source]
----
{
- "TODO": "ADD JSON"
+ "eventType":"view",
+ "scope":"digitall",
+ "flattenedProperties":{
+ "URLParameters":{
+ "utm_source":"source"
+ }
+ },
+ "target":{
+ "scope":"digitall",
+ "itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "itemType":"page",
+ "properties":{
+ "pageInfo":{
+ "pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "nodeType":"jnt:page",
+ "pageName":"Home",
+ ...
+ },
+ "attributes":{},
+ "consentTypes":[]
+ }
+ },
+ "source":{
+ "scope":"digitall",
+ "itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
+ "itemType":"site"
+ }
}
----
-Most objects were refactored as part of this new release.
-
-If using the default Apache 1.x data model, our Unomi 2 migration process will handle the data model changes for you.
-If you are using custom events/objects, please refer to the detailed migration guide for more details.
+If using the default Apache 1.x data model, our Unomi 2 migration process will handle the data model changes for you.
+If you are using custom events/objects, please refer to the detailed migration guide for more details.
==== Removal of the Web Tracker
@@ -82,8 +145,8 @@
==== GraphQL API - beta
-Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API.
-Available behind a feature flag (the API disabled by default), the GraphQL API is available for you to play with.
+Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API.
+Available behind a feature flag (the API disabled by default), the GraphQL API is available for you to play with.
More details about how to enable/disable the GraphQL API are available in the <<GraphQL API,corresponding section>> of the documentation.
@@ -91,20 +154,20 @@
==== Migrate from Unomi 1.x
-To facilitate migration we prepared a set of scripts that will automatically handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.
+To facilitate migration we prepared a set of scripts that will automatically handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.
-It is worth keeping in mind that for Apache Unomi 2.0 we do not support “hot” migration,
+It is worth keeping in mind that for Apache Unomi 2.0 we do not support “hot” migration,
the migration process will require a shutdown of your cluster to guarantee that no new events will be collected while data migration is in progress.
-Special caution must be taken if you declared custom events as our migration scripts can only handle objects we know of.
+Special caution must be taken if you declared custom events as our migration scripts can only handle objects we know of.
More details about migration (incl. of custom events) is available in the corresponding section <<Migrations,corresponding section>> of the documentation.
==== Elasticsearch compatibility
-We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0,
+We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0,
this ensure you are on a recent version that is not impacted by the log4j vulnerabilities (fixed in Elasticsearch 7.16.3).
-This version increase is releated to Apache Unomi 2.0 makeing use of a new Elasticsearch field type
-called https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html[Flattened],
-and although it was available in prior versions of Elasticsearch, we do not recommend using those
+This version increase is releated to Apache Unomi 2.0 makeing use of a new Elasticsearch field type
+called https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html[Flattened],
+and although it was available in prior versions of Elasticsearch, we do not recommend using those
due to the above-mentioned log4j vulnerabilities.
