UNOMI-203, UNOMI-632, UNOMI-633, UNOMI-634: improve integrations test with possibility to test migrations (#464)
* UNOMI-203: improve integration test to use KarafTestSupport
* UNOMI-203: improve integration test to use KarafTestSupport
* UNOMI-633: implement migration configuration handling and silent migration option
* UNOMI-632: implement migration test for 2.0.0 with coverage on actual migrated data
* UNOMI-632: migrate remaining indices that need a reindexation by generify the segment migration script to handle others indices in the same scenario
* Oops
* UNOMI-632: add a bit of documentation on migration testing, for Unomi developers
* UNOMI-633: update documentation regarding migration command
* UNOMI-633: rename 'silent' into 'skipConfirmation' sounds more clear
diff --git a/itests/README.md b/itests/README.md
index 845ea66..7be12dc 100644
--- a/itests/README.md
+++ b/itests/README.md
@@ -38,16 +38,16 @@
If possible clean what your test created at the end of its execution or at the very least make sure to use unique IDs
-When you need a service from Unomi to execute your test inject it with a filer:
+When you need a service from Unomi to execute your test, you can add it to the BaseIT code:
```java
-@Inject @Filter(timeout = 60000)
-protected ProfileService profileService;
+@Before
+public void waitForStartup() throws InterruptedException {
+ // ...
+ // init unomi services that are available once unomi:start have been called
+ persistenceService=getOsgiService(PersistenceService.class, 600000);
+}
```
-This will ensure the service is available before starting the test and if you need a resource like an URL you can do something like this:
-```java
-@Inject @Filter(value="(configDiscriminator=IMPORT)", timeout = 60000)
-protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
-```
+This will ensure the service is available before starting the test.
## Running integration tests
You can run the integration tests along with the build by doing:
@@ -82,4 +82,164 @@
Here's an example:
- mvn clean install -Dit.karaf.debug=hold:true -Dit.test=org.apache.unomi.itests.graphql.GraphQLEventIT
\ No newline at end of file
+ mvn clean install -Dit.karaf.debug=hold:true -Dit.test=org.apache.unomi.itests.graphql.GraphQLEventIT
+
+## Migration tests
+
+Migration can now be tested, by reusing an ElasticSearch snapshot.
+The snapshot should be from a Unomi version where you want to start the migration from.
+
+The snapshot is copied to the /target folder using a maven ant plugin:
+
+ <plugin>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.8</version>
+ <executions>
+ <execution>
+ <phase>generate-resources</phase>
+ <configuration>
+ <tasks>
+ <unzip src="${project.basedir}/src/test/resources/migration/snapshots_repository.zip" dest="${project.build.directory}" />
+ </tasks>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+Also the ElasticSearch maven plugin is configured to allow this snapshot repository using conf:
+
+ <path.repo>${project.build.directory}/snapshots_repository</path.repo>
+
+Now that migration accept configuration file we can provide it, this allows to avoid the migration process to prompt questions (in BaseIT configuration):
+
+ replaceConfigurationFile("etc/org.apache.unomi.migration.cfg", new File("src/test/resources/migration/org.apache.unomi.migration.cfg")),
+
+The config should contain all the required prop for the migration you want to do, example:
+
+ esAddress = http://localhost:9400
+ httpClient.trustAllCertificates = true
+ indexPrefix = context
+
+Then in the first Test of the suite you can restore the Snapshot and run the migration cmd, like this:
+
+```java
+public class Migrate16xTo200IT extends BaseIT {
+
+ @Override
+ @Before
+ public void waitForStartup() throws InterruptedException {
+
+ // Restore snapshot from 1.6.x
+ try (CloseableHttpClient httpClient = HttpUtils.initHttpClient(true)) {
+ // Create snapshot repo
+ HttpUtils.executePutRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/", resourceAsString("migration/create_snapshots_repository.json"), null);
+ // Get snapshot, insure it exists
+ String snapshot = HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/snapshot_1.6.x", null);
+ if (snapshot == null || !snapshot.contains("snapshot_1.6.x")) {
+ throw new RuntimeException("Unable to retrieve 1.6.x snapshot for ES restore");
+ }
+ // Restore the snapshot
+ HttpUtils.executePostRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/snapshot_1.6.x/_restore?wait_for_completion=true", "{}", null);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ // Do migrate the data set
+ executeCommand("unomi:migrate 1.6.0 true");
+ // Call super for starting Unomi and wait for the complete startup
+ super.waitForStartup();
+ }
+
+ @After
+ public void cleanup() throws InterruptedException {
+ // Do some cleanup for next tests
+ }
+
+ @Test
+ public void checkMigratedData() throws Exception {
+ // call Unomi services to check the migrated data is correct.
+ }
+}
+```
+
+### How to update a migration test ElasticSearch Snapshot ?
+
+In the following example we want to modify the snapshot: `snapshot_1.6.x`.
+This snapshot has been done on Unomi 1.6.x using ElasticSearch 7.11.0.
+So we will set up locally those servers in the exact same versions.
+(For now just download them and do not start them yet.)
+
+First we need to extract the zip of the snapshot repository from the test resources:
+
+ /src/test/resources/migration/snapshots_repository.zip
+
+In my case I unzip it to:
+
+ /servers/elasticsearch-7.11.0/
+
+So I have the following folders structure:
+
+ /servers/elasticsearch-7.11.0/snapshots_repository/snapshots
+
+Now we need to configure our ElasticSearch server to allow this path as repo, edit the `elasticsearch.yml` to add this:
+
+ path:
+ repo:
+ - /servers/elasticsearch-7.11.0/snapshots_repository
+
+Start ElasticSearch server.
+Now we have to add the snapshot repository, do the following request on your ElasticSearch instance:
+
+ PUT /_snapshot/snapshots_repository/
+ {
+ "type": "fs",
+ "settings": {
+ "location": "snapshots"
+ }
+ }
+
+Now we need to restore the snapshot we want to modify,
+but first let's try to see if the snapshot with the id `snapshot_1.6.x` correctly exists:
+
+ GET /_snapshot/snapshots_repository/snapshot_1.6.x
+
+If the snapshot exists we can restore it:
+
+ POST /_snapshot/snapshots_repository/snapshot_1.6.x/_restore?wait_for_completion=true
+ {}
+
+At the end of the previous request ElasticSearch should be ready and our Unomi snapshot is restored to version `1.6.x`.
+Now make sure your Unomi server is correctly configured to connect to your running ElasticSearch, then start the Unomi server.
+In my case it's Unomi version 1.6.0.
+
+Once Unomi started you can perform all the operations you want to be able to add the required data to the next snapshot, like:
+- creating new events
+- creating new profiles with new data to be migrated
+- create rules/segments etc ...
+- anything you want to be part of the new snapshot.
+
+(NOTE: that it is important to add new data to the existing snapshot, but try to not removing things,
+they are probably used by the actual migration tests already.)
+
+Once you data updated we need to recreate the snapshot, first we delete the old snapshot:
+
+ DELETE /_snapshot/snapshots_repository/snapshot_1.6.x
+
+Then we recreate it:
+
+ PUT /_snapshot/snapshots_repository/snapshot_1.6.x
+
+Once the process finished (check the ElasticSearch logs to see that the snapshot is correctly created),
+we need to remove the snapshot repository from our local ElasticSearch
+
+ DELETE /_snapshot/snapshots_repository
+
+And the final step is, zipping the new version of the snapshot repository and replace it in the test resources:
+
+ zip -r snapshots_repository.zip /servers/elasticsearch-7.11.0/snapshots_repository
+ cp /servers/elasticsearch-7.11.0/snapshots_repository.zip src/test/resources/migration/snapshots_repository.zip
+
+Now you can modify the migration test class to test that your added data in 1.6.x is correctly migrated in 2.0.0
diff --git a/itests/pom.xml b/itests/pom.xml
index b65bce8..fb17992 100644
--- a/itests/pom.xml
+++ b/itests/pom.xml
@@ -28,41 +28,93 @@
<name>Apache Unomi :: Integration Tests</name>
<description>Apache Unomi Context Server integration tests</description>
+ <dependencyManagement>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.karaf</groupId>
+ <artifactId>karaf-bom</artifactId>
+ <version>${karaf.version}</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+
<dependencies>
+ <!-- Provide the KarafTestSupport -->
+ <dependency>
+ <groupId>org.apache.karaf.itests</groupId>
+ <artifactId>common</artifactId>
+ <version>${karaf.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <!-- Define the Apache Karaf version to download and use for the test -->
+ <!-- We use a released version here to avoid SNAPSHOT resolution -->
<dependency>
<groupId>org.apache.unomi</groupId>
<artifactId>unomi</artifactId>
<version>${project.version}</version>
<type>tar.gz</type>
<scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.karaf</groupId>
+ <artifactId>org.apache.karaf.client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient-osgi</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
+ <!-- Required to use shell commands in the tests -->
<dependency>
- <groupId>org.apache.unomi</groupId>
- <artifactId>unomi-router-karaf-feature</artifactId>
- <classifier>features</classifier>
- <version>${project.version}</version>
- <type>xml</type>
+ <groupId>org.apache.karaf.shell</groupId>
+ <artifactId>org.apache.karaf.shell.core</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <!-- Provide the PaxExam Karaf support -->
+ <dependency>
+ <groupId>org.ops4j.pax.exam</groupId>
+ <artifactId>pax-exam-container-karaf</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <!-- Provide the PaxExam JUnit extension -->
+ <dependency>
+ <groupId>org.ops4j.pax.exam</groupId>
+ <artifactId>pax-exam-junit4</artifactId>
<scope>test</scope>
</dependency>
<dependency>
- <groupId>org.apache.unomi</groupId>
- <artifactId>cdp-graphql-feature</artifactId>
- <classifier>features</classifier>
- <version>${project.version}</version>
- <type>xml</type>
+ <groupId>org.apache.geronimo.specs</groupId>
+ <artifactId>geronimo-atinject_1.0_spec</artifactId>
+ <version>1.2</version>
<scope>test</scope>
</dependency>
<dependency>
- <groupId>org.apache.unomi</groupId>
- <artifactId>unomi-persistence-spi</artifactId>
- <version>${project.version}</version>
- <scope>provided</scope>
+ <groupId>org.awaitility</groupId>
+ <artifactId>awaitility</artifactId>
</dependency>
<dependency>
- <groupId>org.apache.unomi</groupId>
- <artifactId>unomi-wab</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
+ <groupId>org.apache.servicemix.bundles</groupId>
+ <artifactId>org.apache.servicemix.bundles.hamcrest</artifactId>
+ <version>1.3_1</version>
+ <scope>runtime</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient-osgi</artifactId>
+ <version>4.5.5</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpcore-osgi</artifactId>
+ <version>4.4.9</version>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty.websocket</groupId>
+ <artifactId>websocket-client</artifactId>
+ <version>9.4.28.v20200408</version>
</dependency>
<dependency>
<groupId>org.apache.unomi</groupId>
@@ -70,11 +122,17 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
-
<dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <scope>provided</scope>
+ <groupId>org.apache.unomi</groupId>
+ <artifactId>unomi-persistence-spi</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.unomi</groupId>
+ <artifactId>shell-commands</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
@@ -82,73 +140,6 @@
<version>${groovy.version}</version>
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpclient-osgi</artifactId>
- <type>bundle</type>
- </dependency>
- <dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpcore-osgi</artifactId>
- </dependency>
- <dependency>
- <groupId>org.eclipse.jetty.websocket</groupId>
- <artifactId>websocket-client</artifactId>
- <version>9.4.28.v20200408</version>
- <type>bundle</type>
- </dependency>
-
-
- <!-- Dependencies for pax exam karaf container -->
- <dependency>
- <groupId>org.ops4j.pax.exam</groupId>
- <artifactId>pax-exam-container-karaf</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.ops4j.pax.exam</groupId>
- <artifactId>pax-exam-junit4</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.ops4j.pax.exam</groupId>
- <artifactId>pax-exam</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.ops4j.pax.url</groupId>
- <artifactId>pax-url-aether</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>javax.inject</groupId>
- <artifactId>javax.inject</artifactId>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.osgi</groupId>
- <artifactId>osgi.core</artifactId>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>1.6.6</version>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.ops4j.pax.url</groupId>
- <artifactId>pax-url-wrap</artifactId>
- <classifier>uber</classifier>
- <version>2.5.4</version>
- <scope>test</scope>
- </dependency>
</dependencies>
<profiles>
@@ -197,6 +188,23 @@
</executions>
</plugin>
<plugin>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.8</version>
+ <executions>
+ <execution>
+ <phase>generate-resources</phase>
+ <configuration>
+ <tasks>
+ <unzip src="${project.basedir}/src/test/resources/migration/snapshots_repository.zip" dest="${project.build.directory}" />
+ </tasks>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<!-- REPLACE THE FOLLOWING WITH THE PLUGIN VERSION YOU NEED -->
@@ -213,6 +221,7 @@
</environmentVariables>
<instanceSettings>
<properties>
+ <path.repo>${project.build.directory}/snapshots_repository</path.repo>
<cluster.routing.allocation.disk.threshold_enabled>false</cluster.routing.allocation.disk.threshold_enabled>
<http.cors.allow-origin>*</http.cors.allow-origin>
<http.cors.allow-methods>OPTIONS,HEAD,GET,POST,PUT,DELETE</http.cors.allow-methods>
@@ -249,6 +258,9 @@
<includes>
<include>**/*AllITs.java</include>
</includes>
+ <systemPropertyVariables>
+ <my.system.property>foo</my.system.property>
+ </systemPropertyVariables>
</configuration>
<executions>
<execution>
diff --git a/itests/src/test/java/org/apache/unomi/itests/AllITs.java b/itests/src/test/java/org/apache/unomi/itests/AllITs.java
index c0f858e..1db1662 100644
--- a/itests/src/test/java/org/apache/unomi/itests/AllITs.java
+++ b/itests/src/test/java/org/apache/unomi/itests/AllITs.java
@@ -17,6 +17,7 @@
package org.apache.unomi.itests;
+import org.apache.unomi.itests.migration.Migrate16xTo200IT;
import org.apache.unomi.itests.graphql.*;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
@@ -29,6 +30,7 @@
*/
@RunWith(Suite.class)
@SuiteClasses({
+ Migrate16xTo200IT.class,
BasicIT.class,
ConditionEvaluatorIT.class,
ConditionESQueryBuilderIT.class,
diff --git a/itests/src/test/java/org/apache/unomi/itests/BaseIT.java b/itests/src/test/java/org/apache/unomi/itests/BaseIT.java
index 0c400b9..dc10395 100644
--- a/itests/src/test/java/org/apache/unomi/itests/BaseIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/BaseIT.java
@@ -42,14 +42,20 @@
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
+import org.apache.karaf.itests.KarafTestSupport;
import org.apache.unomi.api.Item;
import org.apache.unomi.api.conditions.Condition;
import org.apache.unomi.api.rules.Rule;
-import org.apache.unomi.api.services.DefinitionsService;
-import org.apache.unomi.api.services.RulesService;
+import org.apache.unomi.api.services.*;
+import org.apache.unomi.groovy.actions.services.GroovyActionsService;
import org.apache.unomi.lifecycle.BundleWatcher;
import org.apache.unomi.persistence.spi.CustomObjectMapper;
import org.apache.unomi.persistence.spi.PersistenceService;
+import org.apache.unomi.router.api.ExportConfiguration;
+import org.apache.unomi.router.api.ImportConfiguration;
+import org.apache.unomi.router.api.services.ImportExportConfigurationService;
+import org.apache.unomi.schema.api.SchemaService;
+import org.apache.unomi.services.UserListService;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
@@ -58,14 +64,12 @@
import org.ops4j.pax.exam.CoreOptions;
import org.ops4j.pax.exam.Option;
import org.ops4j.pax.exam.junit.PaxExam;
-import org.ops4j.pax.exam.karaf.container.internal.JavaVersionUtil;
import org.ops4j.pax.exam.karaf.options.LogLevelOption.LogLevel;
import org.ops4j.pax.exam.options.MavenArtifactUrlReference;
import org.ops4j.pax.exam.options.extra.VMOption;
import org.ops4j.pax.exam.spi.reactors.ExamReactorStrategy;
import org.ops4j.pax.exam.spi.reactors.PerSuite;
import org.ops4j.pax.exam.util.Filter;
-import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceEvent;
import org.osgi.framework.ServiceListener;
@@ -97,15 +101,10 @@
import java.util.concurrent.CountDownLatch;
import java.util.function.Predicate;
import java.util.function.Supplier;
+import java.util.stream.Stream;
-import static org.ops4j.pax.exam.CoreOptions.maven;
import static org.ops4j.pax.exam.CoreOptions.systemProperty;
-import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.debugConfiguration;
-import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.editConfigurationFilePut;
-import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.karafDistributionConfiguration;
-import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.keepRuntimeFolder;
-import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.logLevel;
-import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.replaceConfigurationFile;
+import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.*;
/**
* Base class for integration tests.
@@ -114,13 +113,10 @@
*/
@RunWith(PaxExam.class)
@ExamReactorStrategy(PerSuite.class)
-public abstract class BaseIT {
+public abstract class BaseIT extends KarafTestSupport {
private final static Logger LOGGER = LoggerFactory.getLogger(BaseIT.class);
- protected static final String HTTP_PORT = "8181";
- protected static final String URL = "http://localhost:" + HTTP_PORT;
- protected static final String KARAF_DIR = "target/exam";
protected static final String UNOMI_KEY = "670c26d1cc413346c3b2fd9ce65dab41";
protected static final ContentType JSON_CONTENT_TYPE = ContentType.create("application/json");
protected static final String BASE_URL = "http://localhost";
@@ -131,6 +127,7 @@
protected static final int DEFAULT_TRYING_TRIES = 30;
protected final static ObjectMapper objectMapper;
+ protected static boolean unomiStarted = false;
static {
objectMapper = new ObjectMapper();
@@ -139,37 +136,67 @@
objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
}
- @Inject
- @Filter(timeout = 600000)
protected PersistenceService persistenceService;
-
- @Inject
- @Filter(timeout = 600000)
protected RulesService rulesService;
-
- @Inject
- @Filter(timeout = 600000)
protected DefinitionsService definitionsService;
+ protected ProfileService profileService;
+ protected EventService eventService;
+ protected BundleWatcher bundleWatcher;
+ protected GroovyActionsService groovyActionsService;
+ protected SegmentService segmentService;
+ protected SchemaService schemaService;
+ protected ScopeService scopeService;
+ protected PatchService patchService;
+ protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
+ protected ImportExportConfigurationService<ExportConfiguration> exportConfigurationService;
+ protected UserListService userListService;
+ protected TopicService topicService;
@Inject
protected BundleContext bundleContext;
@Inject
@Filter(timeout = 600000)
- protected BundleWatcher bundleWatcher;
-
- @Inject
- @Filter(timeout = 600000)
protected ConfigurationAdmin configurationAdmin;
- private CloseableHttpClient httpClient;
+ protected CloseableHttpClient httpClient;
@Before
public void waitForStartup() throws InterruptedException {
+ // disable retry
+ retry = new KarafTestSupport.Retry(false);
+
+ // Start Unomi if not already done
+ if (!unomiStarted) {
+ executeCommand("unomi:start");
+ unomiStarted = true;
+ }
+
+ // Wait for startup complete
+ bundleWatcher = getOsgiService(BundleWatcher.class, 600000);
while (!bundleWatcher.isStartupComplete() || !bundleWatcher.allAdditionalBundleStarted()) {
LOGGER.info("Waiting for startup to complete...");
Thread.sleep(1000);
}
+
+ // init unomi services that are available once unomi:start have been called
+ persistenceService = getOsgiService(PersistenceService.class, 600000);
+ rulesService = getOsgiService(RulesService.class, 600000);
+ definitionsService = getOsgiService(DefinitionsService.class, 600000);
+ profileService = getOsgiService(ProfileService.class, 600000);
+ eventService = getOsgiService(EventService.class, 600000);
+ groovyActionsService = getOsgiService(GroovyActionsService.class, 600000);
+ segmentService = getOsgiService(SegmentService.class, 600000);
+ schemaService = getOsgiService(SchemaService.class, 600000);
+ scopeService = getOsgiService(ScopeService.class, 600000);
+ patchService = getOsgiService(PatchService.class, 600000);
+ userListService = getOsgiService(UserListService.class, 600000);
+ topicService = getOsgiService(TopicService.class, 600000);
+ patchService = getOsgiService(PatchService.class, 600000);
+ importConfigurationService = getOsgiService(ImportExportConfigurationService.class, "(configDiscriminator=IMPORT)", 600000);
+ exportConfigurationService = getOsgiService(ImportExportConfigurationService.class, "(configDiscriminator=EXPORT)", 600000);
+
+ // init httpClient
httpClient = initHttpClient();
}
@@ -198,55 +225,53 @@
Thread.sleep(1000);
}
+ @Override
+ public MavenArtifactUrlReference getKarafDistribution() {
+ return CoreOptions.maven().groupId("org.apache.unomi").artifactId("unomi").versionAsInProject().type("tar.gz");
+ }
+
@Configuration
- public Option[] config() throws InterruptedException {
-
- MavenArtifactUrlReference karafUrl = maven().groupId("org.apache.unomi").artifactId("unomi").type("tar.gz").versionAsInProject();
-
- List<Option> options = new ArrayList<>();
-
- Option[] commonOptions = new Option[] {
- karafDistributionConfiguration().frameworkUrl(karafUrl).unpackDirectory(new File(KARAF_DIR)).useDeployFolder(true),
+ public Option[] config() {
+ System.out.println("==== Configuring container");
+ Option[] options = new Option[]{
replaceConfigurationFile("etc/org.apache.unomi.router.cfg", new File("src/test/resources/org.apache.unomi.router.cfg")),
+ replaceConfigurationFile("etc/org.apache.unomi.migration.cfg", new File("src/test/resources/migration/org.apache.unomi.migration.cfg")),
+
replaceConfigurationFile("data/tmp/1-basic-test.csv", new File("src/test/resources/1-basic-test.csv")),
replaceConfigurationFile("data/tmp/recurrent_import/2-surfers-test.csv", new File("src/test/resources/2-surfers-test.csv")),
- replaceConfigurationFile("data/tmp/recurrent_import/3-surfers-overwrite-test.csv",
- new File("src/test/resources/3-surfers-overwrite-test.csv")),
- replaceConfigurationFile("data/tmp/recurrent_import/4-surfers-delete-test.csv",
- new File("src/test/resources/4-surfers-delete-test.csv")),
+ replaceConfigurationFile("data/tmp/recurrent_import/3-surfers-overwrite-test.csv", new File("src/test/resources/3-surfers-overwrite-test.csv")),
+ replaceConfigurationFile("data/tmp/recurrent_import/4-surfers-delete-test.csv", new File("src/test/resources/4-surfers-delete-test.csv")),
replaceConfigurationFile("data/tmp/recurrent_import/5-ranking-test.csv", new File("src/test/resources/5-ranking-test.csv")),
replaceConfigurationFile("data/tmp/recurrent_import/6-actors-test.csv", new File("src/test/resources/6-actors-test.csv")),
replaceConfigurationFile("data/tmp/testLogin.json", new File("src/test/resources/testLogin.json")),
replaceConfigurationFile("data/tmp/testCopyProperties.json", new File("src/test/resources/testCopyProperties.json")),
- replaceConfigurationFile("data/tmp/testCopyPropertiesWithoutSystemTags.json",
- new File("src/test/resources/testCopyPropertiesWithoutSystemTags.json")),
- replaceConfigurationFile("data/tmp/testLoginEventCondition.json",
- new File("src/test/resources/testLoginEventCondition.json")),
- replaceConfigurationFile("data/tmp/testClickEventCondition.json",
- new File("src/test/resources/testClickEventCondition.json")),
+ replaceConfigurationFile("data/tmp/testCopyPropertiesWithoutSystemTags.json", new File("src/test/resources/testCopyPropertiesWithoutSystemTags.json")),
+ replaceConfigurationFile("data/tmp/testLoginEventCondition.json", new File("src/test/resources/testLoginEventCondition.json")),
+ replaceConfigurationFile("data/tmp/testClickEventCondition.json", new File("src/test/resources/testClickEventCondition.json")),
replaceConfigurationFile("data/tmp/testRuleGroovyAction.json", new File("src/test/resources/testRuleGroovyAction.json")),
- replaceConfigurationFile("data/tmp/groovy/UpdateAddressAction.groovy",
- new File("src/test/resources/groovy/UpdateAddressAction.groovy")), keepRuntimeFolder(),
- // configureConsole().ignoreLocalConsole(),
- logLevel(LogLevel.INFO),
- editConfigurationFilePut("etc/custom.system.properties", "org.apache.unomi.graphql.feature.activated", "true"),
+ replaceConfigurationFile("data/tmp/groovy/UpdateAddressAction.groovy", new File("src/test/resources/groovy/UpdateAddressAction.groovy")),
+
editConfigurationFilePut("etc/org.ops4j.pax.logging.cfg", "log4j2.rootLogger.level", "INFO"),
editConfigurationFilePut("etc/org.apache.karaf.features.cfg", "serviceRequirements", "disable"),
- // editConfigurationFilePut("etc/org.ops4j.pax.web.cfg", "org.osgi.service.http.port", HTTP_PORT),
- // systemProperty("org.osgi.service.http.port").value(HTTP_PORT),
+ editConfigurationFilePut("etc/system.properties", "my.system.property", System.getProperty("my.system.property")),
+ editConfigurationFilePut("etc/custom.system.properties", "org.apache.unomi.graphql.feature.activated", "true"),
+ editConfigurationFilePut("etc/custom.system.properties", "org.apache.unomi.elasticsearch.cluster.name", "contextElasticSearchITests"),
+ editConfigurationFilePut("etc/custom.system.properties", "org.apache.unomi.elasticsearch.addresses", "localhost:9400"),
+
systemProperty("org.ops4j.pax.exam.rbc.rmi.port").value("1199"),
- systemProperty("org.apache.unomi.itests.elasticsearch.transport.port").value("9500"),
- systemProperty("org.apache.unomi.itests.elasticsearch.cluster.name").value("contextElasticSearchITests"),
- systemProperty("org.apache.unomi.itests.elasticsearch.http.port").value("9400"),
- systemProperty("org.apache.unomi.itests.elasticsearch.bootstrap.seccomp").value("false"),
systemProperty("org.apache.unomi.hazelcast.group.name").value("cellar"),
systemProperty("org.apache.unomi.hazelcast.group.password").value("pass"),
systemProperty("org.apache.unomi.hazelcast.network.port").value("5701"),
systemProperty("org.apache.unomi.hazelcast.tcp-ip.members").value("127.0.0.1"),
systemProperty("org.apache.unomi.hazelcast.tcp-ip.interface").value("127.0.0.1"),
- systemProperty("unomi.autoStart").value("true"), CoreOptions.bundleStartLevel(100), CoreOptions.frameworkStartLevel(100) };
- options.addAll(Arrays.asList(commonOptions));
+ logLevel(LogLevel.INFO),
+ keepRuntimeFolder(),
+ CoreOptions.bundleStartLevel(100),
+ CoreOptions.frameworkStartLevel(100)
+ };
+ List<Option> karafOptions = new ArrayList<>();
+ karafOptions.addAll(Arrays.asList(options));
String karafDebug = System.getProperty("it.karaf.debug");
if (karafDebug != null) {
@@ -265,7 +290,7 @@
}
}
}
- options.add(0, debugConfiguration(port, hold));
+ karafOptions.add(0, debugConfiguration(port, hold));
}
// Jacoco setup
@@ -275,7 +300,7 @@
final String jacocoOption = "-javaagent:" + agentFile + "=destfile=" + System.getProperty("user.dir")
+ "/target/jacoco.exec,includes=org.apache.unomi.*";
LOGGER.info("set jacoco java agent: {}", jacocoOption);
- options.add(new VMOption(jacocoOption));
+ karafOptions.add(new VMOption(jacocoOption));
} else {
LOGGER.warn("Unable to set jacoco agent as {} was not found", agentFile);
}
@@ -283,34 +308,11 @@
String customLogging = System.getProperty("it.karaf.customLogging");
if (customLogging != null) {
String[] customLoggingParts = customLogging.split(":");
- options.add(editConfigurationFilePut("etc/org.ops4j.pax.logging.cfg", "log4j2.logger.customLogging.name", customLoggingParts[0]));
- options.add(editConfigurationFilePut("etc/org.ops4j.pax.logging.cfg", "log4j2.logger.customLogging.level", customLoggingParts[1]));
+ karafOptions.add(editConfigurationFilePut("etc/org.ops4j.pax.logging.cfg", "log4j2.logger.customLogging.name", customLoggingParts[0]));
+ karafOptions.add(editConfigurationFilePut("etc/org.ops4j.pax.logging.cfg", "log4j2.logger.customLogging.level", customLoggingParts[1]));
}
- if (JavaVersionUtil.getMajorVersion() >= 9) {
- Option[] jdk9PlusOptions = new Option[] { new VMOption("--add-reads=java.xml=java.logging"),
- new VMOption("--add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED"),
- new VMOption("--patch-module"),
- new VMOption("java.base=lib/endorsed/org.apache.karaf.specs.locator-" + System.getProperty("karaf.version") + ".jar"),
- new VMOption("--patch-module"),
- new VMOption("java.xml=lib/endorsed/org.apache.karaf.specs.java.xml-" + System.getProperty("karaf.version") + ".jar"),
- new VMOption("--add-opens"), new VMOption("java.base/java.security=ALL-UNNAMED"), new VMOption("--add-opens"),
- new VMOption("java.base/java.net=ALL-UNNAMED"), new VMOption("--add-opens"),
- new VMOption("java.base/java.lang=ALL-UNNAMED"), new VMOption("--add-opens"),
- new VMOption("java.base/java.util=ALL-UNNAMED"), new VMOption("--add-opens"),
- new VMOption("java.naming/javax.naming.spi=ALL-UNNAMED"), new VMOption("--add-opens"),
- new VMOption("java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED"),
- new VMOption("--add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED"),
- new VMOption("--add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED"),
- new VMOption("--add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED"),
- new VMOption("--add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED"), new VMOption("-classpath"),
- new VMOption("lib/jdk9plus/*" + File.pathSeparator + "lib/boot/*")
-
- };
- options.addAll(Arrays.asList(jdk9PlusOptions));
- }
-
- return options.toArray(new Option[0]);
+ return Stream.of(super.config(), karafOptions.toArray(new Option[karafOptions.size()])).flatMap(Stream::of).toArray(Option[]::new);
}
protected <T> T keepTrying(String failMessage, Supplier<T> call, Predicate<T> predicate, int timeout, int retries)
@@ -355,7 +357,11 @@
protected String bundleResourceAsString(final String resourcePath) throws IOException {
final java.net.URL url = bundleContext.getBundle().getResource(resourcePath);
if (url != null) {
- return IOUtils.toString(url);
+ try (InputStream stream = url.openStream()) {
+ return IOUtils.toString(stream);
+ } catch (final Exception e) {
+ throw new RuntimeException(e);
+ }
} else {
return null;
}
@@ -445,7 +451,7 @@
}
public String getFullUrl(String url) throws Exception {
- return BASE_URL + ":" + HTTP_PORT + url;
+ return BASE_URL + ":" + getHttpPort() + url;
}
protected <T> T get(final String url, Class<T> clazz) {
diff --git a/itests/src/test/java/org/apache/unomi/itests/BasicIT.java b/itests/src/test/java/org/apache/unomi/itests/BasicIT.java
index 819ce44..c20a463 100644
--- a/itests/src/test/java/org/apache/unomi/itests/BasicIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/BasicIT.java
@@ -92,14 +92,6 @@
private static final String EMAIL_VISITOR_1 = "visitor1@apache.unomi.org";
private static final String EMAIL_VISITOR_2 = "visitor2@apache.unomi.org";
- @Inject @Filter(timeout = 600000)
- protected ProfileService profileService;
- @Inject @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
- @Inject @Filter(timeout = 600000)
- protected ScopeService scopeService;
-
@Before
public void setUp() throws InterruptedException {
TestUtils.createScope(TEST_SCOPE, "Test scope", scopeService);
@@ -113,9 +105,15 @@
}
@Test
- public void testContextJS() throws IOException {
+ public void simpleTest() throws Exception {
+ System.out.println("==== System Property in probe bundle: " + System.getProperty("my.system.property"));
+ assertContains("foo", System.getProperty("my.system.property"));
+ }
+
+ @Test
+ public void testContextJS() throws Exception {
LOGGER.info("Start test testContextJS");
- HttpUriRequest request = new HttpGet(URL + "/cxs/context.js?sessionId=" + SESSION_ID_0);
+ HttpUriRequest request = new HttpGet(getFullUrl("/cxs/context.js?sessionId=" + SESSION_ID_0));
request.setHeader("Content-Type", "application/json");
// The underlying HTTP connection is still held by the response object
// to allow the response content to be streamed directly from the network socket.
@@ -137,10 +135,10 @@
}
@Test
- public void testContextJSONWithUrlParameter() throws IOException, InterruptedException {
+ public void testContextJSONWithUrlParameter() throws Exception {
LOGGER.info("Start test testContextJSONWithUrlParameter");
ContextRequest contextRequest = new ContextRequest();
- HttpPost request = new HttpPost(URL + "/cxs/context.json?sessionId=" + SESSION_ID_1);
+ HttpPost request = new HttpPost(getFullUrl("/cxs/context.json?sessionId=" + SESSION_ID_1));
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.create("application/json")));
executeContextJSONRequest(request, SESSION_ID_1);
@@ -148,11 +146,11 @@
}
@Test
- public void testContextJSON() throws IOException, InterruptedException {
+ public void testContextJSON() throws Exception {
LOGGER.info("Start test testContextJSON");
ContextRequest contextRequest = new ContextRequest();
contextRequest.setSessionId(SESSION_ID_2);
- HttpPost request = new HttpPost(URL + "/cxs/context.json");
+ HttpPost request = new HttpPost(getFullUrl("/cxs/context.json"));
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.create("application/json")));
executeContextJSONRequest(request, SESSION_ID_2);
@@ -160,7 +158,7 @@
}
@Test
- public void testMultipleLoginOnSameBrowser() throws IOException, InterruptedException {
+ public void testMultipleLoginOnSameBrowser() throws Exception {
LOGGER.info("Start test testMultipleLoginOnSameBrowser");
// Add login event condition
@@ -180,7 +178,7 @@
// First page view with the first visitor aka VISITOR_1 and SESSION_ID_3
ContextRequest contextRequestPageViewSession1 = getContextRequestWithPageViewEvent(sourceSite, SESSION_ID_3);
- HttpPost requestPageView1 = new HttpPost(URL + "/cxs/context.json");
+ HttpPost requestPageView1 = new HttpPost(getFullUrl("/cxs/context.json"));
requestPageView1.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequestPageViewSession1),
ContentType.create("application/json")));
TestUtils.RequestResponse requestResponsePageView1 = executeContextJSONRequest(requestPageView1, SESSION_ID_3);
@@ -198,7 +196,7 @@
// Create login event with VISITOR_1
ContextRequest contextRequestLoginVisitor1 = getContextRequestWithLoginEvent(sourceSite, loginEventPropertiesVisitor1,
EMAIL_VISITOR_1, SESSION_ID_3);
- HttpPost requestLoginVisitor1 = new HttpPost(URL + "/cxs/context.json");
+ HttpPost requestLoginVisitor1 = new HttpPost(getFullUrl("/cxs/context.json"));
requestLoginVisitor1.addHeader("Cookie", requestResponsePageView1.getCookieHeaderValue());
requestLoginVisitor1.addHeader("X-Unomi-Peer", UNOMI_KEY);
requestLoginVisitor1.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequestLoginVisitor1),
@@ -212,7 +210,7 @@
Thread.sleep(1000);
// Lets add a page view with VISITOR_1 to simulate reloading the page after login and be able to check the profile properties
- HttpPost requestPageView2 = new HttpPost(URL + "/cxs/context.json");
+ HttpPost requestPageView2 = new HttpPost(getFullUrl("/cxs/context.json"));
requestPageView2.addHeader("Cookie", requestResponsePageView1.getCookieHeaderValue());
requestPageView2.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequestPageViewSession1),
ContentType.create("application/json")));
@@ -227,7 +225,7 @@
// Lets simulate a logout by requesting the context with a new page view event and a new session id
// but we will send the cookie of the profile id from VISITOR_1
ContextRequest contextRequestPageViewSession2 = getContextRequestWithPageViewEvent(sourceSite, SESSION_ID_4);
- HttpPost requestPageView3 = new HttpPost(URL + "/cxs/context.json");
+ HttpPost requestPageView3 = new HttpPost(getFullUrl("/cxs/context.json"));
requestPageView3.addHeader("Cookie", requestResponsePageView1.getCookieHeaderValue());
requestPageView3.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequestPageViewSession2),
ContentType.create("application/json")));
@@ -252,7 +250,7 @@
// Create login event with VISITOR_2
ContextRequest contextRequestLoginVisitor2 = getContextRequestWithLoginEvent(sourceSite, loginEventPropertiesVisitor2,
EMAIL_VISITOR_2, SESSION_ID_4);
- HttpPost requestLoginVisitor2 = new HttpPost(URL + "/cxs/context.json");
+ HttpPost requestLoginVisitor2 = new HttpPost(getFullUrl("/cxs/context.json"));
requestLoginVisitor2.addHeader("Cookie", requestResponsePageView1.getCookieHeaderValue());
requestLoginVisitor2.addHeader("X-Unomi-Peer", UNOMI_KEY);
requestLoginVisitor2.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequestLoginVisitor2),
@@ -266,7 +264,7 @@
Thread.sleep(1000);
// Lets add a page view with VISITOR_2 to simulate reloading the page after login
- HttpPost requestPageView4 = new HttpPost(URL + "/cxs/context.json");
+ HttpPost requestPageView4 = new HttpPost(getFullUrl("/cxs/context.json"));
requestPageView4.addHeader("Cookie", requestResponseLoginVisitor2.getCookieHeaderValue());
requestPageView4.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequestPageViewSession2),
ContentType.create("application/json")));
diff --git a/itests/src/test/java/org/apache/unomi/itests/ConditionEvaluatorIT.java b/itests/src/test/java/org/apache/unomi/itests/ConditionEvaluatorIT.java
index c9eb401..d74672c 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ConditionEvaluatorIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ConditionEvaluatorIT.java
@@ -50,11 +50,6 @@
protected Item emptyItem;
protected Date lastVisit;
- @Inject @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
- @Inject @Filter(timeout = 600000)
- private DefinitionsService definitionsService;
-
protected boolean eval(Condition c) {
return persistenceService.testMatch(c, item);
}
diff --git a/itests/src/test/java/org/apache/unomi/itests/ContextServletIT.java b/itests/src/test/java/org/apache/unomi/itests/ContextServletIT.java
index 830e995..7593175 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ContextServletIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ContextServletIT.java
@@ -93,34 +93,6 @@
private static final int DEFAULT_TRYING_TRIES = 30;
public static final String TEST_SCOPE = "test-scope";
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
- @Inject
- @Filter(timeout = 600000)
- protected SegmentService segmentService;
-
- @Inject
- @Filter(timeout = 600000)
- protected SchemaService schemaService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ScopeService scopeService;
-
private Profile profile;
@Before
@@ -179,7 +151,7 @@
}
@Test
- public void testUpdateEventFromContextAuthorizedThirdParty_Success() throws IOException, InterruptedException {
+ public void testUpdateEventFromContextAuthorizedThirdParty_Success() throws Exception {
//Arrange
String eventId = "test-event-id-" + System.currentTimeMillis();
String sessionId = "test-session-id";
@@ -205,7 +177,7 @@
ContextRequest contextRequest = new ContextRequest();
contextRequest.setSessionId(session.getItemId());
contextRequest.setEvents(Arrays.asList(event));
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.addHeader(THIRD_PARTY_HEADER_NAME, UNOMI_KEY);
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request, sessionId);
@@ -217,7 +189,7 @@
}
@Test
- public void testUpdateEventFromContextUnAuthorizedThirdParty_Fail() throws IOException, InterruptedException {
+ public void testUpdateEventFromContextUnAuthorizedThirdParty_Fail() throws Exception {
//Arrange
String eventId = "test-event-id-" + System.currentTimeMillis();
String sessionId = "test-session-id";
@@ -244,7 +216,7 @@
ContextRequest contextRequest = new ContextRequest();
contextRequest.setSessionId(session.getItemId());
contextRequest.setEvents(Arrays.asList(event));
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request, sessionId);
@@ -255,7 +227,7 @@
}
@Test
- public void testUpdateEventFromContextAuthorizedThirdPartyNoItemID_Fail() throws IOException, InterruptedException {
+ public void testUpdateEventFromContextAuthorizedThirdPartyNoItemID_Fail() throws Exception {
//Arrange
String eventId = "test-event-id-" + System.currentTimeMillis();
String sessionId = "test-session-id";
@@ -276,7 +248,7 @@
ContextRequest contextRequest = new ContextRequest();
contextRequest.setSessionId(session.getItemId());
contextRequest.setEvents(Arrays.asList(event));
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request, sessionId);
@@ -288,7 +260,7 @@
}
@Test
- public void testCreateEventsWithNoTimestampParam_profileAddedToSegment() throws IOException, InterruptedException {
+ public void testCreateEventsWithNoTimestampParam_profileAddedToSegment() throws Exception {
//Arrange
String sessionId = "test-session-id";
String scope = TEST_SCOPE;
@@ -301,7 +273,7 @@
contextRequest.setSessionId(sessionId);
contextRequest.setRequireSegments(true);
contextRequest.setEvents(Arrays.asList(event));
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
String cookieHeaderValue = TestUtils.executeContextJSONRequest(request, sessionId).getCookieHeaderValue();
@@ -320,14 +292,14 @@
}
@Test
- public void testCreateEventWithTimestampParam_pastEvent_profileIsNotAddedToSegment() throws IOException, InterruptedException {
+ public void testCreateEventWithTimestampParam_pastEvent_profileIsNotAddedToSegment() throws Exception {
//Arrange
String sessionId = "test-session-id";
String scope = TEST_SCOPE;
Event event = new Event();
event.setEventType(TEST_EVENT_TYPE);
event.setScope(scope);
- String regularURI = URL + CONTEXT_URL;
+ String regularURI = getFullUrl(CONTEXT_URL);
long oldTimestamp = LocalDateTime.now(ZoneId.of("UTC")).minusDays(SEGMENT_NUMBER_OF_DAYS + 1).toInstant(ZoneOffset.UTC)
.toEpochMilli();
String customTimestampURI = regularURI + "?timestamp=" + oldTimestamp;
@@ -353,14 +325,14 @@
}
@Test
- public void testCreateEventWithTimestampParam_futureEvent_profileIsNotAddedToSegment() throws IOException, InterruptedException {
+ public void testCreateEventWithTimestampParam_futureEvent_profileIsNotAddedToSegment() throws Exception {
//Arrange
String sessionId = "test-session-id";
String scope = TEST_SCOPE;
Event event = new Event();
event.setEventType(TEST_EVENT_TYPE);
event.setScope(scope);
- String regularURI = URL + CONTEXT_URL;
+ String regularURI = getFullUrl(CONTEXT_URL);
long futureTimestamp = LocalDateTime.now(ZoneId.of("UTC")).plusDays(1).toInstant(ZoneOffset.UTC).toEpochMilli();
String customTimestampURI = regularURI + "?timestamp=" + futureTimestamp;
@@ -386,7 +358,7 @@
}
@Test
- public void testCreateEventWithProfileId_Success() throws IOException, InterruptedException {
+ public void testCreateEventWithProfileId_Success() throws Exception {
//Arrange
String eventId = "test-event-id-" + System.currentTimeMillis();
String eventType = "test-event-type";
@@ -399,7 +371,7 @@
contextRequest.setEvents(Arrays.asList(event));
//Act
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.addHeader(THIRD_PARTY_HEADER_NAME, UNOMI_KEY);
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request);
@@ -409,7 +381,7 @@
}
@Test
- public void testCreateEventWithPropertiesValidation_Success() throws IOException, InterruptedException {
+ public void testCreateEventWithPropertiesValidation_Success() throws Exception {
//Arrange
String eventId = "valid-event-id-" + System.currentTimeMillis();
String profileId = "valid-profile-id";
@@ -426,7 +398,7 @@
contextRequest.setEvents(Arrays.asList(event));
//Act
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.addHeader(THIRD_PARTY_HEADER_NAME, UNOMI_KEY);
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request);
@@ -439,7 +411,7 @@
}
@Test
- public void testCreateEventWithPropertyValueValidation_Failure() throws IOException, InterruptedException {
+ public void testCreateEventWithPropertyValueValidation_Failure() throws Exception {
//Arrange
String eventId = "invalid-event-value-id-" + System.currentTimeMillis();
String profileId = "invalid-profile-id";
@@ -456,7 +428,7 @@
contextRequest.setEvents(Arrays.asList(event));
//Act
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.addHeader(THIRD_PARTY_HEADER_NAME, UNOMI_KEY);
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request);
@@ -467,7 +439,7 @@
}
@Test
- public void testCreateEventWithPropertyNameValidation_Failure() throws IOException, InterruptedException {
+ public void testCreateEventWithPropertyNameValidation_Failure() throws Exception {
//Arrange
String eventId = "invalid-event-prop-id-" + System.currentTimeMillis();
String profileId = "invalid-profile-id";
@@ -483,7 +455,7 @@
contextRequest.setEvents(Arrays.asList(event));
//Act
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.addHeader(THIRD_PARTY_HEADER_NAME, UNOMI_KEY);
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request);
@@ -494,7 +466,7 @@
}
@Test
- public void testOGNLVulnerability() throws IOException, InterruptedException {
+ public void testOGNLVulnerability() throws Exception {
File vulnFile = new File("target/vuln-file.txt");
if (vulnFile.exists()) {
@@ -505,7 +477,7 @@
Map<String, String> parameters = new HashMap<>();
parameters.put("VULN_FILE_PATH", vulnFileCanonicalPath);
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(
new StringEntity(getValidatedBundleJSON("security/ognl-payload-1.json", parameters), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request);
@@ -515,7 +487,7 @@
}
@Test
- public void testMVELVulnerability() throws IOException, InterruptedException {
+ public void testMVELVulnerability() throws Exception {
File vulnFile = new File("target/vuln-file.txt");
if (vulnFile.exists()) {
@@ -526,7 +498,7 @@
Map<String, String> parameters = new HashMap<>();
parameters.put("VULN_FILE_PATH", vulnFileCanonicalPath);
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(
new StringEntity(getValidatedBundleJSON("security/mvel-payload-1.json", parameters), ContentType.APPLICATION_JSON));
TestUtils.executeContextJSONRequest(request);
@@ -536,21 +508,21 @@
}
@Test
- public void testPersonalization() throws IOException, InterruptedException {
+ public void testPersonalization() throws Exception {
Map<String, String> parameters = new HashMap<>();
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(new StringEntity(getValidatedBundleJSON("personalization.json", parameters), ContentType.APPLICATION_JSON));
TestUtils.RequestResponse response = TestUtils.executeContextJSONRequest(request);
assertEquals("Invalid response code", 200, response.getStatusCode());
}
@Test
- public void testPersonalizationWithControlGroup() throws IOException, InterruptedException {
+ public void testPersonalizationWithControlGroup() throws Exception {
Map<String, String> parameters = new HashMap<>();
parameters.put("storeInSession", "false");
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(
new StringEntity(getValidatedBundleJSON("personalization-controlgroup.json", parameters), ContentType.APPLICATION_JSON));
TestUtils.RequestResponse response = TestUtils.executeContextJSONRequest(request);
@@ -576,7 +548,7 @@
// now let's test with session storage
parameters.put("storeInSession", "true");
- request = new HttpPost(URL + CONTEXT_URL);
+ request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(
new StringEntity(getValidatedBundleJSON("personalization-controlgroup.json", parameters), ContentType.APPLICATION_JSON));
response = TestUtils.executeContextJSONRequest(request);
@@ -627,7 +599,7 @@
}
@Test
- public void testRequireScoring() throws IOException, InterruptedException {
+ public void testRequireScoring() throws Exception {
Map<String, String> parameters = new HashMap<>();
String scoringSource = getValidatedBundleJSON("score1.json", parameters);
@@ -639,7 +611,7 @@
// first let's make sure everything works without the requireScoring parameter
parameters = new HashMap<>();
- HttpPost request = new HttpPost(URL + CONTEXT_URL);
+ HttpPost request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(new StringEntity(getValidatedBundleJSON("withoutRequireScores.json", parameters), ContentType.APPLICATION_JSON));
TestUtils.RequestResponse response = TestUtils.executeContextJSONRequest(request);
assertEquals("Invalid response code", 200, response.getStatusCode());
@@ -650,7 +622,7 @@
// now let's test adding it.
parameters = new HashMap<>();
- request = new HttpPost(URL + CONTEXT_URL);
+ request = new HttpPost(getFullUrl(CONTEXT_URL));
request.setEntity(new StringEntity(getValidatedBundleJSON("withRequireScores.json", parameters), ContentType.APPLICATION_JSON));
response = TestUtils.executeContextJSONRequest(request);
assertEquals("Invalid response code", 200, response.getStatusCode());
diff --git a/itests/src/test/java/org/apache/unomi/itests/CopyPropertiesActionIT.java b/itests/src/test/java/org/apache/unomi/itests/CopyPropertiesActionIT.java
index f74b812..edbca99 100644
--- a/itests/src/test/java/org/apache/unomi/itests/CopyPropertiesActionIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/CopyPropertiesActionIT.java
@@ -64,13 +64,6 @@
public static final String PROPERTY_TO_MAP = "PropertyToMap";
public static final String MAPPED_PROPERTY = "MappedProperty";
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
@Before
public void setUp() throws InterruptedException {
Profile profile = new Profile();
diff --git a/itests/src/test/java/org/apache/unomi/itests/EventServiceIT.java b/itests/src/test/java/org/apache/unomi/itests/EventServiceIT.java
index 06622f5..4bde047 100644
--- a/itests/src/test/java/org/apache/unomi/itests/EventServiceIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/EventServiceIT.java
@@ -49,14 +49,6 @@
private final static String TEST_PROFILE_ID = "test-profile-id";
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@After
public void tearDown() {
TestUtils.removeAllEvents(definitionsService, persistenceService);
diff --git a/itests/src/test/java/org/apache/unomi/itests/GroovyActionsServiceIT.java b/itests/src/test/java/org/apache/unomi/itests/GroovyActionsServiceIT.java
index 9c38e76..20c507f 100644
--- a/itests/src/test/java/org/apache/unomi/itests/GroovyActionsServiceIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/GroovyActionsServiceIT.java
@@ -55,22 +55,6 @@
public static final String UPDATE_ADDRESS_ACTION_GROOVY_FILE = "data/tmp/groovy/UpdateAddressAction.groovy";
public static final String UPDATE_ADDRESS_ACTION = "UpdateAddressAction";
- @Inject
- @Filter(timeout = 600000)
- protected GroovyActionsService groovyActionsService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
@Before
public void setUp() throws InterruptedException {
Profile profile = new Profile();
diff --git a/itests/src/test/java/org/apache/unomi/itests/IncrementPropertyIT.java b/itests/src/test/java/org/apache/unomi/itests/IncrementPropertyIT.java
index 985aed0..98fe217 100644
--- a/itests/src/test/java/org/apache/unomi/itests/IncrementPropertyIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/IncrementPropertyIT.java
@@ -55,22 +55,6 @@
private Rule rule;
private Event event;
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
- @Inject
- @Filter(timeout = 600000)
- protected RulesService rulesService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
@Before
public void setup() throws Exception {
profile = createProfile();
diff --git a/itests/src/test/java/org/apache/unomi/itests/InputValidationIT.java b/itests/src/test/java/org/apache/unomi/itests/InputValidationIT.java
index 3ca1887..8248ca3 100644
--- a/itests/src/test/java/org/apache/unomi/itests/InputValidationIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/InputValidationIT.java
@@ -61,14 +61,6 @@
private final static String ERROR_MESSAGE_INVALID_DATA_RECEIVED = "Request rejected by the server because: Invalid received data";
public static final String DUMMY_SCOPE = "dummy_scope";
- @Inject
- @Filter(timeout = 600000)
- protected SchemaService schemaService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ScopeService scopeService;
-
@Before
public void setUp() throws InterruptedException {
TestUtils.createScope(DUMMY_SCOPE, "Dummy scope", scopeService);
@@ -82,25 +74,25 @@
}
@Test
- public void test_param_EventsCollectorRequestNotNull() throws IOException {
+ public void test_param_EventsCollectorRequestNotNull() throws Exception {
doPOSTRequestTest(EVENT_COLLECTOR_URL, null, null, 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doGETRequestTest(EVENT_COLLECTOR_URL, null, null, 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
}
@Test
- public void test_param_EventsNotEmpty() throws IOException {
+ public void test_param_EventsNotEmpty() throws Exception {
doPOSTRequestTest(EVENT_COLLECTOR_URL, null, "/validation/eventcollector_emptyEvents.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doGETRequestTest(EVENT_COLLECTOR_URL, null, "/validation/eventcollector_emptyEvents.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
}
@Test
- public void test_param_SessionIDPattern() throws IOException {
+ public void test_param_SessionIDPattern() throws Exception {
doPOSTRequestTest(EVENT_COLLECTOR_URL, null, "/validation/eventcollector_invalidSessionId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doGETRequestTest(EVENT_COLLECTOR_URL, null, "/validation/eventcollector_invalidSessionId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
}
@Test
- public void test_eventCollector_valid() throws IOException, InterruptedException {
+ public void test_eventCollector_valid() throws Exception {
// needed schema for event to be valid during tests
schemaService.saveSchema(resourceAsString("schemas/schema-dummy.json"));
schemaService.saveSchema(resourceAsString("schemas/schema-dummy-properties.json"));
@@ -122,7 +114,7 @@
}
@Test
- public void test_contextRequest_SessionIDPattern() throws IOException {
+ public void test_contextRequest_SessionIDPattern() throws Exception {
doPOSTRequestTest(CONTEXT_JSON_URL, null, "/validation/contextRequest_invalidSessionId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doPOSTRequestTest(CONTEXT_JS_URL, null, "/validation/contextRequest_invalidSessionId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doGETRequestTest(CONTEXT_JSON_URL, null, "/validation/contextRequest_invalidSessionId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
@@ -130,7 +122,7 @@
}
@Test
- public void test_contextRequest_ProfileIDPattern() throws IOException {
+ public void test_contextRequest_ProfileIDPattern() throws Exception {
doPOSTRequestTest(CONTEXT_JSON_URL, null, "/validation/contextRequest_invalidProfileId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doPOSTRequestTest(CONTEXT_JS_URL, null, "/validation/contextRequest_invalidProfileId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
doGETRequestTest(CONTEXT_JSON_URL, null, "/validation/contextRequest_invalidProfileId.json", 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
@@ -138,7 +130,7 @@
}
@Test
- public void test_contextRequest_valid() throws IOException {
+ public void test_contextRequest_valid() throws Exception {
doPOSTRequestTest(CONTEXT_JSON_URL, null, "/validation/contextRequest_valid.json", 200, null);
doPOSTRequestTest(CONTEXT_JS_URL, null, "/validation/contextRequest_valid.json", 200, null);
doGETRequestTest(CONTEXT_JSON_URL, null, "/validation/contextRequest_valid.json", 200, null);
@@ -146,7 +138,7 @@
}
@Test
- public void test_eventCollector_request_size_exceed_limit() throws IOException, InterruptedException {
+ public void test_eventCollector_request_size_exceed_limit() throws Exception {
// needed schema for event to be valid during tests
schemaService.saveSchema(resourceAsString("schemas/schema-dummy.json"));
schemaService.saveSchema(resourceAsString("schemas/schema-dummy-properties.json"));
@@ -168,7 +160,7 @@
}
@Test
- public void test_contextJSON_SessionIDPattern() throws IOException {
+ public void test_contextJSON_SessionIDPattern() throws Exception {
String baseUrl = CONTEXT_JS_URL;
String queryString = "?sessionId=" + URLEncoder.encode("<script>alert();</script>", StandardCharsets.UTF_8.toString());
doPOSTRequestTest(baseUrl + queryString, null, null, 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
@@ -189,7 +181,7 @@
}
@Test
- public void test_contextJSON_PersonaIdPattern() throws IOException {
+ public void test_contextJSON_PersonaIdPattern() throws Exception {
String baseUrl = CONTEXT_JS_URL;
String queryString = "?personaId=" + URLEncoder.encode("<script>alert();</script>", StandardCharsets.UTF_8.toString());
doPOSTRequestTest(baseUrl + queryString, null, null, 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
@@ -210,7 +202,7 @@
}
@Test
- public void test_cookie_profileIdPattern() throws IOException {
+ public void test_cookie_profileIdPattern() throws Exception {
Map<String, String> headers = new HashMap<>();
headers.put("Cookie", "context-profile-id=<script>alert();</script>");
doPOSTRequestTest(CONTEXT_JSON_URL, headers, null, 400, ERROR_MESSAGE_INVALID_DATA_RECEIVED);
@@ -225,11 +217,11 @@
doGETRequestTest(CONTEXT_JS_URL, headers, null, 200, null);
}
- private void doGETRequestTest(String uri, Map<String, String> headers, String entityResourcePath, int expectedHTTPStatusCode, String expectedErrorMessage) throws IOException {
+ private void doGETRequestTest(String uri, Map<String, String> headers, String entityResourcePath, int expectedHTTPStatusCode, String expectedErrorMessage) throws Exception {
// test old servlets
- performGETRequestTest(URL + uri, headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
+ performGETRequestTest(getFullUrl(uri), headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
// test directly CXS endpoints
- performGETRequestTest(URL + "/cxs" + uri, headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
+ performGETRequestTest(getFullUrl("/cxs" + uri), headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
}
private void performGETRequestTest(String url, Map<String, String> headers, String entityResourcePath, int expectedHTTPStatusCode, String expectedErrorMessage) throws IOException {
@@ -240,11 +232,11 @@
performRequest(new HttpGet(url), headers, expectedHTTPStatusCode, expectedErrorMessage);
}
- private void doPOSTRequestTest(String uri, Map<String, String> headers, String entityResourcePath, int expectedHTTPStatusCode, String expectedErrorMessage) throws IOException {
+ private void doPOSTRequestTest(String uri, Map<String, String> headers, String entityResourcePath, int expectedHTTPStatusCode, String expectedErrorMessage) throws Exception {
// test old servlets
- performPOSTRequestTest(URL + uri, headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
+ performPOSTRequestTest(getFullUrl(uri), headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
// test directly CXS endpoints
- performPOSTRequestTest(URL + "/cxs" + uri, headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
+ performPOSTRequestTest(getFullUrl("/cxs" + uri), headers, entityResourcePath, expectedHTTPStatusCode, expectedErrorMessage);
}
private void performPOSTRequestTest(String url, Map<String, String> headers, String entityResourcePath, int expectedHTTPStatusCode, String expectedErrorMessage) throws IOException {
diff --git a/itests/src/test/java/org/apache/unomi/itests/JSONSchemaIT.java b/itests/src/test/java/org/apache/unomi/itests/JSONSchemaIT.java
index 75f7d13..f30e3f1 100644
--- a/itests/src/test/java/org/apache/unomi/itests/JSONSchemaIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/JSONSchemaIT.java
@@ -61,14 +61,6 @@
private static final int DEFAULT_TRYING_TRIES = 30;
public static final String DUMMY_SCOPE = "dummy_scope";
- @Inject
- @Filter(timeout = 600000)
- protected SchemaService schemaService;
-
- @Inject
- @Filter(timeout = 6000000)
- protected ScopeService scopeService;
-
@Before
public void setUp() throws InterruptedException {
keepTrying("Couldn't find json schema endpoint", () -> get(JSONSCHEMA_URL, List.class), Objects::nonNull, DEFAULT_TRYING_TIMEOUT,
@@ -329,7 +321,7 @@
}
}
- private Event sendEventAndWaitItsIndexed(String eventResourcePath) throws IOException, InterruptedException {
+ private Event sendEventAndWaitItsIndexed(String eventResourcePath) throws Exception {
// build event collector request
String eventMarker = UUID.randomUUID().toString();
HashMap<String, String> eventReplacements = new HashMap<>();
@@ -353,8 +345,8 @@
return events.get(0);
}
- private void eventCollectorPost(String eventCollectorRequest) {
- HttpPost request = new HttpPost(URL + EVENT_COLLECTOR_URL);
+ private void eventCollectorPost(String eventCollectorRequest) throws Exception {
+ HttpPost request = new HttpPost(getFullUrl(EVENT_COLLECTOR_URL));
request.addHeader("Content-Type", "application/json");
request.setEntity(new StringEntity(eventCollectorRequest, ContentType.create("application/json")));
CloseableHttpResponse response;
diff --git a/itests/src/test/java/org/apache/unomi/itests/ModifyConsentIT.java b/itests/src/test/java/org/apache/unomi/itests/ModifyConsentIT.java
index 1deb98c..365da46 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ModifyConsentIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ModifyConsentIT.java
@@ -48,14 +48,6 @@
private final static String PROFILE_TEST_ID = "profile-consent";
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
@Before
public void setUp() throws InterruptedException {
Profile profile = new Profile();
diff --git a/itests/src/test/java/org/apache/unomi/itests/PatchIT.java b/itests/src/test/java/org/apache/unomi/itests/PatchIT.java
index 181a2eb..5058371 100644
--- a/itests/src/test/java/org/apache/unomi/itests/PatchIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/PatchIT.java
@@ -44,25 +44,6 @@
public class PatchIT extends BaseIT {
private Logger logger = LoggerFactory.getLogger(PatchIT.class);
- @Inject
- @Filter(timeout = 600000)
- protected PatchService patchService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
- @Inject
- protected BundleContext bundleContext;
-
@Test
public void testPatch() throws IOException {
PropertyType company = profileService.getPropertyType("company");
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileExportIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileExportIT.java
index 8273353..daa353f 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileExportIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileExportIT.java
@@ -49,13 +49,6 @@
public class ProfileExportIT extends BaseIT {
private Logger logger = LoggerFactory.getLogger(ProfileExportIT.class);
- @Inject
- @Filter(value = "(configDiscriminator=EXPORT)", timeout = 600000)
- protected ImportExportConfigurationService<ExportConfiguration> exportConfigurationService;
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Test
public void testExport() throws InterruptedException {
Date timestamp = new Date();
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileImportActorsIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileImportActorsIT.java
index bfab6a3..f42fbe9 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileImportActorsIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileImportActorsIT.java
@@ -48,13 +48,6 @@
@ExamReactorStrategy(PerSuite.class)
public class ProfileImportActorsIT extends BaseIT {
- @Inject
- @Filter(value = "(configDiscriminator=IMPORT)", timeout = 600000)
- protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Test
public void testImportActors() throws InterruptedException {
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileImportBasicIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileImportBasicIT.java
index 7f7c7c2..cfce700 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileImportBasicIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileImportBasicIT.java
@@ -49,11 +49,6 @@
public class ProfileImportBasicIT extends BaseIT {
private Logger logger = LoggerFactory.getLogger(ProfileImportBasicIT.class);
- @Inject @Filter(value="(configDiscriminator=IMPORT)", timeout = 600000)
- protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
- @Inject @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Test
public void testImportBasic() throws IOException, InterruptedException {
/*** Basic Test ***/
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileImportRankingIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileImportRankingIT.java
index 12737e2..2b321b2 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileImportRankingIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileImportRankingIT.java
@@ -47,13 +47,6 @@
@ExamReactorStrategy(PerSuite.class)
public class ProfileImportRankingIT extends BaseIT {
- @Inject
- @Filter(value = "(configDiscriminator=IMPORT)", timeout = 600000)
- protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Test
public void testImportRanking() throws InterruptedException {
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileImportSurfersIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileImportSurfersIT.java
index 4612b20..e1d478b 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileImportSurfersIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileImportSurfersIT.java
@@ -50,13 +50,6 @@
public class ProfileImportSurfersIT extends BaseIT {
private Logger logger = LoggerFactory.getLogger(ProfileImportSurfersIT.class);
- @Inject
- @Filter(value = "(configDiscriminator=IMPORT)", timeout = 600000)
- protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Test
public void testImportSurfers() throws InterruptedException {
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileMergeIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileMergeIT.java
index ca260ed..206b216 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileMergeIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileMergeIT.java
@@ -51,22 +51,6 @@
@ExamReactorStrategy(PerSuite.class)
public class ProfileMergeIT extends BaseIT {
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
- @Inject
- @Filter(timeout = 600000)
- protected RulesService rulesService;
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
private final static String TEST_EVENT_TYPE = "mergeProfileTestEventType";
private final static String TEST_RULE_ID = "mergeOnPropertyTest";
private final static String TEST_PROFILE_ID = "mergeOnPropertyTestProfileId";
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileServiceIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileServiceIT.java
index 1340816..8e38424 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileServiceIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileServiceIT.java
@@ -58,18 +58,6 @@
private static final String TEST_PROFILE_ALIAS = "test-profile-alias";
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
@Before
public void setUp() {
TestUtils.removeAllProfiles(definitionsService, persistenceService);
diff --git a/itests/src/test/java/org/apache/unomi/itests/ProfileServiceWithoutOverwriteIT.java b/itests/src/test/java/org/apache/unomi/itests/ProfileServiceWithoutOverwriteIT.java
index aa3a34c..79e733c 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ProfileServiceWithoutOverwriteIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ProfileServiceWithoutOverwriteIT.java
@@ -49,7 +49,7 @@
private final static String TEST_PROFILE_ID = "test-profile-id";
@Configuration
- public Option[] config() throws InterruptedException {
+ public Option[] config() {
List<Option> options = new ArrayList<>();
options.addAll(Arrays.asList(super.config()));
options.add(systemProperty("org.apache.unomi.elasticsearch.throwExceptions").value("true"));
@@ -57,18 +57,6 @@
return options.toArray(new Option[0]);
}
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
@Before
public void setUp() {
TestUtils.removeAllProfiles(definitionsService, persistenceService);
diff --git a/itests/src/test/java/org/apache/unomi/itests/PropertiesUpdateActionIT.java b/itests/src/test/java/org/apache/unomi/itests/PropertiesUpdateActionIT.java
index 4628332..21d3f92 100644
--- a/itests/src/test/java/org/apache/unomi/itests/PropertiesUpdateActionIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/PropertiesUpdateActionIT.java
@@ -61,16 +61,6 @@
private final static String PROFILE_TARGET_TEST_ID = "profile-target-event";
private final static String PROFILE_TEST_ID = "profile-to-update-by-event";
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
- @Inject
- @Filter(timeout = 600000)
- protected RulesService rulesService;
-
@Before
public void setUp() throws InterruptedException {
Profile profile = new Profile();
diff --git a/itests/src/test/java/org/apache/unomi/itests/RuleServiceIT.java b/itests/src/test/java/org/apache/unomi/itests/RuleServiceIT.java
index 2943d07..aa88cde 100644
--- a/itests/src/test/java/org/apache/unomi/itests/RuleServiceIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/RuleServiceIT.java
@@ -55,14 +55,6 @@
private final static String TEST_RULE_ID = "test-rule-id";
public static final String TEST_SCOPE = "test-scope";
- @Inject
- @Filter(timeout = 600000)
- protected RulesService rulesService;
-
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
@Before
public void setUp() {
TestUtils.removeAllProfiles(definitionsService, persistenceService);
diff --git a/itests/src/test/java/org/apache/unomi/itests/ScopeIT.java b/itests/src/test/java/org/apache/unomi/itests/ScopeIT.java
index 39f242f..11edf0d 100644
--- a/itests/src/test/java/org/apache/unomi/itests/ScopeIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/ScopeIT.java
@@ -50,14 +50,6 @@
private static final int DEFAULT_TRYING_TIMEOUT = 2000;
private static final int DEFAULT_TRYING_TRIES = 30;
- @Inject
- @Filter(timeout = 600000)
- protected ScopeService scopeService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
@Before
public void setUp() throws InterruptedException {
keepTrying("Couldn't find scope endpoint", () -> get(SCOPE_URL, List.class), Objects::nonNull, DEFAULT_TRYING_TIMEOUT,
diff --git a/itests/src/test/java/org/apache/unomi/itests/SecurityIT.java b/itests/src/test/java/org/apache/unomi/itests/SecurityIT.java
index f76c612..69f64d6 100644
--- a/itests/src/test/java/org/apache/unomi/itests/SecurityIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/SecurityIT.java
@@ -54,7 +54,7 @@
}
@Test
- public void testOGNLInjection() throws IOException {
+ public void testOGNLInjection() throws Exception {
ContextRequest contextRequest = new ContextRequest();
List<PersonalizationService.PersonalizationRequest> personalizations = new ArrayList<>();
PersonalizationService.PersonalizationRequest personalizationRequest = new PersonalizationService.PersonalizationRequest();
@@ -92,7 +92,7 @@
contextRequest.setPersonalizations(personalizations);
contextRequest.setSessionId(SESSION_ID);
- HttpPost request = new HttpPost(URL + "/cxs/context.json");
+ HttpPost request = new HttpPost(getFullUrl("/cxs/context.json"));
request.setEntity(new StringEntity(objectMapper.writeValueAsString(contextRequest), ContentType.create("application/json")));
TestUtils.RequestResponse response = executeContextJSONRequest(request, SESSION_ID);
diff --git a/itests/src/test/java/org/apache/unomi/itests/SegmentIT.java b/itests/src/test/java/org/apache/unomi/itests/SegmentIT.java
index 1eb0371..b1a3c06 100644
--- a/itests/src/test/java/org/apache/unomi/itests/SegmentIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/SegmentIT.java
@@ -58,22 +58,6 @@
private final static Logger LOGGER = LoggerFactory.getLogger(SegmentIT.class);
private final static String SEGMENT_ID = "test-segment-id-2";
- @Inject
- @Filter(timeout = 600000)
- protected SegmentService segmentService;
-
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
@Before
public void setUp() throws InterruptedException {
removeItems(Segment.class);
diff --git a/itests/src/test/java/org/apache/unomi/itests/SendEventActionIT.java b/itests/src/test/java/org/apache/unomi/itests/SendEventActionIT.java
index bae27b3..3b77dae 100644
--- a/itests/src/test/java/org/apache/unomi/itests/SendEventActionIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/SendEventActionIT.java
@@ -48,13 +48,6 @@
private final static String TEST_EVENT_TYPE = "sendEventTestEventType";
private final static String TEST_PROFILE_ID = "sendEventTestProfileId";
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
@After
public void tearDown() throws InterruptedException {
eventService.removeProfileEvents(TEST_PROFILE_ID);
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/BaseGraphQLIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/BaseGraphQLIT.java
index 9b8ff09..3f6ab35 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/BaseGraphQLIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/BaseGraphQLIT.java
@@ -44,25 +44,20 @@
@ExamReactorStrategy(PerSuite.class)
public abstract class BaseGraphQLIT extends BaseIT {
- protected static final String GRAPHQL_ENDPOINT = URL + "/graphql";
-
protected static final ContentType JSON_CONTENT_TYPE = ContentType.create("application/json");
- @Inject
- protected BundleContext bundleContext;
-
- protected CloseableHttpResponse postAnonymous(final String resource) throws IOException {
+ protected CloseableHttpResponse postAnonymous(final String resource) throws Exception {
return postAs(resource, null, null);
}
- protected CloseableHttpResponse post(final String resource) throws IOException {
+ protected CloseableHttpResponse post(final String resource) throws Exception {
return postAs(resource, "karaf", "karaf");
}
- protected CloseableHttpResponse postAs(final String resource, final String username, final String password) throws IOException {
+ protected CloseableHttpResponse postAs(final String resource, final String username, final String password) throws Exception {
final String resourceAsString = resourceAsString(resource);
- final HttpPost request = new HttpPost(GRAPHQL_ENDPOINT);
+ final HttpPost request = new HttpPost(getFullUrl("/graphql"));
request.setEntity(new StringEntity(resourceAsString, JSON_CONTENT_TYPE));
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLEventIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLEventIT.java
index 5c9e02d..385a4ff 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLEventIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLEventIT.java
@@ -35,18 +35,6 @@
public class GraphQLEventIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected EventService eventService;
-
- @Inject
- @Filter(timeout = 600000)
- protected PersistenceService persistenceService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
private final String profileID = "profile-1";
private final String eventID = "event-1";
private Profile profile;
@@ -61,7 +49,7 @@
@Test
- public void testGetEvent_notExists() throws IOException {
+ public void testGetEvent_notExists() throws Exception {
try (CloseableHttpResponse response = post("graphql/event/get-event-not-exists.json")) {
final ResponseContext context = ResponseContext.parse(response.getEntity());
@@ -70,7 +58,7 @@
}
@Test
- public void testGetEvent() throws IOException, InterruptedException {
+ public void testGetEvent() throws Exception {
final Event event = createEvent(eventID, profile);
refreshPersistence();
@@ -84,7 +72,7 @@
}
@Test
- public void testFindEvents() throws IOException, InterruptedException {
+ public void testFindEvents() throws Exception {
createEvent(eventID, profile);
createEvent("event-2", profile);
final Profile profile2 = new Profile("profile-2");
@@ -102,7 +90,7 @@
}
@Test
- public void testProcessEvents() throws IOException {
+ public void testProcessEvents() throws Exception {
final Profile originalProfile = persistenceService.load(profileID, Profile.class);
Assert.assertNull(originalProfile.getProperty("firstName"));
Assert.assertNull(originalProfile.getProperty("lastName"));
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLListIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLListIT.java
index 326d98f..712ce90 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLListIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLListIT.java
@@ -28,10 +28,6 @@
public class GraphQLListIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Test
public void testCRUD() throws Exception {
Profile persistedProfile = null;
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileAliasesIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileAliasesIT.java
index 6cd6275..d7defef 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileAliasesIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileAliasesIT.java
@@ -29,10 +29,6 @@
public class GraphQLProfileAliasesIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Before
public void setUp() throws InterruptedException {
removeItems(ProfileAlias.class);
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileIT.java
index 867b654..5914ac2 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfileIT.java
@@ -30,17 +30,13 @@
public class GraphQLProfileIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Before
public void setUp() throws InterruptedException {
removeItems(Profile.class);
}
@Test
- public void testGetProfile_WithoutCreation() throws IOException {
+ public void testGetProfile_WithoutCreation() throws Exception {
try (CloseableHttpResponse response = post("graphql/profile/get-profile-without-creation.json")) {
final ResponseContext context = ResponseContext.parse(response.getEntity());
@@ -49,7 +45,7 @@
}
@Test
- public void testGetProfile_WithCreation() throws IOException {
+ public void testGetProfile_WithCreation() throws Exception {
try (CloseableHttpResponse response = post("graphql/profile/get-profile-with-creation.json")) {
final ResponseContext context = ResponseContext.parse(response.getEntity());
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfilePropertiesIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfilePropertiesIT.java
index a10853e..ea90614 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfilePropertiesIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLProfilePropertiesIT.java
@@ -39,10 +39,6 @@
public class GraphQLProfilePropertiesIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
private final static Logger LOGGER = LoggerFactory.getLogger(GraphQLProfilePropertiesIT.class);
@Test
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSegmentIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSegmentIT.java
index bed6367..2636733 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSegmentIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSegmentIT.java
@@ -33,10 +33,6 @@
public class GraphQLSegmentIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected ProfileService profileService;
-
@Before
public void setUp() throws InterruptedException {
removeItems(Segment.class);
@@ -48,7 +44,7 @@
}
@Test
- public void testCreateThenGetAndDeleteSegment() throws IOException, InterruptedException {
+ public void testCreateThenGetAndDeleteSegment() throws Exception {
try (CloseableHttpResponse response = post("graphql/segment/create-or-update-segment.json")) {
final ResponseContext context = ResponseContext.parse(response.getEntity());
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSourceIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSourceIT.java
index a5a828f..985ad26 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSourceIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLSourceIT.java
@@ -28,12 +28,8 @@
public class GraphQLSourceIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- ScopeService scopeService;
-
@Test
- public void testCRUD() throws IOException, InterruptedException {
+ public void testCRUD() throws Exception {
try (CloseableHttpResponse response = post("graphql/source/create-source.json")) {
final ResponseContext context = ResponseContext.parse(response.getEntity());
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLTopicIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLTopicIT.java
index 96274c4..69c1439 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLTopicIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLTopicIT.java
@@ -27,12 +27,8 @@
public class GraphQLTopicIT extends BaseGraphQLIT {
- @Inject
- @Filter(timeout = 600000)
- protected TopicService topicService;
-
@Test
- public void testCRUD() throws IOException, InterruptedException {
+ public void testCRUD() throws Exception {
try (CloseableHttpResponse response = post("graphql/topic/create-topic.json")) {
final ResponseContext context = ResponseContext.parse(response.getEntity());
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLViewIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLViewIT.java
index 5c12b83..6065b8b 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLViewIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLViewIT.java
@@ -40,22 +40,6 @@
extends BaseGraphQLIT
{
- @Inject
- @Filter(timeout = 600000)
- protected UserListService userListService;
-
- @Inject
- @Filter(timeout = 600000)
- protected TopicService topicService;
-
- @Inject
- @Filter(timeout = 600000)
- protected SegmentService segmentService;
-
- @Inject
- @Filter(timeout = 600000)
- protected DefinitionsService definitionsService;
-
@Test
public void test()
throws Exception
diff --git a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLWebSocketIT.java b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLWebSocketIT.java
index 74a4ad8..bcef7eb 100644
--- a/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLWebSocketIT.java
+++ b/itests/src/test/java/org/apache/unomi/itests/graphql/GraphQLWebSocketIT.java
@@ -42,8 +42,6 @@
private final static Logger LOGGER = LoggerFactory.getLogger(GraphQLWebSocketIT.class);
- private static final String SUBSCRIPTION_ENDPOINT = "ws://localhost:" + HTTP_PORT + "/graphql";
-
@Test
public void testWebSocketConnectionSegment() throws Exception {
WebSocketClient client = new WebSocketClient();
@@ -52,7 +50,7 @@
LOGGER.info("Starting web socket client...");
client.start();
- URI echoUri = new URI(SUBSCRIPTION_ENDPOINT);
+ URI echoUri = new URI("ws://localhost:" + getHttpPort() + "/graphql");
ClientUpgradeRequest request = new ClientUpgradeRequest();
Future<Session> onConnected = client.connect(socket, echoUri, request);
diff --git a/itests/src/test/java/org/apache/unomi/itests/migration/Migrate16xTo200IT.java b/itests/src/test/java/org/apache/unomi/itests/migration/Migrate16xTo200IT.java
new file mode 100644
index 0000000..1372ba2
--- /dev/null
+++ b/itests/src/test/java/org/apache/unomi/itests/migration/Migrate16xTo200IT.java
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+package org.apache.unomi.itests.migration;
+
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.unomi.api.*;
+import org.apache.unomi.itests.BaseIT;
+import org.apache.unomi.persistence.spi.aggregate.TermsAggregate;
+import org.apache.unomi.shell.migration.utils.HttpUtils;
+import org.apache.unomi.shell.migration.utils.MigrationUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+public class Migrate16xTo200IT extends BaseIT {
+
+ @Override
+ @Before
+ public void waitForStartup() throws InterruptedException {
+
+ // Restore snapshot from 1.6.x
+ try (CloseableHttpClient httpClient = HttpUtils.initHttpClient(true)) {
+ // Create snapshot repo
+ HttpUtils.executePutRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/", resourceAsString("migration/create_snapshots_repository.json"), null);
+ // Get snapshot, insure it exists
+ String snapshot = HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/snapshot_1.6.x", null);
+ if (snapshot == null || !snapshot.contains("snapshot_1.6.x")) {
+ throw new RuntimeException("Unable to retrieve 1.6.x snapshot for ES restore");
+ }
+ // Restore the snapshot
+ HttpUtils.executePostRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/snapshot_1.6.x/_restore?wait_for_completion=true", "{}", null);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ // Do migrate the data set
+ executeCommand("unomi:migrate 1.6.0 true");
+ // Call super for starting Unomi and wait for the complete startup
+ super.waitForStartup();
+ }
+
+ @After
+ public void cleanup() throws InterruptedException {
+ removeItems(Profile.class);
+ removeItems(Session.class);
+ removeItems(Event.class);
+ removeItems(Scope.class);
+ }
+
+ @Test
+ public void checkMigratedData() throws Exception {
+ checkProfileInterests();
+ checkScopeHaveBeenCreated();
+ checkFormEventRestructured();
+ checkViewEventRestructured();
+ checkEventTypesNotPersistedAnymore();
+ checkForMappingUpdates();
+ }
+
+ /**
+ * Multiple index mappings have been update, check a simple check that after migration those mappings contains the latest modifications.
+ */
+ private void checkForMappingUpdates() throws IOException {
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-scope/_mapping", null).contains("\"match\":\"*\",\"match_mapping_type\":\"string\",\"mapping\":{\"analyzer\":\"folding\""));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-segment/_mapping", null).contains("\"condition\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-scoring/_mapping", null).contains("\"condition\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-campaign/_mapping", null).contains("\"entryCondition\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-conditiontype/_mapping", null).contains("\"parentCondition\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-goal/_mapping", null).contains("\"startEvent\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-patch/_mapping", null).contains("\"data\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-rule/_mapping", null).contains("\"condition\":{\"type\":\"object\",\"enabled\":false}"));
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/context-profile/_mapping", null).contains("\"interests\":{\"type\":\"nested\""));
+ for (String eventIndex : MigrationUtils.getIndexesPrefixedBy(httpClient, "http://localhost:9400", "context-event-")) {
+ Assert.assertTrue(HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/" + eventIndex + "/_mapping", null).contains("\"flattenedProperties\":{\"type\":\"flattened\"}"));
+ }
+ }
+
+ /**
+ * Data set contains a form event (id: 7b55b4fd-5ff0-4a85-9dc4-ffde322a1de6) with this data:
+ * {
+ * "properties": {
+ * "pets": "cat",
+ * "firstname": "foo",
+ * "sports": [
+ * "football",
+ * "tennis"
+ * ],
+ * "city": "Berlin",
+ * "age": "15",
+ * "email": "foo@bar.fr",
+ * "drone": "dewey",
+ * "lastname": "bar",
+ * "contactMethod": [
+ * "postalMethod",
+ * "phoneMethod"
+ * ]
+ * }
+ * }
+ */
+ private void checkFormEventRestructured() {
+ List<Event> events = persistenceService.query("eventType", "form", null, Event.class);
+ for (Event formEvent : events) {
+ Assert.assertEquals(0, formEvent.getProperties().size());
+ Map<String, Object> fields = (Map<String, Object>) formEvent.getFlattenedProperties().get("fields");
+ Assert.assertTrue(fields.size() > 0);
+
+ if (Objects.equals(formEvent.getItemId(), "7b55b4fd-5ff0-4a85-9dc4-ffde322a1de6")) {
+ // check singled valued
+ Assert.assertEquals("cat", fields.get("pets"));
+ // check multi-valued
+ List<String> sports = (List<String>) fields.get("sports");
+ Assert.assertEquals(2, sports.size());
+ Assert.assertTrue(sports.contains("football"));
+ Assert.assertTrue(sports.contains("tennis"));
+ }
+ }
+ }
+
+ /**
+ * Data set contains a view event (id: a4aa836b-c437-48ef-be02-6fbbcba3a1de) with two interests: football:50 and basketball:30
+ * Data set contains a view event (id: 34d53399-f173-451f-8d48-f34f5d9618a9) with two URL Parameters: paramerter_test:value, multiple_paramerter_test:[value1, value2]
+ */
+ private void checkViewEventRestructured() {
+ List<Event> events = persistenceService.query("eventType", "view", null, Event.class);
+ for (Event viewEvent : events) {
+
+ // check interests
+ if (Objects.equals(viewEvent.getItemId(), "a4aa836b-c437-48ef-be02-6fbbcba3a1de")) {
+ CustomItem target = (CustomItem) viewEvent.getTarget();
+ Assert.assertNull(target.getProperties().get("interests"));
+ Map<String, Object> interests = (Map<String, Object>) viewEvent.getFlattenedProperties().get("interests");
+ Assert.assertEquals(30, interests.get("basketball"));
+ Assert.assertEquals(50, interests.get("football"));
+ }
+
+ // check URL parameters
+ if (Objects.equals(viewEvent.getItemId(), "34d53399-f173-451f-8d48-f34f5d9618a9")) {
+ CustomItem target = (CustomItem) viewEvent.getTarget();
+ Map<String, Object> pageInfo = (Map<String, Object>) target.getProperties().get("pageInfo");
+ Assert.assertNull(pageInfo.get("parameters"));
+ Map<String, Object> parameters = (Map<String, Object>) viewEvent.getFlattenedProperties().get("URLParameters");
+ Assert.assertEquals("value", parameters.get("paramerter_test"));
+ List<String> multipleParameterTest = (List<String>) parameters.get("multiple_paramerter_test");
+ Assert.assertEquals(2, multipleParameterTest.size());
+ Assert.assertTrue(multipleParameterTest.contains("value1"));
+ Assert.assertTrue(multipleParameterTest.contains("value2"));
+ }
+ }
+ }
+
+
+ /**
+ * Data set contains 2 events that are not persisted anymore:
+ * One updateProperties event
+ * One sessionCreated event
+ * This test ensures that both have been removed.
+ */
+ private void checkEventTypesNotPersistedAnymore() {
+ Assert.assertEquals(0, persistenceService.query("eventType", "updateProperties", null, Event.class).size());
+ Assert.assertEquals(0, persistenceService.query("eventType", "sessionCreated", null, Event.class).size());
+ }
+
+ /**
+ * Data set contains multiple events, this test is generic enough to ensure all existing events have the scope created correctly
+ * So the data set can contain multiple different scope it's not a problem.
+ */
+ private void checkScopeHaveBeenCreated() {
+ // check that the scope mySite have been created based on the previous existings events
+ Map<String, Long> existingScopesFromEvents = persistenceService.aggregateWithOptimizedQuery(null, new TermsAggregate("scope"), Event.ITEM_TYPE);
+ for (String scopeFromEvents : existingScopesFromEvents.keySet()) {
+ if (!Objects.equals(scopeFromEvents, "_filtered")) {
+ Scope scope = scopeService.getScope(scopeFromEvents);
+ Assert.assertNotNull(scope);
+ }
+ }
+ }
+
+ /**
+ * Data set contains a profile (id: e67ecc69-a7b3-47f1-b91f-5d6e7b90276e) with two interests: football:50 and basketball:30
+ * Also it's first name is test_profile
+ */
+ private void checkProfileInterests() {
+ // check that the test_profile interests have been migrated to new data structure
+ Profile profile = persistenceService.load("e67ecc69-a7b3-47f1-b91f-5d6e7b90276e", Profile.class);
+ Assert.assertEquals("test_profile", profile.getProperty("firstName"));
+
+ List<Map<String, Object>> interests = (List<Map<String, Object>>) profile.getProperty("interests");
+ Assert.assertEquals(2, interests.size());
+ for (Map<String, Object> interest : interests) {
+ if ("basketball".equals(interest.get("key"))) {
+ Assert.assertEquals(30, interest.get("value"));
+ }
+ if ("football".equals(interest.get("key"))) {
+ Assert.assertEquals(50, interest.get("value"));
+ }
+ }
+ }
+}
diff --git a/itests/src/test/resources/migration/create_snapshots_repository.json b/itests/src/test/resources/migration/create_snapshots_repository.json
new file mode 100644
index 0000000..5c80ddb
--- /dev/null
+++ b/itests/src/test/resources/migration/create_snapshots_repository.json
@@ -0,0 +1,6 @@
+{
+ "type": "fs",
+ "settings": {
+ "location": "snapshots"
+ }
+}
\ No newline at end of file
diff --git a/itests/src/test/resources/migration/org.apache.unomi.migration.cfg b/itests/src/test/resources/migration/org.apache.unomi.migration.cfg
new file mode 100644
index 0000000..92214cf
--- /dev/null
+++ b/itests/src/test/resources/migration/org.apache.unomi.migration.cfg
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Migration config used for silent migration
+
+esAddress = http://localhost:9400
+httpClient.trustAllCertificates = true
+indexPrefix = context
diff --git a/itests/src/test/resources/migration/snapshots_repository.zip b/itests/src/test/resources/migration/snapshots_repository.zip
new file mode 100644
index 0000000..e0a81a7
--- /dev/null
+++ b/itests/src/test/resources/migration/snapshots_repository.zip
Binary files differ
diff --git a/kar/src/main/feature/feature.xml b/kar/src/main/feature/feature.xml
index b0c8db5..541314a 100644
--- a/kar/src/main/feature/feature.xml
+++ b/kar/src/main/feature/feature.xml
@@ -91,6 +91,7 @@
<bundle start-level="85" start="false">mvn:org.apache.unomi/cxs-lists-extension-actions/${project.version}</bundle>
<bundle start-level="85" start="false">mvn:org.apache.unomi/shell-dev-commands/${project.version}</bundle>
+ <configfile finalname="/etc/org.apache.unomi.migration.cfg">mvn:org.apache.unomi/shell-commands/${project.version}/cfg/migration</configfile>
<bundle start-level="99">mvn:org.apache.unomi/shell-commands/${project.version}</bundle>
</feature>
diff --git a/manual/src/main/asciidoc/shell-commands.adoc b/manual/src/main/asciidoc/shell-commands.adoc
index e951a30..2de339d 100644
--- a/manual/src/main/asciidoc/shell-commands.adoc
+++ b/manual/src/main/asciidoc/shell-commands.adoc
@@ -41,15 +41,21 @@
DESCRIPTION
unomi:migrate
- This will Migrate your date in ES to be compliant with current version
+ This will Migrate your date in ES to be compliant with current version.
+ It's possible to configure the migration using OSGI configuration file: org.apache.unomi.migration.cfg,
+ if no configuration is provided then questions will be prompted during the migration process.
SYNTAX
- unomi:migrate [fromVersionWithoutSuffix]
+ unomi:migrate [fromVersionWithoutSuffix] [skipConfirmation]
ARGUMENTS
fromVersionWithoutSuffix
Origin version without suffix/qualifier (e.g: 1.2.0)
(defaults to 1.2.0)
+ skipConfirmation
+ Should the confirmation before starting the migration process be skipped ?
+ (defaults to false)
+
```
==== Lifecycle commands
diff --git a/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java b/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java
index bf6efd9..cd51832 100644
--- a/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java
+++ b/persistence-elasticsearch/core/src/main/java/org/apache/unomi/persistence/elasticsearch/ElasticSearchPersistenceServiceImpl.java
@@ -459,17 +459,6 @@
bulkProcessorFlushInterval = System.getProperty(BULK_PROCESSOR_FLUSH_INTERVAL, bulkProcessorFlushInterval);
bulkProcessorBackoffPolicy = System.getProperty(BULK_PROCESSOR_BACKOFF_POLICY, bulkProcessorBackoffPolicy);
itemsMonthlyIndexed = itemsMonthlyIndexedOverride.equals("none") ? Collections.emptyList() : Arrays.asList(System.getProperty(MONTHLY_INDEX_ITEMS_MONTHLY_INDEXED, itemsMonthlyIndexedOverride).split(",").clone());
- // this property is used for integration tests, to make sure we don't conflict with an already running ElasticSearch instance.
- if (System.getProperty("org.apache.unomi.itests.elasticsearch.http.port") != null) {
- elasticSearchAddressList.clear();
- elasticSearchAddressList.add("localhost:" + System.getProperty("org.apache.unomi.itests.elasticsearch.http.port"));
- logger.info("Overriding ElasticSearch address list from system property=" + elasticSearchAddressList);
- }
- // this property is used for integration tests, to make sure we don't conflict with an already running ElasticSearch instance.
- if (System.getProperty("org.apache.unomi.itests.elasticsearch.cluster.name") != null) {
- clusterName = System.getProperty("org.apache.unomi.itests.elasticsearch.cluster.name");
- logger.info("Overriding cluster name from system property=" + clusterName);
- }
buildClient();
diff --git a/pom.xml b/pom.xml
index f9d867c..e801b30 100644
--- a/pom.xml
+++ b/pom.xml
@@ -63,6 +63,7 @@
<inceptionYear>2014</inceptionYear>
<properties>
+ <karaf.version>4.2.15</karaf.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<cxf.version>3.3.11</cxf.version>
<version.jackson.core>2.10.5</version.jackson.core>
diff --git a/tools/shell-commands/pom.xml b/tools/shell-commands/pom.xml
index 57ae9dc..e3e863d 100644
--- a/tools/shell-commands/pom.xml
+++ b/tools/shell-commands/pom.xml
@@ -104,6 +104,30 @@
</instructions>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>attach-artifacts</id>
+ <phase>package</phase>
+ <goals>
+ <goal>attach-artifact</goal>
+ </goals>
+ <configuration>
+ <artifacts>
+ <artifact>
+ <file>
+ src/main/resources/org.apache.unomi.migration.cfg
+ </file>
+ <type>cfg</type>
+ <classifier>migration</classifier>
+ </artifact>
+ </artifacts>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/Migration.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/Migration.java
index 8a64882..c7d9ece 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/Migration.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/Migration.java
@@ -37,5 +37,5 @@
* @throws IOException if there was an error while executing the migration
* @deprecated do groovy script for implementing new migrations
*/
- void execute(Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig, BundleContext bundleContext) throws IOException;
+ void execute(Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig, BundleContext bundleContext) throws IOException;
}
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/MigrationConfig.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/MigrationConfig.java
new file mode 100644
index 0000000..2e6b995
--- /dev/null
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/MigrationConfig.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.unomi.shell.migration;
+
+import org.apache.karaf.shell.api.console.Session;
+import org.apache.unomi.shell.migration.utils.ConsoleUtils;
+import org.osgi.service.component.annotations.Activate;
+import org.osgi.service.component.annotations.Component;
+import org.osgi.service.component.annotations.Modified;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Service uses to aggregate different configuration needed by the migrations
+ * Source of config:
+ * - file system in OSGI config file: org.apache.unomi.migration.cfg
+ * - user interactions in the console during the migration process
+ */
+@Component(immediate = true, service = MigrationConfig.class, configurationPid = {"org.apache.unomi.migration"})
+public class MigrationConfig {
+
+ public static final String CONFIG_ES_ADDRESS = "esAddress";
+ public static final String CONFIG_TRUST_ALL_CERTIFICATES = "httpClient.trustAllCertificates";
+ public static final String INDEX_PREFIX = "indexPrefix";
+ public static final String NUMBER_OF_SHARDS = "number_of_shards";
+ public static final String NUMBER_OF_REPLICAS = "number_of_replicas";
+ public static final String TOTAL_FIELDS_LIMIT = "mapping.total_fields.limit";
+ public static final String MAX_DOC_VALUE_FIELDS_SEARCH = "max_docvalue_fields_search";
+
+ private static final Map<String, MigrationConfigProperty> configProperties;
+ static {
+ Map<String, MigrationConfigProperty> m = new HashMap<>();
+ m.put(CONFIG_ES_ADDRESS, new MigrationConfigProperty("Enter ElasticSearch TARGET address (default: http://localhost:9200): ", "http://localhost:9200"));
+ m.put(CONFIG_TRUST_ALL_CERTIFICATES, new MigrationConfigProperty("We need to initialize a HttpClient, do we need to trust all certificates ?", null));
+ m.put(INDEX_PREFIX, new MigrationConfigProperty("Enter ElasticSearch Unomi indices prefix (default: context): ", "context"));
+ m.put(NUMBER_OF_SHARDS, new MigrationConfigProperty("Enter ElasticSearch index mapping configuration: number_of_shards (default: 3): ", "3"));
+ m.put(NUMBER_OF_REPLICAS, new MigrationConfigProperty("Enter ElasticSearch index mapping configuration: number_of_replicas (default: 0): ", "0"));
+ m.put(TOTAL_FIELDS_LIMIT, new MigrationConfigProperty("Enter ElasticSearch index mapping configuration: mapping.total_fields.limit (default: 1000): ", "1000"));
+ m.put(MAX_DOC_VALUE_FIELDS_SEARCH, new MigrationConfigProperty("Enter ElasticSearch index mapping configuration: max_docvalue_fields_search (default: 1000): ", "1000"));
+ configProperties = Collections.unmodifiableMap(m);
+ }
+
+ Map<String, String> initialConfig = new HashMap<>();
+ Map<String, String> computeConfig = new HashMap<>();
+
+ @Activate
+ @Modified
+ public void modified(Map<String, String> config) {
+ initialConfig = config;
+ reset();
+ }
+
+ /**
+ * Used reset user choices to initial file system config (useful at the beginning of each new migrate session)
+ */
+ public void reset() {
+ computeConfig.clear();
+ computeConfig.putAll(initialConfig);
+ }
+
+ public String getString(String name, Session session) throws IOException {
+ if (computeConfig.containsKey(name)) {
+ return computeConfig.get(name);
+ }
+ if (configProperties.containsKey(name)) {
+ MigrationConfigProperty migrateConfigProperty = configProperties.get(name);
+ String answer = ConsoleUtils.askUserWithDefaultAnswer(session, migrateConfigProperty.getDescription(), migrateConfigProperty.getDefaultValue());
+ computeConfig.put(name, answer);
+ return answer;
+ }
+ return null;
+ }
+
+ public boolean getBoolean(String name, Session session) throws IOException {
+ if (computeConfig.containsKey(name)) {
+ return Boolean.parseBoolean(computeConfig.get(name));
+ }
+ if (configProperties.containsKey(name)) {
+ MigrationConfigProperty migrateConfigProperty = configProperties.get(name);
+ boolean answer = ConsoleUtils.askUserWithAuthorizedAnswer(session, migrateConfigProperty.getDescription(), Arrays.asList("yes", "no")).equalsIgnoreCase("yes");
+ computeConfig.put(name, answer ? "true" : "false");
+ return answer;
+ }
+ return false;
+ }
+}
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/MigrationConfigProperty.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/MigrationConfigProperty.java
new file mode 100644
index 0000000..58f4bee
--- /dev/null
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/MigrationConfigProperty.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.unomi.shell.migration;
+
+/**
+ * Just a bean for a configuration property to be used during migration process
+ */
+public class MigrationConfigProperty {
+ String description;
+ String defaultValue;
+
+ public MigrationConfigProperty(String description, String defaultValue) {
+ this.description = description;
+ this.defaultValue = defaultValue;
+ }
+
+ public String getDescription() {
+ return description;
+ }
+
+ public String getDefaultValue() {
+ return defaultValue;
+ }
+}
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java
index 80778ec..dd4946d 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java
@@ -26,6 +26,7 @@
import org.apache.karaf.shell.api.action.lifecycle.Reference;
import org.apache.karaf.shell.api.action.lifecycle.Service;
import org.apache.karaf.shell.api.console.Session;
+import org.apache.unomi.shell.migration.MigrationConfig;
import org.apache.unomi.shell.migration.MigrationScript;
import org.apache.unomi.shell.migration.utils.ConsoleUtils;
import org.apache.unomi.shell.migration.utils.HttpUtils;
@@ -41,12 +42,13 @@
import java.util.stream.Collectors;
import java.util.stream.Stream;
-@Command(scope = "unomi", name = "migrate", description = "This will Migrate your data in ES to be compliant with current version")
+import static org.apache.unomi.shell.migration.MigrationConfig.CONFIG_TRUST_ALL_CERTIFICATES;
+
+@Command(scope = "unomi", name = "migrate", description = "This will Migrate your data in ES to be compliant with current version. " +
+ "It's possible to configure the migration using OSGI configuration file: org.apache.unomi.migration.cfg, if no configuration is provided then questions will be prompted during the migration process.")
@Service
public class Migrate implements Action {
- public static final String CONFIG_ES_ADDRESS = "esAddress";
- public static final String CONFIG_TRUST_ALL_CERTIFICATES = "httpClient.trustAllCertificates";
- public static final String INDEX_PREFIX = "indexPrefix";
+
@Reference
Session session;
@@ -54,9 +56,15 @@
@Reference
BundleContext bundleContext;
+ @Reference
+ MigrationConfig migrationConfig;
+
@Argument(name = "originVersion", description = "Origin version without suffix/qualifier (e.g: 1.2.0)", valueToShowInHelp = "1.2.0")
private String originVersion;
+ @Argument(index = 1, name = "skipConfirmation", description = "Should the confirmation before starting the migration process be skipped ?", valueToShowInHelp = "false")
+ private boolean skipConfirmation = false;
+
public Object execute() throws Exception {
// Load migration scrips
Set<MigrationScript> scripts = loadOSGIScripts();
@@ -80,20 +88,17 @@
}
// Check for user approval before migrate
- if (ConsoleUtils.askUserWithAuthorizedAnswer(session,
+ if (!skipConfirmation && ConsoleUtils.askUserWithAuthorizedAnswer(session,
"[WARNING] You are about to execute a migration, this a very sensitive operation, are you sure? (yes/no): ",
Arrays.asList("yes", "no")).equalsIgnoreCase("no")) {
ConsoleUtils.printMessage(session, "Migration process aborted");
return null;
}
- // Build conf
- Map<String, Object> migrationConfig = new HashMap<>();
- migrationConfig.put(CONFIG_ES_ADDRESS, ConsoleUtils.askUserWithDefaultAnswer(session, "Enter ElasticSearch 7 TARGET address (default = http://localhost:9200): ", "http://localhost:9200"));
- migrationConfig.put(CONFIG_TRUST_ALL_CERTIFICATES, ConsoleUtils.askUserWithAuthorizedAnswer(session,"We need to initialize a HttpClient, do we need to trust all certificates? (yes/no): ", Arrays.asList("yes", "no")).equalsIgnoreCase("yes"));
- migrationConfig.put(INDEX_PREFIX, ConsoleUtils.askUserWithDefaultAnswer(session, "SOURCE index name (default: context) : ", "context"));
-
- try (CloseableHttpClient httpClient = HttpUtils.initHttpClient((Boolean) migrationConfig.get(CONFIG_TRUST_ALL_CERTIFICATES))) {
+ // reset migration config from previous stored users choices.
+ migrationConfig.reset();
+
+ try (CloseableHttpClient httpClient = HttpUtils.initHttpClient(migrationConfig.getBoolean(CONFIG_TRUST_ALL_CERTIFICATES, session))) {
// Compile scripts
scripts = parseScripts(scripts, session, httpClient, migrationConfig);
@@ -133,7 +138,7 @@
.collect(Collectors.toCollection(TreeSet::new));
}
- private Set<MigrationScript> parseScripts(Set<MigrationScript> scripts, Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig) {
+ private Set<MigrationScript> parseScripts(Set<MigrationScript> scripts, Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig) {
Map<String, GroovyShell> shellsPerBundle = new HashMap<>();
return scripts.stream()
@@ -187,7 +192,7 @@
return migrationScripts;
}
- private GroovyShell buildShellForBundle(Bundle bundle, Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig) {
+ private GroovyShell buildShellForBundle(Bundle bundle, Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig) {
GroovyClassLoader groovyLoader = new GroovyClassLoader(bundle.adapt(BundleWiring.class).getClassLoader());
GroovyScriptEngine groovyScriptEngine = new GroovyScriptEngine((URL[]) null, groovyLoader);
GroovyShell groovyShell = new GroovyShell(groovyScriptEngine.getGroovyClassLoader());
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo121.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo121.java
index 0618329..072e970 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo121.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo121.java
@@ -20,6 +20,7 @@
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.karaf.shell.api.console.Session;
import org.apache.unomi.shell.migration.Migration;
+import org.apache.unomi.shell.migration.MigrationConfig;
import org.apache.unomi.shell.migration.utils.ConsoleUtils;
import org.apache.unomi.shell.migration.utils.MigrationUtils;
import org.json.JSONArray;
@@ -41,10 +42,10 @@
private List propsTaggedAsPersonalIdentifier = Arrays.asList("firstName", "lastName", "email", "phoneNumber", "address", "facebookId", "googleId", "linkedInId", "twitterId");
@Override
- public void execute(Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig, BundleContext bundleContext) throws IOException {
+ public void execute(Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig, BundleContext bundleContext) throws IOException {
this.httpClient = httpClient;
this.session = session;
- this.esAddress = (String) migrationConfig.get("esAddress");
+ this.esAddress = migrationConfig.getString(MigrationConfig.CONFIG_ES_ADDRESS, session);
migrateTags();
}
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo122.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo122.java
index b371f8a..5ad9c28 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo122.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo122.java
@@ -19,6 +19,7 @@
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.karaf.shell.api.console.Session;
import org.apache.unomi.shell.migration.Migration;
+import org.apache.unomi.shell.migration.MigrationConfig;
import org.apache.unomi.shell.migration.utils.ConsoleUtils;
import org.apache.unomi.shell.migration.utils.HttpRequestException;
import org.apache.unomi.shell.migration.utils.HttpUtils;
@@ -34,10 +35,10 @@
private String esAddress;
@Override
- public void execute(Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig, BundleContext bundleContext) throws IOException {
+ public void execute(Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig, BundleContext bundleContext) throws IOException {
this.httpClient = httpClient;
this.session = session;
- this.esAddress = (String) migrationConfig.get("esAddress");
+ this.esAddress = migrationConfig.getString(MigrationConfig.CONFIG_ES_ADDRESS, session);
deleteOldIndexTemplate();
}
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo150.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo150.java
index 4eb84c8..074a7fa 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo150.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo150.java
@@ -19,6 +19,7 @@
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.karaf.shell.api.console.Session;
import org.apache.unomi.shell.migration.Migration;
+import org.apache.unomi.shell.migration.MigrationConfig;
import org.apache.unomi.shell.migration.utils.ConsoleUtils;
import org.apache.unomi.shell.migration.utils.HttpUtils;
import org.json.JSONArray;
@@ -37,10 +38,10 @@
public static final String INDEX_DATE_PREFIX = "date-";
@Override
- public void execute(Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig, BundleContext bundleContext) throws IOException {
- String esAddress = (String) migrationConfig.get("esAddress");
+ public void execute(Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig, BundleContext bundleContext) throws IOException {
+ String esAddress = migrationConfig.getString(MigrationConfig.CONFIG_ES_ADDRESS, session);
String es5Address = ConsoleUtils.askUserWithDefaultAnswer(session, "SOURCE Elasticsearch 5.6 cluster address (default: http://localhost:9210) : ", "http://localhost:9210");
- String sourceIndexPrefix = (String) migrationConfig.get("indexPrefix");
+ String sourceIndexPrefix = migrationConfig.getString(MigrationConfig.INDEX_PREFIX, session);
String destIndexPrefix = ConsoleUtils.askUserWithDefaultAnswer(session, "TARGET index prefix (default: context) : ", "context");
int numberOfShards = Integer.parseInt(ConsoleUtils.askUserWithDefaultAnswer(session, "Number of shards for TARGET (default: 5) : ", "5"));
int numberOfReplicas = Integer.parseInt(ConsoleUtils.askUserWithDefaultAnswer(session, "Number of replicas for TARGET (default: 1) : ", "1"));
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo200.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo200.java
index 3e8967e..b1de181 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo200.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/impl/MigrationTo200.java
@@ -27,6 +27,7 @@
import org.apache.http.util.EntityUtils;
import org.apache.karaf.shell.api.console.Session;
import org.apache.unomi.shell.migration.Migration;
+import org.apache.unomi.shell.migration.MigrationConfig;
import org.apache.unomi.shell.migration.utils.ConsoleUtils;
import org.apache.unomi.shell.migration.utils.MigrationUtils;
import org.json.JSONArray;
@@ -48,26 +49,30 @@
private CloseableHttpClient httpClient;
private Session session;
private String esAddress;
+ private String indexPrefix;
private BundleContext bundleContext;
+ private MigrationConfig migrationConfig;
@Override
- public void execute(Session session, CloseableHttpClient httpClient, Map<String, Object> migrationConfig, BundleContext bundleContext) throws IOException {
+ public void execute(Session session, CloseableHttpClient httpClient, MigrationConfig migrationConfig, BundleContext bundleContext) throws IOException {
this.httpClient = httpClient;
this.session = session;
- this.esAddress = (String) migrationConfig.get("esAddress");
+ this.esAddress = migrationConfig.getString(MigrationConfig.CONFIG_ES_ADDRESS, session);
+ this.indexPrefix = migrationConfig.getString(MigrationConfig.INDEX_PREFIX, session);
this.bundleContext = bundleContext;
+ this.migrationConfig = migrationConfig;
- doExecute((String) migrationConfig.get("indexPrefix"));
+ doExecute();
}
- private void doExecute(String indexPrefix) throws IOException {
+ private void doExecute() throws IOException {
Set<String> indexes = MigrationUtils.getIndexesPrefixedBy(httpClient, esAddress, indexPrefix + "-event-");
- createScopeMapping(indexPrefix);
- createScopes(getSetOfScopes(indexes), indexPrefix);
+ createScopeMapping();
+ createScopes(getSetOfScopes(indexes));
createProfileAliasDocumentsFromProfile();
}
- private boolean scopeIndexNotExists(String indexPrefix) throws IOException {
+ private boolean scopeIndexNotExists() throws IOException {
final HttpGet httpGet = new HttpGet(esAddress + "/" + indexPrefix + "-scope");
httpGet.addHeader("Accept", "application/json");
@@ -78,26 +83,19 @@
}
}
- private void createScopeMapping(String indexPrefix) throws IOException {
+ private void createScopeMapping() throws IOException {
- if (scopeIndexNotExists(indexPrefix)) {
+ if (scopeIndexNotExists()) {
System.out.println("Creation for index = \"" + indexPrefix + "-scope\" starting.");
- System.out.println("Specify the following parameters:");
- String numberOfShards = ConsoleUtils.askUserWithDefaultAnswer(session, "number_of_shards: (default: 3)", "3");
- String numberOfReplicas = ConsoleUtils.askUserWithDefaultAnswer(session, "number_of_replicas: (default: 0)", "0");
- String mappingTotalFieldsLimit = ConsoleUtils
- .askUserWithDefaultAnswer(session, "mapping.total_fields.limit: (default: 1000)", "1000");
- String maxDocValueFieldsSearch = ConsoleUtils
- .askUserWithDefaultAnswer(session, "max_docvalue_fields_search: (default: 1000)", "1000");
-
final HttpPut httpPost = new HttpPut(esAddress + "/" + indexPrefix + "-scope");
httpPost.addHeader("Accept", "application/json");
httpPost.addHeader("Content-Type", "application/json");
- String request = MigrationUtils.resourceAsString(bundleContext,"requestBody/scopeMapping.json").replace("$numberOfShards", numberOfShards)
- .replace("$numberOfReplicas", numberOfReplicas).replace("$mappingTotalFieldsLimit", mappingTotalFieldsLimit)
- .replace("$maxDocValueFieldsSearch", maxDocValueFieldsSearch);
+ String baseRequest = MigrationUtils.resourceAsString(bundleContext,"requestBody/2.0.0/base_index_mapping.json");
+ String mapping = MigrationUtils.extractMappingFromBundles(bundleContext, "scope.json");
+ // We intentionally extract setting from profile index, because the scope index doesnt exist yet, and all indices share the same configuration regarding shards, replicas, etc ..
+ String request = MigrationUtils.buildIndexCreationRequest(httpClient, esAddress, baseRequest, indexPrefix + "-profile", mapping);
httpPost.setEntity(new StringEntity(request));
@@ -116,7 +114,7 @@
}
- private void createScopes(Set<String> scopes, String indexPrefix) throws IOException {
+ private void createScopes(Set<String> scopes) throws IOException {
final StringBuilder body = new StringBuilder();
String saveScopeBody = MigrationUtils.resourceAsString(bundleContext,"requestBody/bulkSaveScope.ndjson");
scopes.forEach(scope -> body.append(saveScopeBody.replace("$scope", scope)));
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/utils/MigrationUtils.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/utils/MigrationUtils.java
index 33b0278..e93998b 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/utils/MigrationUtils.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/utils/MigrationUtils.java
@@ -24,6 +24,7 @@
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.util.EntityUtils;
import org.json.JSONObject;
+import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;
import java.io.BufferedReader;
@@ -34,6 +35,7 @@
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
+import java.util.Enumeration;
import java.util.Set;
import java.util.stream.Collectors;
@@ -97,17 +99,43 @@
return Collections.emptySet();
}
+ public static String extractMappingFromBundles(BundleContext bundleContext, String fileName) throws IOException {
+ for (Bundle bundle : bundleContext.getBundles()) {
+ Enumeration<URL> predefinedMappings = bundle.findEntries("META-INF/cxs/mappings", fileName, true);
+ if (predefinedMappings == null) {
+ continue;
+ }
+ while (predefinedMappings.hasMoreElements()) {
+ URL predefinedMappingURL = predefinedMappings.nextElement();
+ return IOUtils.toString(predefinedMappingURL);
+ }
+ }
+
+ throw new RuntimeException("no mapping found in bundles for: " + fileName);
+ }
+
+ public static String buildIndexCreationRequest(CloseableHttpClient httpClient, String esAddress, String baseIndexSettings,
+ String originalIndexForSettingsExtraction, String mapping) throws IOException {
+
+ String settings = baseIndexSettings;
+
+ // Extract existing settings on index that still exists
+ if (originalIndexForSettingsExtraction != null) {
+ JSONObject originalIndexSettings = new JSONObject(HttpUtils.executeGetRequest(httpClient, esAddress + "/" + originalIndexForSettingsExtraction + "/_settings", null));
+ settings = settings
+ .replace("#numberOfShards", originalIndexSettings.getJSONObject(originalIndexForSettingsExtraction).getJSONObject("settings").getJSONObject("index").getString("number_of_shards"))
+ .replace("#numberOfReplicas", originalIndexSettings.getJSONObject(originalIndexForSettingsExtraction).getJSONObject("settings").getJSONObject("index").getString("number_of_replicas"))
+ .replace("#maxDocValueFieldsSearch", originalIndexSettings.getJSONObject(originalIndexForSettingsExtraction).getJSONObject("settings").getJSONObject("index").getString("max_docvalue_fields_search"))
+ .replace("#mappingTotalFieldsLimit", originalIndexSettings.getJSONObject(originalIndexForSettingsExtraction).getJSONObject("settings").getJSONObject("index").getJSONObject("mapping").getJSONObject("total_fields").getString("limit"));
+ }
+
+ return settings.replace("#mappings", mapping);
+ }
+
public static void reIndex(CloseableHttpClient httpClient, BundleContext bundleContext, String esAddress, String indexName,
String newIndexSettings, String painlessScript) throws IOException {
String indexNameCloned = indexName + "-cloned";
- // Init requests
- JSONObject originalIndexSettings = new JSONObject(HttpUtils.executeGetRequest(httpClient, esAddress + "/" + indexName + "/_settings", null));
- String newIndexRequest = newIndexSettings
- .replace("#numberOfShards", originalIndexSettings.getJSONObject(indexName).getJSONObject("settings").getJSONObject("index").getString("number_of_shards"))
- .replace("#numberOfReplicas", originalIndexSettings.getJSONObject(indexName).getJSONObject("settings").getJSONObject("index").getString("number_of_replicas"))
- .replace("#maxDocValueFieldsSearch", originalIndexSettings.getJSONObject(indexName).getJSONObject("settings").getJSONObject("index").getString("max_docvalue_fields_search"))
- .replace("#mappingTotalFieldsLimit", originalIndexSettings.getJSONObject(indexName).getJSONObject("settings").getJSONObject("index").getJSONObject("mapping").getJSONObject("total_fields").getString("limit"));
String reIndexRequest = resourceAsString(bundleContext, "requestBody/2.0.0/base_reindex_request.json")
.replace("#source", indexNameCloned).replace("#dest", indexName)
.replace("#painless", StringUtils.isNotEmpty(painlessScript) ? getScriptPart(painlessScript) : "");
@@ -121,7 +149,7 @@
// Delete original index
HttpUtils.executeDeleteRequest(httpClient, esAddress + "/" + indexName, null);
// Recreate the original index with new mappings
- HttpUtils.executePutRequest(httpClient, esAddress + "/" + indexName, newIndexRequest, null);
+ HttpUtils.executePutRequest(httpClient, esAddress + "/" + indexName, newIndexSettings, null);
// Reindex data from clone
HttpUtils.executePostRequest(httpClient, esAddress + "/_reindex", reIndexRequest, null);
// Remove clone
diff --git a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-01-globalReindex.groovy b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-01-globalReindex.groovy
new file mode 100644
index 0000000..24603d7
--- /dev/null
+++ b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-01-globalReindex.groovy
@@ -0,0 +1,31 @@
+import org.apache.unomi.shell.migration.utils.MigrationUtils
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+String esAddress = migrationConfig.getString("esAddress", session)
+String indexPrefix = migrationConfig.getString("indexPrefix", session)
+
+String baseSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/base_index_mapping.json")
+String[] indicesToReindex = ["segment", "scoring", "campaign", "conditionType", "goal", "patch", "rule"];
+indicesToReindex.each { indexToReindex ->
+ String mappingFileName = "${indexToReindex}.json"
+ String realIndexName = "${indexPrefix}-${indexToReindex.toLowerCase()}"
+ String mapping = MigrationUtils.extractMappingFromBundles(bundleContext, mappingFileName)
+ String newIndexSettings = MigrationUtils.buildIndexCreationRequest(httpClient, esAddress, baseSettings, realIndexName, mapping)
+ MigrationUtils.reIndex(httpClient, bundleContext, esAddress, realIndexName, newIndexSettings, null)
+}
diff --git a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-01-segmentReindex.groovy b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-01-segmentReindex.groovy
deleted file mode 100644
index 96048e3..0000000
--- a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-01-segmentReindex.groovy
+++ /dev/null
@@ -1,22 +0,0 @@
-import org.apache.unomi.shell.migration.utils.MigrationUtils
-
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-String newIndexSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/segment_index.json");
-MigrationUtils.reIndex(httpClient, bundleContext, migrationConfig.get("esAddress"), migrationConfig.get("indexPrefix") + "-segment",
- newIndexSettings, null)
diff --git a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-02-scoringPlanReindex.groovy b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-02-scoringPlanReindex.groovy
deleted file mode 100644
index 46e8981..0000000
--- a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-02-scoringPlanReindex.groovy
+++ /dev/null
@@ -1,22 +0,0 @@
-import org.apache.unomi.shell.migration.utils.MigrationUtils
-
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-String newIndexSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/scoring_index.json");
-MigrationUtils.reIndex(httpClient, bundleContext, migrationConfig.get("esAddress"), migrationConfig.get("indexPrefix") + "-scoring",
- newIndexSettings, null)
diff --git a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-03-profileReindex.groovy b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-03-profileReindex.groovy
index 83ab512..1fae7ff 100644
--- a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-03-profileReindex.groovy
+++ b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-03-profileReindex.groovy
@@ -17,6 +17,11 @@
* limitations under the License.
*/
-String newIndexSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/profile_index.json");
-MigrationUtils.reIndex(httpClient, bundleContext, migrationConfig.get("esAddress"), migrationConfig.get("indexPrefix") + "-profile",
+String esAddress = migrationConfig.getString("esAddress", session)
+String indexPrefix = migrationConfig.getString("indexPrefix", session)
+
+String baseSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/base_index_mapping.json")
+String mapping = MigrationUtils.extractMappingFromBundles(bundleContext, "profile.json")
+String newIndexSettings = MigrationUtils.buildIndexCreationRequest(httpClient, esAddress, baseSettings, indexPrefix + "-profile", mapping)
+MigrationUtils.reIndex(httpClient, bundleContext, esAddress, indexPrefix + "-profile",
newIndexSettings, MigrationUtils.getFileWithoutComments(bundleContext, "requestBody/2.0.0/profile_migrate.painless"))
diff --git a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-04-eventsReindex.groovy b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-04-eventsReindex.groovy
index 10c6464..aaad059 100644
--- a/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-04-eventsReindex.groovy
+++ b/tools/shell-commands/src/main/resources/META-INF/cxs/migration/migrate-2.0.0-04-eventsReindex.groovy
@@ -18,17 +18,19 @@
* limitations under the License.
*/
-String esAddress = migrationConfig.get("esAddress")
-String indexPrefix = migrationConfig.get("indexPrefix")
+String esAddress = migrationConfig.getString("esAddress", session)
+String indexPrefix = migrationConfig.getString("indexPrefix", session)
// Remove all internal events that are no more persisted
String removeInternalEventsRequest = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/event_delete_by_query.json")
HttpUtils.executePostRequest(httpClient, "${esAddress}/${indexPrefix}-event-*/_delete_by_query", removeInternalEventsRequest, null)
// Reindex the rest of the events
-String newIndexSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/event_index.json");
+String baseSettings = MigrationUtils.resourceAsString(bundleContext, "requestBody/2.0.0/base_index_mapping.json")
String reIndexScript = MigrationUtils.getFileWithoutComments(bundleContext, "requestBody/2.0.0/event_migrate.painless");
+String mapping = MigrationUtils.extractMappingFromBundles(bundleContext, "event.json")
Set<String> eventIndices = MigrationUtils.getIndexesPrefixedBy(httpClient, esAddress, "${indexPrefix}-event-")
eventIndices.each { eventIndex ->
+ String newIndexSettings = MigrationUtils.buildIndexCreationRequest(httpClient, esAddress, baseSettings, eventIndex, mapping)
MigrationUtils.reIndex(httpClient, bundleContext, esAddress, eventIndex, newIndexSettings, reIndexScript)
}
\ No newline at end of file
diff --git a/tools/shell-commands/src/main/resources/org.apache.unomi.migration.cfg b/tools/shell-commands/src/main/resources/org.apache.unomi.migration.cfg
new file mode 100644
index 0000000..960020c
--- /dev/null
+++ b/tools/shell-commands/src/main/resources/org.apache.unomi.migration.cfg
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Migration config used for silent migration
+
+# esAddress = http://localhost:9200
+# httpClient.trustAllCertificates = true
+# indexPrefix = context
\ No newline at end of file
diff --git a/tools/shell-commands/src/main/resources/requestBody/2.0.0/base_index_mapping.json b/tools/shell-commands/src/main/resources/requestBody/2.0.0/base_index_mapping.json
new file mode 100644
index 0000000..a5de575
--- /dev/null
+++ b/tools/shell-commands/src/main/resources/requestBody/2.0.0/base_index_mapping.json
@@ -0,0 +1,23 @@
+{
+ "settings": {
+ "index": {
+ "number_of_shards": #numberOfShards,
+ "number_of_replicas": #numberOfReplicas,
+ "mapping.total_fields.limit": #mappingTotalFieldsLimit,
+ "max_docvalue_fields_search": #maxDocValueFieldsSearch
+ },
+ "analysis": {
+ "analyzer": {
+ "folding": {
+ "type": "custom",
+ "tokenizer": "keyword",
+ "filter": [
+ "lowercase",
+ "asciifolding"
+ ]
+ }
+ }
+ }
+ },
+ "mappings": #mappings
+}
\ No newline at end of file
diff --git a/tools/shell-commands/src/main/resources/requestBody/2.0.0/event_index.json b/tools/shell-commands/src/main/resources/requestBody/2.0.0/event_index.json
deleted file mode 100644
index a0e25f9..0000000
--- a/tools/shell-commands/src/main/resources/requestBody/2.0.0/event_index.json
+++ /dev/null
@@ -1,87 +0,0 @@
-{
- "settings": {
- "index": {
- "number_of_shards": #numberOfShards,
- "number_of_replicas": #numberOfReplicas,
- "mapping.total_fields.limit": #mappingTotalFieldsLimit,
- "max_docvalue_fields_search": #maxDocValueFieldsSearch
- },
- "analysis": {
- "analyzer": {
- "folding": {
- "type": "custom",
- "tokenizer": "keyword",
- "filter": [
- "lowercase",
- "asciifolding"
- ]
- }
- }
- }
- },
- "mappings": {
- "dynamic_templates": [
- {
- "all": {
- "match": "*",
- "match_mapping_type": "string",
- "mapping": {
- "type": "text",
- "analyzer": "folding",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
- }
- }
- }
- }
- }
- ],
- "properties": {
- "flattenedProperties": {
- "type": "flattened"
- },
- "timeStamp": {
- "type": "date"
- },
- "target" : {
- "properties" : {
- "lastEventDate" : {
- "type" : "date"
- },
- "profile" : {
- "properties" : {
- "properties" : {
- "properties" : {
- "birthDate" : {
- "type" : "date"
- },
- "firstVisit" : {
- "type" : "date"
- },
- "lastVisit" : {
- "type" : "date"
- },
- "notificationRefreshDate" : {
- "type" : "date"
- },
- "previousVisit" : {
- "type" : "date"
- }
- }
- },
- "systemProperties" : {
- "properties" : {
-
- }
- }
- }
- }
- }
- }
- }
- }
-
-
-}
diff --git a/tools/shell-commands/src/main/resources/requestBody/2.0.0/profile_index.json b/tools/shell-commands/src/main/resources/requestBody/2.0.0/profile_index.json
deleted file mode 100644
index 0808aa0..0000000
--- a/tools/shell-commands/src/main/resources/requestBody/2.0.0/profile_index.json
+++ /dev/null
@@ -1,76 +0,0 @@
-{
- "settings": {
- "index": {
- "number_of_shards": #numberOfShards,
- "number_of_replicas": #numberOfReplicas,
- "mapping.total_fields.limit": #mappingTotalFieldsLimit,
- "max_docvalue_fields_search": #maxDocValueFieldsSearch
- },
- "analysis": {
- "analyzer": {
- "folding": {
- "type": "custom",
- "tokenizer": "keyword",
- "filter": [
- "lowercase",
- "asciifolding"
- ]
- }
- }
- }
- },
- "mappings": {
- "dynamic_templates": [
- {
- "all": {
- "match": "*",
- "match_mapping_type": "string",
- "mapping": {
- "type": "text",
- "analyzer": "folding",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
- }
- }
- }
- }
- }
- ],
- "properties": {
- "properties": {
- "properties": {
- "age": {
- "type": "long"
- },
- "firstVisit": {
- "type": "date"
- },
- "lastVisit": {
- "type": "date"
- },
- "previousVisit": {
- "type": "date"
- },
- "nbOfVisits": {
- "type": "long"
- },
- "interests": {
- "type": "nested"
- }
- }
- },
- "consents": {
- "properties": {
- "statusDate": {
- "type": "date"
- },
- "revokeDate": {
- "type": "date"
- }
- }
- }
- }
- }
-}
diff --git a/tools/shell-commands/src/main/resources/requestBody/2.0.0/scoring_index.json b/tools/shell-commands/src/main/resources/requestBody/2.0.0/scoring_index.json
deleted file mode 100644
index d254186..0000000
--- a/tools/shell-commands/src/main/resources/requestBody/2.0.0/scoring_index.json
+++ /dev/null
@@ -1,68 +0,0 @@
-{
- "settings": {
- "index": {
- "number_of_shards": #numberOfShards,
- "number_of_replicas": #numberOfReplicas,
- "mapping.total_fields.limit": #mappingTotalFieldsLimit,
- "max_docvalue_fields_search": #maxDocValueFieldsSearch
- },
- "analysis": {
- "analyzer": {
- "folding": {
- "type": "custom",
- "tokenizer": "keyword",
- "filter": [
- "lowercase",
- "asciifolding"
- ]
- }
- }
- }
- },
- "mappings": {
- "dynamic_templates": [
- {
- "all": {
- "match": "*",
- "match_mapping_type": "string",
- "mapping": {
- "type": "text",
- "analyzer": "folding",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
- }
- }
- }
- }
- }
- ],
- "properties": {
- "metadata": {
- "properties": {
- "enabled": {
- "type": "boolean"
- },
- "hidden": {
- "type": "boolean"
- },
- "missingPlugins": {
- "type": "boolean"
- },
- "readOnly": {
- "type": "boolean"
- }
- }
- },
- "elements": {
- "properties": {
- "condition": {
- "type": "object",
- "enabled": false
- }
- }
- }
- }
- }
-}
diff --git a/tools/shell-commands/src/main/resources/requestBody/2.0.0/segment_index.json b/tools/shell-commands/src/main/resources/requestBody/2.0.0/segment_index.json
deleted file mode 100644
index ec8a3c3..0000000
--- a/tools/shell-commands/src/main/resources/requestBody/2.0.0/segment_index.json
+++ /dev/null
@@ -1,64 +0,0 @@
-{
- "settings": {
- "index": {
- "number_of_shards": #numberOfShards,
- "number_of_replicas": #numberOfReplicas,
- "mapping.total_fields.limit": #mappingTotalFieldsLimit,
- "max_docvalue_fields_search": #maxDocValueFieldsSearch
- },
- "analysis": {
- "analyzer": {
- "folding": {
- "type": "custom",
- "tokenizer": "keyword",
- "filter": [
- "lowercase",
- "asciifolding"
- ]
- }
- }
- }
- },
- "mappings": {
- "dynamic_templates": [
- {
- "all": {
- "match": "*",
- "match_mapping_type": "string",
- "mapping": {
- "type": "text",
- "analyzer": "folding",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
- }
- }
- }
- }
- }
- ],
- "properties": {
- "metadata": {
- "properties": {
- "enabled": {
- "type": "boolean"
- },
- "hidden": {
- "type": "boolean"
- },
- "missingPlugins": {
- "type": "boolean"
- },
- "readOnly": {
- "type": "boolean"
- }
- }
- },
- "condition": {
- "type": "object",
- "enabled": false
- }
- }
- }
-}
\ No newline at end of file
diff --git a/tools/shell-commands/src/main/resources/requestBody/scopeMapping.json b/tools/shell-commands/src/main/resources/requestBody/scopeMapping.json
deleted file mode 100644
index b45b0a8..0000000
--- a/tools/shell-commands/src/main/resources/requestBody/scopeMapping.json
+++ /dev/null
@@ -1,44 +0,0 @@
-{
- "settings": {
- "index": {
- "number_of_shards": $numberOfShards,
- "number_of_replicas": $numberOfReplicas,
- "mapping.total_fields.limit": $mappingTotalFieldsLimit,
- "max_docvalue_fields_search": $maxDocValueFieldsSearch
- },
- "analysis": {
- "analyzer": {
- "folding": {
- "type": "custom",
- "tokenizer": "keyword",
- "filter": [
- "lowercase",
- "asciifolding"
- ]
- }
- }
- }
- },
- "mappings": {
- "dynamic_templates": [
- {
- "all": {
- "match": "*",
- "match_mapping_type": "string",
- "mapping": {
- "type": "text",
- "analyzer": "folding",
- "fields": {
- "keyword": {
- "type": "keyword",
- "ignore_above": 256
- }
- }
- }
- }
- }
- ],
- "properties": {
- }
- }
-}
