//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
The profile import and export feature in Apache Unomi is based on configurations and consumes or produces CSV files that
contain profiles to be imported and exported.

=== Importing profiles

Only `ftp`, `sftp`, `ftps` and `file` are supported in the source path. For example:

    file:///tmp/?fileName=profiles.csv&move=.done&consumer.delay=25s

Where:

- `fileName` Can be a pattern, for example `include=.*.csv` instead of `fileName=...` to consume all CSV files. 
By default the processed files are moved to `.camel` folder you can change it using the `move` option.
- `consumer.delay` Is the frequency of polling in milliseconds. For example, 20000 milliseconds is 20 seconds. This
frequency can also be 20s. Other possible format are: 2h30m10s = 2 hours and 30 minutes and 10 seconds.

See http://camel.apache.org/ftp.html and http://camel.apache.org/file2.html to build more complex source path. Also be
careful with FTP configuration as most servers no longer support plain text FTP and you should use SFTP or FTPS
instead, but they are a little more difficult to configure properly. It is recommended to test the connection with an
FTP client first before setting up these source paths to ensure that everything works properly. Also on FTP
connections most servers require PASSIVE mode so you can specify that in the path using the `passiveMode=true` parameter.

Here are some examples of FTPS and SFTP source paths:

    sftp://USER@HOST/PATH?password=PASSWORD&include=.*.csv
    ftps://USER@HOST?password=PASSWORD&fileName=profiles.csv&passiveMode=true

Where:

- `USER` is the user name of the SFTP/FTPS user account to login with
- `PASSWORD` is the password for the user account
- `HOST` is the host name (or IP address) of the host server that provides the SFTP / FTPS server
- `PATH` is a path to a directory inside the user's account where the file will be retrieved.

==== Import API

Apache Unomi provides REST endpoints to manage import configurations:

      GET /cxs/importConfiguration
      GET /cxs/importConfiguration/{configId}
      POST /cxs/importConfiguration
      DELETE /cxs/importConfiguration/{configId}

This is how a oneshot import configuration looks like:

    {
        "itemId": "importConfigId",
        "itemType": "importConfig",
        "name": "Import Config Sample",
        "description": "Sample description",
        "configType": "oneshot",        //Config type can be 'oneshot' or 'recurrent'
        "properties": {
            "mapping": {
                "email": 0,                 //<Apache Unomi Property Id> : <Column Index In the CSV>
                "firstName": 2,
                ...
            }
        },
        "columnSeparator": ",",         //Character used to separate columns
        "lineSeparator": "\\n",         //Character used to separate lines (\n or \r)
        "multiValueSeparator": ";",     //Character used to separate values for multivalued columns
        "multiValueDelimiter": "[]",    //Character used to wrap values for multivalued columns
        "status": "SUCCESS",            //Status of last execution
        "executions": [                 //(RETURN) Last executions by default only last 5 are returned
            ...
        ],
        "mergingProperty": "email",         //Apache Unomi Property Id used to check duplicates
        "overwriteExistingProfiles": true,  //Overwrite profiles that have duplicates
        "propertiesToOverwrite": "firstName, lastName, ...",      //If last is set to true, which property to overwrite, 'null' means overwrite all
        "hasHeader": true,                  //CSV file to import contains a header line
        "hasDeleteColumn": false            //CSV file to import doesn't contain a TO DELETE column (if it contains, will be the last column)
    }

A recurrent import configuration is similar to the previous one with some specific information to add to the JSON like:

  {
    ...
    "configType": "recurrent",
    "properties": {
      "source": "ftp://USER@SERVER[:PORT]/PATH?password=xxx&fileName=profiles.csv&move=.done&consumer.delay=20000",
                // Only 'ftp', 'sftp', 'ftps' and 'file' are supported in the 'source' path
                // eg. file:///tmp/?fileName=profiles.csv&move=.done&consumer.delay=25s
                // 'fileName' can be a pattern eg 'include=.*.csv' instead of 'fileName=...' to consume all CSV files
                // By default the processed files are moved to '.camel' folder you can change it using the 'move' option
                // 'consumer.delay' is the frequency of polling. '20000' (in milliseconds) means 20 seconds. Can be also '20s'
                // Other possible format are: '2h30m10s' = 2 hours and 30 minutes and 10 seconds
      "mapping": {
        ...
      }
    },
    ...
    "active": true,  //If true the polling will start according to the 'source' configured above
    ...
  }


=== Exporting profiles

Only `ftp`, `sftp`, `ftps` and `file are supported in the source path. For example:

    file:///tmp/?fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append) 
    sftp://USER@HOST/PATH?password=PASSWORD&binary=true&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append
    ftps://USER@HOST?password=PASSWORD&binary=true&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true

As you can see in the examples above, you can inject variables in the produced file name `${date:now:yyyyMMddHHmm}` is
the current date formatted with the pattern `yyyyMMddHHmm`. `fileExist` option put as `Append` will tell the file writer
to append to the same file for each execution of the export configuration. You cam omit this option to write a profile
per file.

See http://camel.apache.org/ftp.html and http://camel.apache.org/file2.html to build more complex destination path.

==== Export API

Apache Unomi provides REST endpoints to manage export configurations:

      GET /cxs/exportConfiguration
      GET /cxs/exportConfiguration/{configId}
      POST /cxs/exportConfiguration
      DELETE /cxs/exportConfiguration/{configId}

This is how a oneshot export configuration looks like:

    {
        "itemId": "exportConfigId",
        "itemType": "exportConfig",
        "name": "Export configuration sample",
        "description": "Sample description",
        "configType": "oneshot",
        "properties": {
            "period": "2m30s",
            "segment": "contacts",
            "mapping": {
                "0": "firstName",
                "1": "lastName",
                ...
            }
        },
        "columnSeparator": ",",
        "lineSeparator": "\\n",
        "multiValueSeparator": ";",
        "multiValueDelimiter": "[]",
        "status": "RUNNING",
        "executions": [
            ...
        ]
    }

A recurrent export configuration is similar to the previous one with some specific information to add to the JSON like:

    {
        ...
        "configType": "recurrent",
        "properties": {
        "destination": "sftp://USER@SERVER:PORT/PATH?password=XXX&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append",
        "period": "2m30s",      //Same as 'consumer.delay' option in the import source path
        "segment": "contacts",  //Segment ID to use to collect profiles to export
        "mapping": {
            ...
            }
        },
        ...
        "active": true,           //If true the configuration will start polling upon save until the user deactivate it
        ...
    }

=== Configuration in details

First configuration you need to change would be the configuration type of your import / export feature (code name
router) in the `etc/unomi.custom.system.properties` file (creating it if necessary):

    #Configuration Type values {'nobroker', 'kafka'}
    org.apache.unomi.router.config.type=nobroker

By default the feature is configured (as above) to use no external broker,  which means to handle import/export data it
will use in memory queues (In the same JVM as Apache Unomi). If you are clustering Apache Unomi, most important thing
to know about this type of configuration is that each Apache Unomi will handle the import/export task by itself without
the help of other nodes (No Load-Distribution).

Changing this property to kafka means you have to provide the Apache Kafka configuration, and in the opposite of the
nobroker option import/export data will be handled using an external broker (Apache Kafka), this will lighten the burden
on the Apache Unomi machines.

You may use several Apache Kafka instance, 1 per N Apache Unomi nodes for better application scaling.

To enable using Apache Kafka you need to configure the feature as follows:

    #Configuration Type values {'nobroker', 'kafka'}
    org.apache.unomi.router.config.type=kafka

Uncomment and update Kafka settings to use Kafka as a broker

    #Kafka
    org.apache.unomi.router.kafka.host=localhost
    org.apache.unomi.router.kafka.port=9092
    org.apache.unomi.router.kafka.import.topic=import-deposit
    org.apache.unomi.router.kafka.export.topic=export-deposit
    org.apache.unomi.router.kafka.import.groupId=unomi-import-group
    org.apache.unomi.router.kafka.export.groupId=unomi-import-group
    org.apache.unomi.router.kafka.consumerCount=10
    org.apache.unomi.router.kafka.autoCommit=true

There is couple of properties you may want to change to fit your needs, one of them is the *import.oneshot.uploadDir which*
will tell Apache Unomi where to store temporarily the CSV files to import in Oneshot mode, it's a technical property
to allow the choice of the convenient disk space where to store the files to be imported. It defaults to the following path
under the Apache Unomi Karaf (It is recommended to change the path to a more convenient one).

    #Import One Shot upload directory
    org.apache.unomi.router.import.oneshot.uploadDir=${karaf.data}/tmp/unomi_oneshot_import_configs/

Next two properties are max sizes for executions history and error reports, for some reason you don't want Apache Unomi
to report all the executions history and error reports generated by the executions of an import/export configuration.
To change this you have to change the default values of these properties.

    #Import/Export executions history size
    org.apache.unomi.router.executionsHistory.size=5

    #errors report size
    org.apache.unomi.router.executions.error.report.size=200

Final one is about the allowed endpoints you can use when building the source or destionation path, as mentioned above
we can have a path of type `file`, `ftp`, `ftps`, `sftp`. You can make it less if you want to omit some endpoints (eg.
you don't want to permit the use of non secure FTP).

    #Allowed source endpoints
    org.apache.unomi.router.config.allowedEndpoints=file,ftp,sftp,ftps