utils/usergrid-util-python/usergrid_tools/migration/README.md - usergrid - Git at Google

 # Usergrid Data Migrator

 ## Prerequisites
 * Python 2 (not python 3)

 * Install the Usergrid Python SDK: https://github.com/jwest-apigee/usergrid-python

 With Pip (requires python-pip to be installed): `pip install usergrid`

 * Install Usergrid Tools

 With Pip (requires python-pip to be installed): `pip install usergrid-tools`


 ## Overview
 The purpose of this document is to provide an overview of the Python Script provided in the same directory which allows you to migrate data, connections and users from one Usergrid platform / org / app to another.  This can be used in the upgrade process from Usergrid 1.0 to 2.x since there is no upgrade path.

 This script functions by taking source and target endpoint configurations (with credentials) and a set of command-line parameters to read data from one Usergrid instance and write to another.  It is written in Python and requires Python 2.7.6+.

 There are multiple processes at work in the migration to speed the process up.  There is a main thread which reads entities from the API and then publishes the entities with metadata into a Python Queue which has multiple worker processes listening for work.  The number of worker threads is configurable by command line parameters.


 # Process to Migrate Data and Graph (Connections)
 Usergrid is a Graph database and allows for connections between entities.  In order for a connection to be made, both the source entity and the target entity must exist.  Therefore, in order to migrate connections it is adviseable to first migrate all the data and then all the connections associated with that data.

 # Concepts
 As with any migration process there is a source and a target.  The source and target have the following parameters:

 * API URL: The HTTP[S] URL where the platform can be reached
 * Org: You must specify one org at a time to migrate using this script
 * App: You can optinally specify one or more applications to migrate.  If you specify zero applications then all applications will be migrated
 * Collection: You can optionally specify one or more collections to migrate.  If you specify zero collections then all applications will be migrated
 * QL: You can specify a Query Language predicate to be used.  If none is specified, 'select *' will be used which will migrate all data within a given collection
 * Graph: Graph implies traversal of graph edges which necessarily must exist.  This is an alternative to using query which uses the indexing.

 # Graph Loops

 When iterating a graph it is possible to get stuck in a loop.  For example:

 ```
 A --follows--> B
 B --likes--> C
 C --loves--> A
 ```

 There are two options to prevent getting stuck in a loop:
 * `graph_depth` option - this will limit the graph depth which will be traversed from a given entity.
 * And/Or Marking nodes and edges as 'visited'.  This requires a place to store this state.  See Using Redis in the next section

 # Using Redis

 Redis can be used for the following:

 If using Redis, version 2.8+ is needed because TTL is used with the 'ex' parameter.

 * Keeping track of the modified date for each entity.  When running the script subsequent times after this, entiites which were not modified will not be copied.
 * Keeping track of visited nodes for migrating a graph.  This is done with a TTL such that a job can be resumed, but since there is no modified date on an edge you cannot know if there are new edges or not.  Therefore, when the TTL expires the nodes will be visited again
 * Keeping track of the URLs for the connections which are created between entities.  This has no TTL.  Subsequent runs will not create connections which are found in Redis which have already been created.


 # Mapping
 Using this script it is not necessary to keep the same application name, org name and/or collection name as the source at the target.  For example, you could migrate from /myOrg/myApp/myCollection to /org123/app456/collections789.


 # Configuration Files
 Example source/target configuration files:

 ```
 {
   "endpoint": {
     "api_url": "https://api.usergrid.com"
   },
   "credentials": {
     "myOrg1": {
       "client_id": "YXA6lpg9sEaaEeONxG0g3Uz44Q",
       "client_secret": "ZdF66u2h3Hc7csOcsEtgewmxalB1Ygg"
     },
     "myOrg2": {
       "client_id": "ZXf63p239sDaaEeONSG0g3Uz44Z",
       "client_secret": "ZdF66u2h3Hc7csOcsEtgewmxajsadfj32"
     }
   }
 }
 ```
 * api_url: the API URL to access/write data
 * Credentials:
  * For each org, with the org name (case-sensetive) as the key:
   * client_id - the org-level Client ID. This can be retrieved from the BaaS/Usergrid Portal.
   * client_secret - the org-level Client Secret. This can be retrieved from the BaaS/Usergrid Portal.

 # Command Line Parameters

 ```
 Usergrid Org/App Data Migrator

 optional arguments:
   -h, --help            show this help message and exit
   --log_dir LOG_DIR     path to the place where logs will be written
   --log_level LOG_LEVEL
                         log level - DEBUG, INFO, WARN, ERROR, CRITICAL
   -o ORG, --org ORG     Name of the org to migrate
   -a APP, --app APP     Name of one or more apps to include, specify none to
                         include all apps
   -e INCLUDE_EDGE, --include_edge INCLUDE_EDGE
                         Name of one or more edges/connection types to INCLUDE,
                         specify none to include all edges
   --exclude_edge EXCLUDE_EDGE
                         Name of one or more edges/connection types to EXCLUDE,
                         specify none to include all edges
   --exclude_collection EXCLUDE_COLLECTION
                         Name of one or more collections to EXCLUDE, specify
                         none to include all collections
   -c COLLECTION, --collection COLLECTION
                         Name of one or more collections to include, specify
                         none to include all collections
   --use_name_for_collection USE_NAME_FOR_COLLECTION
                         Name of one or more collections to use [name] instead
                         of [uuid] for creating entities and edges
   -m {data,none,reput,credentials,graph}, --migrate {data,none,reput,credentials,graph}
                         Specifies what to migrate: data, connections,
                         credentials, audit or none (just iterate the
                         apps/collections)
   -s SOURCE_CONFIG, --source_config SOURCE_CONFIG
                         The path to the source endpoint/org configuration file
   -d TARGET_CONFIG, --target_config TARGET_CONFIG
                         The path to the target endpoint/org configuration file
   --limit LIMIT         The number of entities to return per query request
   -w ENTITY_WORKERS, --entity_workers ENTITY_WORKERS
                         The number of worker processes to do the migration
   --visit_cache_ttl VISIT_CACHE_TTL
                         The TTL of the cache of visiting nodes in the graph
                         for connections
   --error_retry_sleep ERROR_RETRY_SLEEP
                         The number of seconds to wait between retrieving after
                         an error
   --page_sleep_time PAGE_SLEEP_TIME
                         The number of seconds to wait between retrieving pages
                         from the UsergridQueryIterator
   --entity_sleep_time ENTITY_SLEEP_TIME
                         The number of seconds to wait between retrieving pages
                         from the UsergridQueryIterator
   --collection_workers COLLECTION_WORKERS
                         The number of worker processes to do the migration
   --queue_size_max QUEUE_SIZE_MAX
                         The max size of entities to allow in the queue
   --graph_depth GRAPH_DEPTH
                         The graph depth to traverse to copy
   --queue_watermark_high QUEUE_WATERMARK_HIGH
                         The point at which publishing to the queue will PAUSE
                         until it is at or below low watermark
   --min_modified MIN_MODIFIED
                         Break when encountering a modified date before this,
                         per collection
   --max_modified MAX_MODIFIED
                         Break when encountering a modified date after this,
                         per collection
   --queue_watermark_low QUEUE_WATERMARK_LOW
                         The point at which publishing to the queue will RESUME
                         after it has reached the high watermark
   --ql QL               The QL to use in the filter for reading data from
                         collections
   --skip_cache_read     Skip reading the cache (modified timestamps and graph
                         edges)
   --skip_cache_write    Skip updating the cache with modified timestamps of
                         entities and graph edges
   --create_apps         Create apps at the target if they do not exist
   --nohup               specifies not to use stdout for logging
   --graph               Use GRAPH instead of Query
   --su_username SU_USERNAME
                         Superuser username
   --su_password SU_PASSWORD
                         Superuser Password
   --inbound_connections
                         Name of the org to migrate
   --map_app MAP_APP     Multiple allowed: A colon-separated string such as
                         'apples:oranges' which indicates to put data from the
                         app named 'apples' from the source endpoint into app
                         named 'oranges' in the target endpoint
   --map_collection MAP_COLLECTION
                         One or more colon-separated string such as 'cats:dogs'
                         which indicates to put data from collections named
                         'cats' from the source endpoint into a collection
                         named 'dogs' in the target endpoint, applicable
                         globally to all apps
   --map_org MAP_ORG     One or more colon-separated strings such as 'red:blue'
                         which indicates to put data from org named 'red' from
                         the source endpoint into a collection named 'blue' in
                         the target endpoint
 ```

 ## Example Command Line

 Use the following command to migrate DATA AND GRAPH  (no graph edges or connections between entities).  If there are no graph edges (connections) then using `-m graph` is not necessary.  This will copy all data from all apps in the org 'myorg', creating apps in the target org if they do not already exist.  Note that --create_apps will be required if the Apps in the target org have not been created.

 ```
 $ usergrid_data_migrator -o myorg -m graph -w 4 -s mySourceConfig.json -d myTargetConfiguration.json  --create_apps
 ```

 Use the following command to migrate DATA ONLY (no graph edges or connections between entities).  This will copy all data from all apps in the org 'myorg', creating apps in the target org if they do not already exist.  Note that --create_apps will be required if the Apps in the target org have not been created.

 ```
 $ usergrid_data_migrator -o myorg -m data -w 4 -s mySourceConfig.json -d myTargetConfiguration.json --create_apps
 ```

 Use the following command to migrate CREDENTIALS for Application-level Users.  Note that `usergrid.sysadmin.login.allowed=true` must be set in the `usergrid-deployment.properties` file on the source and target Tomcat nodes.

 ```
 $ usergrid_data_migrator -o myorg -m credentails -w 4 -s mySourceConfig.json -d myTargetConfiguration.json --create_apps --su_username foo --su_password bar
 ```

 This command:

 ```
 $ usergrid_data_migrator -o myorg -a app1 -a app2 -m data -w 4 --map_app app1:appplication_1 --map_app app2:application_2 --map_collection pets:animals --map_org myorg:my_new_org -s mySourceConfig.json -d myTargetConfiguration.json
 ```
 will do the following:

 * migrate Apps named 'app1' and 'app2' in org named 'myorg' from the API endpoint defined in 'mySourceConfig.json' to the API endpoint defined in 'myTargetConfiguration.json'
 * In the process:
 ** data from 'myorg' will ge migrated to the org named 'my_new_org'
 ** data from 'app1' will be migrated to the app named 'application_1'
 ** data from 'app2' will be migrated to the app named 'application_2'
 ** all collections named 'pets' will be overridden at the destination to 'animals'


 # FAQ

 ### Does the process keep the same UUIDs?

 * Yes - with this script the same UUIDs can be kept from the source into the destination.  An exception is if you specify going from one collection to another under the same Org hierarchy.

 ### Does the process keep the ordering of connections by time?

 * Yes ordering of connections is maintained in the process.
	# Usergrid Data Migrator

	## Prerequisites
	* Python 2 (not python 3)

	* Install the Usergrid Python SDK: https://github.com/jwest-apigee/usergrid-python

	With Pip (requires python-pip to be installed): `pip install usergrid`

	* Install Usergrid Tools

	With Pip (requires python-pip to be installed): `pip install usergrid-tools`


	## Overview
	The purpose of this document is to provide an overview of the Python Script provided in the same directory which allows you to migrate data, connections and users from one Usergrid platform / org / app to another. This can be used in the upgrade process from Usergrid 1.0 to 2.x since there is no upgrade path.

	This script functions by taking source and target endpoint configurations (with credentials) and a set of command-line parameters to read data from one Usergrid instance and write to another. It is written in Python and requires Python 2.7.6+.

	There are multiple processes at work in the migration to speed the process up. There is a main thread which reads entities from the API and then publishes the entities with metadata into a Python Queue which has multiple worker processes listening for work. The number of worker threads is configurable by command line parameters.


	# Process to Migrate Data and Graph (Connections)
	Usergrid is a Graph database and allows for connections between entities. In order for a connection to be made, both the source entity and the target entity must exist. Therefore, in order to migrate connections it is adviseable to first migrate all the data and then all the connections associated with that data.

	# Concepts
	As with any migration process there is a source and a target. The source and target have the following parameters:

	* API URL: The HTTP[S] URL where the platform can be reached
	* Org: You must specify one org at a time to migrate using this script
	* App: You can optinally specify one or more applications to migrate. If you specify zero applications then all applications will be migrated
	* Collection: You can optionally specify one or more collections to migrate. If you specify zero collections then all applications will be migrated
	* QL: You can specify a Query Language predicate to be used. If none is specified, 'select *' will be used which will migrate all data within a given collection
	* Graph: Graph implies traversal of graph edges which necessarily must exist. This is an alternative to using query which uses the indexing.

	# Graph Loops

	When iterating a graph it is possible to get stuck in a loop. For example:

	```
	A --follows--> B
	B --likes--> C
	C --loves--> A
	```

	There are two options to prevent getting stuck in a loop:
	* `graph_depth` option - this will limit the graph depth which will be traversed from a given entity.
	* And/Or Marking nodes and edges as 'visited'. This requires a place to store this state. See Using Redis in the next section

	# Using Redis

	Redis can be used for the following:

	If using Redis, version 2.8+ is needed because TTL is used with the 'ex' parameter.

	* Keeping track of the modified date for each entity. When running the script subsequent times after this, entiites which were not modified will not be copied.
	* Keeping track of visited nodes for migrating a graph. This is done with a TTL such that a job can be resumed, but since there is no modified date on an edge you cannot know if there are new edges or not. Therefore, when the TTL expires the nodes will be visited again
	* Keeping track of the URLs for the connections which are created between entities. This has no TTL. Subsequent runs will not create connections which are found in Redis which have already been created.


	# Mapping
	Using this script it is not necessary to keep the same application name, org name and/or collection name as the source at the target. For example, you could migrate from /myOrg/myApp/myCollection to /org123/app456/collections789.


	# Configuration Files
	Example source/target configuration files:

	```
	{
	"endpoint": {
	"api_url": "https://api.usergrid.com"
	},
	"credentials": {
	"myOrg1": {
	"client_id": "YXA6lpg9sEaaEeONxG0g3Uz44Q",
	"client_secret": "ZdF66u2h3Hc7csOcsEtgewmxalB1Ygg"
	},
	"myOrg2": {
	"client_id": "ZXf63p239sDaaEeONSG0g3Uz44Z",
	"client_secret": "ZdF66u2h3Hc7csOcsEtgewmxajsadfj32"
	}
	}
	}
	```
	* api_url: the API URL to access/write data
	* Credentials:
	* For each org, with the org name (case-sensetive) as the key:
	* client_id - the org-level Client ID. This can be retrieved from the BaaS/Usergrid Portal.
	* client_secret - the org-level Client Secret. This can be retrieved from the BaaS/Usergrid Portal.

	# Command Line Parameters

	```
	Usergrid Org/App Data Migrator

	optional arguments:
	-h, --help show this help message and exit
	--log_dir LOG_DIR path to the place where logs will be written
	--log_level LOG_LEVEL
	log level - DEBUG, INFO, WARN, ERROR, CRITICAL
	-o ORG, --org ORG Name of the org to migrate
	-a APP, --app APP Name of one or more apps to include, specify none to
	include all apps
	-e INCLUDE_EDGE, --include_edge INCLUDE_EDGE
	Name of one or more edges/connection types to INCLUDE,
	specify none to include all edges
	--exclude_edge EXCLUDE_EDGE
	Name of one or more edges/connection types to EXCLUDE,
	specify none to include all edges
	--exclude_collection EXCLUDE_COLLECTION
	Name of one or more collections to EXCLUDE, specify
	none to include all collections
	-c COLLECTION, --collection COLLECTION
	Name of one or more collections to include, specify
	none to include all collections
	--use_name_for_collection USE_NAME_FOR_COLLECTION
	Name of one or more collections to use [name] instead
	of [uuid] for creating entities and edges
	-m {data,none,reput,credentials,graph}, --migrate {data,none,reput,credentials,graph}
	Specifies what to migrate: data, connections,
	credentials, audit or none (just iterate the
	apps/collections)
	-s SOURCE_CONFIG, --source_config SOURCE_CONFIG
	The path to the source endpoint/org configuration file
	-d TARGET_CONFIG, --target_config TARGET_CONFIG
	The path to the target endpoint/org configuration file
	--limit LIMIT The number of entities to return per query request
	-w ENTITY_WORKERS, --entity_workers ENTITY_WORKERS
	The number of worker processes to do the migration
	--visit_cache_ttl VISIT_CACHE_TTL
	The TTL of the cache of visiting nodes in the graph
	for connections
	--error_retry_sleep ERROR_RETRY_SLEEP
	The number of seconds to wait between retrieving after
	an error
	--page_sleep_time PAGE_SLEEP_TIME
	The number of seconds to wait between retrieving pages
	from the UsergridQueryIterator
	--entity_sleep_time ENTITY_SLEEP_TIME
	The number of seconds to wait between retrieving pages
	from the UsergridQueryIterator
	--collection_workers COLLECTION_WORKERS
	The number of worker processes to do the migration
	--queue_size_max QUEUE_SIZE_MAX
	The max size of entities to allow in the queue
	--graph_depth GRAPH_DEPTH
	The graph depth to traverse to copy
	--queue_watermark_high QUEUE_WATERMARK_HIGH
	The point at which publishing to the queue will PAUSE
	until it is at or below low watermark
	--min_modified MIN_MODIFIED
	Break when encountering a modified date before this,
	per collection
	--max_modified MAX_MODIFIED
	Break when encountering a modified date after this,
	per collection
	--queue_watermark_low QUEUE_WATERMARK_LOW
	The point at which publishing to the queue will RESUME
	after it has reached the high watermark
	--ql QL The QL to use in the filter for reading data from
	collections
	--skip_cache_read Skip reading the cache (modified timestamps and graph
	edges)
	--skip_cache_write Skip updating the cache with modified timestamps of
	entities and graph edges
	--create_apps Create apps at the target if they do not exist
	--nohup specifies not to use stdout for logging
	--graph Use GRAPH instead of Query
	--su_username SU_USERNAME
	Superuser username
	--su_password SU_PASSWORD
	Superuser Password
	--inbound_connections
	Name of the org to migrate
	--map_app MAP_APP Multiple allowed: A colon-separated string such as
	'apples:oranges' which indicates to put data from the
	app named 'apples' from the source endpoint into app
	named 'oranges' in the target endpoint
	--map_collection MAP_COLLECTION
	One or more colon-separated string such as 'cats:dogs'
	which indicates to put data from collections named
	'cats' from the source endpoint into a collection
	named 'dogs' in the target endpoint, applicable
	globally to all apps
	--map_org MAP_ORG One or more colon-separated strings such as 'red:blue'
	which indicates to put data from org named 'red' from
	the source endpoint into a collection named 'blue' in
	the target endpoint
	```

	## Example Command Line

	Use the following command to migrate DATA AND GRAPH (no graph edges or connections between entities). If there are no graph edges (connections) then using `-m graph` is not necessary. This will copy all data from all apps in the org 'myorg', creating apps in the target org if they do not already exist. Note that --create_apps will be required if the Apps in the target org have not been created.

	```
	$ usergrid_data_migrator -o myorg -m graph -w 4 -s mySourceConfig.json -d myTargetConfiguration.json --create_apps
	```

	Use the following command to migrate DATA ONLY (no graph edges or connections between entities). This will copy all data from all apps in the org 'myorg', creating apps in the target org if they do not already exist. Note that --create_apps will be required if the Apps in the target org have not been created.

	```
	$ usergrid_data_migrator -o myorg -m data -w 4 -s mySourceConfig.json -d myTargetConfiguration.json --create_apps
	```

	Use the following command to migrate CREDENTIALS for Application-level Users. Note that `usergrid.sysadmin.login.allowed=true` must be set in the `usergrid-deployment.properties` file on the source and target Tomcat nodes.

	```
	$ usergrid_data_migrator -o myorg -m credentails -w 4 -s mySourceConfig.json -d myTargetConfiguration.json --create_apps --su_username foo --su_password bar
	```

	This command:

	```
	$ usergrid_data_migrator -o myorg -a app1 -a app2 -m data -w 4 --map_app app1:appplication_1 --map_app app2:application_2 --map_collection pets:animals --map_org myorg:my_new_org -s mySourceConfig.json -d myTargetConfiguration.json
	```
	will do the following:

	* migrate Apps named 'app1' and 'app2' in org named 'myorg' from the API endpoint defined in 'mySourceConfig.json' to the API endpoint defined in 'myTargetConfiguration.json'
	* In the process:
	** data from 'myorg' will ge migrated to the org named 'my_new_org'
	** data from 'app1' will be migrated to the app named 'application_1'
	** data from 'app2' will be migrated to the app named 'application_2'
	** all collections named 'pets' will be overridden at the destination to 'animals'


	# FAQ

	### Does the process keep the same UUIDs?

	* Yes - with this script the same UUIDs can be kept from the source into the destination. An exception is if you specify going from one collection to another under the same Org hierarchy.

	### Does the process keep the ordering of connections by time?

	* Yes ordering of connections is maintained in the process.