AMBARI-23945. Infra Solr Migration - helper command to enable/disable/fix solr kerberos config.
diff --git a/ambari-infra-solr-client/README.md b/ambari-infra-solr-client/README.md
index 2df113b..53c6a3e 100644
--- a/ambari-infra-solr-client/README.md
+++ b/ambari-infra-solr-client/README.md
@@ -764,9 +764,16 @@
#### <a id="if-solr-restarted">What to do if Solr instances restarted right after Ambari upgrade but before upgrade Solr instance packages?</a>
-As Solr instances won't start with the new upgraded configs (only if kerberos is enabled), you can do a small fix to make it work to just add this line to `infra-solr-env/content`:
+If you restarted Solr before backup or upgrade Solr server packages, you can fix the Solr config with the following command:
```bash
-SOLR_KERB_NAME_RULES="{{infra_solr_kerberos_name_rules}}"
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py --ini-file $CONFIG_INI_LOCATION --action fix-solr5-kerberos-config
+```
+
+That is basically add `SOLR_KERB_NAME_RULES` back to `infra-solr-env/content` and disable authorization for Solr. (upload a /security.json to /infra-solr znode without the authorization config, then turn manually managed /security.json on in order to not override /security.json again on Solr restart) After the command finished successfully, you will need to restart Solr instances.
+
+But if you added `SOLR_KERB_NAME_RULES` config to the `infra-solr-env/content`, you will require to delete that after you upgraded Solr package (and before restarting them). You can do that with the `fix-solr7-kerberos-config` action:
+```bash
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py --ini-file $CONFIG_INI_LOCATION --action fix-solr7-kerberos-config
```
#### <a id="get-core-/-shard-names-with-hosts">Get core / shard names with hosts</a>
@@ -775,24 +782,17 @@
#### <a id="turn-off-infra-solr-authorization">Turn off Infra Solr Authorization</a>
-You can turn off Solr authorization plugin with setting `infra-solr-security-json/content` Ambari configuration to `{"authentication": {"class": "org.apache.solr.security.KerberosPlugin"}}` (with that authentication will be still enabled). Then you will need to restart Solr, as that config is uploaded to the `/infra-solr/security.json` znode during startup. Other option is to use zkcli.sh of an Infra Solr to upload the security.json to the right place:
-```bash
-# Setup env for zkcli.sh
-source /etc/ambari-infra-solr/conf/infra-solr-env.sh
-# Run that command only if kerberos is enabled.
-export SOLR_ZK_CREDS_AND_ACLS="${SOLR_AUTHENTICATION_OPTS}"
-ZK_CONN_STRING=... # connection string -> zookeeper server addresses with the znode, e.g.: c7401.ambari.apache.org:2181/infra-solr
-
-/usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh -zkhost $ZK_CONN_STRING -cmd put /security.json
- '{"authentication": {"class": "org.apache.solr.security.KerberosPlugin"}}'
-```
-
-Or you can also use the `migationHelper.py` script to disable the Solr authorization (for that to keep this settings, you can disable the management of the security.json in `infra-solr-security-json` config type)
-
+You can turn off Solr authorization plugin with the `disable-solr-authorization` action (can be executed after config generation [step](#0-gather-params)):
```bash
/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py --ini-file $CONFIG_INI_LOCATION --action disable-solr-authorization
```
+You can re-enable it with the following command: (or set `infra-solr-security-json/infra_solr_security_manually_managed` configuration to `false`, then restart Solr)
+
+```bash
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py --ini-file $CONFIG_INI_LOCATION --action enable-solr-authorization
+```
+
#### <a id="">Solr Migration Helper Script</a>
`/usr/lib/ambari-infra-solr-client/migrationHelper.py --help`
diff --git a/ambari-infra-solr-client/src/main/python/migrationHelper.py b/ambari-infra-solr-client/src/main/python/migrationHelper.py
index 4d63bdd..1e62b98 100755
--- a/ambari-infra-solr-client/src/main/python/migrationHelper.py
+++ b/ambari-infra-solr-client/src/main/python/migrationHelper.py
@@ -61,6 +61,8 @@
REQUESTS_API_URL = '/requests'
BATCH_REQUEST_API_URL = "/api/v1/clusters/{0}/request_schedules"
+GET_ACTUAL_CONFIG_URL = '/configurations/service_config_versions?service_name={0}&is_current=true'
+CREATE_CONFIGURATIONS_URL = '/configurations'
LIST_SOLR_COLLECTION_URL = '{0}/admin/collections?action=LIST&wt=json'
CREATE_SOLR_COLLECTION_URL = '{0}/admin/collections?action=CREATE&name={1}&collection.configName={2}&numShards={3}&replicationFactor={4}&maxShardsPerNode={5}&wt=json'
@@ -295,7 +297,6 @@
return request_schedules
-
def create_command_request(command, parameters, hosts, cluster, context, service=SOLR_SERVICE_NAME, component=SOLR_COMPONENT_NAME):
request = {}
request_info = {}
@@ -635,7 +636,84 @@
else:
return collections
-def get_solr_urls(options, config, collection, collections_json):
+def get_infra_solr_props(config, accessor):
+ cluster = config.get('ambari_server', 'cluster')
+ service_configs = get_json(accessor, CLUSTERS_URL.format(cluster) + GET_ACTUAL_CONFIG_URL.format(SOLR_SERVICE_NAME))
+ infra_solr_props = {}
+ infra_solr_env_properties = {}
+ infra_solr_security_json_properties = {}
+ if 'items' in service_configs and len(service_configs['items']) > 0:
+ if 'configurations' in service_configs['items'][0]:
+ for config in service_configs['items'][0]['configurations']:
+ if 'type' in config and config['type'] == 'infra-solr-env':
+ infra_solr_env_properties = config['properties']
+ if 'type' in config and config['type'] == 'infra-solr-security-json':
+ infra_solr_security_json_properties = config['properties']
+ infra_solr_props['infra-solr-env'] = infra_solr_env_properties
+ infra_solr_props['infra-solr-security-json'] = infra_solr_security_json_properties
+ return infra_solr_props
+
+def insert_string_before(full_str, sub_str, insert_str):
+ idx = full_str.index(sub_str)
+ return full_str[:idx] + insert_str + full_str[idx:]
+
+def set_solr_security_management(infra_solr_props, accessor, enable = True):
+ security_props = infra_solr_props['infra-solr-security-json']
+ check_value = "false" if enable else "true"
+ set_value = "true" if enable else "false"
+ turn_status = "on" if enable else "off"
+ if 'infra_solr_security_manually_managed' in security_props and security_props['infra_solr_security_manually_managed'] == check_value:
+ security_props['infra_solr_security_manually_managed'] = set_value
+ post_configuration = create_configs('infra-solr-security-json', security_props, 'Turn {0} security.json manaul management by migrationHelper.py'.format(turn_status))
+ apply_configs(config, accessor, post_configuration)
+ else:
+ print "Configuration 'infra-solr-security-json/infra_solr_security_manually_managed' has already set to '{0}'".format(set_value)
+
+def set_solr_name_rules(infra_solr_props, accessor, add = False):
+ """
+ Set name rules in infra-solr-env/content if not set in add mode, in non-add mode, remove it if exists
+ :param add: solr kerb name rules needs to be added (if false, it needs to be removed)
+ """
+ infra_solr_env_props = infra_solr_props['infra-solr-env']
+ name_rules_param = "SOLR_KERB_NAME_RULES=\"{{infra_solr_kerberos_name_rules}}\"\n"
+
+ if 'content' in infra_solr_env_props and (name_rules_param not in infra_solr_env_props['content']) is add:
+ if add:
+ print "Adding 'SOLR_KERB_NAME_RULES' to 'infra-solr-env/content'"
+ new_content = insert_string_before(infra_solr_env_props['content'], "SOLR_KERB_KEYTAB", name_rules_param)
+ infra_solr_env_props['content'] = new_content
+ post_configuration = create_configs('infra-solr-env', infra_solr_env_props, 'Add "SOLR_KERB_NAME_RULES" by migrationHelper.py')
+ apply_configs(config, accessor, post_configuration)
+ else:
+ print "Removing 'SOLR_KERB_NAME_RULES' from 'infra-solr-env/content'"
+ new_content = infra_solr_env_props['content'].replace(name_rules_param, '')
+ infra_solr_env_props['content'] = new_content
+ post_configuration = create_configs('infra-solr-env', infra_solr_env_props, 'Remove "SOLR_KERB_NAME_RULES" by migrationHelper.py')
+ apply_configs(config, accessor, post_configuration)
+ else:
+ if add:
+ print "'SOLR_KERB_NAME_RULES' has already set in configuration 'infra-solr-env/content'"
+ else:
+ print "Configuration 'infra-solr-env/content' does not contain 'SOLR_KERB_NAME_RULES'"
+
+def apply_configs(config, accessor, post_configuration):
+ cluster = config.get('ambari_server', 'cluster')
+ desired_configs_post_body = {}
+ desired_configs_post_body["Clusters"] = {}
+ desired_configs_post_body["Clusters"]["desired_configs"] = post_configuration
+ accessor(CLUSTERS_URL.format(cluster), 'PUT', json.dumps(desired_configs_post_body))
+
+def create_configs(config_type, properties, context):
+ configs_for_posts = {}
+ configuration = {}
+ configuration['type'] = config_type
+ configuration['tag'] = "version" + str(int(round(time.time() * 1000)))
+ configuration['properties'] = properties
+ configuration['service_config_version_note'] = context
+ configs_for_posts[config_type] = configuration
+ return configs_for_posts
+
+def get_solr_urls(options, config, collection, collections_json): # TODO: use proper url from collections.json
solr_urls = []
solr_hosts = None
solr_port = "8886"
@@ -1521,7 +1599,7 @@
else:
print "Collection '{0}' does not exist or filtered out. Skipping update collection state operation.".format(backup_fulltext_index_name)
-def disable_solr_authorization(options, accessor, parser, config):
+def set_solr_authorization(options, accessor, parser, config, enable_authorization, fix_kerberos_config = False):
solr_znode='/infra-solr'
if config.has_section('infra_solr') and config.has_option('infra_solr', 'znode'):
solr_znode=config.get('infra_solr', 'znode')
@@ -1529,11 +1607,24 @@
if config.has_section('cluster') and config.has_option('cluster', 'kerberos_enabled'):
kerberos_enabled=config.get('cluster', 'kerberos_enabled')
if kerberos_enabled == 'true':
- print "Disable Solr authorization by uploading a new security.json ..."
- copy_znode(options, config, COLLECTIONS_DATA_JSON_LOCATION.format("security-without-authr.json"),
+ infra_solr_props = get_infra_solr_props(config, accessor)
+ if enable_authorization:
+ print "Enable Solr security.json management by Ambari ... "
+ set_solr_security_management(infra_solr_props, accessor, enable = False)
+ if fix_kerberos_config:
+ set_solr_name_rules(infra_solr_props, accessor, False)
+ else:
+ print "Disable Solr authorization by uploading a new security.json and turn on security.json management by Ambari..."
+ set_solr_security_management(infra_solr_props, accessor, enable = True)
+ copy_znode(options, config, COLLECTIONS_DATA_JSON_LOCATION.format("security-without-authr.json"),
"{0}/security.json".format(solr_znode), copy_from_local=True)
+ if fix_kerberos_config:
+ set_solr_name_rules(infra_solr_props, accessor, True)
else:
- print "Security is not enabled. Skipping disable Solr authorization operation."
+ if fix_kerberos_config:
+ print "Security is not enabled. Skipping enable/disable Solr authorization + fix infra-solr-env kerberos config operation."
+ else:
+ print "Security is not enabled. Skipping enable/disable Solr authorization operation."
def summarize_shard_check_result(check_results, skip_warnings = False, skip_index_size = False):
warnings = 0
@@ -1611,8 +1702,9 @@
parser = optparse.OptionParser("usage: %prog [options]")
parser.add_option("-a", "--action", dest="action", type="string", help="delete-collections | backup | cleanup-znodes | backup-and-cleanup | migrate | restore |' \
- ' rolling-restart-solr | rolling-restart-atlas | rolling-restart-ranger | check-shards | check-backup-shards | disable-solr-authorization |'\
- ' upgrade-solr-clients | upgrade-solr-instances | upgrade-logsearch-portal | upgrade-logfeeders | stop-logsearch | restart-solr |restart-logsearch | restart-ranger | restart-atlas")
+ ' rolling-restart-solr | rolling-restart-atlas | rolling-restart-ranger | check-shards | check-backup-shards | enable-solr-authorization | disable-solr-authorization |'\
+ ' fix-solr5-kerberos-config | fix-solr7-kerberos-config | upgrade-solr-clients | upgrade-solr-instances | upgrade-logsearch-portal | upgrade-logfeeders | stop-logsearch |'\
+ ' restart-solr |restart-logsearch | restart-ranger | restart-atlas")
parser.add_option("-i", "--ini-file", dest="ini_file", type="string", help="Config ini file to parse (required)")
parser.add_option("-f", "--force", dest="force", default=False, action="store_true", help="force index upgrade even if it's the right version")
parser.add_option("-v", "--verbose", dest="verbose", action="store_true", help="use for verbose logging")
@@ -1745,8 +1837,14 @@
print "ATLAS service has not found in the config or filtered out."
elif options.action.lower() == 'rolling-restart-solr':
rolling_restart(options, accessor, parser, config, SOLR_SERVICE_NAME, SOLR_COMPONENT_NAME, "Rolling Restart Infra Solr Instances")
+ elif options.action.lower() == 'enable-solr-authorization':
+ set_solr_authorization(options, accessor, parser, config, True)
elif options.action.lower() == 'disable-solr-authorization':
- disable_solr_authorization(options, accessor, parser, config)
+ set_solr_authorization(options, accessor, parser, config, False)
+ elif options.action.lower() == 'fix-solr5-kerberos-config':
+ set_solr_authorization(options, accessor, parser, config, False, True)
+ elif options.action.lower() == 'fix-solr7-kerberos-config':
+ set_solr_authorization(options, accessor, parser, config, True, True)
elif options.action.lower() == 'check-shards':
check_shards(options, accessor, parser, config)
elif options.action.lower() == 'check-backup-shards':
@@ -1756,11 +1854,10 @@
else:
parser.print_help()
print 'action option is invalid (available actions: delete-collections | backup | cleanup-znodes | backup-and-cleanup | migrate | restore |' \
- ' rolling-restart-solr | rolling-restart-ranger | rolling-restart-atlas | check-shards | check-backup-shards | check-docs | disable-solr-authorization |' \
- ' upgrade-solr-clients | upgrade-solr-instances | upgrade-logsearch-portal | upgrade-logfeeders | stop-logsearch | restart-solr |' \
+ ' rolling-restart-solr | rolling-restart-ranger | rolling-restart-atlas | check-shards | check-backup-shards | check-docs | enable-solr-authorization |'\
+ ' disable-solr-authorization | fix-solr5-kerberos-config | fix-solr7-kerberos-config | upgrade-solr-clients | upgrade-solr-instances | upgrade-logsearch-portal |' \
+ ' upgrade-logfeeders | stop-logsearch | restart-solr |' \
' restart-logsearch | restart-ranger | restart-atlas)'
sys.exit(1)
- if options.verbose:
- print "Migration helper command ({0}) {1}FINISHED{2}".format(str(args), colors.OKGREEN, colors.ENDC)
- else:
- print "Migration helper command {0}FINISHED{1}".format(colors.OKGREEN, colors.ENDC)
\ No newline at end of file
+
+ print "Migration helper command {0}FINISHED{1}".format(colors.OKGREEN, colors.ENDC)
\ No newline at end of file
