Release is built against the Apache Hadoop 2.4.0, HBase-0.98.1 and Accumulo 1.5.1 artifacts
diff --git a/DISCLAIMER b/DISCLAIMER
new file mode 100644
index 0000000..f855d84
--- /dev/null
+++ b/DISCLAIMER
@@ -0,0 +1,10 @@
+
+Apache Slider is an effort undergoing incubation at The Apache Software
+Foundation (ASF), sponsored by Apache Incubator.
+
+Incubation is required of all newly accepted projects until a further review
+indicates that the infrastructure, communications, and decision making process
+have stabilized in a manner consistent with other successful ASF projects.
+While incubation status is not necessarily a reflection of the completeness
+or stability of the code, it does indicate that the project has yet
+to be fully endorsed by the ASF.
diff --git a/LICENSE.TXT b/LICENSE.txt
similarity index 73%
rename from LICENSE.TXT
rename to LICENSE.txt
index 1906a91..b029173 100644
--- a/LICENSE.TXT
+++ b/LICENSE.txt
@@ -209,7 +209,7 @@
licenses.
For Python Mock unit test framework,
- at slider-agent/src/test/python/mock
+ at slider-agent/src/test/python/mock, the BSD license:
Copyright (c) 2003-2012, Michael Foord
All rights reserved.
@@ -239,7 +239,8 @@
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
For Kokki, resource management library,
- at slider-agent/src/main/python/resource_management derived from:
+ which slider-agent/src/main/python/resource_management is derived from,
+ the BSD license:
Copyright (c) 2009 Samuel Stauffer <samuel@descolada.com>
All rights reserved.
@@ -270,7 +271,8 @@
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-For jinja2, used by resource management library as a templating engine:
+For jinja2, used by resource management library as a templating engine,
+ at slider-agent/src/main/python/jinja2, the BSD license:
Copyright (c) 2009 by the Jinja Team, see AUTHORS for more details.
@@ -302,4 +304,120 @@
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+For datatables,
+ at slider-core/src/main/resources/webapps/static/dt-1.9.4,
+ the BSD license:
+
+Copyright (c) 2008-2013, Allan Jardine All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Allan Jardine nor SpryMedia may be used to
+ endorse or promote products derived from this software without
+ specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS "AS IS" AND ANY
+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+THE POSSIBILITY OF SUCH DAMAGE.
+
+For jquery,
+ at slider-core/src/main/resources/webapps/static/jquery/jquery-1.8.2.min.js,
+ the MIT license:
+
+Copyright 2014 jQuery Foundation and other contributors
+http://jquery.com/
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+For jquery-ui,
+ at slider-core/src/main/resources/webapps/static/jquery/jquery-ui-1.9.1.custom.min.js
+ and slider-core/src/main/resources/webapps/static/jquery/themes-1.9.1/,
+ the MIT license:
+
+Copyright 2014 jQuery Foundation and other contributors,
+http://jqueryui.com/
+
+This software consists of voluntary contributions made by many
+individuals (AUTHORS.txt, http://jqueryui.com/about) For exact
+contribution history, see the revision history and logs, available
+at http://jquery-ui.googlecode.com/svn/
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+For jstree,
+ at slider-core/src/main/resources/webapps/static/jt/jquery.jstree.js,
+ the MIT license:
+
+Copyright (c) 2010 Ivan Bozhanov (vakata.com)
+http://jstree.com/
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/NOTICE.txt b/NOTICE.txt
new file mode 100644
index 0000000..86ac395
--- /dev/null
+++ b/NOTICE.txt
@@ -0,0 +1,5 @@
+Apache Slider
+Copyright 2007-2014 The Apache Software Foundation
+
+This product includes software developed at The Apache Software
+Foundation (http://www.apache.org/).
diff --git a/README.md b/README.md
index 854d70e..480502d 100644
--- a/README.md
+++ b/README.md
@@ -36,18 +36,58 @@
in the face of failures of servers in the cluster -as well as parts of the
application itself
+## Open-Source Development
+
+Apache Slider is an effort undergoing incubation at The Apache Software
+Foundation (ASF), sponsored by Apache Incubator.
+
+Incubation is required of all newly accepted projects until a further review
+indicates that the infrastructure, communications, and decision making process
+have stabilized in a manner consistent with other successful ASF projects.
+While incubation status is not necessarily a reflection of the completeness
+or stability of the code, it does indicate that the project has yet
+to be fully endorsed by the ASF.
+
+### Mailing Lists
+
+We have a single mailing list for developers and users of Slider: dev@slider.incubator.apache.org
+
+1. You can subscribe to this by emailing dev-subscribe@slider.incubator.apache.org from the
+email account to which you wish to subscribe from -and follow the instructions returned.
+1. You can unsubscribe later by emailing dev-unsubscribe@slider.incubator.apache.org
+
+There is a mailing list of every commit to the source code repository, commits@slider.incubator.apache.org.
+This is generally only of interest to active developers.
+
+
+### Bug reports
+
+Bug reports and other issues can be filed on the [Apache Jira](https://issues.apache.org/jira/) server.
+Please use the SLIDER project for filing the issues.
+
+### Source code access
+
+Read-only:
+
+* [https://git.apache.org/repos/asf/incubator-slider.git](https://git.apache.org/repos/asf/incubator-slider.git)
+* [https://github.com/apache/incubator-slider.git](https://github.com/apache/incubator-slider.git)
+
+Read-write (for committers):
+
+* [https://git-wip-us.apache.org/repos/asf/incubator-slider.git](https://git-wip-us.apache.org/repos/asf/incubator-slider.git)
+
# License
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- (http://www.apache.org/licenses/LICENSE-2.0)
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ (http://www.apache.org/licenses/LICENSE-2.0)
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
diff --git a/app-packages/accumulo-v1_5/README.txt b/app-packages/accumulo-v1_5/README.txt
index af8960d..c184b0b 100644
--- a/app-packages/accumulo-v1_5/README.txt
+++ b/app-packages/accumulo-v1_5/README.txt
@@ -1,3 +1,20 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
How to create a Slider package?
Replace the placeholder tarball for Accumulo.
diff --git a/app-packages/accumulo-v1_5/jmx_metrics.json b/app-packages/accumulo-v1_5/jmx_metrics.json
new file mode 100644
index 0000000..05a1bd9
--- /dev/null
+++ b/app-packages/accumulo-v1_5/jmx_metrics.json
@@ -0,0 +1,41 @@
+{
+ "Component": {
+ "ACCUMULO_MASTER": {
+ "masterGoalState": {
+ "metric": "/stats/masterGoalState",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "masterState": {
+ "metric": "/stats/masterState",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "deadTabletServers": {
+ "metric": "/stats/deadTabletServers",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "totals_ingestrate": {
+ "metric": "/stats/totals/ingestrate",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "totals_queryrate": {
+ "metric": "/stats/totals/queryrate",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "totals_diskrate": {
+ "metric": "/stats/totals/diskrate",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "badTabletServers": {
+ "metric": "/stats/badTabletServers",
+ "pointInTime": true,
+ "temporal": false
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/app-packages/accumulo-v1_5/metainfo.xml b/app-packages/accumulo-v1_5/metainfo.xml
index db904e2..79bef1d 100644
--- a/app-packages/accumulo-v1_5/metainfo.xml
+++ b/app-packages/accumulo-v1_5/metainfo.xml
@@ -21,7 +21,13 @@
<service>
<name>ACCUMULO</name>
<comment>
- The Apache Accumulo is a robust, scalable, high performance data storage system.
+ The Apache Accumulo sorted, distributed key/value store is a robust,
+ scalable, high performance data storage system that features cell-based
+ access control and customizable server-side processing. It is based on
+ Google's BigTable design and is built on top of Apache Hadoop,
+ Zookeeper, and Thrift.
+ Requirements:
+ 1. Ensure parent dir for path (accumulo-site/instance.dfs.dir) is accessible to the App owner.
</comment>
<version>1.5.1</version>
<exportGroups>
@@ -29,14 +35,22 @@
<name>QuickLinks</name>
<exports>
<export>
- <name>Monitor</name>
+ <name>org.apache.slider.monitor</name>
<value>http://${ACCUMULO_MONITOR_HOST}:${site.accumulo-site.monitor.port.client}</value>
</export>
+ <export>
+ <name>org.apache.slider.jmx</name>
+ <value>http://${ACCUMULO_MONITOR_HOST}:${site.accumulo-site.monitor.port.client}/xml</value>
+ </export>
</exports>
</exportGroup>
</exportGroups>
<commandOrders>
<commandOrder>
+ <command>ACCUMULO_MASTER-START</command>
+ <requires>ACCUMULO_MONITOR-INSTALLED</requires>
+ </commandOrder>
+ <commandOrder>
<command>ACCUMULO_TSERVER-START</command>
<requires>ACCUMULO_MASTER-STARTED</requires>
</commandOrder>
@@ -67,6 +81,7 @@
<component>
<name>ACCUMULO_MONITOR</name>
<category>MASTER</category>
+ <publishConfig>true</publishConfig>
<commandScript>
<script>scripts/accumulo_monitor.py</script>
<scriptType>PYTHON</scriptType>
diff --git a/app-packages/hbase-v0_96/README.txt b/app-packages/hbase-v0_96/README.txt
index 972f3c0..33c93df 100644
--- a/app-packages/hbase-v0_96/README.txt
+++ b/app-packages/hbase-v0_96/README.txt
@@ -1,14 +1,31 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
How to create a Slider package?
-Replace the placeholder tarball for Accumulo.
+Replace the placeholder tarball for HBase.
cp ~/Downloads/hbase-0.96.1-hadoop2-bin.tar.gz package/files/
rm package/files/hbase-0.96.1-hadoop2-bin.tar.gz.REPLACE
Create a zip package at the root of the package (<slider enlistment>/app-packages/hbase-v0_96/)
- zip -r hbase-v096.zip .
+ zip -r hbase_v096.zip .
Verify the content using
- unzip -l "$@" hbase-v096.zip
+ unzip -l "$@" hbase_v096.zip
While appConfig.json and resources.json are not required for the package they work
well as the default configuration for Slider apps. So its advisable that when you
diff --git a/app-packages/hbase-v0_96/appConfig.json b/app-packages/hbase-v0_96/appConfig.json
index 568bbc0..fd884cb 100644
--- a/app-packages/hbase-v0_96/appConfig.json
+++ b/app-packages/hbase-v0_96/appConfig.json
@@ -18,7 +18,8 @@
"site.global.user_group": "hadoop",
"site.global.security_enabled": "false",
"site.global.ganglia_server_host": "${NN_HOST}",
- "site.global.ganglia_server_port": "8663",
+ "site.global.ganglia_server_port": "8667",
+ "site.global.ganglia_server_id": "Application1",
"site.hbase-site.hbase.hstore.flush.retries.number": "120",
"site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
"site.hbase-site.hbase.hstore.compactionThreshold": "3",
diff --git a/app-packages/hbase-v0_96/ganglia_metrics.json b/app-packages/hbase-v0_96/ganglia_metrics.json
new file mode 100644
index 0000000..da73d48
--- /dev/null
+++ b/app-packages/hbase-v0_96/ganglia_metrics.json
@@ -0,0 +1,38 @@
+{
+ "Component": {
+ "HBASE_REGIONSERVER": {
+ "readRequestsCount": {
+ "metric": "regionserver.Server.readRequestCount",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "regions": {
+ "metric": "regionserver.Server.regionCount",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "flushQueueSize": {
+ "metric": "regionserver.Server.flushQueueLength",
+ "pointInTime": false,
+ "temporal": true
+ }
+ },
+ "HBASE_MASTER": {
+ "cluster_requests": {
+ "metric": "master.Server.clusterRequests",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "splitTime_avg_time": {
+ "metric": "master.FileSystem.HlogSplitTime_mean",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "splitSize_avg_time": {
+ "metric": "master.FileSystem.HlogSplitSize_mean",
+ "pointInTime": false,
+ "temporal": true
+ }
+ }
+ }
+}
diff --git a/app-packages/hbase-v0_96/jmx_metrics.json b/app-packages/hbase-v0_96/jmx_metrics.json
new file mode 100644
index 0000000..ca980a9
--- /dev/null
+++ b/app-packages/hbase-v0_96/jmx_metrics.json
@@ -0,0 +1,56 @@
+{
+ "Component": {
+ "HBASE_MASTER": {
+ "metricAverageLoad": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.averageLoad",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "DeadRegionServers": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.numDeadRegionServers",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "ClusterId": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.tag.clusterId",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "IsActiveMaster": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.tag.isActiveMaster",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "MasterActiveTime": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.masterActiveTime",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "MasterStartTime": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.masterStartTime",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "RegionServers": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.numRegionServers",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "ServerName": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.tag.serverName",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "ZookeeperQuorum": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.tag.zookeeperQuorum",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "cluster_requests": {
+ "metric": "Hadoop:service=HBase,name=Master,sub=Server.clusterRequests",
+ "pointInTime": true,
+ "temporal": false
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/app-packages/hbase-v0_96/metainfo.xml b/app-packages/hbase-v0_96/metainfo.xml
index d4da574..e5bcdc9 100644
--- a/app-packages/hbase-v0_96/metainfo.xml
+++ b/app-packages/hbase-v0_96/metainfo.xml
@@ -21,8 +21,10 @@
<service>
<name>HBASE</name>
<comment>
- Apache HBase is the Hadoop database, a distributed, scalable, big data
- store.
+ Apache HBase is the Hadoop database, a distributed, scalable, big data store.
+ Requirements:
+ 1. Ensure parent dir for path (hbase-site/hbase.rootdir) is accessible to the App owner.
+ 2. Ensure ZK root (hbase-site/zookeeper.znode.parent) is unique for the App instance.
</comment>
<version>0.96.0.2.1.1</version>
<type>YARN-APP</type>
@@ -32,13 +34,17 @@
<name>QuickLinks</name>
<exports>
<export>
- <name>JMX_Endpoint</name>
+ <name>org.apache.slider.jmx</name>
<value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>
</export>
<export>
- <name>Master_Status</name>
+ <name>org.apache.slider.monitor</name>
<value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/master-status</value>
</export>
+ <export>
+ <name>org.apache.slider.metrics</name>
+ <value>http://${site.global.ganglia_server_host}/cgi-bin/rrd.py?c=${site.global.ganglia_server_id}</value>
+ </export>
</exports>
</exportGroup>
</exportGroups>
diff --git a/app-packages/storm-v0_91/README.txt b/app-packages/storm-v0_91/README.txt
new file mode 100644
index 0000000..b547424
--- /dev/null
+++ b/app-packages/storm-v0_91/README.txt
@@ -0,0 +1,33 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+How to create a Slider package?
+
+Replace the placeholder tarball for Storm.
+ cp ~/Downloads/apache-storm-0.9.1.2.1.1.0-237.tar.gz package/files/
+ rm package/files/apache-storm-0.9.1.2.1.1.0-237.tar.gz.REPLACE
+
+Create a zip package at the root of the package (<slider enlistment>/app-packages/storm-v0_91/)
+ zip -r storm_v091.zip .
+
+Verify the content using
+ unzip -l "$@" storm_v091.zip
+
+While appConfig.json and resources.json are not required for the package they work
+well as the default configuration for Slider apps. So its advisable that when you
+create an application package for Slider, include sample/default resources.json and
+appConfig.json for a minimal Yarn cluster.
diff --git a/app-packages/storm-v0_91/appConfig.json b/app-packages/storm-v0_91/appConfig.json
index 7bdfd93..d6a6679 100644
--- a/app-packages/storm-v0_91/appConfig.json
+++ b/app-packages/storm-v0_91/appConfig.json
@@ -12,6 +12,12 @@
"site.global.app_root": "${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237",
"site.global.user_group": "hadoop",
"site.global.security_enabled": "false",
+ "site.global.ganglia_server_host": "${NN_HOST}",
+ "site.global.ganglia_server_id": "Application2",
+ "site.global.ganglia_enabled":"true",
+ "site.global.ganglia_server_port": "8668",
+ "site.global.rest_api_port": "${STORM_REST_API.ALLOCATED_PORT}",
+ "site.global.rest_api_admin_port": "${STORM_REST_API.ALLOCATED_PORT}",
"site.storm-site.topology.tuple.serializer": "backtype.storm.serialization.types.ListDelegateSerializer",
"site.storm-site.topology.workers": "1",
"site.storm-site.drpc.worker.threads": "64",
@@ -38,14 +44,14 @@
"site.storm-site.storm.local.mode.zmq": "false",
"site.storm-site.topology.max.task.parallelism": "null",
"site.storm-site.storm.zookeeper.port": "2181",
- "site.storm-site.nimbus.childopts": "-Xmx1024m -Djava.security.auth.login.config=/etc/storm/storm_jaas.conf -javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host={0},port=8649,wireformat31x=true,mode=multicast,config=/tmp/container002/work/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM",
- "site.storm-site.worker.childopts": "-Xmx768m -javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host={0},port=8650,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM",
+ "site.storm-site.nimbus.childopts": "-Xmx1024m -javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM",
+ "site.storm-site.worker.childopts": "-Xmx768m -javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM",
"site.storm-site.drpc.queue.size": "128",
"site.storm-site.storm.zookeeper.retry.times": "5",
"site.storm-site.nimbus.monitor.freq.secs": "10",
"site.storm-site.storm.cluster.mode": "distributed",
"site.storm-site.dev.zookeeper.path": "${AGENT_WORK_ROOT}/app/tmp/dev-storm-zookeeper",
- "site.storm-site.drpc.invocations.port": "${DRPC_SERVER.ALLOCATED_PORT}",
+ "site.storm-site.drpc.invocations.port": "0",
"site.storm-site.storm.zookeeper.root": "/storm",
"site.storm-site.logviewer.childopts": "-Xmx128m",
"site.storm-site.transactional.zookeeper.port": "null",
@@ -62,7 +68,7 @@
"site.storm-site.logviewer.appender.name": "A1",
"site.storm-site.nimbus.host": "${NIMBUS_HOST}",
"site.storm-site.ui.port": "${STORM_UI_SERVER.ALLOCATED_PORT}",
- "site.storm-site.supervisor.slots.ports": "[6700, 6701]",
+ "site.storm-site.supervisor.slots.ports": "[0, 0]",
"site.storm-site.nimbus.file.copy.expiration.secs": "600",
"site.storm-site.supervisor.monitor.frequency.secs": "3",
"site.storm-site.transactional.zookeeper.servers": "null",
@@ -77,14 +83,14 @@
"site.storm-site.topology.executor.receive.buffer.size": "1024",
"site.storm-site.topology.stats.sample.rate": "0.05",
"site.storm-site.topology.fall.back.on.java.serialization": "true",
- "site.storm-site.supervisor.childopts": "-Xmx256m -Djava.security.auth.login.config=/etc/storm/storm_jaas.conf -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=56431 -javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host={0},port=8650,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM",
+ "site.storm-site.supervisor.childopts": "-Xmx256m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=0 -javaagent:${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=${NN_HOST},port=8668,wireformat31x=true,mode=multicast,config=${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.1.2.1.1.0-237/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM",
"site.storm-site.topology.enable.message.timeouts": "true",
"site.storm-site.storm.messaging.netty.max_wait_ms": "1000",
"site.storm-site.nimbus.topology.validator": "backtype.storm.nimbus.DefaultTopologyValidator",
"site.storm-site.nimbus.supervisor.timeout.secs": "60",
"site.storm-site.topology.disruptor.wait.strategy": "com.lmax.disruptor.BlockingWaitStrategy",
"site.storm-site.nimbus.inbox.jar.expiration.secs": "3600",
- "site.storm-site.drpc.port": "3772",
+ "site.storm-site.drpc.port": "0",
"site.storm-site.topology.kryo.factory": "backtype.storm.serialization.DefaultKryoFactory",
"site.storm-site.storm.zookeeper.retry.interval": "1000",
"site.storm-site.storm.messaging.netty.max_retries": "30",
@@ -99,7 +105,7 @@
"site.storm-site.topology.trident.batch.emit.interval.millis": "500",
"site.storm-site.topology.builtin.metrics.bucket.size.secs": "60",
"site.storm-site.storm.thrift.transport": "backtype.storm.security.auth.SimpleTransportPlugin",
- "site.storm-site.logviewer.port": "8000",
+ "site.storm-site.logviewer.port": "0",
"site.storm-site.topology.debug": "false"
},
"components": {
diff --git a/app-packages/storm-v0_91/ganglia_metrics.json b/app-packages/storm-v0_91/ganglia_metrics.json
new file mode 100644
index 0000000..478649b
--- /dev/null
+++ b/app-packages/storm-v0_91/ganglia_metrics.json
@@ -0,0 +1,31 @@
+{
+ "Component": {
+ "NIMBUS_SERVER": {
+ "totalslots": {
+ "metric": "Total Slots",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "totalexecutors": {
+ "metric": "Total Executors",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "topologies": {
+ "metric": "Topologies",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "totaltasks": {
+ "metric": "Total Tasks",
+ "pointInTime": false,
+ "temporal": true
+ },
+ "usedslots": {
+ "metric": "Used Slots",
+ "pointInTime": false,
+ "temporal": true
+ }
+ }
+ }
+}
diff --git a/app-packages/storm-v0_91/jmx_metrics.json b/app-packages/storm-v0_91/jmx_metrics.json
new file mode 100644
index 0000000..fa97527
--- /dev/null
+++ b/app-packages/storm-v0_91/jmx_metrics.json
@@ -0,0 +1,31 @@
+{
+ "Component": {
+ "NIMBUS_SERVER": {
+ "FreeSlots": {
+ "metric": "$['slots.free']",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "Tasks": {
+ "metric": "$['tasks.total']",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "Executors": {
+ "metric": "$['executors.total']",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "Topologies": {
+ "metric": "$['topologies']",
+ "pointInTime": true,
+ "temporal": false
+ },
+ "NimbusUptime": {
+ "metric": "$['nimbus.uptime']",
+ "pointInTime": true,
+ "temporal": false
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/app-packages/storm-v0_91/metainfo.xml b/app-packages/storm-v0_91/metainfo.xml
index 5c77eb5..9913d8c 100644
--- a/app-packages/storm-v0_91/metainfo.xml
+++ b/app-packages/storm-v0_91/metainfo.xml
@@ -24,8 +24,32 @@
<comment>Apache Hadoop Stream processing framework</comment>
<version>0.9.1.2.1</version>
+ <exportGroups>
+ <exportGroup>
+ <name>QuickLinks</name>
+ <exports>
+ <export>
+ <name>org.apache.slider.jmx</name>
+ <value>http://${STORM_REST_API_HOST}:${site.global.rest_api_port}/api/cluster/summary</value>
+ </export>
+ <export>
+ <name>org.apache.slider.monitor</name>
+ <value>http://${STORM_UI_SERVER_HOST}:${site.storm-site.ui.port}</value>
+ </export>
+ <export>
+ <name>org.apache.slider.metrics</name>
+ <value>http://${site.global.ganglia_server_host}/cgi-bin/rrd.py?c=${site.global.ganglia_server_id}</value>
+ </export>
+ </exports>
+ </exportGroup>
+ </exportGroups>
+
<commandOrders>
<commandOrder>
+ <command>NIMBUS-START</command>
+ <requires>SUPERVISOR-INSTALLED,STORM_UI_SERVER-INSTALLED,DRPC_SERVER-INSTALLED,STORM_REST_API-INSTALLED</requires>
+ </commandOrder>
+ <commandOrder>
<command>SUPERVISOR-START</command>
<requires>NIMBUS-STARTED</requires>
</commandOrder>
@@ -78,6 +102,7 @@
<component>
<name>STORM_UI_SERVER</name>
<category>MASTER</category>
+ <publishConfig>true</publishConfig>
<commandScript>
<script>scripts/ui_server.py</script>
<scriptType>PYTHON</scriptType>
diff --git a/app-packages/storm-v0_91/package/scripts/params.py b/app-packages/storm-v0_91/package/scripts/params.py
index 784069d..cf21b27 100644
--- a/app-packages/storm-v0_91/package/scripts/params.py
+++ b/app-packages/storm-v0_91/package/scripts/params.py
@@ -35,14 +35,18 @@
nimbus_host = config['configurations']['storm-site']['nimbus.host']
nimbus_port = config['configurations']['storm-site']['nimbus.thrift.port']
nimbus_host = config['configurations']['storm-site']['nimbus.host']
-rest_api_port = "8745"
-rest_api_admin_port = "8746"
+rest_api_port = config['configurations']['global']['rest_api_port']
+rest_api_admin_port = config['configurations']['global']['rest_api_admin_port']
rest_api_conf_file = format("{conf_dir}/config.yaml")
rest_lib_dir = format("{app_root}/contrib/storm-rest")
storm_bin = format("{app_root}/bin/storm")
-ganglia_installed = False
-
+ganglia_installed = config['configurations']['global']['ganglia_enabled']
+if ganglia_installed:
+ ganglia_report_interval = 60
+ ganglia_server = config['configurations']['global']['ganglia_server_host']
+ ganglia_port = config['configurations']['global']['ganglia_server_port']
+
_authentication = config['configurations']['core-site']['hadoop.security.authentication']
security_enabled = ( not is_empty(_authentication) and _authentication == 'kerberos')
@@ -50,5 +54,5 @@
_hostname_lowercase = config['hostname'].lower()
_kerberos_domain = config['configurations']['global']['kerberos_domain']
_storm_principal_name = config['configurations']['global']['storm_principal_name']
- storm_jaas_principal = _storm_principal_name.replace('_HOST',_hostname_lowercase)
+ storm_jaas_principal = _storm_principal_name.replace('_HOST', _hostname_lowercase)
storm_keytab_path = config['configurations']['global']['storm_keytab']
diff --git a/app-packages/storm-v0_91/package/templates/config.yaml.j2 b/app-packages/storm-v0_91/package/templates/config.yaml.j2
index 89c8e67..32d2c99 100644
--- a/app-packages/storm-v0_91/package/templates/config.yaml.j2
+++ b/app-packages/storm-v0_91/package/templates/config.yaml.j2
@@ -35,7 +35,8 @@
reportInterval: {{ganglia_report_interval}}
# the hostname of the gmond server where storm cluster metrics will be sent
- host: "{{ganglia_server}}"
+ host: {{ganglia_server}}
+ port: {{ganglia_port}}
# address mode
# default is MULTICAST
diff --git a/pom.xml b/pom.xml
index 8810da5..51f0a10 100644
--- a/pom.xml
+++ b/pom.xml
@@ -19,7 +19,7 @@
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
<name>Slider</name>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<packaging>pom</packaging>
<description>
@@ -152,7 +152,7 @@
<groovy.version>2.2.2</groovy.version>
<!--
- options
+ test options
-->
<test.forkedProcessTimeoutInSeconds>18000</test.forkedProcessTimeoutInSeconds>
@@ -218,6 +218,8 @@
<maven.cobertura.version>2.5.2</maven.cobertura.version>
<maven-compiler-plugin.version>3.1</maven-compiler-plugin.version>
<maven-dependency-plugin.version>2.8</maven-dependency-plugin.version>
+ <maven-deploy-plugin.version>2.7</maven-deploy-plugin.version>
+ <maven-doxia-module-markdown.version>1.4</maven-doxia-module-markdown.version>
<maven-enforcer-plugin.version>1.0</maven-enforcer-plugin.version>
<maven-jar-plugin.version>2.3.1</maven-jar-plugin.version>
<maven.javadoc.version>2.8</maven.javadoc.version>
@@ -235,14 +237,12 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<build.dependency.fail.on.warning>true</build.dependency.fail.on.warning>
<build.redirect.test.output.to.file>false</build.redirect.test.output.to.file>
- <maven-doxia-module-markdown.version>1.4</maven-doxia-module-markdown.version>
<!-- github options-->
<github.global.server>github</github.global.server>
<github.site.plugin.version>0.8</github.site.plugin.version>
<maven-site-plugin.skipDeploy>true</maven-site-plugin.skipDeploy>
<github.downloads.plugin.version>0.6</github.downloads.plugin.version>
- <maven-deploy-plugin.version>2.7</maven-deploy-plugin.version>
</properties>
@@ -347,6 +347,7 @@
<exclude>**/httpfs-signature.secret</exclude>
<exclude>**/dfs.exclude</exclude>
<exclude>**/*.iml</exclude>
+ <exclude>DISCLAIMER</exclude>
</excludes>
</configuration>
</plugin>
diff --git a/slider-agent/pom.xml b/slider-agent/pom.xml
index f2fdae1..c0477d3 100644
--- a/slider-agent/pom.xml
+++ b/slider-agent/pom.xml
@@ -19,12 +19,12 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>slider-agent</artifactId>
<packaging>pom</packaging>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider Agent</name>
<description>Slider Agent</description>
<properties>
diff --git a/slider-agent/src/main/python/agent/ActionQueue.py b/slider-agent/src/main/python/agent/ActionQueue.py
index b37c94b..d4d8bc2 100644
--- a/slider-agent/src/main/python/agent/ActionQueue.py
+++ b/slider-agent/src/main/python/agent/ActionQueue.py
@@ -168,6 +168,8 @@
roleResult['configurationTags'] = command['configurationTags']
if Constants.ALLOCATED_PORTS in commandresult:
roleResult['allocatedPorts'] = commandresult[Constants.ALLOCATED_PORTS]
+ if Constants.FOLDERS in commandresult:
+ roleResult['folders'] = commandresult[Constants.FOLDERS]
self.commandStatuses.put_command_status(command, roleResult)
# Store action result to agent response queue
diff --git a/slider-agent/src/main/python/agent/Constants.py b/slider-agent/src/main/python/agent/Constants.py
index b49bda3..b937cd2 100644
--- a/slider-agent/src/main/python/agent/Constants.py
+++ b/slider-agent/src/main/python/agent/Constants.py
@@ -24,3 +24,6 @@
EXIT_CODE = "exitcode"
ALLOCATED_PORTS = "allocated_ports"
+FOLDERS = "folders"
+AGENT_WORK_ROOT = "AGENT_WORK_ROOT"
+AGENT_LOG_ROOT = "AGENT_LOG_ROOT"
diff --git a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
index 20e55df..328084d 100644
--- a/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
+++ b/slider-agent/src/main/python/agent/CustomServiceOrchestrator.py
@@ -125,6 +125,13 @@
if Constants.EXIT_CODE in ret and ret[Constants.EXIT_CODE] == 0:
ret[Constants.ALLOCATED_PORTS] = allocated_port
+
+ # Irrespective of the outcome report the folder paths
+ if command_name == 'INSTALL':
+ ret[Constants.FOLDERS] = {
+ Constants.AGENT_LOG_ROOT : self.config.getLogPath(),
+ Constants.AGENT_WORK_ROOT : self.config.getWorkRootPath()
+ }
return ret
diff --git a/slider-agent/src/main/python/agent/Heartbeat.py b/slider-agent/src/main/python/agent/Heartbeat.py
index dd1e8ab..8192348 100644
--- a/slider-agent/src/main/python/agent/Heartbeat.py
+++ b/slider-agent/src/main/python/agent/Heartbeat.py
@@ -22,6 +22,7 @@
import logging
import time
from pprint import pformat
+import hostname
from ActionQueue import ActionQueue
import AgentConfig
@@ -46,7 +47,8 @@
heartbeat = {'responseId': int(id),
'timestamp': timestamp,
'hostname': self.config.getLabel(),
- 'nodeStatus': nodeStatus
+ 'nodeStatus': nodeStatus,
+ 'fqdn': hostname.public_hostname()
}
commandsInProgress = False
diff --git a/slider-agent/src/test/python/agent/TestActionQueue.py b/slider-agent/src/test/python/agent/TestActionQueue.py
index 3dad2de..2e1e4cf 100644
--- a/slider-agent/src/test/python/agent/TestActionQueue.py
+++ b/slider-agent/src/test/python/agent/TestActionQueue.py
@@ -363,7 +363,8 @@
'taskId': 3,
'structuredOut': '',
'exitcode': 0,
- 'allocatedPorts': {}}
+ 'allocatedPorts': {},
+ 'folders': {'AGENT_LOG_ROOT': tempdir, 'AGENT_WORK_ROOT': tempdir}}
self.assertEqual(len(report['reports']), 1)
self.assertEqual(report['reports'][0], expected)
self.assertTrue(os.path.isfile(configname))
diff --git a/slider-agent/src/test/python/agent/TestHeartbeat.py b/slider-agent/src/test/python/agent/TestHeartbeat.py
index 9ce8a80..b60c14c 100644
--- a/slider-agent/src/test/python/agent/TestHeartbeat.py
+++ b/slider-agent/src/test/python/agent/TestHeartbeat.py
@@ -64,7 +64,7 @@
self.assertEquals(result['nodeStatus']['cause'], "NONE")
self.assertEquals(result['nodeStatus']['status'], "HEALTHY")
# result may or may NOT have an agentEnv structure in it
- self.assertEquals((len(result) is 4) or (len(result) is 5), True)
+ self.assertEquals((len(result) is 5) or (len(result) is 6), True)
self.assertEquals(not heartbeat.reports, True,
"Heartbeat should not contain task in progress")
@@ -132,10 +132,11 @@
hb = heartbeat.build({}, 10)
hb['hostname'] = 'hostname'
hb['timestamp'] = 'timestamp'
+ hb['fqdn'] = 'fqdn'
expected = {'nodeStatus':
{'status': 'HEALTHY',
'cause': 'NONE'},
- 'timestamp': 'timestamp', 'hostname': 'hostname',
+ 'timestamp': 'timestamp', 'hostname': 'hostname', 'fqdn': 'fqdn',
'responseId': 10, 'reports': [
{'status': 'IN_PROGRESS', 'roleCommand': u'INSTALL',
'serviceName': u'HDFS', 'role': u'DATANODE', 'actionId': '1-1',
diff --git a/slider-assembly/pom.xml b/slider-assembly/pom.xml
index cfbf56c..b1beea2 100644
--- a/slider-assembly/pom.xml
+++ b/slider-assembly/pom.xml
@@ -17,7 +17,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>slider-assembly</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider Assembly</name>
<packaging>pom</packaging>
<description>
@@ -28,7 +28,7 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
</parent>
<build>
diff --git a/slider-core/pom.xml b/slider-core/pom.xml
index 100102b..925eb44 100644
--- a/slider-core/pom.xml
+++ b/slider-core/pom.xml
@@ -18,13 +18,13 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>slider-core</artifactId>
<name>Slider Core</name>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<packaging>jar</packaging>
<description>Core Slider Module</description>
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
</parent>
<build>
@@ -419,6 +419,7 @@
</exclusion>
</exclusions>
</dependency>
+<!--
<dependency>
<groupId>org.apache.hbase</groupId>
@@ -432,19 +433,9 @@
<scope>test</scope>
</dependency>
- <!--
- <dependency>
- <groupId>org.apache.accumulo</groupId>
- <artifactId>accumulo</artifactId>
- <version>${accumulo.version}</version>
- <exclusions>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- -->
+-->
+
+<!--
<dependency>
<groupId>org.apache.accumulo</groupId>
@@ -469,6 +460,7 @@
<artifactId>accumulo-trace</artifactId>
<scope>test</scope>
</dependency>
+-->
<dependency>
<groupId>junit</groupId>
diff --git a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
index 8665622..95c120c 100644
--- a/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
+++ b/slider-core/src/main/java/org/apache/slider/client/SliderClient.java
@@ -23,6 +23,7 @@
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.yarn.api.records.ApplicationId;
@@ -93,7 +94,7 @@
import org.apache.slider.core.registry.info.RegisteredEndpoint;
import org.apache.slider.core.registry.info.ServiceInstanceData;
import org.apache.slider.core.registry.retrieve.RegistryRetriever;
-import org.apache.slider.core.registry.zk.ZKPathBuilder;
+import org.apache.slider.core.zk.ZKPathBuilder;
import org.apache.slider.providers.AbstractClientProvider;
import org.apache.slider.providers.SliderProviderFactory;
import org.apache.slider.providers.agent.AgentKeys;
@@ -156,6 +157,8 @@
*/
public SliderClient() {
super("Slider Client");
+ new HdfsConfiguration();
+ new YarnConfiguration();
}
@Override
@@ -2187,7 +2190,7 @@
* @throws IOException
* @throws YarnException
*/
- public List<String> listRegistryInstanceIDs() throws
+ public List<String> listRegistedSliderInstances() throws
IOException,
YarnException {
try {
diff --git a/slider-core/src/main/java/org/apache/slider/common/params/CommonArgs.java b/slider-core/src/main/java/org/apache/slider/common/params/CommonArgs.java
index d35e68e..4cee1d1 100644
--- a/slider-core/src/main/java/org/apache/slider/common/params/CommonArgs.java
+++ b/slider-core/src/main/java/org/apache/slider/common/params/CommonArgs.java
@@ -73,7 +73,7 @@
/**
* fields
*/
- public JCommander commander;
+ public final JCommander commander;
private final String[] args;
private AbstractActionArgs coreAction;
diff --git a/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java b/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
index a083691..714322c 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/CoreFileSystem.java
@@ -542,4 +542,10 @@
clusterSpecPath);
}
}
+
+ public Path fileToPath(File file) {
+ return new Path(file.getAbsoluteFile().toURI());
+
+ }
+
}
diff --git a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
index bdc381e..5b246d0 100644
--- a/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/common/tools/SliderUtils.java
@@ -36,6 +36,7 @@
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.ExitUtil;
import org.apache.hadoop.util.VersionInfo;
+import org.apache.hadoop.yarn.api.ApplicationConstants;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.Container;
import org.apache.hadoop.yarn.api.records.LocalResource;
@@ -1358,6 +1359,27 @@
}
return result;
}
+
+
+ /**
+ * Truncate the given string to a maximum length provided
+ * with a pad (...) added to the end if expected size if more than 10.
+ * @param toTruncate
+ * @param maxSize
+ * @return
+ */
+ public static String truncate(String toTruncate, int maxSize) {
+ if(toTruncate == null || maxSize < 1
+ || toTruncate.length() <= maxSize) {
+ return toTruncate;
+ }
+
+ String pad = "...";
+ if(maxSize < 10) {
+ pad = "";
+ }
+ return toTruncate.substring(0, maxSize - pad.length()).concat(pad);
+ }
/**
diff --git a/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java b/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java
index 54a618a..0580013 100644
--- a/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java
+++ b/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java
@@ -40,8 +40,8 @@
import org.apache.slider.core.persist.InstancePaths;
import org.apache.slider.core.persist.LockAcquireFailedException;
import org.apache.slider.core.persist.LockHeldAction;
-import org.apache.slider.core.registry.zk.ZKPathBuilder;
-import org.apache.slider.core.registry.zk.ZookeeperUtils;
+import org.apache.slider.core.zk.ZKPathBuilder;
+import org.apache.slider.core.zk.ZookeeperUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/slider-core/src/main/java/org/apache/slider/core/conf/AbstractInputPropertiesValidator.java b/slider-core/src/main/java/org/apache/slider/core/conf/AbstractInputPropertiesValidator.java
index 6c27c22..336b4dc 100644
--- a/slider-core/src/main/java/org/apache/slider/core/conf/AbstractInputPropertiesValidator.java
+++ b/slider-core/src/main/java/org/apache/slider/core/conf/AbstractInputPropertiesValidator.java
@@ -35,7 +35,9 @@
protected void validateComponentProperties(ConfTreeOperations props)
throws BadConfigException {
for (String compName : props.getComponentNames()) {
- for (String key : props.getComponent(compName).keySet()) {
+ MapOperations mo = props.getComponent(compName);
+ if (mo == null) continue;
+ for (String key : mo.keySet()) {
validatePropertyNamePrefix(key);
}
}
diff --git a/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java b/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
index 8d90225..e8b6eb1 100644
--- a/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
+++ b/slider-core/src/main/java/org/apache/slider/core/launch/JavaCommandLineBuilder.java
@@ -65,4 +65,14 @@
public void sysprop(String property, String value) {
add("-D" + property + "=" + value);
}
+
+ public JavaCommandLineBuilder forceIPv4() {
+ sysprop("java.net.preferIPv4Stack", "true");
+ return this;
+ }
+
+ public JavaCommandLineBuilder headless() {
+ sysprop("java.awt.headless", "true");
+ return this;
+ }
}
diff --git a/slider-core/src/main/java/org/apache/slider/core/main/ServiceLauncher.java b/slider-core/src/main/java/org/apache/slider/core/main/ServiceLauncher.java
index 8b68e9e..e5e72fa 100644
--- a/slider-core/src/main/java/org/apache/slider/core/main/ServiceLauncher.java
+++ b/slider-core/src/main/java/org/apache/slider/core/main/ServiceLauncher.java
@@ -83,7 +83,7 @@
* Name of the "--conf" argument.
*/
public static final String ARG_CONF = "--conf";
- static int SHUTDOWN_TIME_ON_INTERRUPT = 30 * 1000;
+ static final int SHUTDOWN_TIME_ON_INTERRUPT = 30 * 1000;
private volatile S service;
private int serviceExitCode;
diff --git a/slider-core/src/main/java/org/apache/slider/core/registry/info/ServiceInstanceData.java b/slider-core/src/main/java/org/apache/slider/core/registry/info/ServiceInstanceData.java
index c4154c0..1d8c561 100644
--- a/slider-core/src/main/java/org/apache/slider/core/registry/info/ServiceInstanceData.java
+++ b/slider-core/src/main/java/org/apache/slider/core/registry/info/ServiceInstanceData.java
@@ -26,7 +26,12 @@
import java.util.Map;
/**
- * Service instance data to serialize with JSON
+ * Service instance data to serialize with JSON.
+ *
+ * The equality and hash codes are derived from the
+ * service type and ID, which aren't final so that JSON marshalling
+ * works. Do not change these fields if an instance is stored
+ * in a map
*/
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
@@ -47,11 +52,53 @@
*/
public RegistryView externalView = new RegistryView();
+ public ServiceInstanceData() {
+ }
+
+ public ServiceInstanceData(String id, String serviceType) {
+ this.serviceType = serviceType;
+ this.id = id;
+ }
+
+ /**
+ * Instances are equal if they look after the same service type
+ * and name
+ * @param o other
+ * @return true if id and type match
+ */
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ ServiceInstanceData that = (ServiceInstanceData) o;
+
+ if (!id.equals(that.id)) {
+ return false;
+ }
+ if (!serviceType.equals(that.serviceType)) {
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public int hashCode() {
+ int result = serviceType.hashCode();
+ result = 31 * result + id.hashCode();
+ return result;
+ }
+
@Override
public String toString() {
final StringBuilder sb =
new StringBuilder("ServiceInstanceData{");
- sb.append(", id='").append(id).append('\'');
+ sb.append("id='").append(id).append('\'');
sb.append(", serviceType='").append(serviceType).append('\'');
sb.append('}');
return sb.toString();
diff --git a/slider-core/src/main/java/org/apache/slider/core/registry/zk/BlockingZKWatcher.java b/slider-core/src/main/java/org/apache/slider/core/zk/BlockingZKWatcher.java
similarity index 97%
rename from slider-core/src/main/java/org/apache/slider/core/registry/zk/BlockingZKWatcher.java
rename to slider-core/src/main/java/org/apache/slider/core/zk/BlockingZKWatcher.java
index 68c617c..62ebff3 100644
--- a/slider-core/src/main/java/org/apache/slider/core/registry/zk/BlockingZKWatcher.java
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/BlockingZKWatcher.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.slider.core.registry.zk;
+package org.apache.slider.core.zk;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
diff --git a/slider-core/src/main/java/org/apache/slider/core/zk/MiniZooKeeperCluster.java b/slider-core/src/main/java/org/apache/slider/core/zk/MiniZooKeeperCluster.java
new file mode 100644
index 0000000..c8b3adb
--- /dev/null
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/MiniZooKeeperCluster.java
@@ -0,0 +1,423 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.slider.core.zk;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.service.AbstractService;
+import org.apache.zookeeper.server.NIOServerCnxnFactory;
+import org.apache.zookeeper.server.ZooKeeperServer;
+import org.apache.zookeeper.server.persistence.FileTxnLog;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+
+
+/**
+ * This is a version of the HBase ZK cluster cut out to be standalone.
+ *
+ * <i>Important: keep this Java6 language level for now</i>
+ */
+public class MiniZooKeeperCluster extends AbstractService {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ MiniZooKeeperCluster.class);
+
+ private static final int TICK_TIME = 2000;
+ private static final int CONNECTION_TIMEOUT = 30000;
+ public static final int MAX_CLIENT_CONNECTIONS = 1000;
+
+ private boolean started;
+
+ /** The default port. If zero, we use a random port. */
+ private int defaultClientPort = 0;
+
+ private int clientPort;
+
+ private final List<NIOServerCnxnFactory> standaloneServerFactoryList;
+ private final List<ZooKeeperServer> zooKeeperServers;
+ private final List<Integer> clientPortList;
+
+ private int activeZKServerIndex;
+ private int tickTime = 0;
+ private File baseDir;
+ private final int numZooKeeperServers;
+ private String zkQuorum = "";
+
+ public MiniZooKeeperCluster(int numZooKeeperServers) {
+ super("MiniZooKeeperCluster");
+ this.numZooKeeperServers = numZooKeeperServers;
+ this.started = false;
+ activeZKServerIndex = -1;
+ zooKeeperServers = new ArrayList<ZooKeeperServer>();
+ clientPortList = new ArrayList<Integer>();
+ standaloneServerFactoryList = new ArrayList<NIOServerCnxnFactory>();
+ }
+
+
+ @Override
+ protected void serviceInit(Configuration conf) throws Exception {
+ super.serviceInit(conf);
+ }
+
+ public void setDefaultClientPort(int clientPort) {
+ if (clientPort <= 0) {
+ throw new IllegalArgumentException("Invalid default ZK client port: "
+ + clientPort);
+ }
+ this.defaultClientPort = clientPort;
+ }
+
+ /**
+ * Selects a ZK client port. Returns the default port if specified.
+ * Otherwise, returns a random port. The random port is selected from the
+ * range between 49152 to 65535. These ports cannot be registered with IANA
+ * and are intended for dynamic allocation (see http://bit.ly/dynports).
+ */
+ private int selectClientPort(Random r) {
+ if (defaultClientPort > 0) {
+ return defaultClientPort;
+ }
+ return 0xc000 + r.nextInt(0x3f00);
+ }
+
+ public void setTickTime(int tickTime) {
+ this.tickTime = tickTime;
+ }
+
+ public int getBackupZooKeeperServerNum() {
+ return zooKeeperServers.size() - 1;
+ }
+
+ public int getZooKeeperServerNum() {
+ return zooKeeperServers.size();
+ }
+
+ // / XXX: From o.a.zk.t.ClientBase
+ private static void setupTestEnv() {
+ // during the tests we run with 100K prealloc in the logs.
+ // on windows systems prealloc of 64M was seen to take ~15seconds
+ // resulting in test failure (client timeout on first session).
+ // set env and directly in order to handle static init/gc issues
+ System.setProperty("zookeeper.preAllocSize", "100");
+ FileTxnLog.setPreallocSize(100 * 1024);
+ }
+
+ @Override
+ protected void serviceStart() throws Exception {
+ startup();
+ }
+
+ /**
+ * @param baseDir
+ * @param numZooKeeperServers
+ * @return ClientPort server bound to, -1 if there was a
+ * binding problem and we couldn't pick another port.
+ * @throws IOException
+ * @throws InterruptedException
+ */
+ private int startup() throws IOException,
+ InterruptedException {
+ if (numZooKeeperServers <= 0)
+ return -1;
+
+ setupTestEnv();
+ started = true;
+ baseDir = File.createTempFile("zookeeper", ".dir");
+ recreateDir(baseDir);
+
+ StringBuilder quorumList = new StringBuilder();
+ Random rnd = new Random();
+ int tentativePort = selectClientPort(rnd);
+
+ // running all the ZK servers
+ for (int i = 0; i < numZooKeeperServers; i++) {
+ File dir = new File(baseDir, "zookeeper_" + i).getAbsoluteFile();
+ recreateDir(dir);
+ int tickTimeToUse;
+ if (this.tickTime > 0) {
+ tickTimeToUse = this.tickTime;
+ } else {
+ tickTimeToUse = TICK_TIME;
+ }
+ ZooKeeperServer server = new ZooKeeperServer(dir, dir, tickTimeToUse);
+ NIOServerCnxnFactory standaloneServerFactory;
+ while (true) {
+ try {
+ standaloneServerFactory = new NIOServerCnxnFactory();
+ standaloneServerFactory.configure(
+ new InetSocketAddress(tentativePort),
+ MAX_CLIENT_CONNECTIONS
+ );
+ } catch (BindException e) {
+ LOG.debug("Failed binding ZK Server to client port: " +
+ tentativePort, e);
+ // We're told to use some port but it's occupied, fail
+ if (defaultClientPort > 0) return -1;
+ // This port is already in use, try to use another.
+ tentativePort = selectClientPort(rnd);
+ continue;
+ }
+ break;
+ }
+
+ // Start up this ZK server
+ standaloneServerFactory.startup(server);
+ if (!waitForServerUp(tentativePort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for startup of standalone server");
+ }
+
+ // We have selected this port as a client port.
+ clientPortList.add(tentativePort);
+ standaloneServerFactoryList.add(standaloneServerFactory);
+ zooKeeperServers.add(server);
+ if (quorumList.length() > 0) {
+ quorumList.append(",");
+ }
+ quorumList.append("localhost:").append(tentativePort);
+ tentativePort++; //for the next server
+ }
+
+ // set the first one to be active ZK; Others are backups
+ activeZKServerIndex = 0;
+
+ clientPort = clientPortList.get(activeZKServerIndex);
+ zkQuorum = quorumList.toString();
+ LOG.info("Started MiniZK Cluster and connect 1 ZK server " +
+ "on client port: " + clientPort);
+ return clientPort;
+ }
+
+ private void recreateDir(File dir) throws IOException {
+ if (dir.exists()) {
+ if (!FileUtil.fullyDelete(dir)) {
+ throw new IOException("Could not delete zk base directory: " + dir);
+ }
+ }
+ try {
+ dir.mkdirs();
+ } catch (SecurityException e) {
+ throw new IOException("creating dir: " + dir, e);
+ }
+ }
+
+ /**
+ * Delete the basedir
+ */
+ private void deleteBaseDir() {
+ if (baseDir != null) {
+ baseDir.delete();
+ baseDir = null;
+ }
+
+ }
+
+ @Override
+ protected void serviceStop() throws Exception {
+
+ if (!started) {
+ return;
+ }
+ started = false;
+
+ try {
+ // shut down all the zk servers
+ for (int i = 0; i < standaloneServerFactoryList.size(); i++) {
+ NIOServerCnxnFactory standaloneServerFactory =
+ standaloneServerFactoryList.get(i);
+ int clientPort = clientPortList.get(i);
+
+ standaloneServerFactory.shutdown();
+ if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for shutdown of standalone server");
+ }
+ }
+ for (ZooKeeperServer zkServer : zooKeeperServers) {
+ //explicitly close ZKDatabase since ZookeeperServer does not close them
+ zkServer.getZKDatabase().close();
+ }
+ } finally {
+ // clear everything
+ activeZKServerIndex = 0;
+ standaloneServerFactoryList.clear();
+ clientPortList.clear();
+ zooKeeperServers.clear();
+ }
+
+ LOG.info("Shutdown MiniZK cluster with all ZK servers");
+ }
+
+ /**@return clientPort return clientPort if there is another ZK backup can run
+ * when killing the current active; return -1, if there is no backups.
+ * @throws IOException
+ * @throws InterruptedException
+ */
+ public int killCurrentActiveZooKeeperServer() throws IOException,
+ InterruptedException {
+ if (!started || activeZKServerIndex < 0) {
+ return -1;
+ }
+
+ // Shutdown the current active one
+ NIOServerCnxnFactory standaloneServerFactory =
+ standaloneServerFactoryList.get(activeZKServerIndex);
+ int clientPort = clientPortList.get(activeZKServerIndex);
+
+ standaloneServerFactory.shutdown();
+ if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for shutdown of standalone server");
+ }
+
+ zooKeeperServers.get(activeZKServerIndex).getZKDatabase().close();
+
+ // remove the current active zk server
+ standaloneServerFactoryList.remove(activeZKServerIndex);
+ clientPortList.remove(activeZKServerIndex);
+ zooKeeperServers.remove(activeZKServerIndex);
+ LOG.info("Kill the current active ZK servers in the cluster " +
+ "on client port: " + clientPort);
+
+ if (standaloneServerFactoryList.size() == 0) {
+ // there is no backup servers;
+ return -1;
+ }
+ clientPort = clientPortList.get(activeZKServerIndex);
+ LOG.info("Activate a backup zk server in the cluster " +
+ "on client port: " + clientPort);
+ // return the next back zk server's port
+ return clientPort;
+ }
+
+ /**
+ * Kill one back up ZK servers
+ * @throws IOException
+ * @throws InterruptedException
+ */
+ public void killOneBackupZooKeeperServer() throws IOException,
+ InterruptedException {
+ if (!started || activeZKServerIndex < 0 ||
+ standaloneServerFactoryList.size() <= 1) {
+ return;
+ }
+
+ int backupZKServerIndex = activeZKServerIndex + 1;
+ // Shutdown the current active one
+ NIOServerCnxnFactory standaloneServerFactory =
+ standaloneServerFactoryList.get(backupZKServerIndex);
+ int clientPort = clientPortList.get(backupZKServerIndex);
+
+ standaloneServerFactory.shutdown();
+ if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for shutdown of standalone server");
+ }
+
+ zooKeeperServers.get(backupZKServerIndex).getZKDatabase().close();
+
+ // remove this backup zk server
+ standaloneServerFactoryList.remove(backupZKServerIndex);
+ clientPortList.remove(backupZKServerIndex);
+ zooKeeperServers.remove(backupZKServerIndex);
+ LOG.info("Kill one backup ZK servers in the cluster " +
+ "on client port: " + clientPort);
+ }
+
+ // XXX: From o.a.zk.t.ClientBase
+ private static boolean waitForServerDown(int port, long timeout) throws
+ InterruptedException {
+ long start = System.currentTimeMillis();
+ while (true) {
+ try {
+ Socket sock = null;
+ try {
+ sock = new Socket("localhost", port);
+ OutputStream outstream = sock.getOutputStream();
+ outstream.write("stat".getBytes());
+ outstream.flush();
+ } finally {
+ IOUtils.closeSocket(sock);
+ }
+ } catch (IOException e) {
+ return true;
+ }
+
+ if (System.currentTimeMillis() > start + timeout) {
+ break;
+ }
+ Thread.sleep(250);
+ }
+ return false;
+ }
+
+ // XXX: From o.a.zk.t.ClientBase
+ private static boolean waitForServerUp(int port, long timeout) throws
+ InterruptedException {
+ long start = System.currentTimeMillis();
+ while (true) {
+ try {
+ Socket sock = null;
+ sock = new Socket("localhost", port);
+ BufferedReader reader = null;
+ try {
+ OutputStream outstream = sock.getOutputStream();
+ outstream.write("stat".getBytes());
+ outstream.flush();
+
+ Reader isr = new InputStreamReader(sock.getInputStream());
+ reader = new BufferedReader(isr);
+ String line = reader.readLine();
+ if (line != null && line.startsWith("Zookeeper version:")) {
+ return true;
+ }
+ } finally {
+ IOUtils.closeSocket(sock);
+ IOUtils.closeStream(reader);
+ }
+ } catch (IOException e) {
+ // ignore as this is expected
+ LOG.debug("server localhost:" + port + " not up " + e);
+ }
+
+ if (System.currentTimeMillis() > start + timeout) {
+ break;
+ }
+ Thread.sleep(250);
+ }
+ return false;
+ }
+
+ public int getClientPort() {
+ return clientPort;
+ }
+
+ public String getZkQuorum() {
+ return zkQuorum;
+ }
+}
diff --git a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKCallback.java b/slider-core/src/main/java/org/apache/slider/core/zk/ZKCallback.java
similarity index 95%
rename from slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKCallback.java
rename to slider-core/src/main/java/org/apache/slider/core/zk/ZKCallback.java
index 0fd137e..045b72c 100644
--- a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKCallback.java
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/ZKCallback.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.slider.core.registry.zk;
+package org.apache.slider.core.zk;
import org.apache.zookeeper.Watcher;
diff --git a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKIntegration.java b/slider-core/src/main/java/org/apache/slider/core/zk/ZKIntegration.java
similarity index 98%
rename from slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKIntegration.java
rename to slider-core/src/main/java/org/apache/slider/core/zk/ZKIntegration.java
index c3ab0a5..6270123 100644
--- a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKIntegration.java
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/ZKIntegration.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.slider.core.registry.zk;
+package org.apache.slider.core.zk;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
@@ -49,7 +49,7 @@
public static String SVC_SLIDER = "/" + ZK_SERVICES + "/" + ZK_SLIDER;
public static String SVC_SLIDER_USERS = SVC_SLIDER + "/" + ZK_USERS;
- public static List<String> ZK_USERS_PATH_LIST = new ArrayList<>();
+ public static final List<String> ZK_USERS_PATH_LIST = new ArrayList<>();
static {
ZK_USERS_PATH_LIST.add(ZK_SERVICES);
ZK_USERS_PATH_LIST.add(ZK_SLIDER);
diff --git a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKPathBuilder.java b/slider-core/src/main/java/org/apache/slider/core/zk/ZKPathBuilder.java
similarity index 97%
rename from slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKPathBuilder.java
rename to slider-core/src/main/java/org/apache/slider/core/zk/ZKPathBuilder.java
index c822749..b088568 100644
--- a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZKPathBuilder.java
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/ZKPathBuilder.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.slider.core.registry.zk;
+package org.apache.slider.core.zk;
import java.util.Locale;
diff --git a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZookeeperUtils.java b/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
similarity index 96%
rename from slider-core/src/main/java/org/apache/slider/core/registry/zk/ZookeeperUtils.java
rename to slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
index 84a9321..ea56bc2 100644
--- a/slider-core/src/main/java/org/apache/slider/core/registry/zk/ZookeeperUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/core/zk/ZookeeperUtils.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.slider.core.registry.zk;
+package org.apache.slider.core.zk;
import com.google.common.net.HostAndPort;
import org.apache.hadoop.util.StringUtils;
@@ -32,9 +32,7 @@
String zkPort = Integer.toString(port);
//parse the hosts
String[] hostlist = zkHosts.split(",", 0);
- String quorum = SliderUtils.join(hostlist, ":" + zkPort + ",");
- //this quorum has a trailing comma
- quorum = quorum.substring(0, quorum.length() - 1);
+ String quorum = SliderUtils.join(hostlist, ":" + zkPort + ",", false);
return quorum;
}
diff --git a/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java b/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
index 1ff56a9..f8008a4 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/AbstractClientProvider.java
@@ -63,7 +63,6 @@
/**
* Validate the instance definition.
- * @param clusterSpec
*/
public void validateInstanceDefinition(AggregateConf instanceDefinition) throws
SliderException {
diff --git a/slider-core/src/main/java/org/apache/slider/providers/AbstractProviderService.java b/slider-core/src/main/java/org/apache/slider/providers/AbstractProviderService.java
index 8d3146a..a06134b 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/AbstractProviderService.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/AbstractProviderService.java
@@ -87,9 +87,9 @@
@Override
public void bind(StateAccessForProviders stateAccessor,
- RegistryViewForProviders registry) {
+ RegistryViewForProviders reg) {
this.amState = stateAccessor;
- this.registry = registry;
+ this.registry = reg;
}
@Override
diff --git a/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java b/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
index 7502267..5b85f7b 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/ProviderRole.java
@@ -49,10 +49,7 @@
}
ProviderRole that = (ProviderRole) o;
- if (!name.equals(that.name)) {
- return false;
- }
- return true;
+ return name.equals(that.name);
}
@Override
diff --git a/slider-core/src/main/java/org/apache/slider/providers/ProviderService.java b/slider-core/src/main/java/org/apache/slider/providers/ProviderService.java
index 47a8cc3..8d2462e 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/ProviderService.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/ProviderService.java
@@ -33,7 +33,6 @@
import org.apache.slider.core.registry.info.ServiceInstanceData;
import org.apache.slider.server.appmaster.state.StateAccessForProviders;
import org.apache.slider.server.appmaster.web.rest.agent.AgentRestOperations;
-import org.apache.slider.server.services.curator.RegistryBinderService;
import org.apache.slider.server.services.registry.RegistryViewForProviders;
import org.apache.slider.server.services.utility.EventCallback;
diff --git a/slider-core/src/main/java/org/apache/slider/providers/ProviderUtils.java b/slider-core/src/main/java/org/apache/slider/providers/ProviderUtils.java
index 4a9cb6d..cb7d27a 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/ProviderUtils.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/ProviderUtils.java
@@ -229,7 +229,7 @@
String configName,
Map<String,String> tokenMap) {
String prefix = OptionKeys.SITE_XML_PREFIX +
- (configName.length() > 0 ? configName + "." : "");
+ (!configName.isEmpty() ? configName + "." : "");
for (Map.Entry<String, String> entry : options.entrySet()) {
String key = entry.getKey();
if (key.startsWith(prefix)) {
@@ -294,7 +294,7 @@
FileNotFoundException {
String path;
File scriptFile;
- if (imagePath!=null) {
+ if (imagePath != null) {
File tarball = new File(SliderKeys.LOCAL_TARBALL_INSTALL_SUBDIR);
scriptFile = findBinScriptInExpandedArchive(tarball, bindir, script);
// now work back from the script to build the relative path
@@ -418,7 +418,7 @@
}
log.debug("Found {} entries in {}", ls.length, base);
- List<File> directories = new LinkedList<File>();
+ List<File> directories = new LinkedList<>();
StringBuilder dirs = new StringBuilder();
for (File file : ls) {
log.debug("{}", false);
diff --git a/slider-core/src/main/java/org/apache/slider/providers/SliderProviderFactory.java b/slider-core/src/main/java/org/apache/slider/providers/SliderProviderFactory.java
index 4fc617f..f3bf0b1 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/SliderProviderFactory.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/SliderProviderFactory.java
@@ -87,10 +87,6 @@
SliderProviderFactory providerFactory = providerClass.newInstance();
providerFactory.setConf(conf);
return providerFactory;
- } catch (InstantiationException e) {
- ex = e;
- } catch (IllegalAccessException e) {
- ex = e;
} catch (Exception e) {
ex = e;
}
diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
index 9f082f8..1946ebd 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentClientProvider.java
@@ -208,7 +208,7 @@
tags = new HashSet<>();
tags.add("Name: " + service.getName());
tags.add("Version: " + service.getVersion());
- tags.add("Description: " + service.getComment());
+ tags.add("Description: " + SliderUtils.truncate(service.getComment(), 80));
} catch (IOException e) {
log.error("error retrieving metainfo from {}", appDef, e);
throw new SliderException("error retrieving metainfo", e);
diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
index f7f2b21..6d3d0e1 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/AgentProviderService.java
@@ -78,8 +78,10 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -102,13 +104,22 @@
private static final String LABEL_MAKER = "___";
private static final String CONTAINER_ID = "container_id";
private static final String GLOBAL_CONFIG_TAG = "global";
+ private static final String LOG_FOLDERS_TAG = "LogFolders";
+ private static final int MAX_LOG_ENTRIES = 20;
private final Object syncLock = new Object();
private final Map<String, String> allocatedPorts = new ConcurrentHashMap<>();
private AgentClientProvider clientProvider;
private Map<String, ComponentInstanceState> componentStatuses = new HashMap<>();
private AtomicInteger taskId = new AtomicInteger(0);
- private Metainfo metainfo = null;
+ private volatile Metainfo metainfo = null;
private ComponentCommandOrder commandOrder = null;
+ private Map<String, String> workFolders =
+ Collections.synchronizedMap(new LinkedHashMap<String, String>(MAX_LOG_ENTRIES, 0.75f, false) {
+ protected boolean removeEldestEntry(Map.Entry eldest) {
+ return size() > MAX_LOG_ENTRIES;
+ }
+ });
+ private Boolean canAnyMasterPublish = null;
public AgentProviderService() {
super("AgentProviderService");
@@ -364,6 +375,7 @@
String label = heartBeat.getHostname();
String roleName = getRoleName(label);
+
String containerId = getContainerId(label);
StateAccessForProviders accessor = getAmState();
String scriptPath = getScriptPathFromMetainfo(roleName);
@@ -379,8 +391,10 @@
Boolean isMaster = isMaster(roleName);
ComponentInstanceState componentStatus = componentStatuses.get(label);
- // TODO: Currently only process configurations from Master
- if (isMaster) {
+ // If no Master can explicitly publish then publish if its a master
+ // Otherwise, wait till the master that can publish is ready
+ if (isMaster &&
+ (canAnyMasterPublishConfig() == false || canPublishConfig(roleName))) {
processReturnedStatus(heartBeat, componentStatus);
}
@@ -398,6 +412,10 @@
Command command = getCommand(report.getRoleCommand());
componentStatus.applyCommandResult(result, command);
log.info("Component operation. Status: {}", result);
+
+ if (command == Command.INSTALL && report.getFolders() != null && report.getFolders().size() > 0) {
+ processFolderPaths(report.getFolders(), containerId, heartBeat.getFqdn());
+ }
}
int waitForCount = accessor.getInstanceDefinitionSnapshot().
@@ -443,6 +461,14 @@
return response;
}
+ private void processFolderPaths(Map<String, String> folders, String containerId, String hostFqdn) {
+ for(String key : folders.keySet()) {
+ workFolders.put(String.format("%s-%s-%s", hostFqdn, containerId, key), folders.get(key));
+ }
+
+ publishComponentConfiguration(LOG_FOLDERS_TAG, LOG_FOLDERS_TAG, (new HashMap<>(this.workFolders)).entrySet());
+ }
+
protected void processReturnedStatus(HeartBeat heartBeat, ComponentInstanceState componentStatus) {
List<ComponentStatus> statuses = heartBeat.getComponentStatus();
if (statuses != null && !statuses.isEmpty()) {
@@ -519,7 +545,7 @@
return scriptPath;
}
- protected Boolean isMaster(String roleName) {
+ protected boolean isMaster(String roleName) {
List<Service> services = getMetainfo().getServices();
if (services.size() != 1) {
log.error("Malformed app definition: Expect only one service in the metainfo.xml");
@@ -538,6 +564,43 @@
return false;
}
+ protected boolean canPublishConfig(String roleName) {
+ List<Service> services = getMetainfo().getServices();
+ if (services.size() != 1) {
+ log.error("Malformed app definition: Expect only one service in the metainfo.xml");
+ } else {
+ Service service = services.get(0);
+ for (Component component : service.getComponents()) {
+ if (component.getName().equals(roleName)) {
+ return Boolean.TRUE.toString().equals(component.getPublishConfig());
+ }
+ }
+ }
+ return false;
+ }
+
+ protected boolean canAnyMasterPublishConfig() {
+ if (canAnyMasterPublish == null) {
+ List<Service> services = getMetainfo().getServices();
+ if (services.size() != 1) {
+ log.error("Malformed app definition: Expect only one service in the metainfo.xml");
+ } else {
+ Service service = services.get(0);
+ for (Component component : service.getComponents()) {
+ if (Boolean.TRUE.toString().equals(component.getPublishConfig()) &&
+ component.getCategory().equals("MASTER")) {
+ canAnyMasterPublish = true;
+ }
+ }
+ }
+ }
+
+ if (canAnyMasterPublish == null) {
+ canAnyMasterPublish = false;
+ }
+ return canAnyMasterPublish;
+ }
+
private String getRoleName(String label) {
return label.substring(label.indexOf(LABEL_MAKER) + LABEL_MAKER.length());
}
@@ -742,7 +805,7 @@
//apply any port updates
if (!this.getAllocatedPorts().isEmpty()) {
for (String key : config.keySet()) {
- if (this.getAllocatedPorts().keySet().contains(key)) {
+ if (this.getAllocatedPorts().containsKey(key)) {
config.put(key, getAllocatedPorts().get(key));
}
}
diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/ComponentCommandOrder.java b/slider-core/src/main/java/org/apache/slider/providers/agent/ComponentCommandOrder.java
index 6808d63..0dce4bb 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/ComponentCommandOrder.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/ComponentCommandOrder.java
@@ -93,7 +93,7 @@
Command cmd = Command.valueOf(cmdStr);
- if(cmd != Command.START) {
+ if (cmd != Command.START) {
throw new IllegalArgumentException("Dependency order can only be specified for START.");
}
return new ComponentCommand(compStr, cmd);
@@ -113,8 +113,8 @@
String stateStr = compStStr.substring(splitIndex + 1);
State state = State.valueOf(stateStr);
- if(state != State.STARTED) {
- throw new IllegalArgumentException("Dependency order can only be specified against STARTED.");
+ if (state != State.STARTED && state != State.INSTALLED) {
+ throw new IllegalArgumentException("Dependency order can only be specified against STARTED/INSTALLED.");
}
return new ComponentState(compStr, state);
}
@@ -129,9 +129,18 @@
component, command, currState.getCompName(), currState.getState());
if (currState.getCompName().equals(stateToMatch.componentName)) {
if (currState.getState() != stateToMatch.state) {
- log.info("Cannot schedule {} {} as dependency {} is {}",
- component, command, currState.getCompName(), currState.getState());
- canExecute = false;
+ if (stateToMatch.state == State.STARTED) {
+ log.info("Cannot schedule {} {} as dependency {} is {}",
+ component, command, currState.getCompName(), currState.getState());
+ canExecute = false;
+ } else {
+ //state is INSTALLED
+ if (currState.getState() != State.STARTING && currState.getState() != State.STARTED) {
+ log.info("Cannot schedule {} {} as dependency {} is {}",
+ component, command, currState.getCompName(), currState.getState());
+ canExecute = false;
+ }
+ }
}
}
if (!canExecute) {
diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
index d34a223..6cd08e0 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/Component.java
@@ -22,9 +22,13 @@
public class Component {
String name;
String category;
+ String publishConfig;
+ String minInstanceCount;
+ String maxInstanceCount;
CommandScript commandScript;
public Component() {
+ publishConfig = Boolean.FALSE.toString();
}
public String getName() {
@@ -43,6 +47,30 @@
this.category = category;
}
+ public String getPublishConfig() {
+ return publishConfig;
+ }
+
+ public void setPublishConfig(String publishConfig) {
+ this.publishConfig = publishConfig;
+ }
+
+ public String getMinInstanceCount() {
+ return minInstanceCount;
+ }
+
+ public void setMinInstanceCount(String minInstanceCount) {
+ this.minInstanceCount = minInstanceCount;
+ }
+
+ public String getMaxInstanceCount() {
+ return maxInstanceCount;
+ }
+
+ public void setMaxInstanceCount(String maxInstanceCount) {
+ this.maxInstanceCount = maxInstanceCount;
+ }
+
public CommandScript getCommandScript() {
return commandScript;
}
diff --git a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/MetainfoParser.java b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/MetainfoParser.java
index 554540c..a97c879 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/MetainfoParser.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/agent/application/metadata/MetainfoParser.java
@@ -55,6 +55,9 @@
digester.addObjectCreate("*/component", Component.class);
digester.addBeanPropertySetter("*/component/name");
digester.addBeanPropertySetter("*/component/category");
+ digester.addBeanPropertySetter("*/component/publishConfig");
+ digester.addBeanPropertySetter("*/component/minInstanceCount");
+ digester.addBeanPropertySetter("*/component/maxInstanceCount");
digester.addSetNext("*/component", "addComponent");
digester.addObjectCreate("*/commandScript", CommandScript.class);
diff --git a/slider-core/src/main/java/org/apache/slider/providers/slideram/SliderAMClientProvider.java b/slider-core/src/main/java/org/apache/slider/providers/slideram/SliderAMClientProvider.java
index 66cfeba..6b40856 100644
--- a/slider-core/src/main/java/org/apache/slider/providers/slideram/SliderAMClientProvider.java
+++ b/slider-core/src/main/java/org/apache/slider/providers/slideram/SliderAMClientProvider.java
@@ -39,16 +39,11 @@
import org.apache.slider.core.exceptions.BadConfigException;
import org.apache.slider.core.exceptions.SliderException;
import org.apache.slider.core.launch.AbstractLauncher;
-import org.apache.slider.core.launch.CommandLineBuilder;
import org.apache.slider.core.launch.JavaCommandLineBuilder;
import org.apache.slider.providers.AbstractClientProvider;
import org.apache.slider.providers.PlacementPolicy;
import org.apache.slider.providers.ProviderRole;
import org.apache.slider.providers.ProviderUtils;
-import org.codehaus.jackson.JsonFactory;
-import org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider;
-import org.codehaus.jackson.node.JsonNodeFactory;
-import org.codehaus.jackson.xc.JaxbAnnotationIntrospector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -87,7 +82,7 @@
* List of roles
*/
public static final List<ProviderRole> ROLES =
- new ArrayList<ProviderRole>();
+ new ArrayList<>();
public static final int KEY_AM = ROLE_AM_PRIORITY_INDEX;
@@ -151,7 +146,7 @@
throws IOException, SliderException {
Map<String, LocalResource> providerResources =
- new HashMap<String, LocalResource>();
+ new HashMap<>();
ProviderUtils.addProviderJar(providerResources,
@@ -171,11 +166,6 @@
ServiceInstance.class,
ServiceNames.class,
- JacksonJaxbJsonProvider.class,
- JsonFactory.class,
- JsonNodeFactory.class,
- JaxbAnnotationIntrospector.class,
-
};
String[] jars =
{
@@ -186,12 +176,6 @@
"curator-client.jar",
"curator-x-discovery.jar",
"curator-x-discovery-service.jar",
-
- "jackson-jaxrs",
- "jackson-core-asl",
- "jackson-mapper-asl",
- "jackson-xc",
-
};
ProviderUtils.addDependencyJars(providerResources, fileSystem, tempPath,
libdir, jars,
@@ -228,8 +212,7 @@
MapOperations sliderAM =
aggregateConf.getAppConfOperations().getMandatoryComponent(
SliderKeys.COMPONENT_AM);
- cmdLine.sysprop("java.net.preferIPv4Stack", "true");
- cmdLine.sysprop("java.awt.headless", "true");
+ cmdLine.forceIPv4().headless();
String heap = sliderAM.getOption(RoleKeys.JVM_HEAP,
DEFAULT_JVM_HEAP);
cmdLine.setJVMHeap(heap);
diff --git a/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java b/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
index 25037ea..3f54e27 100644
--- a/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
+++ b/slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
@@ -725,10 +725,9 @@
List<String> serviceInstancesRunning = registry.instanceIDs(serviceName);
log.info("service instances already running: {}", serviceInstancesRunning);
-
- ServiceInstanceData instanceData = new ServiceInstanceData();
- instanceData.id = registryId;
- instanceData.serviceType = appServiceType;
+
+ ServiceInstanceData instanceData = new ServiceInstanceData(registryId,
+ appServiceType);
// IPC services
@@ -750,10 +749,8 @@
// push the registration info to ZK
registry.registerSelf(
- appServiceType,
- registryId,
- amWebAPI,
- instanceData);
+ instanceData, amWebAPI
+ );
}
/**
diff --git a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/CommandReport.java b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/CommandReport.java
index ff5e19a..a37e490 100644
--- a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/CommandReport.java
+++ b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/CommandReport.java
@@ -40,6 +40,7 @@
private String serviceName;
private long taskId;
private String roleCommand;
+ private Map<String, String> folders;
private Map<String, String> allocatedPorts;
private Map<String, Map<String, String>> configurationTags;
@@ -165,7 +166,7 @@
configurationTags = tags;
}
- /** @return the config tags that match this command, or <code>null</code> if none are present */
+ /** @return the allocated ports, or <code>null</code> if none are present */
@JsonProperty("allocatedPorts")
public Map<String, String> getAllocatedPorts() {
return allocatedPorts;
@@ -174,7 +175,19 @@
/** @param ports allocated ports */
@JsonProperty("allocatedPorts")
public void setAllocatedPorts(Map<String, String> ports) {
- allocatedPorts = ports;
+ this.allocatedPorts = ports;
+ }
+
+ /** @return the folders, or <code>null</code> if none are present */
+ @JsonProperty("folders")
+ public Map<String, String> getFolders() {
+ return folders;
+ }
+
+ /** @param folders allocated ports */
+ @JsonProperty("folders")
+ public void setFolders(Map<String, String> folders) {
+ this.folders = folders;
}
@Override
diff --git a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/HeartBeat.java b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/HeartBeat.java
index fa149b8..d3388f5 100644
--- a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/HeartBeat.java
+++ b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/agent/HeartBeat.java
@@ -43,6 +43,7 @@
private List<DiskInfo> mounts = new ArrayList<DiskInfo>();
HostStatus nodeStatus;
private AgentEnv agentEnv = null;
+ private String fqdn;
public long getResponseId() {
return responseId;
@@ -56,18 +57,26 @@
return timestamp;
}
- public String getHostname() {
- return hostname;
- }
-
public void setTimestamp(long timestamp) {
this.timestamp = timestamp;
}
+ public String getHostname() {
+ return hostname;
+ }
+
public void setHostname(String hostname) {
this.hostname = hostname;
}
+ public String getFqdn() {
+ return fqdn;
+ }
+
+ public void setFqdn(String fqdn) {
+ this.fqdn = fqdn;
+ }
+
@JsonProperty("reports")
public List<CommandReport> getReports() {
return this.reports;
diff --git a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/publisher/PublisherResource.java b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/publisher/PublisherResource.java
index b139f74..e6b2664 100644
--- a/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/publisher/PublisherResource.java
+++ b/slider-core/src/main/java/org/apache/slider/server/appmaster/web/rest/publisher/PublisherResource.java
@@ -150,7 +150,7 @@
}
@GET
- @Path("/" + CONFIG+ ".json")
+ @Path("/" + CONFIG + ".json")
@Produces({MediaType.APPLICATION_JSON})
public String getConfigurationContentJson(
@PathParam("setname") String setname,
diff --git a/slider-core/src/main/java/org/apache/slider/server/services/curator/RegistryBinderService.java b/slider-core/src/main/java/org/apache/slider/server/services/curator/RegistryBinderService.java
index f4bd9bd..14b78a7 100644
--- a/slider-core/src/main/java/org/apache/slider/server/services/curator/RegistryBinderService.java
+++ b/slider-core/src/main/java/org/apache/slider/server/services/curator/RegistryBinderService.java
@@ -108,12 +108,6 @@
Preconditions.checkNotNull(name, "null `name` arg");
Preconditions.checkState(isInState(STATE.STARTED), "Not started: " + this);
- if (lookup(id) != null) {
- throw new BadClusterStateException(
- "existing entry for service id %s name %s %s",
- id, name, url);
- }
-
ServiceInstanceBuilder<Payload> instanceBuilder = builder()
.name(name)
.id(id)
@@ -141,16 +135,6 @@
}
/**
- * Get the registered instance by its ID
- * @param id ID
- * @return instance or null
- */
- public synchronized ServiceInstance<Payload> lookup(String id) {
- Preconditions.checkNotNull(id, "null `id` arg");
- return entries.get(id);
- }
-
- /**
* Create a builder. This is already pre-prepared with address, registration
* time and a (random) UUID
* @return a builder
@@ -270,6 +254,19 @@
}
/**
+ * Find a single instance -return that value or raise an exception
+ * @param serviceType service type
+ * @param name the name (required(
+ * @return the instance that matches the criteria
+ * @throws FileNotFoundException if there were no matches
+ * @throws IOException any network problem
+ */
+ public CuratorServiceInstance<Payload> findInstance(String serviceType,
+ String name) throws IOException {
+ Preconditions.checkArgument(StringUtils.isNotEmpty(name), "name");
+ return findInstances(serviceType, name).get(0);
+ }
+ /**
* List registry entries. If a name was given, then the single match is returned
* -otherwise all entries matching the service type
* @param serviceType service type
diff --git a/slider-core/src/main/java/org/apache/slider/server/services/registry/RegistryViewForProviders.java b/slider-core/src/main/java/org/apache/slider/server/services/registry/RegistryViewForProviders.java
index e3d642b..22ba066 100644
--- a/slider-core/src/main/java/org/apache/slider/server/services/registry/RegistryViewForProviders.java
+++ b/slider-core/src/main/java/org/apache/slider/server/services/registry/RegistryViewForProviders.java
@@ -39,15 +39,10 @@
/**
* Register the service, raising IOExceptions when anything fails
- * @param serviceType service type
- * @param instanceName ID -must be unique
- * @param url URL to register
* @param instanceData instance data
+ * @param url URL to register
* @throws IOException on registration problems
*/
void registerServiceInstance(
- String serviceType,
- String instanceName,
- URL url,
- ServiceInstanceData instanceData) throws IOException;
+ ServiceInstanceData instanceData, URL url) throws IOException;
}
diff --git a/slider-core/src/main/java/org/apache/slider/server/services/registry/SliderRegistryService.java b/slider-core/src/main/java/org/apache/slider/server/services/registry/SliderRegistryService.java
index 0d4a357..1904df5 100644
--- a/slider-core/src/main/java/org/apache/slider/server/services/registry/SliderRegistryService.java
+++ b/slider-core/src/main/java/org/apache/slider/server/services/registry/SliderRegistryService.java
@@ -72,29 +72,24 @@
/**
* register an instance -only valid once the service is started.
* This sets the selfRegistration field
- * @param serviceType service type
- * @param instanceName ID -must be unique
- * @param url URL to register
* @param instanceData instance data
+ * @param url URL to register
* @throws IOException on registration problems
*/
- public void registerSelf(String serviceType,
- String instanceName,
- URL url,
- ServiceInstanceData instanceData) throws IOException {
- registerServiceInstance(serviceType, instanceName, url, instanceData);
+ public void registerSelf(ServiceInstanceData instanceData, URL url) throws IOException {
+ registerServiceInstance(instanceData, url);
setSelfRegistration(instanceData);
}
@Override
public void registerServiceInstance(
- String serviceType,
- String instanceName,
- URL url,
- ServiceInstanceData instanceData) throws IOException {
+ ServiceInstanceData instanceData, URL url) throws IOException {
Preconditions.checkNotNull(instanceData);
+ Preconditions.checkNotNull(instanceData.id);
+ Preconditions.checkNotNull(instanceData.serviceType);
+
try {
- register(serviceType, instanceName, url, instanceData);
+ register(instanceData.serviceType, instanceData.id, url, instanceData);
} catch (IOException e) {
throw e;
} catch (Exception e) {
diff --git a/slider-core/src/main/java/org/apache/slider/server/services/utility/AbstractSliderLaunchedService.java b/slider-core/src/main/java/org/apache/slider/server/services/utility/AbstractSliderLaunchedService.java
index aa594b3..5d37c32 100644
--- a/slider-core/src/main/java/org/apache/slider/server/services/utility/AbstractSliderLaunchedService.java
+++ b/slider-core/src/main/java/org/apache/slider/server/services/utility/AbstractSliderLaunchedService.java
@@ -23,7 +23,7 @@
import org.apache.slider.common.SliderXmlConfKeys;
import org.apache.slider.common.tools.SliderUtils;
import org.apache.slider.core.exceptions.BadConfigException;
-import org.apache.slider.core.registry.zk.ZookeeperUtils;
+import org.apache.slider.core.zk.ZookeeperUtils;
import org.apache.slider.server.services.curator.CuratorHelper;
import org.apache.slider.server.services.registry.SliderRegistryService;
import org.slf4j.Logger;
diff --git a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAgentAM.groovy b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAgentAM.groovy
index 6794327..d700fea 100644
--- a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAgentAM.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneAgentAM.groovy
@@ -98,10 +98,10 @@
describe "service registry names"
SliderRegistryService registry = client.registry
def names = registry.getServiceTypes();
- dumpRegistryNames(names)
+ dumpRegistryServiceTypes(names)
describe "service registry instance IDs"
- def instanceIds = client.listRegistryInstanceIDs()
+ def instanceIds = client.listRegistedSliderInstances()
log.info("number of instanceIds: ${instanceIds.size()}")
instanceIds.each { String it -> log.info(it) }
diff --git a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneRegistryAM.groovy b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneRegistryAM.groovy
index 7582b72..7639375 100644
--- a/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneRegistryAM.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/agent/standalone/TestStandaloneRegistryAM.groovy
@@ -115,9 +115,9 @@
describe "service registry names"
SliderRegistryService registryService = client.registry
def serviceTypes = registryService.serviceTypes;
- dumpRegistryNames(serviceTypes)
+ dumpRegistryServiceTypes(serviceTypes)
- List<String> instanceIds = client.listRegistryInstanceIDs()
+ List<String> instanceIds = client.listRegistedSliderInstances()
dumpRegistryInstanceIDs(instanceIds)
diff --git a/slider-core/src/test/groovy/org/apache/slider/common/tools/GroovyZKIntegration.groovy b/slider-core/src/test/groovy/org/apache/slider/common/tools/GroovyZKIntegration.groovy
index 49701e3..541e426 100644
--- a/slider-core/src/test/groovy/org/apache/slider/common/tools/GroovyZKIntegration.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/common/tools/GroovyZKIntegration.groovy
@@ -18,7 +18,7 @@
package org.apache.slider.common.tools
-import org.apache.slider.core.registry.zk.ZKCallback
+import org.apache.slider.core.zk.ZKCallback
import org.apache.zookeeper.WatchedEvent
class GroovyZKIntegration {
diff --git a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
index 0b4c66f..3930864 100644
--- a/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/common/tools/TestZKIntegration.groovy
@@ -18,10 +18,9 @@
package org.apache.slider.common.tools
-import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.apache.slider.test.KeysForTests
import org.apache.slider.test.YarnZKMiniClusterTestBase
import org.apache.zookeeper.CreateMode
diff --git a/slider-core/src/test/groovy/org/apache/slider/server/appmaster/web/rest/publisher/TestPublisherRestResources.groovy b/slider-core/src/test/groovy/org/apache/slider/server/appmaster/web/rest/publisher/TestPublisherRestResources.groovy
index 0100d7a..bc7c79a 100644
--- a/slider-core/src/test/groovy/org/apache/slider/server/appmaster/web/rest/publisher/TestPublisherRestResources.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/server/appmaster/web/rest/publisher/TestPublisherRestResources.groovy
@@ -101,32 +101,35 @@
Client client = createTestClient();
// test the available GET URIs
- WebResource webResource = client.resource(publisher_url + "/dummy-site");
+ String sliderConfigset = publisher_url +"/"+ RestPaths.SLIDER_CONFIGSET + "/"
+ WebResource webResource
+ webResource = client.resource(sliderConfigset);
+ webResource = client.resource(sliderConfigset + "dummy-site");
PublishedConfiguration config = webResource.type(MediaType.APPLICATION_JSON)
.get(PublishedConfiguration.class);
assert config != null
Map<String,String> entries = config.entries
log.info("entries are {}", entries)
- assert entries.get("prop1").equals("val1")
- assert entries.get("prop2").equals("val2")
+ assert entries.get("prop1") =="val1"
+ assert entries.get("prop2")== "val2"
- webResource = client.resource(publisher_url + "/dummy-site/prop1");
+ webResource = client.resource(sliderConfigset + "dummy-site/prop1");
Map<String,String> val = webResource.type(MediaType.APPLICATION_JSON).get(Map.class);
assert "val1".equals(val.get("prop1"))
// some negative tests...
- webResource = client.resource(appendToURL(publisher_url,
- "/foobar-site"));
+ webResource = client.resource(appendToURL(sliderConfigset,
+ "foobar-site"));
ClientResponse response = webResource.type(MediaType.APPLICATION_JSON)
.get(ClientResponse.class);
- assert response.getStatus() == 404
+ assert 404 == response.status
- webResource = client.resource(publisher_url + "/dummy-site/missing.prop");
+ webResource = client.resource(sliderConfigset + "dummy-site/missing.prop");
response = webResource.type(MediaType.TEXT_PLAIN).get(ClientResponse.class);
- assert response.getStatus() == 404
+ assert 404 == response.status
- }
+ }
}
diff --git a/slider-core/src/test/groovy/org/apache/slider/test/MicroZKCluster.groovy b/slider-core/src/test/groovy/org/apache/slider/test/MicroZKCluster.groovy
index d0da914..cb1d9b5 100644
--- a/slider-core/src/test/groovy/org/apache/slider/test/MicroZKCluster.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/test/MicroZKCluster.groovy
@@ -21,8 +21,8 @@
import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
-import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster
import org.apache.slider.common.tools.SliderUtils
+import org.apache.slider.core.zk.MiniZooKeeperCluster
@Slf4j
@CompileStatic
@@ -30,7 +30,6 @@
public static final String HOSTS = "127.0.0.1"
MiniZooKeeperCluster zkCluster
- File baseDir
String zkBindingString
Configuration conf
int port
@@ -44,23 +43,20 @@
}
void createCluster() {
- zkCluster = new MiniZooKeeperCluster(conf)
- baseDir = File.createTempFile("zookeeper", ".dir")
- baseDir.delete()
- baseDir.mkdirs()
- port = zkCluster.startup(baseDir)
- zkBindingString = HOSTS + ":" + port
+ zkCluster = new MiniZooKeeperCluster(1)
+ zkCluster.init(conf)
+ zkCluster.start()
+ zkBindingString = zkCluster.zkQuorum
log.info("Created $this")
}
@Override
void close() throws IOException {
- zkCluster?.shutdown();
- baseDir?.deleteDir()
+ zkCluster?.stop()
}
@Override
String toString() {
- return "Micro ZK cluster as $zkBindingString data=$baseDir"
+ return "Micro ZK cluster as $zkBindingString"
}
}
diff --git a/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy b/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
index 8588a53..2045f11 100644
--- a/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/test/SliderTestUtils.groovy
@@ -695,14 +695,17 @@
public static void dumpRegistryInstanceIDs(List<String> instanceIds) {
describe "service registry instance IDs"
- log.info("number of instanceIds: ${instanceIds.size()}")
- instanceIds.each { String it -> log.info(it) }
+ dumpCollection(instanceIds)
}
- public static void dumpRegistryNames(Collection<String> names) {
- describe "service registry names"
- log.info("number of names: ${names.size()}")
- names.each { String it -> log.info(it) }
+ public static void dumpRegistryServiceTypes(Collection<String> entries) {
+ describe "service registry types"
+ dumpCollection(entries)
+ }
+
+ def static void dumpCollection(Collection<String> entries) {
+ log.info("number of entries: ${entries.size()}")
+ entries.each { String it -> log.info(it) }
}
/**
diff --git a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
index fe1ffc1..d1cc5ca 100644
--- a/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/test/YarnMiniClusterTestBase.groovy
@@ -55,7 +55,6 @@
import static org.apache.slider.common.SliderXMLConfKeysForTesting.*
import static org.apache.slider.test.KeysForTests.*
-import static org.apache.slider.test.SliderTestUtils.log
/**
* Base class for mini cluster tests -creates a field for the
@@ -88,7 +87,6 @@
SLIDER_CONFIG.setBoolean(YarnConfiguration.NM_PMEM_CHECK_ENABLED, false)
SLIDER_CONFIG.setBoolean(YarnConfiguration.NM_VMEM_CHECK_ENABLED, false)
SLIDER_CONFIG.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 1)
-
}
@@ -194,8 +192,8 @@
}
public void stopMiniCluster() {
- Log l = LogFactory.getLog(this.getClass())
- ServiceOperations.stopQuietly(l, miniCluster)
+ Log commonslog = LogFactory.getLog(this.class)
+ ServiceOperations.stopQuietly(commonslog, miniCluster)
hdfsCluster?.shutdown();
}
diff --git a/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy b/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
index dcc07c6..691cd2f 100644
--- a/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
+++ b/slider-core/src/test/groovy/org/apache/slider/test/YarnZKMiniClusterTestBase.groovy
@@ -21,10 +21,11 @@
import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.io.IOUtils
import org.apache.hadoop.yarn.conf.YarnConfiguration
import org.apache.slider.common.SliderXmlConfKeys
-import org.apache.slider.core.registry.zk.BlockingZKWatcher
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.BlockingZKWatcher
+import org.apache.slider.core.zk.ZKIntegration
import java.util.concurrent.atomic.AtomicBoolean
@@ -41,7 +42,7 @@
public void stopMiniCluster() {
super.stopMiniCluster()
- microZKCluster?.close()
+ IOUtils.closeStream(microZKCluster);
}
public ZKIntegration createZKIntegrationInstance(String zkQuorum,
diff --git a/slider-core/src/test/java/org/apache/slider/common/tools/TestSliderUtils.java b/slider-core/src/test/java/org/apache/slider/common/tools/TestSliderUtils.java
index 863a1e6..7cac439 100644
--- a/slider-core/src/test/java/org/apache/slider/common/tools/TestSliderUtils.java
+++ b/slider-core/src/test/java/org/apache/slider/common/tools/TestSliderUtils.java
@@ -20,6 +20,7 @@
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.junit.Test;
+import org.junit.Assert;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,7 +45,20 @@
sliderFileSystem.getFileSystem(),
new Path("target/test-classes/org/apache/slider/common/tools/test.zip"),
"metainfo.xml");
- assert stream != null;
- assert stream.available() > 0;
+ Assert.assertTrue(stream != null);
+ Assert.assertTrue(stream.available() > 0);
+ }
+
+ @Test
+ public void testTruncate () {
+ Assert.assertEquals(SliderUtils.truncate(null, 5), null);
+ Assert.assertEquals(SliderUtils.truncate("323", -1), "323");
+ Assert.assertEquals(SliderUtils.truncate("3232", 5), "3232");
+ Assert.assertEquals(SliderUtils.truncate("1234567890", 0), "1234567890");
+ Assert.assertEquals(SliderUtils.truncate("123456789012345", 15), "123456789012345");
+ Assert.assertEquals(SliderUtils.truncate("123456789012345", 14), "12345678901...");
+ Assert.assertEquals(SliderUtils.truncate("1234567890", 1), "1");
+ Assert.assertEquals(SliderUtils.truncate("1234567890", 10), "1234567890");
+ Assert.assertEquals(SliderUtils.truncate("", 10), "");
}
}
diff --git a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
index 11339c1..b0ac967 100644
--- a/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
+++ b/slider-core/src/test/java/org/apache/slider/providers/agent/TestAgentProviderService.java
@@ -168,6 +168,44 @@
+ " </services>\n"
+ "</metainfo>";
+
+ private static final String metainfo_2_str = "<metainfo>\n"
+ + " <schemaVersion>2.0</schemaVersion>\n"
+ + " <services>\n"
+ + " <service>\n"
+ + " <name>HBASE</name>\n"
+ + " <comment>\n"
+ + " Apache HBase\n"
+ + " </comment>\n"
+ + " <version>0.96.0.2.1.1</version>\n"
+ + " <type>YARN-APP</type>\n"
+ + " <minHadoopVersion>2.1.0</minHadoopVersion>\n"
+ + " <components>\n"
+ + " <component>\n"
+ + " <name>HBASE_MASTER</name>\n"
+ + " <category>MASTER</category>\n"
+ + " <publishConfig>true</publishConfig>\n"
+ + " <minInstanceCount>1</minInstanceCount>\n"
+ + " <maxInstanceCount>2</maxInstanceCount>\n"
+ + " <commandScript>\n"
+ + " <script>scripts/hbase_master.py</script>\n"
+ + " <scriptType>PYTHON</scriptType>\n"
+ + " <timeout>600</timeout>\n"
+ + " </commandScript>\n"
+ + " </component>\n"
+ + " <component>\n"
+ + " <name>HBASE_REGIONSERVER</name>\n"
+ + " <category>SLAVE</category>\n"
+ + " <minInstanceCount>1</minInstanceCount>\n"
+ + " <commandScript>\n"
+ + " <script>scripts/hbase_regionserver.py</script>\n"
+ + " <scriptType>PYTHON</scriptType>\n"
+ + " </commandScript>\n"
+ + " </component>\n"
+ + " </components>\n"
+ + " </service>\n"
+ + " </services>\n"
+ + "</metainfo>";
@Test
public void testRegistration() throws IOException {
@@ -367,26 +405,30 @@
public void testMetainfoParsing() throws Exception {
InputStream metainfo_1 = new ByteArrayInputStream(metainfo_1_str.getBytes());
Metainfo metainfo = new MetainfoParser().parse(metainfo_1);
- assert metainfo.getServices().size() == 1;
+ Assert.assertEquals(metainfo.getServices().size(), 1);
Service service = metainfo.getServices().get(0);
log.info("Service: " + service.toString());
- assert service.getName().equals("HBASE");
- assert service.getComponents().size() == 2;
+ Assert.assertEquals(service.getName(), "HBASE");
+ Assert.assertEquals(service.getComponents().size(), 2);
List<Component> components = service.getComponents();
int found = 0;
for (Component component : components) {
if (component.getName().equals("HBASE_MASTER")) {
- assert component.getCommandScript().getScript().equals("scripts/hbase_master.py");
- assert component.getCategory().equals("MASTER");
+ Assert.assertEquals(component.getMinInstanceCount(), "1");
+ Assert.assertEquals(component.getMaxInstanceCount(), "2");
+ Assert.assertEquals(component.getCommandScript().getScript(), "scripts/hbase_master.py");
+ Assert.assertEquals(component.getCategory(), "MASTER");
found++;
}
if (component.getName().equals("HBASE_REGIONSERVER")) {
- assert component.getCommandScript().getScript().equals("scripts/hbase_regionserver.py");
- assert component.getCategory().equals("SLAVE");
+ Assert.assertEquals(component.getMinInstanceCount(), "1");
+ Assert.assertNull(component.getMaxInstanceCount());
+ Assert.assertEquals(component.getCommandScript().getScript(), "scripts/hbase_regionserver.py");
+ Assert.assertEquals(component.getCategory(), "SLAVE");
found++;
}
}
- assert found == 2;
+ Assert.assertEquals(found, 2);
assert service.getExportGroups().size() == 1;
List<ExportGroup> egs = service.getExportGroups();
@@ -398,19 +440,19 @@
for (Export export : eg.getExports()) {
if (export.getName().equals("JMX_Endpoint")) {
found++;
- assert export.getValue().equals(
+ Assert.assertEquals(export.getValue(),
"http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx");
}
if (export.getName().equals("Master_Status")) {
found++;
- assert export.getValue().equals(
+ Assert.assertEquals(export.getValue(),
"http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/master-status");
}
}
- assert found == 2;
+ Assert.assertEquals(found, 2);
List<CommandOrder> cmdOrders = service.getCommandOrder();
- assert cmdOrders.size() == 2;
+ Assert.assertEquals(cmdOrders.size(), 2);
found = 0;
for (CommandOrder co : service.getCommandOrder()) {
if (co.getCommand().equals("HBASE_REGIONSERVER-START")) {
@@ -418,17 +460,17 @@
found++;
}
if (co.getCommand().equals("A-START")) {
- assert co.getRequires().equals("B-STARTED");
+ Assert.assertEquals(co.getRequires(), "B-STARTED");
found++;
}
}
- assert found == 2;
+ Assert.assertEquals(found, 2);
AgentProviderService aps = new AgentProviderService();
AgentProviderService mockAps = Mockito.spy(aps);
doReturn(metainfo).when(mockAps).getMetainfo();
String scriptPath = mockAps.getScriptPathFromMetainfo("HBASE_MASTER");
- assert scriptPath.equals("scripts/hbase_master.py");
+ Assert.assertEquals(scriptPath, "scripts/hbase_master.py");
String metainfo_1_str_bad = "<metainfo>\n"
+ " <schemaVersion>2.0</schemaVersion>\n"
@@ -441,7 +483,36 @@
metainfo_1 = new ByteArrayInputStream(metainfo_1_str_bad.getBytes());
metainfo = new MetainfoParser().parse(metainfo_1);
- assert metainfo == null;
+ Assert.assertNull(metainfo);
+ }
+
+ @Test
+ public void testMetaInfoRelatedOperations() throws Exception {
+ InputStream metainfo_1 = new ByteArrayInputStream(metainfo_1_str.getBytes());
+ Metainfo metainfo = new MetainfoParser().parse(metainfo_1);
+ InputStream metainfo_2 = new ByteArrayInputStream(metainfo_2_str.getBytes());
+ Metainfo metainfo2 = new MetainfoParser().parse(metainfo_2);
+ String role_hm = "HBASE_MASTER";
+ String role_hrs = "HBASE_REGIONSERVER";
+
+ AgentProviderService aps = new AgentProviderService();
+ AgentProviderService mockAps = Mockito.spy(aps);
+ doReturn(metainfo).when(mockAps).getMetainfo();
+
+ AgentProviderService mockAps2 = Mockito.spy(aps);
+ doReturn(metainfo2).when(mockAps2).getMetainfo();
+
+ Assert.assertTrue(mockAps.isMaster(role_hm));
+ Assert.assertFalse(mockAps.isMaster(role_hrs));
+ Assert.assertFalse(mockAps.canPublishConfig(role_hm));
+ Assert.assertFalse(mockAps.canPublishConfig(role_hrs));
+ Assert.assertFalse(mockAps.canAnyMasterPublishConfig());
+
+ Assert.assertTrue(mockAps2.isMaster(role_hm));
+ Assert.assertFalse(mockAps2.isMaster(role_hrs));
+ Assert.assertTrue(mockAps2.canPublishConfig(role_hm));
+ Assert.assertFalse(mockAps2.canPublishConfig(role_hrs));
+ Assert.assertTrue(mockAps2.canAnyMasterPublishConfig());
}
@Test
@@ -502,6 +573,11 @@
anyString(),
anyString(),
any(HeartBeatResponse.class));
+ doNothing().when(mockAps).publishComponentConfiguration(
+ anyString(),
+ anyString(),
+ anyCollection());
+
} catch (SliderException e) {
}
@@ -584,6 +660,7 @@
cr.setRole("HBASE_REGIONSERVER");
cr.setRoleCommand("INSTALL");
cr.setStatus("COMPLETED");
+ cr.setFolders(new HashMap<String, String>() {{put("a", "b");}});
hb.setReports(Arrays.asList(cr));
hbr = mockAps.handleHeartBeat(hb);
Assert.assertEquals(3, hbr.getResponseId());
@@ -662,6 +739,11 @@
} catch (SliderException | IOException he) {
log.warn(he.getMessage());
}
+
+ Mockito.verify(mockAps, Mockito.times(1)).publishComponentConfiguration(
+ anyString(),
+ anyString(),
+ anyCollection());
}
diff --git a/slider-core/src/test/java/org/apache/slider/providers/agent/TestComponentCommandOrder.java b/slider-core/src/test/java/org/apache/slider/providers/agent/TestComponentCommandOrder.java
index 181371f..3ef1839 100644
--- a/slider-core/src/test/java/org/apache/slider/providers/agent/TestComponentCommandOrder.java
+++ b/slider-core/src/test/java/org/apache/slider/providers/agent/TestComponentCommandOrder.java
@@ -40,7 +40,7 @@
co2.setRequires("C-STARTED");
CommandOrder co3 = new CommandOrder();
co3.setCommand("B-START");
- co3.setRequires("C-STARTED,D-STARTED,E-STARTED");
+ co3.setRequires("C-STARTED,D-STARTED,E-INSTALLED");
ComponentCommandOrder cco = new ComponentCommandOrder(Arrays.asList(co1, co2, co3));
ComponentInstanceState cisB = new ComponentInstanceState("B", "cid", "aid");
@@ -73,11 +73,17 @@
cisD.setState(State.STARTED);
Assert.assertTrue(cco.canExecute("B", Command.START, Arrays.asList(cisB, cisC, cisD, cisE)));
- cisE2.setState(State.INSTALLED);
+ cisE2.setState(State.INSTALLING);
Assert.assertFalse(cco.canExecute("B", Command.START, Arrays.asList(cisE, cisE2)));
+ cisE2.setState(State.INSTALLED);
+ Assert.assertTrue(cco.canExecute("B", Command.START, Arrays.asList(cisE, cisE2)));
+
cisE2.setState(State.STARTED);
Assert.assertTrue(cco.canExecute("B", Command.START, Arrays.asList(cisE, cisE2)));
+
+ cisE2.setState(State.STARTING);
+ Assert.assertTrue(cco.canExecute("B", Command.START, Arrays.asList(cisE, cisE2)));
}
@Test
@@ -138,14 +144,5 @@
} catch (IllegalArgumentException ie) {
log.info(ie.getMessage());
}
-
- co.setCommand(" A-START");
- co.setRequires("B-INSTALLED");
- try {
- cco = new ComponentCommandOrder(Arrays.asList(co));
- Assert.fail("Instantiation should have failed.");
- } catch (IllegalArgumentException ie) {
- log.info(ie.getMessage());
- }
}
}
diff --git a/slider-core/src/test/java/org/apache/slider/server/appmaster/web/rest/publisher/TestAgentProviderService.java b/slider-core/src/test/java/org/apache/slider/server/appmaster/web/rest/publisher/TestAgentProviderService.java
index 6aa7732..97199f4 100644
--- a/slider-core/src/test/java/org/apache/slider/server/appmaster/web/rest/publisher/TestAgentProviderService.java
+++ b/slider-core/src/test/java/org/apache/slider/server/appmaster/web/rest/publisher/TestAgentProviderService.java
@@ -39,8 +39,8 @@
@Override
public void bind(StateAccessForProviders stateAccessor,
- RegistryViewForProviders registry) {
- super.bind(stateAccessor, registry);
+ RegistryViewForProviders reg) {
+ super.bind(stateAccessor, reg);
Map<String,String> dummyProps = new HashMap<>();
dummyProps.put("prop1", "val1");
dummyProps.put("prop2", "val2");
diff --git a/slider-core/src/test/java/org/apache/slider/test/MiniZooKeeperCluster.java b/slider-core/src/test/java/org/apache/slider/test/MiniZooKeeperCluster.java
new file mode 100644
index 0000000..cc2cc9b
--- /dev/null
+++ b/slider-core/src/test/java/org/apache/slider/test/MiniZooKeeperCluster.java
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.slider.test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.zookeeper.server.NIOServerCnxnFactory;
+import org.apache.zookeeper.server.ZooKeeperServer;
+import org.apache.zookeeper.server.persistence.FileTxnLog;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.net.BindException;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+
+
+/**
+ * This is a version of the HBase ZK cluster cut out to be standalone
+ */
+public class MiniZooKeeperCluster {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ MiniZooKeeperCluster.class);
+
+ private static final int TICK_TIME = 2000;
+ private static final int CONNECTION_TIMEOUT = 30000;
+ public static final int MAX_CLIENT_CONNECTIONS = 1000;
+
+ private boolean started;
+
+ /** The default port. If zero, we use a random port. */
+ private int defaultClientPort = 0;
+
+ private int clientPort;
+
+ private List<NIOServerCnxnFactory> standaloneServerFactoryList;
+ private List<ZooKeeperServer> zooKeeperServers;
+ private List<Integer> clientPortList;
+
+ private int activeZKServerIndex;
+ private int tickTime = 0;
+
+ private Configuration configuration;
+
+ public MiniZooKeeperCluster() {
+ this(new Configuration());
+ }
+
+ public MiniZooKeeperCluster(Configuration configuration) {
+ this.started = false;
+ this.configuration = configuration;
+ activeZKServerIndex = -1;
+ zooKeeperServers = new ArrayList<>();
+ clientPortList = new ArrayList<>();
+ standaloneServerFactoryList = new ArrayList<>();
+ }
+
+ public void setDefaultClientPort(int clientPort) {
+ if (clientPort <= 0) {
+ throw new IllegalArgumentException("Invalid default ZK client port: "
+ + clientPort);
+ }
+ this.defaultClientPort = clientPort;
+ }
+
+ /**
+ * Selects a ZK client port. Returns the default port if specified.
+ * Otherwise, returns a random port. The random port is selected from the
+ * range between 49152 to 65535. These ports cannot be registered with IANA
+ * and are intended for dynamic allocation (see http://bit.ly/dynports).
+ */
+ private int selectClientPort() {
+ if (defaultClientPort > 0) {
+ return defaultClientPort;
+ }
+ return 0xc000 + new Random().nextInt(0x3f00);
+ }
+
+ public void setTickTime(int tickTime) {
+ this.tickTime = tickTime;
+ }
+
+ public int getBackupZooKeeperServerNum() {
+ return zooKeeperServers.size() - 1;
+ }
+
+ public int getZooKeeperServerNum() {
+ return zooKeeperServers.size();
+ }
+
+ // / XXX: From o.a.zk.t.ClientBase
+ private static void setupTestEnv() {
+ // during the tests we run with 100K prealloc in the logs.
+ // on windows systems prealloc of 64M was seen to take ~15seconds
+ // resulting in test failure (client timeout on first session).
+ // set env and directly in order to handle static init/gc issues
+ System.setProperty("zookeeper.preAllocSize", "100");
+ FileTxnLog.setPreallocSize(100 * 1024);
+ }
+
+ public int startup(File baseDir) throws IOException, InterruptedException {
+ return startup(baseDir, 1);
+ }
+
+ /**
+ * @param baseDir
+ * @param numZooKeeperServers
+ * @return ClientPort server bound to, -1 if there was a
+ * binding problem and we couldn't pick another port.
+ * @throws IOException
+ * @throws InterruptedException
+ */
+ public int startup(File baseDir, int numZooKeeperServers) throws IOException,
+ InterruptedException {
+ if (numZooKeeperServers <= 0)
+ return -1;
+
+ setupTestEnv();
+ shutdown();
+
+ int tentativePort = selectClientPort();
+
+ // running all the ZK servers
+ for (int i = 0; i < numZooKeeperServers; i++) {
+ File dir = new File(baseDir, "zookeeper_" + i).getAbsoluteFile();
+ recreateDir(dir);
+ int tickTimeToUse;
+ if (this.tickTime > 0) {
+ tickTimeToUse = this.tickTime;
+ } else {
+ tickTimeToUse = TICK_TIME;
+ }
+ ZooKeeperServer server = new ZooKeeperServer(dir, dir, tickTimeToUse);
+ NIOServerCnxnFactory standaloneServerFactory;
+ while (true) {
+ try {
+ standaloneServerFactory = new NIOServerCnxnFactory();
+ standaloneServerFactory.configure(
+ new InetSocketAddress(tentativePort),
+ MAX_CLIENT_CONNECTIONS
+ );
+ } catch (BindException e) {
+ LOG.debug("Failed binding ZK Server to client port: " +
+ tentativePort, e);
+ // We're told to use some port but it's occupied, fail
+ if (defaultClientPort > 0) return -1;
+ // This port is already in use, try to use another.
+ tentativePort = selectClientPort();
+ continue;
+ }
+ break;
+ }
+
+ // Start up this ZK server
+ standaloneServerFactory.startup(server);
+ if (!waitForServerUp(tentativePort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for startup of standalone server");
+ }
+
+ // We have selected this port as a client port.
+ clientPortList.add(tentativePort);
+ standaloneServerFactoryList.add(standaloneServerFactory);
+ zooKeeperServers.add(server);
+ tentativePort++; //for the next server
+ }
+
+ // set the first one to be active ZK; Others are backups
+ activeZKServerIndex = 0;
+ started = true;
+ clientPort = clientPortList.get(activeZKServerIndex);
+ LOG.info("Started MiniZK Cluster and connect 1 ZK server " +
+ "on client port: " + clientPort);
+ return clientPort;
+ }
+
+ private void recreateDir(File dir) throws IOException {
+ if (dir.exists()) {
+ if (!FileUtil.fullyDelete(dir)) {
+ throw new IOException("Could not delete zk base directory: " + dir);
+ }
+ }
+ try {
+ dir.mkdirs();
+ } catch (SecurityException e) {
+ throw new IOException("creating dir: " + dir, e);
+ }
+ }
+
+ /**
+ * @throws IOException
+ */
+ public void shutdown() throws IOException {
+ if (!started) {
+ return;
+ }
+
+ // shut down all the zk servers
+ for (int i = 0; i < standaloneServerFactoryList.size(); i++) {
+ NIOServerCnxnFactory standaloneServerFactory =
+ standaloneServerFactoryList.get(i);
+ int clientPort = clientPortList.get(i);
+
+ standaloneServerFactory.shutdown();
+ if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for shutdown of standalone server");
+ }
+ }
+ for (ZooKeeperServer zkServer : zooKeeperServers) {
+ //explicitly close ZKDatabase since ZookeeperServer does not close them
+ zkServer.getZKDatabase().close();
+ }
+
+ // clear everything
+ started = false;
+ activeZKServerIndex = 0;
+ standaloneServerFactoryList.clear();
+ clientPortList.clear();
+ zooKeeperServers.clear();
+
+ LOG.info("Shutdown MiniZK cluster with all ZK servers");
+ }
+
+ /**@return clientPort return clientPort if there is another ZK backup can run
+ * when killing the current active; return -1, if there is no backups.
+ * @throws IOException
+ * @throws InterruptedException
+ */
+ public int killCurrentActiveZooKeeperServer() throws IOException,
+ InterruptedException {
+ if (!started || activeZKServerIndex < 0) {
+ return -1;
+ }
+
+ // Shutdown the current active one
+ NIOServerCnxnFactory standaloneServerFactory =
+ standaloneServerFactoryList.get(activeZKServerIndex);
+ int clientPort = clientPortList.get(activeZKServerIndex);
+
+ standaloneServerFactory.shutdown();
+ if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for shutdown of standalone server");
+ }
+
+ zooKeeperServers.get(activeZKServerIndex).getZKDatabase().close();
+
+ // remove the current active zk server
+ standaloneServerFactoryList.remove(activeZKServerIndex);
+ clientPortList.remove(activeZKServerIndex);
+ zooKeeperServers.remove(activeZKServerIndex);
+ LOG.info("Kill the current active ZK servers in the cluster " +
+ "on client port: " + clientPort);
+
+ if (standaloneServerFactoryList.size() == 0) {
+ // there is no backup servers;
+ return -1;
+ }
+ clientPort = clientPortList.get(activeZKServerIndex);
+ LOG.info("Activate a backup zk server in the cluster " +
+ "on client port: " + clientPort);
+ // return the next back zk server's port
+ return clientPort;
+ }
+
+ /**
+ * Kill one back up ZK servers
+ * @throws IOException
+ * @throws InterruptedException
+ */
+ public void killOneBackupZooKeeperServer() throws IOException,
+ InterruptedException {
+ if (!started || activeZKServerIndex < 0 ||
+ standaloneServerFactoryList.size() <= 1) {
+ return;
+ }
+
+ int backupZKServerIndex = activeZKServerIndex + 1;
+ // Shutdown the current active one
+ NIOServerCnxnFactory standaloneServerFactory =
+ standaloneServerFactoryList.get(backupZKServerIndex);
+ int clientPort = clientPortList.get(backupZKServerIndex);
+
+ standaloneServerFactory.shutdown();
+ if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) {
+ throw new IOException("Waiting for shutdown of standalone server");
+ }
+
+ zooKeeperServers.get(backupZKServerIndex).getZKDatabase().close();
+
+ // remove this backup zk server
+ standaloneServerFactoryList.remove(backupZKServerIndex);
+ clientPortList.remove(backupZKServerIndex);
+ zooKeeperServers.remove(backupZKServerIndex);
+ LOG.info("Kill one backup ZK servers in the cluster " +
+ "on client port: " + clientPort);
+ }
+
+ // XXX: From o.a.zk.t.ClientBase
+ private static boolean waitForServerDown(int port, long timeout) {
+ long start = System.currentTimeMillis();
+ while (true) {
+ try {
+ Socket sock = new Socket("localhost", port);
+ try {
+ OutputStream outstream = sock.getOutputStream();
+ outstream.write("stat".getBytes());
+ outstream.flush();
+ } finally {
+ sock.close();
+ }
+ } catch (IOException e) {
+ return true;
+ }
+
+ if (System.currentTimeMillis() > start + timeout) {
+ break;
+ }
+ try {
+ Thread.sleep(250);
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ }
+ return false;
+ }
+
+ // XXX: From o.a.zk.t.ClientBase
+ private static boolean waitForServerUp(int port, long timeout) {
+ long start = System.currentTimeMillis();
+ while (true) {
+ try {
+ Socket sock = new Socket("localhost", port);
+ BufferedReader reader = null;
+ try {
+ OutputStream outstream = sock.getOutputStream();
+ outstream.write("stat".getBytes());
+ outstream.flush();
+
+ Reader isr = new InputStreamReader(sock.getInputStream());
+ reader = new BufferedReader(isr);
+ String line = reader.readLine();
+ if (line != null && line.startsWith("Zookeeper version:")) {
+ return true;
+ }
+ } finally {
+ sock.close();
+ if (reader != null) {
+ reader.close();
+ }
+ }
+ } catch (IOException e) {
+ // ignore as this is expected
+ LOG.info("server localhost:" + port + " not up " + e);
+ }
+
+ if (System.currentTimeMillis() > start + timeout) {
+ break;
+ }
+ try {
+ Thread.sleep(250);
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ }
+ return false;
+ }
+
+ public int getClientPort() {
+ return clientPort;
+ }
+}
diff --git a/slider-funtest/pom.xml b/slider-funtest/pom.xml
index 71898b3..529c496 100644
--- a/slider-funtest/pom.xml
+++ b/slider-funtest/pom.xml
@@ -17,7 +17,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>slider-funtest</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider Functional Tests</name>
<packaging>jar</packaging>
<description>
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
</parent>
<build>
diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/abstracttests/AbstractTestBuildSetup.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/abstracttests/AbstractTestBuildSetup.groovy
index fba7461..c42589a 100644
--- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/abstracttests/AbstractTestBuildSetup.groovy
+++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/abstracttests/AbstractTestBuildSetup.groovy
@@ -87,7 +87,7 @@
* @return
*/
public Configuration loadSliderConf() {
- Configuration conf = (new ConfLoader()).loadSliderConf(confXML)
+ Configuration conf = ConfLoader.loadSliderConf(confXML)
return conf
}
diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
index c69b8ab..d0d7fc9 100644
--- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
+++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/CommandTestBase.groovy
@@ -75,7 +75,7 @@
static {
- SLIDER_CONFIG = new ConfLoader().loadSliderConf(SLIDER_CONF_XML);
+ SLIDER_CONFIG = ConfLoader.loadSliderConf(SLIDER_CONF_XML);
THAW_WAIT_TIME = getTimeOptionMillis(SLIDER_CONFIG,
KEY_TEST_THAW_WAIT_TIME,
1000 * DEFAULT_THAW_WAIT_TIME_SECONDS)
@@ -172,7 +172,7 @@
* @return
*/
public static Configuration loadSliderConf() {
- Configuration conf = (new ConfLoader()).loadSliderConf(SLIDER_CONF_XML)
+ Configuration conf = ConfLoader.loadSliderConf(SLIDER_CONF_XML)
return conf
}
diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ConfLoader.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ConfLoader.groovy
index 8ef06cd..447ea87 100644
--- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ConfLoader.groovy
+++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ConfLoader.groovy
@@ -18,12 +18,11 @@
package org.apache.slider.funtest.framework
-import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.yarn.conf.YarnConfiguration
public class ConfLoader {
- YarnConfiguration loadSliderConf(File confFile) {
+ public static YarnConfiguration loadSliderConf(File confFile) {
URI confURI = confFile.toURI();
YarnConfiguration conf = new YarnConfiguration()
def confXmlUrl = confURI.toURL()
diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FuntestProperties.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FuntestProperties.groovy
index 636da7b..8cbc098 100644
--- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FuntestProperties.groovy
+++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/FuntestProperties.groovy
@@ -1,9 +1,3 @@
-package org.apache.slider.funtest.framework
-
-import groovy.transform.CompileStatic
-import org.apache.slider.common.SliderXMLConfKeysForTesting
-import org.apache.slider.common.SliderKeys
-
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@@ -21,6 +15,11 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.slider.funtest.framework
+
+import groovy.transform.CompileStatic
+import org.apache.slider.common.SliderXMLConfKeysForTesting
+import org.apache.slider.common.SliderKeys
/**
* Properties unique to the functional tests
diff --git a/slider-providers/accumulo/accumulo-funtests/pom.xml b/slider-providers/accumulo/accumulo-funtests/pom.xml
index b0bc17b..3513526 100644
--- a/slider-providers/accumulo/accumulo-funtests/pom.xml
+++ b/slider-providers/accumulo/accumulo-funtests/pom.xml
@@ -18,7 +18,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>accumulo-funtests</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider Accumulo Provider Functional Tests</name>
<packaging>jar</packaging>
<description>
@@ -28,7 +28,7 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<relativePath>../../../</relativePath>
</parent>
diff --git a/slider-providers/accumulo/slider-accumulo-provider/pom.xml b/slider-providers/accumulo/slider-accumulo-provider/pom.xml
index e660360..14cbae6 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/pom.xml
+++ b/slider-providers/accumulo/slider-accumulo-provider/pom.xml
@@ -18,7 +18,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>slider-accumulo-provider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider Accumulo Provider</name>
<packaging>jar</packaging>
<description>
@@ -29,7 +29,7 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<relativePath>../../../</relativePath>
</parent>
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/main/java/org/apache/slider/providers/accumulo/AccumuloProviderService.java b/slider-providers/accumulo/slider-accumulo-provider/src/main/java/org/apache/slider/providers/accumulo/AccumuloProviderService.java
index 8396ff1..d35ed66 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/main/java/org/apache/slider/providers/accumulo/AccumuloProviderService.java
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/main/java/org/apache/slider/providers/accumulo/AccumuloProviderService.java
@@ -44,7 +44,7 @@
import org.apache.slider.providers.ProviderUtils;
import org.apache.slider.common.tools.SliderFileSystem;
import org.apache.slider.common.tools.SliderUtils;
-import org.apache.slider.core.registry.zk.BlockingZKWatcher;
+import org.apache.slider.core.zk.BlockingZKWatcher;
import org.apache.slider.common.tools.ConfigHelper;
import org.apache.slider.server.services.utility.EventCallback;
import org.apache.slider.server.services.utility.EventNotifyingService;
@@ -56,7 +56,6 @@
import java.io.File;
import java.io.IOException;
-import java.net.URL;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccCorrectInstanceName.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccCorrectInstanceName.groovy
index ad8981f..89dc90c 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccCorrectInstanceName.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccCorrectInstanceName.groovy
@@ -30,7 +30,7 @@
import org.apache.slider.common.params.Arguments
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccFreezeThaw.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccFreezeThaw.groovy
index b3eecd2..3983a5d 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccFreezeThaw.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccFreezeThaw.groovy
@@ -25,7 +25,7 @@
import org.apache.slider.providers.accumulo.AccumuloKeys
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveHDFSArchive.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveHDFSArchive.groovy
index 7f16a6f..2220cdc 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveHDFSArchive.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveHDFSArchive.groovy
@@ -27,7 +27,7 @@
import org.apache.slider.providers.accumulo.AccumuloKeys
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveLocalArchive.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveLocalArchive.groovy
index e8f6edb..1e3adbd 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveLocalArchive.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccLiveLocalArchive.groovy
@@ -27,7 +27,7 @@
import org.apache.slider.providers.accumulo.AccumuloKeys
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM1T1GC1Mon1.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM1T1GC1Mon1.groovy
index 0db2602..9d2cee6 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM1T1GC1Mon1.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM1T1GC1Mon1.groovy
@@ -26,7 +26,7 @@
import org.apache.slider.providers.accumulo.AccumuloKeys
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM2T2GC1Mon1.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM2T2GC1Mon1.groovy
index bf403b8..2494a56 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM2T2GC1Mon1.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccM2T2GC1Mon1.groovy
@@ -27,7 +27,7 @@
import org.apache.slider.providers.accumulo.AccumuloKeys
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccumuloAMWebApp.groovy b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccumuloAMWebApp.groovy
index 9eb2717..bd975e4 100644
--- a/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccumuloAMWebApp.groovy
+++ b/slider-providers/accumulo/slider-accumulo-provider/src/test/groovy/org/apache/slider/providers/accumulo/live/TestAccumuloAMWebApp.groovy
@@ -29,7 +29,7 @@
import org.apache.slider.server.appmaster.web.SliderAMWebApp
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.accumulo.AccumuloTestBase
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.junit.Test
@CompileStatic
diff --git a/slider-providers/hbase/hbase-funtests/pom.xml b/slider-providers/hbase/hbase-funtests/pom.xml
index a2f3082..28dba7f 100644
--- a/slider-providers/hbase/hbase-funtests/pom.xml
+++ b/slider-providers/hbase/hbase-funtests/pom.xml
@@ -18,7 +18,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>hbase-funtests</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider HBase Provider Functional Tests</name>
<packaging>jar</packaging>
<description>
@@ -28,7 +28,7 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<relativePath>../../../</relativePath>
</parent>
diff --git a/slider-providers/hbase/slider-hbase-provider/pom.xml b/slider-providers/hbase/slider-hbase-provider/pom.xml
index 88d1eca..15319cf 100644
--- a/slider-providers/hbase/slider-hbase-provider/pom.xml
+++ b/slider-providers/hbase/slider-hbase-provider/pom.xml
@@ -18,7 +18,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>slider-hbase-provider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<name>Slider HBase Provider</name>
<packaging>jar</packaging>
<description>
@@ -30,7 +30,7 @@
<parent>
<groupId>org.apache.slider</groupId>
<artifactId>slider</artifactId>
- <version>0.23.0-SNAPSHOT</version>
+ <version>0.30</version>
<relativePath>../../../</relativePath>
</parent>
diff --git a/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseClientProvider.java b/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseClientProvider.java
index 7627829..c40c5f2 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseClientProvider.java
+++ b/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseClientProvider.java
@@ -37,7 +37,7 @@
import org.apache.slider.common.tools.ConfigHelper;
import org.apache.slider.common.tools.SliderFileSystem;
import org.apache.slider.common.tools.SliderUtils;
-import org.apache.slider.core.registry.zk.ZookeeperUtils;
+import org.apache.slider.core.zk.ZookeeperUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -107,7 +107,7 @@
MapOperations worker = appconf.getMandatoryComponent(HBaseKeys.ROLE_WORKER);
- Map<String, String> sitexml = new HashMap<String, String>();
+ Map<String, String> sitexml = new HashMap<>();
//map all cluster-wide site. options
providerUtils.propagateSiteOptions(globalAppOptions, sitexml);
@@ -209,7 +209,7 @@
}
}
- private static Set<String> knownRoleNames = new HashSet<String>();
+ private static Set<String> knownRoleNames = new HashSet<>();
static {
List<ProviderRole> roles = HBaseRoles.getRoles();
knownRoleNames.add(SliderKeys.COMPONENT_AM);
@@ -256,7 +256,7 @@
// add any and all dependency files
Map<String, LocalResource> providerResources =
- new HashMap<String, LocalResource>();
+ new HashMap<>();
ProviderUtils.addProviderJar(providerResources,
this,
diff --git a/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseKeys.java b/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseKeys.java
index af799c0..1d6ca70 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseKeys.java
+++ b/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseKeys.java
@@ -70,6 +70,10 @@
String HBASE_GC_OPTS = "SERVER_GC_OPTS";
String PROPAGATED_CONFDIR = "PROPAGATED_CONFDIR";
+
+ /**
+ * Service type used in registry
+ */
String HBASE_SERVICE_TYPE = "org.apache.hbase";
String HBASE_SITE_PUBLISHED_CONFIG = "hbase-site";
}
diff --git a/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseProviderService.java b/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseProviderService.java
index 236ec62..fc23970 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseProviderService.java
+++ b/slider-providers/hbase/slider-hbase-provider/src/main/java/org/apache/slider/providers/hbase/HBaseProviderService.java
@@ -204,32 +204,25 @@
@Override
protected void serviceStart() throws Exception {
registerHBaseServiceEntry();
-
-
super.serviceStart();
}
private void registerHBaseServiceEntry() throws IOException {
- // not a URL, but needed
- URL hbaseURL = new URL("http://localhost:0");
- ServiceInstanceData instanceData = new ServiceInstanceData();
+ String name = amState.getApplicationName() ;
+// name += ".hbase";
+ ServiceInstanceData instanceData = new ServiceInstanceData(name,
+ HBASE_SERVICE_TYPE);
+ log.info("registering {}/{}", name, HBASE_SERVICE_TYPE );
PublishedConfiguration publishedSite =
- new PublishedConfiguration("HBase site",
- siteConf);
+ new PublishedConfiguration("HBase site", siteConf);
PublishedConfigSet configSet =
amState.getOrCreatePublishedConfigSet(HBASE_SERVICE_TYPE);
instanceData.externalView.configurationsURL = SliderUtils.appendToURL(
amWebAPI.toExternalForm(), SLIDER_PATH_PUBLISHER, HBASE_SERVICE_TYPE);
- configSet.put(HBASE_SITE_PUBLISHED_CONFIG,
- publishedSite);
- String name = amState.getApplicationName()+".hbase";
- log.info("registering {}/{}", name, HBASE_SERVICE_TYPE);
- registry.registerServiceInstance(HBASE_SERVICE_TYPE,
- name,
- null,
- instanceData
- );
+ configSet.put(HBASE_SITE_PUBLISHED_CONFIG, publishedSite);
+
+ registry.registerServiceInstance(instanceData, null);
}
/**
diff --git a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMaster.groovy b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMaster.groovy
index 01996ee..97714d6 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMaster.groovy
+++ b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMaster.groovy
@@ -27,7 +27,7 @@
import org.apache.slider.core.registry.info.ServiceInstanceData
import org.apache.slider.core.registry.retrieve.RegistryRetriever
import org.apache.slider.providers.hbase.HBaseKeys
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.apache.slider.common.params.Arguments
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
@@ -80,15 +80,28 @@
describe "service registry names"
SliderRegistryService registryService = client.registry
def names = registryService.getServiceTypes();
- dumpRegistryNames(names)
+ dumpRegistryServiceTypes(names)
List<CuratorServiceInstance<ServiceInstanceData>> instances =
client.listRegistryInstances();
- def hbaseService = registryService.findByID(
- instances,
- HBaseKeys.HBASE_SERVICE_TYPE)
+
+ def hbaseInstances = registryService.findInstances( HBaseKeys.HBASE_SERVICE_TYPE, null)
+ assert hbaseInstances.size() == 1
+ def hbaseService = hbaseInstances[0]
assert hbaseService
- RegistryRetriever retriever = new RegistryRetriever(hbaseService.payload)
+ def hbaseServiceData = hbaseService.payload
+ log.info "HBase service 0 == $hbaseServiceData"
+ assert hbaseServiceData.id
+ assert hbaseServiceData.serviceType == HBaseKeys.HBASE_SERVICE_TYPE
+
+ hbaseInstances = registryService.findInstances(
+ HBaseKeys.HBASE_SERVICE_TYPE,
+ clustername)
+ assert hbaseInstances.size() == 1
+ def hbaseServiceData2 = hbaseInstances[0].payload
+ assert hbaseServiceData == hbaseServiceData2
+
+ RegistryRetriever retriever = new RegistryRetriever(hbaseServiceData)
log.info retriever.toString()
assert retriever.hasConfigurations(true)
PublishedConfigSet externalConfSet = retriever.getConfigurations(true)
diff --git a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMasterWithBadHeap.groovy b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMasterWithBadHeap.groovy
index 5515eac..9ca5f45 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMasterWithBadHeap.groovy
+++ b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestHBaseMasterWithBadHeap.groovy
@@ -21,16 +21,13 @@
import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.hadoop.hbase.ClusterStatus
-import org.apache.slider.api.ClusterDescription
import org.apache.slider.api.RoleKeys
import org.apache.slider.client.SliderClient
import org.apache.slider.common.SliderExitCodes
-import org.apache.slider.common.SliderXmlConfKeys
import org.apache.slider.common.params.Arguments
import org.apache.slider.core.conf.AggregateConf
import org.apache.slider.core.main.ServiceLaunchException
import org.apache.slider.core.main.ServiceLauncher
-import org.apache.slider.core.registry.zk.ZKIntegration
import org.apache.slider.providers.hbase.HBaseKeys
import org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
import org.junit.Test
diff --git a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestLiveRegionServiceOnHDFS.groovy b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestLiveRegionServiceOnHDFS.groovy
index eb94580..26292fb 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestLiveRegionServiceOnHDFS.groovy
+++ b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestLiveRegionServiceOnHDFS.groovy
@@ -22,7 +22,7 @@
import groovy.util.logging.Slf4j
import org.apache.hadoop.hbase.ClusterStatus
import org.apache.slider.api.ClusterDescription
-import org.apache.slider.core.registry.zk.ZKIntegration
+import org.apache.slider.core.zk.ZKIntegration
import org.apache.slider.client.SliderClient
import org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
import org.apache.slider.core.main.ServiceLauncher
diff --git a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestTwoLiveClusters.groovy b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestTwoLiveClusters.groovy
index dba52ba..a58f99a 100644
--- a/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestTwoLiveClusters.groovy
+++ b/slider-providers/hbase/slider-hbase-provider/src/test/groovy/org/apache/slider/providers/hbase/minicluster/live/TestTwoLiveClusters.groovy
@@ -22,6 +22,7 @@
import groovy.util.logging.Slf4j
import org.apache.slider.common.params.Arguments
import org.apache.slider.client.SliderClient
+import org.apache.slider.providers.hbase.HBaseKeys
import org.apache.slider.providers.hbase.minicluster.HBaseMiniClusterTestBase
import org.apache.slider.core.main.ServiceLauncher
import org.apache.slider.core.registry.info.ServiceInstanceData
@@ -41,8 +42,10 @@
*/
@Test
public void testTwoLiveClusters() throws Throwable {
- createMiniCluster("TestTwoLiveClusters", configuration, 1, true)
- String clustername1 = "testtwoliveclusters-a"
+ def name = "test_two_live_clusters"
+ createMiniCluster(name, configuration, 1, true)
+
+ String clustername1 = name + "-1"
//now launch the cluster
int regionServerCount = 1
ServiceLauncher<SliderClient> launcher = createHBaseCluster(clustername1, regionServerCount, [], true, true)
@@ -59,7 +62,7 @@
waitForHBaseRegionServerCount(sliderClient, clustername1, 1, hbaseClusterStartupToLiveTime)
//now here comes cluster #2
- String clustername2 = "testtwoliveclusters-b"
+ String clustername2 = name + "-2"
String zkpath = "/$clustername2"
@@ -83,21 +86,30 @@
describe "service registry names"
SliderRegistryService registry = cluster2Client.registry
def names = registry.getServiceTypes();
- dumpRegistryNames(names)
+ dumpRegistryServiceTypes(names)
- List<String> instanceIds = sliderClient.listRegistryInstanceIDs()
+ List<String> instanceIds = sliderClient.listRegistedSliderInstances()
dumpRegistryInstanceIDs(instanceIds)
- assert names.size() == 1
+ assert names.size() == 2
assert instanceIds.size() == 2
- List<CuratorServiceInstance<ServiceInstanceData>> instances = sliderClient.listRegistryInstances(
- )
+ List<CuratorServiceInstance<ServiceInstanceData>> instances =
+ sliderClient.listRegistryInstances()
dumpRegistryInstances(instances)
assert instances.size() == 2
+ def hbaseInstances = registry.findInstances(
+ HBaseKeys.HBASE_SERVICE_TYPE, null)
+ assert hbaseInstances.size() == 2
+ def hbase1ServiceData = registry.findInstance(
+ HBaseKeys.HBASE_SERVICE_TYPE, clustername1).payload
+ def hbase2ServiceData = registry.findInstance(
+ HBaseKeys.HBASE_SERVICE_TYPE, clustername2).payload
+ assert !(hbase1ServiceData == hbase2ServiceData)
+
clusterActionFreeze(cluster2Client, clustername2,"freeze cluster 2")
clusterActionFreeze(sliderClient, clustername1,"Freeze cluster 1")
diff --git a/src/site/markdown/architecture/index.md b/src/site/markdown/architecture/index.md
index 4333dcc..d77a58e 100644
--- a/src/site/markdown/architecture/index.md
+++ b/src/site/markdown/architecture/index.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# Architecture
@@ -21,4 +24,4 @@
* [Role history](rolehistory.html)
-
\ No newline at end of file
+
diff --git a/src/site/markdown/developing/index.md b/src/site/markdown/developing/index.md
index 0bd0659..7d8cf93 100644
--- a/src/site/markdown/developing/index.md
+++ b/src/site/markdown/developing/index.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# Developing Slider
@@ -29,4 +32,4 @@
* [Releasing](releasing.html)
-
\ No newline at end of file
+
diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md
index 4ec5cfd..7a5d6e2 100644
--- a/src/site/markdown/index.md
+++ b/src/site/markdown/index.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
diff --git a/src/site/markdown/registry/p2p_service_registries.md b/src/site/markdown/registry/p2p_service_registries.md
index a30698f..6b2724a 100644
--- a/src/site/markdown/registry/p2p_service_registries.md
+++ b/src/site/markdown/registry/p2p_service_registries.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# P2P Service Registries
diff --git a/src/site/markdown/registry/references.md b/src/site/markdown/registry/references.md
index ade4f4f..bf0f068 100644
--- a/src/site/markdown/registry/references.md
+++ b/src/site/markdown/registry/references.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# References
@@ -43,4 +46,4 @@
-
\ No newline at end of file
+
diff --git a/src/site/markdown/release_notes/release-0.24.0.md b/src/site/markdown/release_notes/release-0.30.0.md
similarity index 70%
rename from src/site/markdown/release_notes/release-0.24.0.md
rename to src/site/markdown/release_notes/release-0.30.0.md
index 608b8e0..e88a709 100644
--- a/src/site/markdown/release_notes/release-0.24.0.md
+++ b/src/site/markdown/release_notes/release-0.30.0.md
@@ -15,7 +15,7 @@
limitations under the License.
-->
-# Slider Release 0.24.0
+# Slider Release 0.30.0
May 2014
@@ -26,7 +26,10 @@
## Key changes
-
+1. Slider application registry that allow registration and discovery of application configuration and URLs (such as jmx endpoints and management UI) for client consumption.
+2. Move to a .zip packaging for Slider application packages.
+3. Richer metainfo support to provide start ordering and arbitrary template that can be published.
+4. Updated application definition packages for HBase, Accumulo, and Storm.
## Other changes
diff --git a/src/site/markdown/slider_specs/application_definition.md b/src/site/markdown/slider_specs/application_definition.md
index 52f8106..6e417d2 100644
--- a/src/site/markdown/slider_specs/application_definition.md
+++ b/src/site/markdown/slider_specs/application_definition.md
@@ -17,7 +17,7 @@
# Application Definition
-App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the [AppPackage](application_package.md).
+App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the [AppPackage](application_package.md). Application definition is an XML file and is packaged as metainfo.xml.
## Structure
@@ -180,13 +180,3 @@
</application>
</metainfo>
-
-
-## Open Questions
-
-1. Applications may need some information from other applications or base services such as ZK, YARN, HDFS. Additionally, they may need a dedicated ZK node, a HDFS working folder, etc. How do we capture this requirement? There needs to be a well-known way to ask for these information e.g. fs.default.name, zk_hosts.
-
-2. Similar to the above there are common parameters such as `JAVA_HOME` and other environment variables. Application should be able to refer to these parameters and Slider should be able to provide them.
-
-3. Composite application definition: Composite application definition would require a spec that refers to this spec and binds multiple applications together.
-
diff --git a/src/site/markdown/slider_specs/application_instance_configuration.md b/src/site/markdown/slider_specs/application_instance_configuration.md
index dad7f4e..764d3ce 100644
--- a/src/site/markdown/slider_specs/application_instance_configuration.md
+++ b/src/site/markdown/slider_specs/application_instance_configuration.md
@@ -23,38 +23,82 @@
{
- "configurations": {
- "app-global-config": {
- },
- "config-type-1": {
- },
- "config-type-2": {
- },
- }
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ },
}
-
-
-The configuration overrides are organized in a two level structure where name-value pairs are grouped on the basis of config types they belong to. App instantiator can provide arbitrary custom name-value pairs within a config type defined in the AppPackage or can create a completely new config type that does not exist in the AppAPackage. The interpretation of the configuration is entirely up to the command implementations present in the AppPackage. Slider will simply merge the configs with the InstanceConfiguration being higher priority than that default configuration and hand it off to the app commands.
-
-A sample config for hbase may be as follows:
-
+An appConfig.json contains the application configuration. The sample below shows configuration for HBase.
{
- "configurations": {
- "hbase-log4j": {
- "log4j.logger.org.apache.zookeeper": "INFO",
- "log4j.logger.org.apache.hadoop.hbase": "DEBUG"
- },
- "hbase-site": {
- "hbase.hstore.flush.retries.number": "120",
- "hbase.regionserver.info.port": "",
- "hbase.master.info.port": "60010"
- }
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ "config_types": "core-site,hdfs-site,hbase-site",
+
+ "java_home": "/usr/jdk64/jdk1.7.0_45",
+ "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
+
+ "site.global.app_user": "yarn",
+ "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
+ "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
+ "site.global.security_enabled": "false",
+
+ "site.hbase-site.hbase.hstore.flush.retries.number": "120",
+ "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
+ "site.hbase-site.hbase.hstore.compactionThreshold": "3",
+ "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
+ "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
+ "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
+ "site.hbase-site.hbase.regionserver.port": "0",
+
+ "site.core-site.fs.defaultFS": "${NN_URI}",
+ "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
+ "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
+ }
}
-
+
+appConf.json allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
+
+**Variable naming convention**
+In order to understand how the naming convention work, lets look at how the config is passed on to component commands. Slider agent recevies a structured bag of commands as input for all commands, INSTALL, CONFIGURE, START, etc. The command includes a section "configuration" which has config properties arranged into named property bags.
+
+* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app definition scripts can access all variables under `hbase-site` as a single property bag.
+* Similarly, `site.core-site.fs.defaultFS` allows you to pass in the default fs. *This specific variable is automatically made available by Slider but its shown here as an example.*
+* Variables of the form `site.global.zz` are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as "global variable".
+
+**slider variables**
+
+* Any config not of the form `site.xx.yy` are consumed by Slider itself. Some of the manadatory configuration are:
+ * `agent.conf`: location of the agent config file (typically, "/slider/agent/conf/agent.ini")
+ * `application.def`: location of the application definition package (typically, "/slider/hbase_v096.zip")
+ * `config_types`: list of config types sent to the containers (e.g. "core-site,hdfs-site,hbase-site")
+ * `java_home`: java home path (e.g. "/usr/jdk64/jdk1.7.0_45")
+ * `package_list`: location of the package relative to the root where AppPackage is installed (e.g. "files/hbase-0.96.1-hadoop2-bin.tar.gz"
+
+**dynamically allocated ports**
+
+Apps can ask port to be dynamically assigned by Slider or they can leave it as "0". If there is a need for advertising any listening endpoint then the ports can be marked such.
+
+For example, HBase master info port needs to be advertised so that jmx endpoint can be accessed. This is indicated by using a special value, of the form, ${COMPONENT_NAME.ALLOCATED_PORT}. E.g. "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}"
+
+[Application Definition](application_definition.md) describes how to advertise arbitrary set of properties that are dynamically finalized when application is activated.
+
+**configuraing an app for ganglia metrics**
+
+There is no set guideline for doing so. How an application emits metrics and how the metrics are emitted to the right place is completely defined by the application. In the following example, we hso how HBase app is configured to emit metrics to a ganglia server.
+
+Ganglia server lifecycle is not controlled by the app instance. So the app instance only needs to know where to emit the metrics. This is achieved by three global variables
+
+* "site.global.ganglia_server_host": "gangliaserver.my.org"
+* "site.global.ganglia_server_port": "8663"
+* "site.global.ganglia_server_id": "HBaseApplicationCluster3"
+
+All three variable values are user provided. It is also expected that a gmond server is available on host gangliaserver.my.org listening for metrics at port 8663 and is named "HBaseApplicationCluster3". Its the reponsibility of the ganglia server admin to ensure that the server is unique and is only receving metrics from the application instance.
-The above config overwrites few parameters in hbase-site and hbase-log4j files. Several config properties such as `hbase.zookeeper.quorum` for hbase may not be known to the user at the time of app instantiation. These configurations will be provided by the Slider infrastructure in a well-known form so that the app implementation can read and set them while instantiating component instances..
diff --git a/src/site/markdown/slider_specs/creating_app_definitions.md b/src/site/markdown/slider_specs/creating_app_definitions.md
index 6015eb1..b89df62 100644
--- a/src/site/markdown/slider_specs/creating_app_definitions.md
+++ b/src/site/markdown/slider_specs/creating_app_definitions.md
@@ -15,7 +15,7 @@
limitations under the License.
-->
-# Define and use Slider AppPackage
+# Slider AppPackage
Slider AppPackages are a declarative definition of an application for application management. AppPackage is not a packaging scheme for application binaries and artifacts. Tarball, zip files, rpms etc. are available for that purpose. Instead AppPackage includes the application binaries along with other artifacts necessary for application management.
@@ -31,18 +31,16 @@
yarn 9085 -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver
yarn 9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...
-Shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. Three of these together constitute the container.
+The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. *Three of these together constitute the container*.
## Using an AppPackage
The following command creates an HBase application using the AppPackage for HBase.
- ./slider create cl1 --zkhosts zk1,zk2 --image hdfs://NN:8020/slider/agent/slider-agent-0.21.tar --option agent.conf hdfs://NN:8020/slider/agent/conf/agent.ini --template /work/appConf.json --resources /work/resources.json --option application.def hdfs://NN:8020/slider/hbase_v096.zip
+ ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz --template /work/appConf.json --resources /work/resources.json
Lets analyze various parameters from the perspective of app creation:
* `--image`: its the slider agent tarball
-* `--option agent.conf`: the configuration file for the agent instance
-* `--option app.def`: app def (AppPackage)
* `--template`: app configuration
* `--resources`: yarn resource requests
* … other parameters are described in accompanying docs.
@@ -50,13 +48,13 @@
### AppPackage
The structure of an AppPackage is described at [AppPackage](application_package.md).
-In the enlistment there are three example AppPackages
+In the enlistment, there are three example AppPackages:
* `app-packages/hbase-v0_96`
* `app-packages/accumulo-v1_5`
* `app-packages/storm-v0_91`
-The application zip file, containing the binaries/artifacts of the application itself is a component within the AppPackage. They are:
+The above folders, with minor edits, can be packaged as *zip* files to get the corresponding AppPackages. The application tarball file, containing the binaries/artifacts of the application itself is a component within the AppPackage. They are:
* For hbase - `app-packages/hbase-v0_96/package/files/hbase-0.96.1-hadoop2-bin.tar.gz.REPLACE`
* For accumulo - `app-packages/accumulo-v1_5/package/files/accumulo-1.5.1-bin.tar.gz.REPLACE`
@@ -64,8 +62,6 @@
**They are placehoder files**, mostly because the files themselves are too large as well as users are free to use their own version of the package. To create a Slider AppPackage - replace the file with an actual application tarball and then ensure that the metainfo.xml has the correct file name. After that create a zip file using standard zip commands and ensure that the package has the metainfo.xml file at the root folder.
-Sample **resources.json** and **appConfig.json** files are also included in the enlistment.
-
For example:
* cd slider/app-packages/hbase-v0_96
@@ -109,79 +105,18 @@
83219742 29 files
```
-
-### appConf.json
-An appConf.json contains the application configuration. The sample below shows configuration for HBase.
+Sample **resources.json** and **appConfig.json** files are also included in the enlistment. These are samples and are typically tested on one node test installations.
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- "config_types": "core-site,hdfs-site,hbase-site",
-
- "java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
-
- "site.global.app_user": "yarn",
- "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
- "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
- "site.global.security_enabled": "false",
-
- "site.hbase-site.hbase.hstore.flush.retries.number": "120",
- "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
- "site.hbase-site.hbase.hstore.compactionThreshold": "3",
- "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
- "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
- "site.hbase-site.hbase.regionserver.port": "0",
-
- "site.core-site.fs.defaultFS": "${NN_URI}",
- "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
- "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
- }
- }
+### --template appConfig.json
+An appConfig.json contains the application configuration. See [Specifications InstanceConfiguration](application_instance_configuration.md) for details on how to create a template config file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
-appConf.jso allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
-
-* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app def scripts can access all variables under `hbase-site` as a single property bag.
-* Similarly, `site.core-site.fs.defaultFS` allows you to pass in the default fs. *This specific variable is automatically made available by Slider but its shown here as an example.*
-* Variables of the form `site.global.zz` are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as "global variable".
### --resources resources.json
-The resources.json file encodes the Yarn resource count requirement for the application instance.
+Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application. [Specification of Resources](resource_specification.html) describes how to write a resource config json file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
-The components section lists the two application component for an HBase application.
-* `wait.heartbeat`: a crude mechanism to control the order of component activation. A heartbeat is ~10 seconds.
-* `role.priority`: each component must be assigned unique priority
-* `component.instances`: number of instances for this component type
-* `role.script`: the script path for the role *a temporary work-around as this will eventually be gleaned from metadata.xml*
-
-Sample:
-
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- },
- "components" : {
- "HBASE_MASTER" : {
- "yarn.role.priority" : "1",
- "yarn.component.instances" : "1"
- },
- "slider-appmaster" : {
- "jvm.heapsize" : "256M"
- },
- "HBASE_REGIONSERVER" : {
- "yarn.role.priority" : "2",
- "yarn.component.instances" : "1"
- }
- }
- }
-
-## Creating AppPackage
+## Scripting for AppPackage
Refer to [App Command Scripts](writing_app_command_scripts) for details on how to write scripts for a AppPackage. These scripts are in the package/script folder within the AppPackage. *Use the checked in samples for HBase/Storm/Accumulo as reference for script development.*
diff --git a/src/site/markdown/slider_specs/index.md b/src/site/markdown/slider_specs/index.md
index d960588..114950c 100644
--- a/src/site/markdown/slider_specs/index.md
+++ b/src/site/markdown/slider_specs/index.md
@@ -41,11 +41,13 @@
The entry points to leverage Slider are:
-- [Specifications for AppPackage](application_package.html)
-- [Application Needs](application_needs.html)
-- [Specifications for Application Definition](application_definition.html)
-- [Specifications for Configuration](application_configuration.html)
-- [Specification of Resources](resource_specification.html)
-- [Specifications InstanceConfiguration](application_instance_configuration.html)
+- [Application Needs](application_needs.html) What it takes to be deployable by Slider.
+- [Slider AppPackage](creating_app_definitions.html) Overview of how to create an Slider AppPackage.
+- [Specifications for AppPackage](application_package.html) Describes the structure of an AppPackage
+- [Specifications for Application Definition](application_definition.html) How to write metainfo.xml?
+- [Specifications for Configuration](application_configuration.html) How to create a template config file for an app?
+- [Specification of Resources](resource_specification.html) How to write a resource spec for an app?
+- [Specifications InstanceConfiguration](application_instance_configuration.html) How to write a template config for an app?
- [Guidelines for Clients and Client Applications](canonical_scenarios.html)
- [Documentation for "General Developer Guidelines"](app_developer_guideline.html)
+
\ No newline at end of file
diff --git a/src/site/markdown/slider_specs/resource_specification.md b/src/site/markdown/slider_specs/resource_specification.md
index aef2877..6d41f9e 100644
--- a/src/site/markdown/slider_specs/resource_specification.md
+++ b/src/site/markdown/slider_specs/resource_specification.md
@@ -18,34 +18,36 @@
#Resource Specification
Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application.
+An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider-appmaster".
+
Some parameters that can be specified for a component instance include:
-* yarn.memory: amount of memory requried for the component instance
-* env.MALLOC_ARENA_MAX: maximum number of memory pools used, arbitrary environment settings can be provided through format env.NAME_OF_THE_VARIABLE
-* component.instances: number of instances requested
-* component.name: name of the component
-* yarn.vcores: number of vcores requested
+* `yarn.memory`: amount of memory requried for the component instance
+* `yarn.vcores`: number of vcores requested
+* `yarn.role.priority`: each component must be assigned unique priority. Component with higher priority come up earlier than components with lower priority
+* `yarn.component.instances`: number of instances for this component type
-An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider".
+Sample:
- "components" : {
- "worker" : {
- "yarn.memory" : "768",
- "yarn.component.instances" : "1",
- "yarn.component.name" : "worker",
- "yarn.vcores" : "1"
+ {
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
},
- "slider" : {
- "yarn.memory" : "256",
- "yarn.component.instances" : "1",
- "yarn.component.name" : "slider",
- "yarn.vcores" : "1"
+ "global" : {
},
- "master" : {
- "yarn.memory" : "1024",
- "yarn.component.instances" : "1",
- "yarn.component.name" : "master",
- "yarn.vcores" : "1"
+ "components" : {
+ "HBASE_MASTER" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "1"
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1"
+ },
+ "slider-appmaster" : {
+ },
+ "HBASE_REGIONSERVER" : {
+ "yarn.role.priority" : "2",
+ "yarn.component.instances" : "1"
+ }
}
}
diff --git a/src/site/site.xml b/src/site/site.xml
index 9d5a598..d7f574d 100644
--- a/src/site/site.xml
+++ b/src/site/site.xml
@@ -1,16 +1,19 @@
<?xml version="1.0"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<project name="Slider ${project.version}">
<!--
diff --git a/src/test/clusters/configs.md b/src/test/clusters/configs.md
index 3fb7adb..a2ff2ff 100644
--- a/src/test/clusters/configs.md
+++ b/src/test/clusters/configs.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# test/configs
@@ -17,4 +20,4 @@
This package just contains some configurations for Hadoop clusters/miniclusters
used in some demo, test & jenkins situations, such as machine-local VM clusters.
-These can be generally ignored by others
\ No newline at end of file
+These can be generally ignored by others
diff --git a/src/test/clusters/local/README.md b/src/test/clusters/local/README.md
index 181def7..7493908 100644
--- a/src/test/clusters/local/README.md
+++ b/src/test/clusters/local/README.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# README
@@ -38,4 +41,4 @@
--manager localhost:9080 --filesystem hdfs://localhost:9000 --zkhosts localhost
-
\ No newline at end of file
+
diff --git a/src/test/clusters/local/capacity-scheduler.xml b/src/test/clusters/local/capacity-scheduler.xml
index 80a9fec..f0e7303 100644
--- a/src/test/clusters/local/capacity-scheduler.xml
+++ b/src/test/clusters/local/capacity-scheduler.xml
@@ -1,15 +1,18 @@
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<configuration>
diff --git a/src/test/clusters/local/core-site.xml b/src/test/clusters/local/core-site.xml
index 2ca4a77..5203d06 100644
--- a/src/test/clusters/local/core-site.xml
+++ b/src/test/clusters/local/core-site.xml
@@ -1,17 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
diff --git a/src/test/clusters/local/hadoop-env.sh b/src/test/clusters/local/hadoop-env.sh
index 5836a8a..3d0ff1c 100644
--- a/src/test/clusters/local/hadoop-env.sh
+++ b/src/test/clusters/local/hadoop-env.sh
@@ -1,5 +1,3 @@
-# Copyright 2011 The Apache Software Foundation
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
diff --git a/src/test/clusters/local/hadoop-policy.xml b/src/test/clusters/local/hadoop-policy.xml
index 491dbe7..16c25fc 100644
--- a/src/test/clusters/local/hadoop-policy.xml
+++ b/src/test/clusters/local/hadoop-policy.xml
@@ -1,9 +1,6 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
-
- Copyright 2011 The Apache Software Foundation
-
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
diff --git a/src/test/clusters/local/hbase/hbase-env.sh b/src/test/clusters/local/hbase/hbase-env.sh
index da53a27..ecc1318 100644
--- a/src/test/clusters/local/hbase/hbase-env.sh
+++ b/src/test/clusters/local/hbase/hbase-env.sh
@@ -1,23 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-#/**
-# * Copyright 2007 The Apache Software Foundation
-# *
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
# Set environment variables here.
diff --git a/src/test/clusters/local/hdfs-site.xml b/src/test/clusters/local/hdfs-site.xml
index f6d72b9..65a9e7f 100644
--- a/src/test/clusters/local/hdfs-site.xml
+++ b/src/test/clusters/local/hdfs-site.xml
@@ -1,17 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
@@ -34,4 +37,4 @@
</property>
-</configuration>
\ No newline at end of file
+</configuration>
diff --git a/src/test/clusters/local/httpfs-env.sh b/src/test/clusters/local/httpfs-env.sh
index 84c67b7..266f6fd 100644
--- a/src/test/clusters/local/httpfs-env.sh
+++ b/src/test/clusters/local/httpfs-env.sh
@@ -1,17 +1,20 @@
#!/bin/bash
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License. See accompanying LICENSE file.
-#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
# Set httpfs specific environment variables here.
diff --git a/src/test/clusters/local/httpfs-log4j.properties b/src/test/clusters/local/httpfs-log4j.properties
index 284a819..64149d9 100644
--- a/src/test/clusters/local/httpfs-log4j.properties
+++ b/src/test/clusters/local/httpfs-log4j.properties
@@ -1,16 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
-# limitations under the License. See accompanying LICENSE file.
-#
+# limitations under the License.
# If the Java System property 'httpfs.log.dir' is not defined at HttpFSServer start up time
# Setup sets its value to '${httpfs.home}/logs'
diff --git a/src/test/clusters/local/httpfs-site.xml b/src/test/clusters/local/httpfs-site.xml
index 4a718e1..de47830 100644
--- a/src/test/clusters/local/httpfs-site.xml
+++ b/src/test/clusters/local/httpfs-site.xml
@@ -1,16 +1,19 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<configuration>
diff --git a/src/test/clusters/local/log4j.properties b/src/test/clusters/local/log4j.properties
index 2ca667f..6a38e03 100644
--- a/src/test/clusters/local/log4j.properties
+++ b/src/test/clusters/local/log4j.properties
@@ -1,5 +1,3 @@
-# Copyright 2011 The Apache Software Foundation
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
diff --git a/src/test/clusters/local/mapred-site.xml b/src/test/clusters/local/mapred-site.xml
index 1be6975..3ebd138 100644
--- a/src/test/clusters/local/mapred-site.xml
+++ b/src/test/clusters/local/mapred-site.xml
@@ -1,17 +1,20 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
diff --git a/src/test/clusters/local/yarn-site.xml b/src/test/clusters/local/yarn-site.xml
index 553b632..24ee630 100644
--- a/src/test/clusters/local/yarn-site.xml
+++ b/src/test/clusters/local/yarn-site.xml
@@ -1,16 +1,19 @@
<?xml version="1.0"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<configuration>
diff --git a/src/test/clusters/offline/hbase/hbase-env.sh b/src/test/clusters/offline/hbase/hbase-env.sh
index da53a27..ecc1318 100644
--- a/src/test/clusters/offline/hbase/hbase-env.sh
+++ b/src/test/clusters/offline/hbase/hbase-env.sh
@@ -1,23 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-#/**
-# * Copyright 2007 The Apache Software Foundation
-# *
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
# Set environment variables here.
diff --git a/src/test/clusters/offline/readme.md b/src/test/clusters/offline/readme.md
index 23d9d80..f3c3e91 100644
--- a/src/test/clusters/offline/readme.md
+++ b/src/test/clusters/offline/readme.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# Offline configuration
@@ -19,4 +22,4 @@
that the assembly and scripts are valid.
-
\ No newline at end of file
+
diff --git a/src/test/clusters/offline/slider/slider-client.xml b/src/test/clusters/offline/slider/slider-client.xml
index 31d5812..a49dd29 100644
--- a/src/test/clusters/offline/slider/slider-client.xml
+++ b/src/test/clusters/offline/slider/slider-client.xml
@@ -1,18 +1,21 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- ~ Licensed under the Apache License, Version 2.0 (the "License");
- ~ you may not use this file except in compliance with the License.
- ~ You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License. See accompanying LICENSE file.
- -->
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<!--
Properties set here are picked up in the client.
They are not passed to the AM
diff --git a/src/test/clusters/remote/slider/slider-client.xml b/src/test/clusters/remote/slider/slider-client.xml
index dcf3ecb..5bd2edb 100644
--- a/src/test/clusters/remote/slider/slider-client.xml
+++ b/src/test/clusters/remote/slider/slider-client.xml
@@ -1,18 +1,21 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- ~ Licensed under the Apache License, Version 2.0 (the "License");
- ~ you may not use this file except in compliance with the License.
- ~ You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License. See accompanying LICENSE file.
- -->
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<!--
Properties set here are picked up in the client.
They are not passed to the AM
diff --git a/src/test/clusters/sandbox/README.md b/src/test/clusters/sandbox/README.md
index dae27e6..2b90908 100644
--- a/src/test/clusters/sandbox/README.md
+++ b/src/test/clusters/sandbox/README.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# README
@@ -46,4 +49,4 @@
--manager sandbox:8032 --filesystem hdfs://sandbox:9090
-
\ No newline at end of file
+
diff --git a/src/test/clusters/sandbox/conf/hadoop-policy.xml b/src/test/clusters/sandbox/conf/hadoop-policy.xml
index 491dbe7..16c25fc 100644
--- a/src/test/clusters/sandbox/conf/hadoop-policy.xml
+++ b/src/test/clusters/sandbox/conf/hadoop-policy.xml
@@ -1,9 +1,6 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
-
- Copyright 2011 The Apache Software Foundation
-
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
diff --git a/src/test/clusters/sandbox/conf/log4j.properties b/src/test/clusters/sandbox/conf/log4j.properties
index d1de1bf..f16dad3 100644
--- a/src/test/clusters/sandbox/conf/log4j.properties
+++ b/src/test/clusters/sandbox/conf/log4j.properties
@@ -1,5 +1,3 @@
-# Copyright 2011 The Apache Software Foundation
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
diff --git a/src/test/clusters/sandbox/conf/mapred-site.xml.template b/src/test/clusters/sandbox/conf/mapred-site.xml.template
index 761c352..77dd7ef 100644
--- a/src/test/clusters/sandbox/conf/mapred-site.xml.template
+++ b/src/test/clusters/sandbox/conf/mapred-site.xml.template
@@ -1,17 +1,20 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
diff --git a/src/test/clusters/sandbox/hbase/hbase-env.sh b/src/test/clusters/sandbox/hbase/hbase-env.sh
index da53a27..ecc1318 100644
--- a/src/test/clusters/sandbox/hbase/hbase-env.sh
+++ b/src/test/clusters/sandbox/hbase/hbase-env.sh
@@ -1,23 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-#/**
-# * Copyright 2007 The Apache Software Foundation
-# *
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
# Set environment variables here.
diff --git a/src/test/clusters/sandbox/operations.md b/src/test/clusters/sandbox/operations.md
index 02d2979..0ff5a3a 100644
--- a/src/test/clusters/sandbox/operations.md
+++ b/src/test/clusters/sandbox/operations.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# Just some operations for manual runs against steve's secure VM
@@ -181,4 +184,4 @@
bin/slider destroy cl1
-
\ No newline at end of file
+
diff --git a/src/test/clusters/sandbox/slider/slider-client.xml b/src/test/clusters/sandbox/slider/slider-client.xml
index 3a1658b..f15f71d 100644
--- a/src/test/clusters/sandbox/slider/slider-client.xml
+++ b/src/test/clusters/sandbox/slider/slider-client.xml
@@ -1,18 +1,21 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- ~ Licensed under the Apache License, Version 2.0 (the "License");
- ~ you may not use this file except in compliance with the License.
- ~ You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License. See accompanying LICENSE file.
- -->
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<!--
Properties set here are picked up in the client.
They are not passed to the AM
diff --git a/src/test/clusters/script.md b/src/test/clusters/script.md
index 7dc9054..634658a 100644
--- a/src/test/clusters/script.md
+++ b/src/test/clusters/script.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# script
@@ -51,4 +54,4 @@
-
\ No newline at end of file
+
diff --git a/src/test/clusters/ubuntu-secure/README.md b/src/test/clusters/ubuntu-secure/README.md
index bdf6be3..e730c08 100644
--- a/src/test/clusters/ubuntu-secure/README.md
+++ b/src/test/clusters/ubuntu-secure/README.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# README
@@ -42,4 +45,4 @@
--manager ubuntu:8032 --filesystem hdfs://ubuntu:9090 --zkhosts localhost
-
\ No newline at end of file
+
diff --git a/src/test/clusters/ubuntu-secure/capacity-scheduler.xml b/src/test/clusters/ubuntu-secure/capacity-scheduler.xml
index 80a9fec..f0e7303 100644
--- a/src/test/clusters/ubuntu-secure/capacity-scheduler.xml
+++ b/src/test/clusters/ubuntu-secure/capacity-scheduler.xml
@@ -1,15 +1,18 @@
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<configuration>
diff --git a/src/test/clusters/ubuntu-secure/core-site.xml b/src/test/clusters/ubuntu-secure/core-site.xml
index 2c8ac92..1c8e8c2 100644
--- a/src/test/clusters/ubuntu-secure/core-site.xml
+++ b/src/test/clusters/ubuntu-secure/core-site.xml
@@ -1,17 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
diff --git a/src/test/clusters/ubuntu-secure/hadoop-env.sh b/src/test/clusters/ubuntu-secure/hadoop-env.sh
index b852a02..132c648 100644
--- a/src/test/clusters/ubuntu-secure/hadoop-env.sh
+++ b/src/test/clusters/ubuntu-secure/hadoop-env.sh
@@ -1,5 +1,3 @@
-# Copyright 2011 The Apache Software Foundation
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
diff --git a/src/test/clusters/ubuntu-secure/hadoop-policy.xml b/src/test/clusters/ubuntu-secure/hadoop-policy.xml
index 491dbe7..16c25fc 100644
--- a/src/test/clusters/ubuntu-secure/hadoop-policy.xml
+++ b/src/test/clusters/ubuntu-secure/hadoop-policy.xml
@@ -1,9 +1,6 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
-
- Copyright 2011 The Apache Software Foundation
-
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
diff --git a/src/test/clusters/ubuntu-secure/hbase/hbase-env.sh b/src/test/clusters/ubuntu-secure/hbase/hbase-env.sh
index da53a27..ecc1318 100644
--- a/src/test/clusters/ubuntu-secure/hbase/hbase-env.sh
+++ b/src/test/clusters/ubuntu-secure/hbase/hbase-env.sh
@@ -1,23 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-#/**
-# * Copyright 2007 The Apache Software Foundation
-# *
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
# Set environment variables here.
diff --git a/src/test/clusters/ubuntu-secure/hdfs-site.xml b/src/test/clusters/ubuntu-secure/hdfs-site.xml
index 2878c45..cd8070c 100644
--- a/src/test/clusters/ubuntu-secure/hdfs-site.xml
+++ b/src/test/clusters/ubuntu-secure/hdfs-site.xml
@@ -1,17 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
diff --git a/src/test/clusters/ubuntu-secure/httpfs-env.sh b/src/test/clusters/ubuntu-secure/httpfs-env.sh
index 84c67b7..01a79f4 100644
--- a/src/test/clusters/ubuntu-secure/httpfs-env.sh
+++ b/src/test/clusters/ubuntu-secure/httpfs-env.sh
@@ -1,16 +1,20 @@
#!/bin/bash
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License. See accompanying LICENSE file.
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
#
# Set httpfs specific environment variables here.
diff --git a/src/test/clusters/ubuntu-secure/httpfs-log4j.properties b/src/test/clusters/ubuntu-secure/httpfs-log4j.properties
index 284a819..64149d9 100644
--- a/src/test/clusters/ubuntu-secure/httpfs-log4j.properties
+++ b/src/test/clusters/ubuntu-secure/httpfs-log4j.properties
@@ -1,16 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
-# limitations under the License. See accompanying LICENSE file.
-#
+# limitations under the License.
# If the Java System property 'httpfs.log.dir' is not defined at HttpFSServer start up time
# Setup sets its value to '${httpfs.home}/logs'
diff --git a/src/test/clusters/ubuntu-secure/httpfs-site.xml b/src/test/clusters/ubuntu-secure/httpfs-site.xml
index 4a718e1..de47830 100644
--- a/src/test/clusters/ubuntu-secure/httpfs-site.xml
+++ b/src/test/clusters/ubuntu-secure/httpfs-site.xml
@@ -1,16 +1,19 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<configuration>
diff --git a/src/test/clusters/ubuntu-secure/log4j.properties b/src/test/clusters/ubuntu-secure/log4j.properties
index 2730884..aa172b8 100644
--- a/src/test/clusters/ubuntu-secure/log4j.properties
+++ b/src/test/clusters/ubuntu-secure/log4j.properties
@@ -1,5 +1,3 @@
-# Copyright 2011 The Apache Software Foundation
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
diff --git a/src/test/clusters/ubuntu-secure/mapred-site.xml b/src/test/clusters/ubuntu-secure/mapred-site.xml
index b4471d7..4b4b52b 100644
--- a/src/test/clusters/ubuntu-secure/mapred-site.xml
+++ b/src/test/clusters/ubuntu-secure/mapred-site.xml
@@ -1,17 +1,20 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<!-- Put site-specific property overrides in this file. -->
diff --git a/src/test/clusters/ubuntu-secure/operations.md b/src/test/clusters/ubuntu-secure/operations.md
index 8180639..cb14dfe 100644
--- a/src/test/clusters/ubuntu-secure/operations.md
+++ b/src/test/clusters/ubuntu-secure/operations.md
@@ -1,15 +1,18 @@
<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
# Just some operations for manual runs against a kerberized Ubunt VM
@@ -34,7 +37,7 @@
yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
-
+ ~/zookeeper/bin/zkServer.sh stop
export SLIDER_JVM_OPTS="-Djava.security.krb5.realm=COTHAM -Djava.security.krb5.kdc=ubuntu -Djava.net.preferIPv4Stack=true"
diff --git a/src/test/clusters/ubuntu-secure/slider/slider-client.xml b/src/test/clusters/ubuntu-secure/slider/slider-client.xml
index 062bc24..3ad4b67 100644
--- a/src/test/clusters/ubuntu-secure/slider/slider-client.xml
+++ b/src/test/clusters/ubuntu-secure/slider/slider-client.xml
@@ -1,18 +1,21 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
- ~ Licensed under the Apache License, Version 2.0 (the "License");
- ~ you may not use this file except in compliance with the License.
- ~ You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License. See accompanying LICENSE file.
- -->
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<!--
Properties set here are picked up in the client.
diff --git a/src/test/clusters/ubuntu-secure/yarn-site.xml b/src/test/clusters/ubuntu-secure/yarn-site.xml
index 8cac9ef..4e36fdb 100644
--- a/src/test/clusters/ubuntu-secure/yarn-site.xml
+++ b/src/test/clusters/ubuntu-secure/yarn-site.xml
@@ -1,16 +1,19 @@
<?xml version="1.0"?>
<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
-->
<configuration>
