Merge branch 'cassandra-3.0' into cassandra-3.3
diff --git a/CHANGES.txt b/CHANGES.txt
index 6e313c3..2b67b90 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,8 @@
-3.0.3
+3.3
+ * Avoid infinite loop if owned range is smaller than number of
+ data dirs (CASSANDRA-11034)
+ * Avoid bootstrap hanging when existing nodes have no data to stream (CASSANDRA-11010)
+Merged from 3.0:
* Remove double initialization of newly added tables (CASSANDRA-11027)
* Filter keys searcher results by target range (CASSANDRA-11104)
* Fix deserialization of legacy read commands (CASSANDRA-11087)
@@ -19,25 +23,17 @@
* Remove checksum files after replaying hints (CASSANDRA-10947)
* Support passing base table metadata to custom 2i validation (CASSANDRA-10924)
* Ensure stale index entries are purged during reads (CASSANDRA-11013)
+ * (cqlsh) Also apply --connect-timeout to control connection
+ timeout (CASSANDRA-10959)
* Fix AssertionError when removing from list using UPDATE (CASSANDRA-10954)
* Fix UnsupportedOperationException when reading old sstable with range
tombstone (CASSANDRA-10743)
* MV should use the maximum timestamp of the primary key (CASSANDRA-10910)
* Fix potential assertion error during compaction (CASSANDRA-10944)
- * Fix counting of received sstables in streaming (CASSANDRA-10949)
- * Implement hints compression (CASSANDRA-9428)
- * Fix potential assertion error when reading static columns (CASSANDRA-10903)
- * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711)
- * Avoid building PartitionUpdate in toString (CASSANDRA-10897)
- * Reduce heap spent when receiving many SSTables (CASSANDRA-10797)
- * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873)
- * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653)
- * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837)
- * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806)
Merged from 2.2:
-2.2.5
* maxPurgeableTimestamp needs to check memtables too (CASSANDRA-9949)
* Apply change to compaction throughput in real time (CASSANDRA-10025)
+ * (cqlsh) encode input correctly when saving history
* Fix potential NPE on ORDER BY queries with IN (CASSANDRA-10955)
* Start L0 STCS-compactions even if there is a L0 -> L1 compaction
going (CASSANDRA-10979)
@@ -45,22 +41,11 @@
* Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980)
* Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465)
* Fix error streaming section more than 2GB (CASSANDRA-10961)
- * (cqlsh) Also apply --connect-timeout to control connection
- timeout (CASSANDRA-10959)
* Histogram buckets exposed in jmx are sorted incorrectly (CASSANDRA-10975)
* Enable GC logging by default (CASSANDRA-10140)
* Optimize pending range computation (CASSANDRA-9258)
* Skip commit log and saved cache directories in SSTable version startup check (CASSANDRA-10902)
* drop/alter user should be case sensitive (CASSANDRA-10817)
- * jemalloc detection fails due to quoting issues in regexv (CASSANDRA-10946)
- * (cqlsh) show correct column names for empty result sets (CASSANDRA-9813)
- * Add new types to Stress (CASSANDRA-9556)
- * Add property to allow listening on broadcast interface (CASSANDRA-9748)
- * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
- * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
- * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
- * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
- * (cqlsh) encode input correctly when saving history
Merged from 2.1:
* test_bulk_round_trip_blogposts is failing occasionally (CASSANDRA-10938)
* Fix isJoined return true only after becoming cluster member (CASANDRA-11007)
@@ -76,6 +61,54 @@
* Retry sending gossip syn multiple times during shadow round (CASSANDRA-8072)
* Fix pending range calculation during moves (CASSANDRA-10887)
* Sane default (200Mbps) for inter-DC streaming througput (CASSANDRA-8708)
+
+
+3.2
+ * Make sure tokens don't exist in several data directories (CASSANDRA-6696)
+ * Add requireAuthorization method to IAuthorizer (CASSANDRA-10852)
+ * Move static JVM options to conf/jvm.options file (CASSANDRA-10494)
+ * Fix CassandraVersion to accept x.y version string (CASSANDRA-10931)
+ * Add forceUserDefinedCleanup to allow more flexible cleanup (CASSANDRA-10708)
+ * (cqlsh) allow setting TTL with COPY (CASSANDRA-9494)
+ * Fix counting of received sstables in streaming (CASSANDRA-10949)
+ * Implement hints compression (CASSANDRA-9428)
+ * Fix potential assertion error when reading static columns (CASSANDRA-10903)
+ * Fix EstimatedHistogram creation in nodetool tablehistograms (CASSANDRA-10859)
+ * Establish bootstrap stream sessions sequentially (CASSANDRA-6992)
+ * Sort compactionhistory output by timestamp (CASSANDRA-10464)
+ * More efficient BTree removal (CASSANDRA-9991)
+ * Make tablehistograms accept the same syntax as tablestats (CASSANDRA-10149)
+ * Group pending compactions based on table (CASSANDRA-10718)
+ * Add compressor name in sstablemetadata output (CASSANDRA-9879)
+ * Fix type casting for counter columns (CASSANDRA-10824)
+ * Prevent running Cassandra as root (CASSANDRA-8142)
+ * bound maximum in-flight commit log replay mutation bytes to 64 megabytes (CASSANDRA-8639)
+ * Normalize all scripts (CASSANDRA-10679)
+ * Make compression ratio much more accurate (CASSANDRA-10225)
+ * Optimize building of Clustering object when only one is created (CASSANDRA-10409)
+ * Make index building pluggable (CASSANDRA-10681)
+ * Add sstable flush observer (CASSANDRA-10678)
+ * Improve NTS endpoints calculation (CASSANDRA-10200)
+ * Improve performance of the folderSize function (CASSANDRA-10677)
+ * Add support for type casting in selection clause (CASSANDRA-10310)
+ * Added graphing option to cassandra-stress (CASSANDRA-7918)
+ * Abort in-progress queries that time out (CASSANDRA-7392)
+ * Add transparent data encryption core classes (CASSANDRA-9945)
+Merged from 3.0:
+ * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
+ * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711)
+ * Avoid building PartitionUpdate in toString (CASSANDRA-10897)
+ * Reduce heap spent when receiving many SSTables (CASSANDRA-10797)
+ * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873)
+ * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653)
+ * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837)
+ * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806)
+Merged from 2.2:
+ * jemalloc detection fails due to quoting issues in regexv (CASSANDRA-10946)
+ * (cqlsh) show correct column names for empty result sets (CASSANDRA-9813)
+ * Add new types to Stress (CASSANDRA-9556)
+ * Add property to allow listening on broadcast interface (CASSANDRA-9748)
+Merged from 2.1:
* Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
* Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
* cqlsh fails to decode utf-8 characters for text typed columns (CASSANDRA-10875)
@@ -88,12 +121,14 @@
* Allow cancellation of index summary redistribution (CASSANDRA-8805)
-3.0.2
- * Fix upgrade data loss due to range tombstone deleting more data than then should
- (CASSANDRA-10822)
+3.1.1
+Merged from 3.0:
+ * Fix upgrade data loss due to range tombstone deleting more data than then should
+ (CASSANDRA-10822)
-3.0.1
+3.1
+Merged from 3.0:
* Avoid MV race during node decommission (CASSANDRA-10674)
* Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
* Handle single-column deletions correction in materialized views
diff --git a/NEWS.txt b/NEWS.txt
index 89fc4a7..92f3d3e 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -13,21 +13,42 @@
'sstableloader' tool. You can upgrade the file format of your snapshots
using the provided 'sstableupgrade' tool.
-3.0.3
-=====
+3.3
+===
+
+Upgrading
+---------
+ - Nothing specific to this release, but please see 3.0 if you are upgrading
+ from a previous version.
+
+3.2
+===
New features
------------
+ - We now make sure that a token does not exist in several data directories. This
+ means that we run one compaction strategy per data_file_directory and we use
+ one thread per directory to flush. Use nodetool relocatesstables to make sure your
+ tokens are in the correct place, or just wait and compaction will handle it. See
+ CASSANDRA-6696 for more details.
+ - bound maximum in-flight commit log replay mutation bytes to 64 megabytes
+ tunable via cassandra.commitlog_max_outstanding_replay_bytes
+ - Support for type casting has been added to the selection clause.
- Hinted handoff now supports compression. Reference cassandra.yaml:hints_compression.
Note: hints compression is currently disabled by default.
Upgrading
---------
- - Nothing specific to 3.0.3 but please see previous versions upgrading section,
- especially if you are upgrading from 2.2.
+ - The compression ratio metrics computation has been modified to be more accurate.
+ - Running Cassandra as root is prevented by default.
+ - JVM options are moved from cassandra-env.(sh|ps1) to jvm.options file
+
+Deprecation
+-----------
+ - The Thrift API is deprecated and will be removed in Cassandra 4.0.
-3.0.1
+3.1
=====
Upgrading
diff --git a/bin/cassandra b/bin/cassandra
index c15c1ab..98de893 100755
--- a/bin/cassandra
+++ b/bin/cassandra
@@ -211,7 +211,7 @@
}
# Parse any command line options.
-args=`getopt vfhp:bD:H:E: "$@"`
+args=`getopt vRfhp:bD:H:E: "$@"`
eval set -- "$args"
classname="org.apache.cassandra.service.CassandraDaemon"
@@ -234,6 +234,10 @@
"$JAVA" -cp "$CLASSPATH" org.apache.cassandra.tools.GetVersion
exit 0
;;
+ -R)
+ allow_root="yes"
+ shift
+ ;;
-D)
properties="$properties -D$2"
shift 2
@@ -248,15 +252,27 @@
;;
--)
shift
+ if [ "x$*" != "x" ] ; then
+ echo "Error parsing arguments! Unknown argument \"$*\"" >&2
+ exit 1
+ fi
break
;;
*)
- echo "Error parsing arguments!" >&2
+ echo "Error parsing arguments! Unknown argument \"$1\"" >&2
exit 1
;;
esac
done
+if [ "x$allow_root" != "xyes" ] ; then
+ if [ "`id -u`" = "1" ] || [ "`id -g`" = "0" ] ; then
+ echo "Running Cassandra as root user or group is not recommended - please start Cassandra using a different system user."
+ echo "If you really want to force running Cassandra as root, use -R command line option."
+ exit 1
+ fi
+fi
+
# see CASSANDRA-7254
"$JAVA" -cp "$CLASSPATH" $JVM_OPTS 2>&1 | grep -q 'Error: Exception thrown by the agent : java.lang.NullPointerException'
if [ $? -ne "1" ]; then
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 74e7103..adc0bbb 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -68,7 +68,7 @@
CQL_LIB_PREFIX = 'cassandra-driver-internal-only-'
CASSANDRA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), '..')
-CASSANDRA_CQL_HTML_FALLBACK = 'https://cassandra.apache.org/doc/cql3/CQL-3.0.html'
+CASSANDRA_CQL_HTML_FALLBACK = 'https://cassandra.apache.org/doc/cql3/CQL-3.2.html'
# default location of local CQL.html
if os.path.exists(CASSANDRA_PATH + '/doc/cql3/CQL.html'):
@@ -464,7 +464,7 @@
'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 'THOUSANDSSEP', 'BOOLSTYLE',
'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE', 'MAXROWS',
- 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 'MAXINSERTERRORS', 'ERRFILE']
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 'MAXINSERTERRORS', 'ERRFILE', 'TTL']
COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
@@ -1817,6 +1817,7 @@
MAXINSERTERRORS=-1 - the maximum global number of insert errors, -1 means no maximum
ERRFILE='' - a file where to store all rows that could not be imported, by default this is
import_ks_table.err where <ks> is your keyspace and <table> is your table name.
+ TTL=3600 - the time to live in seconds, by default data will not expire
Available COPY TO options and defaults:
diff --git a/bin/nodetool b/bin/nodetool
index d211459..0ea078f 100755
--- a/bin/nodetool
+++ b/bin/nodetool
@@ -1,4 +1,5 @@
#!/bin/sh
+
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -22,11 +23,11 @@
fi
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
- for include in "`dirname "$0"`/cassandra.in.sh" \
- "$HOME/.cassandra.in.sh" \
- /usr/share/cassandra/cassandra.in.sh \
+ for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
- /opt/cassandra/cassandra.in.sh; do
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
break
@@ -43,6 +44,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CASSANDRA_CONF" -o -z "$CLASSPATH" ]; then
echo "You must set the CASSANDRA_CONF and CLASSPATH vars" >&2
exit 1
@@ -50,7 +56,9 @@
# Run cassandra-env.sh to pick up JMX_PORT
if [ -f "$CASSANDRA_CONF/cassandra-env.sh" ]; then
+ JVM_OPTS_SAVE=$JVM_OPTS
. "$CASSANDRA_CONF/cassandra-env.sh"
+ JVM_OPTS=$JVM_OPTS_SAVE
fi
# JMX Port passed via cmd line args (-p 9999 / --port 9999 / --port=9999)
@@ -90,20 +98,14 @@
shift
done
-# Special-case path variables.
-case "`uname`" in
- CYGWIN*)
- CLASSPATH="`cygpath -p -w "$CLASSPATH"`"
- CASSANDRA_CONF="`cygpath -p -w "$CASSANDRA_CONF"`"
- ;;
-esac
+if [ "x$MAX_HEAP_SIZE" = "x" ]; then
+ MAX_HEAP_SIZE="128m"
+fi
-"$JAVA" $JAVA_AGENT -cp "$CLASSPATH" \
- -Xmx128m \
- -Dcassandra.storagedir="$cassandra_storagedir" \
- -Dlogback.configurationFile=logback-tools.xml \
- -Dstorage-config="$CASSANDRA_CONF" \
- $JVM_ARGS \
- org.apache.cassandra.tools.NodeTool -p $JMX_PORT $ARGS
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
+ -Dcassandra.storagedir="$cassandra_storagedir" \
+ -Dlogback.configurationFile=logback-tools.xml \
+ $JVM_ARGS \
+ org.apache.cassandra.tools.NodeTool -p $JMX_PORT $ARGS
# vi:ai sw=4 ts=4 tw=0 et
diff --git a/bin/sstableloader b/bin/sstableloader
index 67829b8..b6a6d83 100755
--- a/bin/sstableloader
+++ b/bin/sstableloader
@@ -17,11 +17,11 @@
# limitations under the License.
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
- for include in "`dirname "$0"`/cassandra.in.sh" \
- "$HOME/.cassandra.in.sh" \
- /usr/share/cassandra/cassandra.in.sh \
+ for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
- /opt/cassandra/cassandra.in.sh; do
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
break
@@ -38,6 +38,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
diff --git a/bin/sstablescrub b/bin/sstablescrub
index ea784ef..58bd21b 100755
--- a/bin/sstablescrub
+++ b/bin/sstablescrub
@@ -20,7 +20,7 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
"`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
@@ -38,6 +38,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
diff --git a/bin/sstableupgrade b/bin/sstableupgrade
index 91fa422..a722cdc 100755
--- a/bin/sstableupgrade
+++ b/bin/sstableupgrade
@@ -20,22 +20,27 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
- `dirname $0`/cassandra.in.sh; do
- if [ -r $include ]; then
- . $include
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
+ if [ -r "$include" ]; then
+ . "$include"
break
fi
done
-elif [ -r $CASSANDRA_INCLUDE ]; then
- . $CASSANDRA_INCLUDE
+elif [ -r "$CASSANDRA_INCLUDE" ]; then
+ . "$CASSANDRA_INCLUDE"
fi
# Use JAVA_HOME if set, otherwise look for java in PATH
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
+if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
else
- JAVA=`which java`
+ JAVA="`which java`"
+fi
+
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
fi
if [ -z "$CLASSPATH" ]; then
@@ -53,4 +58,3 @@
org.apache.cassandra.tools.StandaloneUpgrader "$@"
# vi:ai sw=4 ts=4 tw=0 et
-
diff --git a/bin/sstableutil b/bin/sstableutil
index 2860729..9f07785 100755
--- a/bin/sstableutil
+++ b/bin/sstableutil
@@ -20,7 +20,7 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
"`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
@@ -38,6 +38,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
diff --git a/bin/sstableverify b/bin/sstableverify
index c3e40c7..892750b 100755
--- a/bin/sstableverify
+++ b/bin/sstableverify
@@ -20,7 +20,7 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
"`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
@@ -38,6 +38,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
@@ -47,7 +52,7 @@
MAX_HEAP_SIZE="256M"
fi
-"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" -Xmx$MAX_HEAP_SIZE \
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.StandaloneVerifier "$@"
diff --git a/build.xml b/build.xml
index f9ff00d..c5d1b5f 100644
--- a/build.xml
+++ b/build.xml
@@ -25,7 +25,7 @@
<property name="debuglevel" value="source,lines,vars"/>
<!-- default version and SCM information -->
- <property name="base.version" value="3.0.3"/>
+ <property name="base.version" value="3.3"/>
<property name="scm.connection" value="scm:git://git.apache.org/cassandra.git"/>
<property name="scm.developerConnection" value="scm:git://git.apache.org/cassandra.git"/>
<property name="scm.url" value="http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/>
@@ -783,6 +783,9 @@
</path>
</classpath>
</javac>
+ <copy todir="${stress.build.classes}">
+ <fileset dir="${stress.build.src}/resources" />
+ </copy>
</target>
<target name="_write-poms" depends="maven-declare-dependencies">
diff --git a/conf/cassandra-env.ps1 b/conf/cassandra-env.ps1
index 5eefb04..2a9acce 100644
--- a/conf/cassandra-env.ps1
+++ b/conf/cassandra-env.ps1
@@ -419,10 +419,6 @@
exit
}
- # enable assertions. disabling this in production will give a modest
- # performance benefit (around 5%).
- $env:JVM_OPTS = "$env:JVM_OPTS -ea"
-
# Specifies the default port over which Cassandra will be available for
# JMX connections.
$JMX_PORT="7199"
@@ -430,50 +426,11 @@
# store in env to check if it's avail in verification
$env:JMX_PORT=$JMX_PORT
- # enable thread priorities, primarily so we can give periodic tasks
- # a lower priority to avoid interfering with client workload
- $env:JVM_OPTS="$env:JVM_OPTS -XX:+UseThreadPriorities"
- # allows lowering thread priority without being root on linux - probably
- # not necessary on Windows but doesn't harm anything.
- # see http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workar
- $env:JVM_OPTS="$env:JVM_OPTS -XX:ThreadPriorityPolicy=42"
-
- $env:JVM_OPTS="$env:JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
-
- # Per-thread stack size.
- $env:JVM_OPTS="$env:JVM_OPTS -Xss256k"
-
- # Larger interned string table, for gossip's benefit (CASSANDRA-6410)
- $env:JVM_OPTS="$env:JVM_OPTS -XX:StringTableSize=1000003"
-
- # Make sure all memory is faulted and zeroed on startup.
- # This helps prevent soft faults in containers and makes
- # transparent hugepage allocation more effective.
- #$env:JVM_OPTS="$env:JVM_OPTS -XX:+AlwaysPreTouch"
-
- # Biased locking does not benefit Cassandra.
- $env:JVM_OPTS="$env:JVM_OPTS -XX:-UseBiasedLocking"
-
- # Enable thread-local allocation blocks and allow the JVM to automatically
- # resize them at runtime.
- $env:JVM_OPTS="$env:JVM_OPTS -XX:+UseTLAB -XX:+ResizeTLAB"
-
- # http://www.evanjones.ca/jvm-mmap-pause.html
- $env:JVM_OPTS="$env:JVM_OPTS -XX:+PerfDisableSharedMem"
-
# Configure the following for JEMallocAllocator and if jemalloc is not available in the system
# library path.
# set LD_LIBRARY_PATH=<JEMALLOC_HOME>/lib/
# $env:JVM_OPTS="$env:JVM_OPTS -Djava.library.path=<JEMALLOC_HOME>/lib/"
- # uncomment to have Cassandra JVM listen for remote debuggers/profilers on port 1414
- # $env:JVM_OPTS="$env:JVM_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414"
-
- # Prefer binding to IPv4 network intefaces (when net.ipv6.bindv6only=1). See
- # http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6342561 (short version:
- # comment out this entry to enable IPv6 support).
- $env:JVM_OPTS="$env:JVM_OPTS -Djava.net.preferIPv4Stack=true"
-
# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
@@ -495,6 +452,4 @@
$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC"
$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"
-
- #$env:JVM_OPTS="$env:JVM_OPTS -XX:+UnlockCommercialFeatures -XX:+FlightRecorder"
}
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index 83fe4c5..6e1910c 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -203,62 +203,17 @@
JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
fi
-# enable assertions. disabling this in production will give a modest
-# performance benefit (around 5%).
-JVM_OPTS="$JVM_OPTS -ea"
-
-# Per-thread stack size.
-JVM_OPTS="$JVM_OPTS -Xss256k"
-
-# Make sure all memory is faulted and zeroed on startup.
-# This helps prevent soft faults in containers and makes
-# transparent hugepage allocation more effective.
-JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
-
-# Biased locking does not benefit Cassandra.
-JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
-
-# Larger interned string table, for gossip's benefit (CASSANDRA-6410)
-JVM_OPTS="$JVM_OPTS -XX:StringTableSize=1000003"
-
-# Enable thread-local allocation blocks and allow the JVM to automatically
-# resize them at runtime.
-JVM_OPTS="$JVM_OPTS -XX:+UseTLAB -XX:+ResizeTLAB"
-
-# http://www.evanjones.ca/jvm-mmap-pause.html
-JVM_OPTS="$JVM_OPTS -XX:+PerfDisableSharedMem"
-
# provides hints to the JIT compiler
JVM_OPTS="$JVM_OPTS -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler"
# add the jamm javaagent
JVM_OPTS="$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.3.0.jar"
-# enable thread priorities, primarily so we can give periodic tasks
-# a lower priority to avoid interfering with client workload
-JVM_OPTS="$JVM_OPTS -XX:+UseThreadPriorities"
-# allows lowering thread priority without being root. see
-# http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html
-JVM_OPTS="$JVM_OPTS -XX:ThreadPriorityPolicy=42"
-
# set jvm HeapDumpPath with CASSANDRA_HEAPDUMP_DIR
-JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
if [ "x$CASSANDRA_HEAPDUMP_DIR" != "x" ]; then
JVM_OPTS="$JVM_OPTS -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
fi
-# uncomment to have Cassandra JVM listen for remote debuggers/profilers on port 1414
-# JVM_OPTS="$JVM_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414"
-
-# uncomment to have Cassandra JVM log internal method compilation (developers only)
-# JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation"
-# JVM_OPTS="$JVM_OPTS -XX:+UnlockCommercialFeatures -XX:+FlightRecorder"
-
-# Prefer binding to IPv4 network intefaces (when net.ipv6.bindv6only=1). See
-# http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6342561 (short version:
-# comment out this entry to enable IPv6 support).
-JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
-
# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index 76aa4dd..779575c 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -413,14 +413,13 @@
# This sets the amount of memtable flush writer threads. These will
# be blocked by disk io, and each one will hold a memtable in memory
-# while blocked.
+# while blocked.
#
-# memtable_flush_writers defaults to the smaller of (number of disks,
-# number of cores), with a minimum of 2 and a maximum of 8.
-#
-# If your data directories are backed by SSD, you should increase this
-# to the number of cores.
-#memtable_flush_writers: 8
+# memtable_flush_writers defaults to one per data_file_directory.
+#
+# If your data directories are backed by SSD, you can increase this, but
+# avoid having memtable_flush_writers * data_file_directories > number of cores
+#memtable_flush_writers: 1
# A fixed memory pool size in MB for for SSTable index summaries. If left
# empty, this will default to 5% of the heap size. If the memory usage of
diff --git a/conf/jvm.options b/conf/jvm.options
index a7b3bd8..4aec619 100644
--- a/conf/jvm.options
+++ b/conf/jvm.options
@@ -8,6 +8,133 @@
# - dynamic flags will be appended to these on cassandra-env #
###########################################################################
+######################
+# STARTUP PARAMETERS #
+######################
+
+# Uncomment any of the following properties to enable specific startup parameters
+
+# In a multi-instance deployment, multiple Cassandra instances will independently assume that all
+# CPU processors are available to it. This setting allows you to specify a smaller set of processors
+# and perhaps have affinity.
+#-Dcassandra.available_processors=number_of_processors
+
+# The directory location of the cassandra.yaml file.
+#-Dcassandra.config=directory
+
+# Sets the initial partitioner token for a node the first time the node is started.
+#-Dcassandra.initial_token=token
+
+# Set to false to start Cassandra on a node but not have the node join the cluster.
+#-Dcassandra.join_ring=true|false
+
+# Set to false to clear all gossip state for the node on restart. Use when you have changed node
+# information in cassandra.yaml (such as listen_address).
+#-Dcassandra.load_ring_state=true|false
+
+# Enable pluggable metrics reporter. See Pluggable metrics reporting in Cassandra 2.0.2.
+#-Dcassandra.metricsReporterConfigFile=file
+
+# Set the port on which the CQL native transport listens for clients. (Default: 9042)
+#-Dcassandra.native_transport_port=port
+
+# Overrides the partitioner. (Default: org.apache.cassandra.dht.Murmur3Partitioner)
+#-Dcassandra.partitioner=partitioner
+
+# To replace a node that has died, restart a new node in its place specifying the address of the
+# dead node. The new node must not have any data in its data directory, that is, it must be in the
+# same state as before bootstrapping.
+#-Dcassandra.replace_address=listen_address or broadcast_address of dead node
+
+# Allow restoring specific tables from an archived commit log.
+#-Dcassandra.replayList=table
+
+# Allows overriding of the default RING_DELAY (1000ms), which is the amount of time a node waits
+# before joining the ring.
+#-Dcassandra.ring_delay_ms=ms
+
+# Set the port for the Thrift RPC service, which is used for client connections. (Default: 9160)
+#-Dcassandra.rpc_port=port
+
+# Set the SSL port for encrypted communication. (Default: 7001)
+#-Dcassandra.ssl_storage_port=port
+
+# Enable or disable the native transport server. See start_native_transport in cassandra.yaml.
+# cassandra.start_native_transport=true|false
+
+# Enable or disable the Thrift RPC server. (Default: true)
+#-Dcassandra.start_rpc=true/false
+
+# Set the port for inter-node communication. (Default: 7000)
+#-Dcassandra.storage_port=port
+
+# Set the default location for the trigger JARs. (Default: conf/triggers)
+#-Dcassandra.triggers_dir=directory
+
+# For testing new compaction and compression strategies. It allows you to experiment with different
+# strategies and benchmark write performance differences without affecting the production workload.
+#-Dcassandra.write_survey=true
+
+########################
+# GENERAL JVM SETTINGS #
+########################
+
+# enable assertions. disabling this in production will give a modest
+# performance benefit (around 5%).
+-ea
+
+# enable thread priorities, primarily so we can give periodic tasks
+# a lower priority to avoid interfering with client workload
+-XX:+UseThreadPriorities
+
+# allows lowering thread priority without being root on linux - probably
+# not necessary on Windows but doesn't harm anything.
+# see http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workar
+-XX:ThreadPriorityPolicy=42
+
+# Enable heap-dump if there's an OOM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# Per-thread stack size.
+-Xss256k
+
+# Larger interned string table, for gossip's benefit (CASSANDRA-6410)
+-XX:StringTableSize=1000003
+
+# Make sure all memory is faulted and zeroed on startup.
+# This helps prevent soft faults in containers and makes
+# transparent hugepage allocation more effective.
+-XX:+AlwaysPreTouch
+
+# Disable biased locking as it does not benefit Cassandra.
+-XX:-UseBiasedLocking
+
+# Enable thread-local allocation blocks and allow the JVM to automatically
+# resize them at runtime.
+-XX:+UseTLAB
+-XX:+ResizeTLAB
+
+# http://www.evanjones.ca/jvm-mmap-pause.html
+-XX:+PerfDisableSharedMem
+
+# Prefer binding to IPv4 network intefaces (when net.ipv6.bindv6only=1). See
+# http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6342561 (short version:
+# comment out this entry to enable IPv6 support).
+-Djava.net.preferIPv4Stack=true
+
+### Debug options
+
+# uncomment to enable flight recorder
+#-XX:+UnlockCommercialFeatures
+#-XX:+FlightRecorder
+
+# uncomment to have Cassandra JVM listen for remote debuggers/profilers on port 1414
+#-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414
+
+# uncomment to have Cassandra JVM log internal method compilation (developers only)
+#-XX:+UnlockDiagnosticVMOptions
+#-XX:+LogCompilation
+
#################
# HEAP SETTINGS #
#################
diff --git a/debian/changelog b/debian/changelog
index 0f3a46c..b975a5e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,14 +1,20 @@
-cassandra (3.0.3) unstable; urgency=medium
-
- * New release
-
- -- Jake Luciani <jake@apache.org> Wed, 03 Feb 2016 08:54:57 -0500
-
-cassandra (3.0.1) unstable; urgency=medium
+cassandra (3.3) unstable; urgency=medium
* New release
- -- Jake Luciani <jake@apache.org> Fri, 04 Dec 2015 15:56:02 -0500
+ -- Jake Luciani <jake@apache.org> Wed, 03 Feb 2016 09:00:11 -0500
+
+cassandra (3.2) unstable; urgency=medium
+
+ * New release
+
+ -- Jake Luciani <jake@apache.org> Tue, 05 Jan 2016 10:24:04 -0500
+
+cassandra (3.1) unstable; urgency=medium
+
+ * New release
+
+ -- Jake Luciani <jake@apache.org> Fri, 04 Dec 2015 16:00:04 -0500
cassandra (3.0.0) unstable; urgency=medium
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index e48e890..891b1d0 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -1,6 +1,6 @@
<link rel="StyleSheet" href="CQL.css" type="text/css" media="screen">
-h1. Cassandra Query Language (CQL) v3.4.0
+h1. Cassandra Query Language (CQL) v3.4.1
@@ -1031,6 +1031,7 @@
<selector> ::= <identifier>
| WRITETIME '(' <identifier> ')'
| TTL '(' <identifier> ')'
+ | CAST '(' <selector> AS <type> ')'
| <function> '(' (<selector> (',' <selector>)*)? ')'
<where-clause> ::= <relation> ( AND <relation> )*
@@ -1072,7 +1073,7 @@
The @<select-clause>@ determines which columns needs to be queried and returned in the result-set. It consists of either the comma-separated list of <selector> or the wildcard character (@*@) to select all the columns defined for the table.
-A @<selector>@ is either a column name to retrieve or a @<function>@ of one or more @<term>@s. The function allowed are the same as for @<term>@ and are described in the "function section":#functions. In addition to these generic functions, the @WRITETIME@ (resp. @TTL@) function allows to select the timestamp of when the column was inserted (resp. the time to live (in seconds) for the column (or null if the column has no expiration set)).
+A @<selector>@ is either a column name to retrieve or a @<function>@ of one or more @<term>@s. The function allowed are the same as for @<term>@ and are described in the "function section":#functions. In addition to these generic functions, the @WRITETIME@ (resp. @TTL@) function allows to select the timestamp of when the column was inserted (resp. the time to live (in seconds) for the column (or null if the column has no expiration set)) and the "@CAST@":#castFun function can be used to convert one data type to another.
Any @<selector>@ can be aliased using @AS@ keyword (see examples). Please note that @<where-clause>@ and @<order-by>@ clause should refer to the columns by their original names and not by their aliases.
@@ -1855,6 +1856,37 @@
CQL3 distinguishes between built-in functions (so called 'native functions') and "user-defined functions":#udfs. CQL3 includes several native functions, described below:
+h3(#castFun). Cast
+
+The @cast@ function can be used to converts one native datatype to another.
+
+The following table describes the conversions supported by the @cast@ function. Cassandra will silently ignore any cast converting a datatype into its own datatype.
+
+|_. from |_. to |
+|@ascii@ |@text@, @varchar@ |
+|@bigint@ |@tinyint@, @smallint@, @int@, @float@, @double@, @decimal@, @varint@, @text@, @varchar@ |
+|@boolean@ |@text@, @varchar@ |
+|@counter@ |@tinyint@, @smallint@, @int@, @bigint@, @float@, @double@, @decimal@, @varint@, @text@, @varchar@ |
+|@date@ |@timestamp@ |
+|@decimal@ |@tinyint@, @smallint@, @int@, @bigint@, @float@, @double@, @varint@, @text@, @varchar@ |
+|@double@ |@tinyint@, @smallint@, @int@, @bigint@, @float@, @decimal@, @varint@, @text@, @varchar@ |
+|@float@ |@tinyint@, @smallint@, @int@, @bigint@, @double@, @decimal@, @varint@, @text@, @varchar@ |
+|@inet@ |@text@, @varchar@ |
+|@int@ |@tinyint@, @smallint@, @bigint@, @float@, @double@, @decimal@, @varint@, @text@, @varchar@ |
+|@smallint@ |@tinyint@, @int@, @bigint@, @float@, @double@, @decimal@, @varint@, @text@, @varchar@ |
+|@time@ |@text@, @varchar@ |
+|@timestamp@|@date@, @text@, @varchar@ |
+|@timeuuid@ |@timestamp@, @date@, @text@, @varchar@ |
+|@tinyint@ |@tinyint@, @smallint@, @int@, @bigint@, @float@, @double@, @decimal@, @varint@, @text@, @varchar@ |
+|@uuid@ |@text@, @varchar@ |
+|@varint@ |@tinyint@, @smallint@, @int@, @bigint@, @float@, @double@, @decimal@, @text@, @varchar@ |
+
+
+The conversions rely strictly on Java's semantics. For example, the double value 1 will be converted to the text value '1.0'.
+
+bc(sample).
+SELECT avg(cast(count as double)) FROM myTable
+
h3(#tokenFun). Token
The @token@ function allows to compute the token for a given partition key. The exact signature of the token function depends on the table concerned and of the partitioner used by the cluster.
@@ -2269,6 +2301,10 @@
The following describes the changes in each version of CQL.
+h3. 3.4.1
+
+* Adds @CAST@ functions. See "@Cast@":#castFun.
+
h3. 3.4.0
* Support for "materialized views":#createMVStmt
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index aeb2d0b..97f1a98 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -187,6 +187,7 @@
copy_options['errfile'] = safe_normpath(opts.pop('errfile', 'import_%s_%s.err' % (self.ks, self.table,)))
copy_options['ratefile'] = safe_normpath(opts.pop('ratefile', ''))
copy_options['maxoutputsize'] = int(opts.pop('maxoutputsize', '-1'))
+ copy_options['ttl'] = int(opts.pop('ttl', -1))
self.check_options(copy_options)
return CopyOptions(copy=copy_options, dialect=dialect_options, unrecognized=opts)
@@ -1651,6 +1652,7 @@
self.max_attempts = options.copy['maxattempts']
self.min_batch_size = options.copy['minbatchsize']
self.max_batch_size = options.copy['maxbatchsize']
+ self.ttl = options.copy['ttl']
self._session = None
@property
@@ -1727,6 +1729,8 @@
protect_name(self.table),
', '.join(protect_names(self.valid_columns),),
', '.join(['?' for _ in self.valid_columns]))
+ if self.ttl >= 0:
+ query += 'USING TTL %s' % (self.ttl,)
query_statement = self.session.prepare(query)
query_statement.consistency_level = self.consistency_level
diff --git a/pylib/cqlshlib/cql3handling.py b/pylib/cqlshlib/cql3handling.py
index 25cf427..5d4a9ae 100644
--- a/pylib/cqlshlib/cql3handling.py
+++ b/pylib/cqlshlib/cql3handling.py
@@ -687,6 +687,7 @@
| "WRITETIME" "(" [colname]=<cident> ")"
| "TTL" "(" [colname]=<cident> ")"
| "COUNT" "(" star=( "*" | "1" ) ")"
+ | "CAST" "(" <selector> "AS" <storageType> ")"
| <functionName> <selectionFunctionArguments>
;
<selectionFunctionArguments> ::= "(" ( <selector> ( "," <selector> )* )? ")"
diff --git a/src/java/org/apache/cassandra/auth/AllowAllAuthorizer.java b/src/java/org/apache/cassandra/auth/AllowAllAuthorizer.java
index bc6fee4..3b40979 100644
--- a/src/java/org/apache/cassandra/auth/AllowAllAuthorizer.java
+++ b/src/java/org/apache/cassandra/auth/AllowAllAuthorizer.java
@@ -22,6 +22,12 @@
public class AllowAllAuthorizer implements IAuthorizer
{
+ @Override
+ public boolean requireAuthorization()
+ {
+ return false;
+ }
+
public Set<Permission> authorize(AuthenticatedUser user, IResource resource)
{
return resource.applicablePermissions();
diff --git a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
index b34b648..826e89d 100644
--- a/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java
@@ -64,7 +64,7 @@
* in CREATE/ALTER ROLE statements.
*
* Such a configuration could be implemented using a custom IRoleManager that
- * extends CassandraRoleManager and which includes Option.PASSWORD in the Set<Option>
+ * extends CassandraRoleManager and which includes Option.PASSWORD in the {@code Set<Option>}
* returned from supportedOptions/alterableOptions. Any additional processing
* of the password itself (such as storing it in an alternative location) would
* be added in overridden createRole and alterRole implementations.
diff --git a/src/java/org/apache/cassandra/auth/DataResource.java b/src/java/org/apache/cassandra/auth/DataResource.java
index f64ed93..0aa24db 100644
--- a/src/java/org/apache/cassandra/auth/DataResource.java
+++ b/src/java/org/apache/cassandra/auth/DataResource.java
@@ -54,31 +54,22 @@
Permission.MODIFY,
Permission.AUTHORIZE);
private static final String ROOT_NAME = "data";
- private static final DataResource ROOT_RESOURCE = new DataResource();
+ private static final DataResource ROOT_RESOURCE = new DataResource(Level.ROOT, null, null);
private final Level level;
private final String keyspace;
private final String table;
- private DataResource()
- {
- level = Level.ROOT;
- keyspace = null;
- table = null;
- }
+ // memoized hashcode since DataRessource is immutable and used in hashmaps often
+ private final transient int hash;
- private DataResource(String keyspace)
+ private DataResource(Level level, String keyspace, String table)
{
- level = Level.KEYSPACE;
- this.keyspace = keyspace;
- table = null;
- }
-
- private DataResource(String keyspace, String table)
- {
- level = Level.TABLE;
+ this.level = level;
this.keyspace = keyspace;
this.table = table;
+
+ this.hash = Objects.hashCode(level, keyspace, table);
}
/**
@@ -97,7 +88,7 @@
*/
public static DataResource keyspace(String keyspace)
{
- return new DataResource(keyspace);
+ return new DataResource(Level.KEYSPACE, keyspace, null);
}
/**
@@ -109,7 +100,7 @@
*/
public static DataResource table(String keyspace, String table)
{
- return new DataResource(keyspace, table);
+ return new DataResource(Level.TABLE, keyspace, table);
}
/**
@@ -272,6 +263,6 @@
@Override
public int hashCode()
{
- return Objects.hashCode(level, keyspace, table);
+ return hash;
}
}
diff --git a/src/java/org/apache/cassandra/auth/IAuthorizer.java b/src/java/org/apache/cassandra/auth/IAuthorizer.java
index 01c05af..a023e3e 100644
--- a/src/java/org/apache/cassandra/auth/IAuthorizer.java
+++ b/src/java/org/apache/cassandra/auth/IAuthorizer.java
@@ -29,6 +29,15 @@
public interface IAuthorizer
{
/**
+ * Whether or not the authorizer will attempt authorization.
+ * If false the authorizer will not be called for authorization of resources.
+ */
+ default boolean requireAuthorization()
+ {
+ return true;
+ }
+
+ /**
* Returns a set of permissions of a user on a resource.
* Since Roles were introduced in version 2.2, Cassandra does not distinguish in any
* meaningful way between users and roles. A role may or may not have login privileges
diff --git a/src/java/org/apache/cassandra/auth/IRoleManager.java b/src/java/org/apache/cassandra/auth/IRoleManager.java
index 5afc7f3..b27681d 100644
--- a/src/java/org/apache/cassandra/auth/IRoleManager.java
+++ b/src/java/org/apache/cassandra/auth/IRoleManager.java
@@ -170,7 +170,7 @@
/**
* Where an implementation supports OPTIONS in CREATE and ALTER operations
- * this method should return the Map<String, String> representing the custom
+ * this method should return the {@code Map<String, String>} representing the custom
* options associated with the role, as supplied to CREATE or ALTER.
* It should never return null; if the implementation does not support
* OPTIONS or if none were supplied then it should return an empty map.
diff --git a/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java b/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java
index 0482199..3bee0e3 100644
--- a/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java
+++ b/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java
@@ -213,10 +213,10 @@
}
}
- if (user == null)
- throw new AuthenticationException("Authentication ID must not be null");
if (pass == null)
throw new AuthenticationException("Password must not be null");
+ if (user == null)
+ throw new AuthenticationException("Authentication ID must not be null");
username = new String(user, StandardCharsets.UTF_8);
password = new String(pass, StandardCharsets.UTF_8);
diff --git a/src/java/org/apache/cassandra/auth/PermissionsCache.java b/src/java/org/apache/cassandra/auth/PermissionsCache.java
index 8746b36..95aa398 100644
--- a/src/java/org/apache/cassandra/auth/PermissionsCache.java
+++ b/src/java/org/apache/cassandra/auth/PermissionsCache.java
@@ -107,7 +107,7 @@
private LoadingCache<Pair<AuthenticatedUser, IResource>, Set<Permission>> initCache(
LoadingCache<Pair<AuthenticatedUser, IResource>, Set<Permission>> existing)
{
- if (authorizer instanceof AllowAllAuthorizer)
+ if (!authorizer.requireAuthorization())
return null;
if (DatabaseDescriptor.getPermissionsValidity() <= 0)
diff --git a/src/java/org/apache/cassandra/auth/RoleOptions.java b/src/java/org/apache/cassandra/auth/RoleOptions.java
index 9609ff3..1205d34 100644
--- a/src/java/org/apache/cassandra/auth/RoleOptions.java
+++ b/src/java/org/apache/cassandra/auth/RoleOptions.java
@@ -90,7 +90,7 @@
}
/**
- * Return a Map<String, String> representing custom options
+ * Return a {@code Map<String, String>} representing custom options
* It is the responsiblity of IRoleManager implementations which support
* IRoleManager.Option.OPTION to handle type checking and conversion of these
* values, if present
diff --git a/src/java/org/apache/cassandra/auth/Roles.java b/src/java/org/apache/cassandra/auth/Roles.java
index da6804b..854d1f2 100644
--- a/src/java/org/apache/cassandra/auth/Roles.java
+++ b/src/java/org/apache/cassandra/auth/Roles.java
@@ -28,7 +28,7 @@
/**
* Get all roles granted to the supplied Role, including both directly granted
* and inherited roles.
- * The returned roles may be cached if roles_validity_in_ms > 0
+ * The returned roles may be cached if {@code roles_validity_in_ms > 0}
*
* @param primaryRole the Role
* @return set of all granted Roles for the primary Role
diff --git a/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java b/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
index 5935669..35469cc 100644
--- a/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
+++ b/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
@@ -23,6 +23,11 @@
public class ScheduledExecutors
{
/**
+ * This pool is used for periodic fast (sub-microsecond) tasks.
+ */
+ public static final DebuggableScheduledThreadPoolExecutor scheduledFastTasks = new DebuggableScheduledThreadPoolExecutor("ScheduledFastTasks");
+
+ /**
* This pool is used for periodic short (sub-second) tasks.
*/
public static final DebuggableScheduledThreadPoolExecutor scheduledTasks = new DebuggableScheduledThreadPoolExecutor("ScheduledTasks");
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java b/src/java/org/apache/cassandra/config/CFMetaData.java
index cb6d3b8..e722742 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -25,6 +25,7 @@
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
import java.util.stream.Collectors;
import com.google.common.annotations.VisibleForTesting;
@@ -38,6 +39,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.cassandra.auth.DataResource;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.cql3.QueryProcessor;
import org.apache.cassandra.cql3.statements.CFStatement;
@@ -68,6 +70,8 @@
SUPER, COUNTER, DENSE, COMPOUND
}
+ private static final Pattern PATTERN_WORD_CHARS = Pattern.compile("\\w+");
+
private static final Logger logger = LoggerFactory.getLogger(CFMetaData.class);
public static final Serializer serializer = new Serializer();
@@ -118,6 +122,8 @@
// for those tables in practice).
private volatile ColumnDefinition compactValueColumn;
+ public final DataResource resource;
+
/*
* All of these methods will go away once CFMetaData becomes completely immutable.
*/
@@ -288,6 +294,7 @@
this.partitionColumns = partitionColumns;
this.serializers = new Serializers(this);
+ this.resource = DataResource.table(ksName, cfName);
rebuild();
}
@@ -415,7 +422,7 @@
/**
* Generates deterministic UUID from keyspace/columnfamily name pair.
- * This is used to generate the same UUID for C* version < 2.1
+ * This is used to generate the same UUID for {@code C* version < 2.1}
*
* Since 2.1, this is only used for system columnfamilies and tests.
*/
@@ -826,9 +833,9 @@
return columnMetadata.get(name);
}
- public static boolean isNameValid(String name)
- {
- return name != null && !name.isEmpty() && name.length() <= Schema.NAME_LENGTH && name.matches("\\w+");
+ public static boolean isNameValid(String name) {
+ return name != null && !name.isEmpty()
+ && name.length() <= Schema.NAME_LENGTH && PATTERN_WORD_CHARS.matcher(name).matches();
}
public CFMetaData validate() throws ConfigurationException
diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java
index ca5bcea..9277df9 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -98,7 +98,7 @@
@Deprecated
public Integer concurrent_replicates = null;
- public Integer memtable_flush_writers = null;
+ public Integer memtable_flush_writers = 1;
public Integer memtable_heap_space_in_mb;
public Integer memtable_offheap_space_in_mb;
public Float memtable_cleanup_threshold = null;
@@ -171,6 +171,7 @@
public int commitlog_segment_size_in_mb = 32;
public ParameterizedClass commitlog_compression;
public int commitlog_max_compression_buffers_in_pool = 3;
+ public TransparentDataEncryptionOptions transparent_data_encryption_options = new TransparentDataEncryptionOptions();
public Integer max_mutation_size_in_kb;
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index c903775..c82e930 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -51,11 +51,12 @@
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.scheduler.IRequestScheduler;
import org.apache.cassandra.scheduler.NoScheduler;
+import org.apache.cassandra.security.EncryptionContext;
import org.apache.cassandra.service.CacheService;
import org.apache.cassandra.thrift.ThriftServer;
-import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.memory.*;
+import org.apache.commons.lang3.StringUtils;
public class DatabaseDescriptor
{
@@ -101,6 +102,7 @@
private static String localDC;
private static Comparator<InetAddress> localComparator;
+ private static EncryptionContext encryptionContext;
public static void forceStaticInitialization() {}
static
@@ -322,8 +324,8 @@
if (conf.authorizer != null)
authorizer = FBUtilities.newAuthorizer(conf.authorizer);
- if (authenticator instanceof AllowAllAuthenticator && !(authorizer instanceof AllowAllAuthorizer))
- throw new ConfigurationException("AllowAllAuthenticator can't be used with " + conf.authorizer, false);
+ if (!authenticator.requireAuthentication() && authorizer.requireAuthorization())
+ throw new ConfigurationException(conf.authenticator + " can't be used with " + conf.authorizer, false);
if (conf.role_manager != null)
roleManager = FBUtilities.newRoleManager(conf.role_manager);
@@ -580,9 +582,6 @@
if (conf.hints_directory.equals(conf.saved_caches_directory))
throw new ConfigurationException("saved_caches_directory must not be the same as the hints_directory", false);
- if (conf.memtable_flush_writers == null)
- conf.memtable_flush_writers = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length)));
-
if (conf.memtable_flush_writers < 1)
throw new ConfigurationException("memtable_flush_writers must be at least 1, but was " + conf.memtable_flush_writers, false);
@@ -685,6 +684,10 @@
if (conf.user_defined_function_fail_timeout < conf.user_defined_function_warn_timeout)
throw new ConfigurationException("user_defined_function_warn_timeout must less than user_defined_function_fail_timeout", false);
+ // always attempt to load the cipher factory, as we could be in the situation where the user has disabled encryption,
+ // but has existing commitlogs and sstables on disk that are still encrypted (and still need to be read)
+ encryptionContext = new EncryptionContext(config.transparent_data_encryption_options);
+
if (conf.max_mutation_size_in_kb == null)
conf.max_mutation_size_in_kb = conf.commitlog_segment_size_in_mb * 1024 / 2;
else if (conf.commitlog_segment_size_in_mb * 1024 < 2 * conf.max_mutation_size_in_kb)
@@ -921,8 +924,8 @@
{
List<String> tokens = new ArrayList<String>();
if (tokenString != null)
- for (String token : tokenString.split(","))
- tokens.add(token.replaceAll("^\\s+", "").replaceAll("\\s+$", ""));
+ for (String token : StringUtils.split(tokenString, ','))
+ tokens.add(token.trim());
return tokens;
}
@@ -1942,9 +1945,19 @@
conf.user_function_timeout_policy = userFunctionTimeoutPolicy;
}
+ public static EncryptionContext getEncryptionContext()
+ {
+ return encryptionContext;
+ }
+
public static long getGCWarnThreshold()
{
return conf.gc_warn_threshold_in_ms;
}
+ @VisibleForTesting
+ public static void setEncryptionContext(EncryptionContext ec)
+ {
+ encryptionContext = ec;
+ }
}
diff --git a/src/java/org/apache/cassandra/config/TransparentDataEncryptionOptions.java b/src/java/org/apache/cassandra/config/TransparentDataEncryptionOptions.java
new file mode 100644
index 0000000..4ad0305
--- /dev/null
+++ b/src/java/org/apache/cassandra/config/TransparentDataEncryptionOptions.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.config;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Objects;
+
+public class TransparentDataEncryptionOptions
+{
+ public boolean enabled = false;
+ public int chunk_length_kb = 64;
+ public String cipher = "AES/CBC/PKCS5Padding";
+ public String key_alias;
+ public int iv_length = 16;
+
+ public ParameterizedClass key_provider;
+
+ public TransparentDataEncryptionOptions()
+ { }
+
+ public TransparentDataEncryptionOptions(boolean enabled)
+ {
+ this.enabled = enabled;
+ }
+
+ public TransparentDataEncryptionOptions(String cipher, String keyAlias, ParameterizedClass keyProvider)
+ {
+ this(true, cipher, keyAlias, keyProvider);
+ }
+
+ public TransparentDataEncryptionOptions(boolean enabled, String cipher, String keyAlias, ParameterizedClass keyProvider)
+ {
+ this.enabled = enabled;
+ this.cipher = cipher;
+ key_alias = keyAlias;
+ key_provider = keyProvider;
+ }
+
+ public String get(String key)
+ {
+ return key_provider.parameters.get(key);
+ }
+
+ @VisibleForTesting
+ public void remove(String key)
+ {
+ key_provider.parameters.remove(key);
+ }
+
+ public boolean equals(Object o)
+ {
+ return o instanceof TransparentDataEncryptionOptions && equals((TransparentDataEncryptionOptions) o);
+ }
+
+ public boolean equals(TransparentDataEncryptionOptions other)
+ {
+ // not sure if this is a great equals() impl....
+ return Objects.equal(cipher, other.cipher) &&
+ Objects.equal(key_alias, other.key_alias);
+ }
+}
diff --git a/src/java/org/apache/cassandra/config/ViewDefinition.java b/src/java/org/apache/cassandra/config/ViewDefinition.java
index b29a8f9..89d69ee 100644
--- a/src/java/org/apache/cassandra/config/ViewDefinition.java
+++ b/src/java/org/apache/cassandra/config/ViewDefinition.java
@@ -129,8 +129,10 @@
}
/**
- * Replace the column {@param from} with {@param to} in this materialized view definition's partition,
+ * Replace the column 'from' with 'to' in this materialized view definition's partition,
* clustering, or included columns.
+ * @param from the existing column
+ * @param to the new column
*/
public void renameColumn(ColumnIdentifier from, ColumnIdentifier to)
{
diff --git a/src/java/org/apache/cassandra/cql3/CQL3Type.java b/src/java/org/apache/cassandra/cql3/CQL3Type.java
index 4e67346..989fcf8 100644
--- a/src/java/org/apache/cassandra/cql3/CQL3Type.java
+++ b/src/java/org/apache/cassandra/cql3/CQL3Type.java
@@ -815,11 +815,6 @@
return true;
}
- public boolean isCollection()
- {
- return false;
- }
-
public void freeze() throws InvalidRequestException
{
for (CQL3Type.Raw t : types)
diff --git a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
index 93734e9..74d0d28 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
@@ -46,7 +46,8 @@
public class ColumnIdentifier extends Selectable implements IMeasurableMemory, Comparable<ColumnIdentifier>
{
private static final Pattern PATTERN_DOUBLE_QUOTE = Pattern.compile("\"", Pattern.LITERAL);
-
+ private static final String ESCAPED_DOUBLE_QUOTE = Matcher.quoteReplacement("\"\"");
+
public final ByteBuffer bytes;
private final String text;
/**
@@ -332,6 +333,6 @@
{
if (UNQUOTED_IDENTIFIER.matcher(text).matches())
return text;
- return '"' + PATTERN_DOUBLE_QUOTE.matcher(text).replaceAll(Matcher.quoteReplacement("\"\"")) + '"';
+ return '"' + PATTERN_DOUBLE_QUOTE.matcher(text).replaceAll(ESCAPED_DOUBLE_QUOTE) + '"';
}
}
diff --git a/src/java/org/apache/cassandra/cql3/Cql.g b/src/java/org/apache/cassandra/cql3/Cql.g
index 453a03d..6b4eff6 100644
--- a/src/java/org/apache/cassandra/cql3/Cql.g
+++ b/src/java/org/apache/cassandra/cql3/Cql.g
@@ -328,9 +328,10 @@
unaliasedSelector returns [Selectable.Raw s]
@init { Selectable.Raw tmp = null; }
: ( c=cident { tmp = c; }
- | K_COUNT '(' countArgument ')' { tmp = new Selectable.WithFunction.Raw(FunctionName.nativeFunction("countRows"), Collections.<Selectable.Raw>emptyList());}
+ | K_COUNT '(' countArgument ')' { tmp = Selectable.WithFunction.Raw.newCountRowsFunction();}
| K_WRITETIME '(' c=cident ')' { tmp = new Selectable.WritetimeOrTTL.Raw(c, true); }
| K_TTL '(' c=cident ')' { tmp = new Selectable.WritetimeOrTTL.Raw(c, false); }
+ | K_CAST '(' sn=unaliasedSelector K_AS t=native_type ')' {tmp = new Selectable.WithCast.Raw(sn, t);}
| f=functionName args=selectionFunctionArgs { tmp = new Selectable.WithFunction.Raw(f, args); }
) ( '.' fi=cident { tmp = new Selectable.WithFieldSelection.Raw(tmp, fi); } )* { $s = tmp; }
;
@@ -1047,7 +1048,7 @@
RoleOptions opts = new RoleOptions();
RoleName name = new RoleName();
}
- : K_ALTER K_USER u=username { name.setName($u.text, true); }
+ : K_ALTER K_USER u=username { name.setName($u.text, false); }
( K_WITH userPassword[opts] )?
( K_SUPERUSER { opts.setOption(IRoleManager.Option.SUPERUSER, true); }
| K_NOSUPERUSER { opts.setOption(IRoleManager.Option.SUPERUSER, false); } ) ?
@@ -1062,7 +1063,7 @@
boolean ifExists = false;
RoleName name = new RoleName();
}
- : K_DROP K_USER (K_IF K_EXISTS { ifExists = true; })? u=username { name.setName($u.text, true); $stmt = new DropRoleStatement(name, ifExists); }
+ : K_DROP K_USER (K_IF K_EXISTS { ifExists = true; })? u=username { name.setName($u.text, false); $stmt = new DropRoleStatement(name, ifExists); }
;
/**
@@ -1574,7 +1575,7 @@
unreserved_keyword returns [String str]
: u=unreserved_function_keyword { $str = u; }
- | k=(K_TTL | K_COUNT | K_WRITETIME | K_KEY) { $str = $k.text; }
+ | k=(K_TTL | K_COUNT | K_WRITETIME | K_KEY | K_CAST) { $str = $k.text; }
;
unreserved_function_keyword returns [String str]
@@ -1674,6 +1675,7 @@
K_VALUES: V A L U E S;
K_TIMESTAMP: T I M E S T A M P;
K_TTL: T T L;
+K_CAST: C A S T;
K_ALTER: A L T E R;
K_RENAME: R E N A M E;
K_ADD: A D D;
diff --git a/src/java/org/apache/cassandra/cql3/Lists.java b/src/java/org/apache/cassandra/cql3/Lists.java
index 18b382b..55208ea 100644
--- a/src/java/org/apache/cassandra/cql3/Lists.java
+++ b/src/java/org/apache/cassandra/cql3/Lists.java
@@ -358,13 +358,9 @@
CellPath elementPath = existingRow.getComplexColumnData(column).getCellByIndex(idx).path();
if (value == null)
- {
params.addTombstone(column, elementPath);
- }
else if (value != ByteBufferUtil.UNSET_BYTE_BUFFER)
- {
params.addCell(column, elementPath, value);
- }
}
}
diff --git a/src/java/org/apache/cassandra/cql3/QueryOptions.java b/src/java/org/apache/cassandra/cql3/QueryOptions.java
index 6324911..e6cda89 100644
--- a/src/java/org/apache/cassandra/cql3/QueryOptions.java
+++ b/src/java/org/apache/cassandra/cql3/QueryOptions.java
@@ -152,7 +152,7 @@
throw new UnsupportedOperationException();
}
- /** The pageSize for this query. Will be <= 0 if not relevant for the query. */
+ /** The pageSize for this query. Will be {@code <= 0} if not relevant for the query. */
public int getPageSize()
{
return getSpecificOptions().pageSize;
diff --git a/src/java/org/apache/cassandra/cql3/SingleColumnRelation.java b/src/java/org/apache/cassandra/cql3/SingleColumnRelation.java
index e2c0b79..867a919 100644
--- a/src/java/org/apache/cassandra/cql3/SingleColumnRelation.java
+++ b/src/java/org/apache/cassandra/cql3/SingleColumnRelation.java
@@ -37,7 +37,7 @@
/**
* Relations encapsulate the relationship between an entity of some kind, and
- * a value (term). For example, <key> > "start" or "colname1" = "somevalue".
+ * a value (term). For example, {@code <key> > "start" or "colname1" = "somevalue"}.
*
*/
public final class SingleColumnRelation extends Relation
diff --git a/src/java/org/apache/cassandra/cql3/UntypedResultSet.java b/src/java/org/apache/cassandra/cql3/UntypedResultSet.java
index ada1e0f..fb8d567 100644
--- a/src/java/org/apache/cassandra/cql3/UntypedResultSet.java
+++ b/src/java/org/apache/cassandra/cql3/UntypedResultSet.java
@@ -187,7 +187,8 @@
if (pager.isExhausted())
return endOfData();
- try (ReadOrderGroup orderGroup = pager.startOrderGroup(); PartitionIterator iter = pager.fetchPageInternal(pageSize, orderGroup))
+ try (ReadExecutionController executionController = pager.executionController();
+ PartitionIterator iter = pager.fetchPageInternal(pageSize, executionController))
{
currentPage = select.process(iter, nowInSec).rows.iterator();
}
diff --git a/src/java/org/apache/cassandra/cql3/functions/AbstractFunction.java b/src/java/org/apache/cassandra/cql3/functions/AbstractFunction.java
index b77f4d5..d36a492 100644
--- a/src/java/org/apache/cassandra/cql3/functions/AbstractFunction.java
+++ b/src/java/org/apache/cassandra/cql3/functions/AbstractFunction.java
@@ -25,6 +25,7 @@
import org.apache.cassandra.cql3.AssignmentTestable;
import org.apache.cassandra.cql3.ColumnSpecification;
import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.commons.lang3.text.StrBuilder;
/**
* Base class for our native/hardcoded functions.
@@ -116,4 +117,13 @@
sb.append(") -> ").append(returnType.asCQL3Type());
return sb.toString();
}
+
+ @Override
+ public String columnName(List<String> columnNames)
+ {
+ return new StrBuilder(name().toString()).append('(')
+ .appendWithSeparators(columnNames, ", ")
+ .append(')')
+ .toString();
+ }
}
diff --git a/src/java/org/apache/cassandra/cql3/functions/AggregateFcts.java b/src/java/org/apache/cassandra/cql3/functions/AggregateFcts.java
index a1b67e1..864e3e5 100644
--- a/src/java/org/apache/cassandra/cql3/functions/AggregateFcts.java
+++ b/src/java/org/apache/cassandra/cql3/functions/AggregateFcts.java
@@ -83,17 +83,6 @@
}
/**
- * Checks if the specified function is the count rows (e.g. COUNT(*) or COUNT(1)) function.
- *
- * @param function the function to check
- * @return <code>true</code> if the specified function is the count rows one, <code>false</code> otherwise.
- */
- public static boolean isCountRows(Function function)
- {
- return function == countRowsFunction;
- }
-
- /**
* The function used to count the number of rows of a result set. This function is called when COUNT(*) or COUNT(1)
* is specified.
*/
@@ -122,6 +111,12 @@
}
};
}
+
+ @Override
+ public String columnName(List<String> columnNames)
+ {
+ return "count";
+ }
};
/**
diff --git a/src/java/org/apache/cassandra/cql3/functions/CastFcts.java b/src/java/org/apache/cassandra/cql3/functions/CastFcts.java
new file mode 100644
index 0000000..8669225
--- /dev/null
+++ b/src/java/org/apache/cassandra/cql3/functions/CastFcts.java
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.cql3.functions;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.cassandra.cql3.CQL3Type;
+import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.BooleanType;
+import org.apache.cassandra.db.marshal.ByteType;
+import org.apache.cassandra.db.marshal.CounterColumnType;
+import org.apache.cassandra.db.marshal.DecimalType;
+import org.apache.cassandra.db.marshal.DoubleType;
+import org.apache.cassandra.db.marshal.FloatType;
+import org.apache.cassandra.db.marshal.InetAddressType;
+import org.apache.cassandra.db.marshal.Int32Type;
+import org.apache.cassandra.db.marshal.IntegerType;
+import org.apache.cassandra.db.marshal.LongType;
+import org.apache.cassandra.db.marshal.ShortType;
+import org.apache.cassandra.db.marshal.SimpleDateType;
+import org.apache.cassandra.db.marshal.TimeType;
+import org.apache.cassandra.db.marshal.TimeUUIDType;
+import org.apache.cassandra.db.marshal.TimestampType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.db.marshal.UUIDType;
+import org.apache.commons.lang3.text.WordUtils;
+
+/**
+ * Casting functions
+ *
+ */
+public final class CastFcts
+{
+ private static final String FUNCTION_NAME_PREFIX = "castAs";
+
+ public static Collection<Function> all()
+ {
+ List<Function> functions = new ArrayList<>();
+
+ @SuppressWarnings("unchecked")
+ final AbstractType<? extends Number>[] numericTypes = new AbstractType[] {ByteType.instance,
+ ShortType.instance,
+ Int32Type.instance,
+ LongType.instance,
+ FloatType.instance,
+ DoubleType.instance,
+ DecimalType.instance,
+ CounterColumnType.instance,
+ IntegerType.instance};
+
+ for (AbstractType<? extends Number> inputType : numericTypes)
+ {
+ addFunctionIfNeeded(functions, inputType, ByteType.instance, Number::byteValue);
+ addFunctionIfNeeded(functions, inputType, ShortType.instance, Number::shortValue);
+ addFunctionIfNeeded(functions, inputType, Int32Type.instance, Number::intValue);
+ addFunctionIfNeeded(functions, inputType, LongType.instance, Number::longValue);
+ addFunctionIfNeeded(functions, inputType, FloatType.instance, Number::floatValue);
+ addFunctionIfNeeded(functions, inputType, DoubleType.instance, Number::doubleValue);
+ addFunctionIfNeeded(functions, inputType, DecimalType.instance, p -> BigDecimal.valueOf(p.doubleValue()));
+ addFunctionIfNeeded(functions, inputType, IntegerType.instance, p -> BigInteger.valueOf(p.longValue()));
+ functions.add(CastAsTextFunction.create(inputType, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(inputType, UTF8Type.instance));
+ }
+
+ functions.add(JavaFunctionWrapper.create(AsciiType.instance, UTF8Type.instance, p -> p));
+
+ functions.add(CastAsTextFunction.create(InetAddressType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(InetAddressType.instance, UTF8Type.instance));
+
+ functions.add(CastAsTextFunction.create(BooleanType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(BooleanType.instance, UTF8Type.instance));
+
+ functions.add(CassandraFunctionWrapper.create(TimeUUIDType.instance, SimpleDateType.instance, TimeFcts.timeUuidtoDate));
+ functions.add(CassandraFunctionWrapper.create(TimeUUIDType.instance, TimestampType.instance, TimeFcts.timeUuidToTimestamp));
+ functions.add(CastAsTextFunction.create(TimeUUIDType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(TimeUUIDType.instance, UTF8Type.instance));
+ functions.add(CassandraFunctionWrapper.create(TimestampType.instance, SimpleDateType.instance, TimeFcts.timestampToDate));
+ functions.add(CastAsTextFunction.create(TimestampType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(TimestampType.instance, UTF8Type.instance));
+ functions.add(CassandraFunctionWrapper.create(SimpleDateType.instance, TimestampType.instance, TimeFcts.dateToTimestamp));
+ functions.add(CastAsTextFunction.create(SimpleDateType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(SimpleDateType.instance, UTF8Type.instance));
+ functions.add(CastAsTextFunction.create(TimeType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(TimeType.instance, UTF8Type.instance));
+
+ functions.add(CastAsTextFunction.create(UUIDType.instance, AsciiType.instance));
+ functions.add(CastAsTextFunction.create(UUIDType.instance, UTF8Type.instance));
+
+ return functions;
+ }
+
+ /**
+ * Creates the name of the cast function use to cast to the specified type.
+ *
+ * @param outputType the output type
+ * @return the name of the cast function use to cast to the specified type
+ */
+ public static String getFunctionName(AbstractType<?> outputType)
+ {
+ return getFunctionName(outputType.asCQL3Type());
+ }
+
+ /**
+ * Creates the name of the cast function use to cast to the specified type.
+ *
+ * @param outputType the output type
+ * @return the name of the cast function use to cast to the specified type
+ */
+ public static String getFunctionName(CQL3Type outputType)
+ {
+ return FUNCTION_NAME_PREFIX + WordUtils.capitalize(toLowerCaseString(outputType));
+ }
+
+ /**
+ * Adds to the list a function converting the input type in to the output type if they are not the same.
+ *
+ * @param functions the list to add to
+ * @param inputType the input type
+ * @param outputType the output type
+ * @param converter the function use to convert the input type into the output type
+ */
+ private static <I, O> void addFunctionIfNeeded(List<Function> functions,
+ AbstractType<I> inputType,
+ AbstractType<O> outputType,
+ java.util.function.Function<I, O> converter)
+ {
+ if (!inputType.equals(outputType))
+ functions.add(wrapJavaFunction(inputType, outputType, converter));
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <O, I> Function wrapJavaFunction(AbstractType<I> inputType,
+ AbstractType<O> outputType,
+ java.util.function.Function<I, O> converter)
+ {
+ return inputType.equals(CounterColumnType.instance)
+ ? JavaCounterFunctionWrapper.create(outputType, (java.util.function.Function<Long, O>) converter)
+ : JavaFunctionWrapper.create(inputType, outputType, converter);
+ }
+
+ private static String toLowerCaseString(CQL3Type type)
+ {
+ return type.toString().toLowerCase();
+ }
+
+ /**
+ * Base class for the CAST functions.
+ *
+ * @param <I> the input type
+ * @param <O> the output type
+ */
+ private static abstract class CastFunction<I, O> extends NativeScalarFunction
+ {
+ public CastFunction(AbstractType<I> inputType, AbstractType<O> outputType)
+ {
+ super(getFunctionName(outputType), outputType, inputType);
+ }
+
+ @Override
+ public String columnName(List<String> columnNames)
+ {
+ return String.format("cast(%s as %s)", columnNames.get(0), toLowerCaseString(outputType().asCQL3Type()));
+ }
+
+ @SuppressWarnings("unchecked")
+ protected AbstractType<O> outputType()
+ {
+ return (AbstractType<O>) returnType;
+ }
+
+ @SuppressWarnings("unchecked")
+ protected AbstractType<I> inputType()
+ {
+ return (AbstractType<I>) argTypes.get(0);
+ }
+ }
+
+ /**
+ * <code>CastFunction</code> that implements casting by wrapping a java <code>Function</code>.
+ *
+ * @param <I> the input parameter
+ * @param <O> the output parameter
+ */
+ private static class JavaFunctionWrapper<I, O> extends CastFunction<I, O>
+ {
+ /**
+ * The java function used to convert the input type into the output one.
+ */
+ private final java.util.function.Function<I, O> converter;
+
+ public static <I, O> JavaFunctionWrapper<I, O> create(AbstractType<I> inputType,
+ AbstractType<O> outputType,
+ java.util.function.Function<I, O> converter)
+ {
+ return new JavaFunctionWrapper<I, O>(inputType, outputType, converter);
+ }
+
+ protected JavaFunctionWrapper(AbstractType<I> inputType,
+ AbstractType<O> outputType,
+ java.util.function.Function<I, O> converter)
+ {
+ super(inputType, outputType);
+ this.converter = converter;
+ }
+
+ public final ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
+ {
+ ByteBuffer bb = parameters.get(0);
+ if (bb == null)
+ return null;
+
+ return outputType().decompose(converter.apply(compose(bb)));
+ }
+
+ protected I compose(ByteBuffer bb)
+ {
+ return inputType().compose(bb);
+ }
+ }
+
+ /**
+ * <code>JavaFunctionWrapper</code> for counter columns.
+ *
+ * <p>Counter columns need to be handled in a special way because their binary representation is converted into
+ * the one of a BIGINT before functions are applied.</p>
+ *
+ * @param <O> the output parameter
+ */
+ private static class JavaCounterFunctionWrapper<O> extends JavaFunctionWrapper<Long, O>
+ {
+ public static <O> JavaFunctionWrapper<Long, O> create(AbstractType<O> outputType,
+ java.util.function.Function<Long, O> converter)
+ {
+ return new JavaCounterFunctionWrapper<O>(outputType, converter);
+ }
+
+ protected JavaCounterFunctionWrapper(AbstractType<O> outputType,
+ java.util.function.Function<Long, O> converter)
+ {
+ super(CounterColumnType.instance, outputType, converter);
+ }
+
+ protected Long compose(ByteBuffer bb)
+ {
+ return LongType.instance.compose(bb);
+ }
+ }
+
+ /**
+ * <code>CastFunction</code> that implements casting by wrapping an existing <code>NativeScalarFunction</code>.
+ *
+ * @param <I> the input parameter
+ * @param <O> the output parameter
+ */
+ private static final class CassandraFunctionWrapper<I, O> extends CastFunction<I, O>
+ {
+ /**
+ * The native scalar function used to perform the conversion.
+ */
+ private final NativeScalarFunction delegate;
+
+ public static <I, O> CassandraFunctionWrapper<I, O> create(AbstractType<I> inputType,
+ AbstractType<O> outputType,
+ NativeScalarFunction delegate)
+ {
+ return new CassandraFunctionWrapper<I, O>(inputType, outputType, delegate);
+ }
+
+ private CassandraFunctionWrapper(AbstractType<I> inputType,
+ AbstractType<O> outputType,
+ NativeScalarFunction delegate)
+ {
+ super(inputType, outputType);
+ assert delegate.argTypes().size() == 1 && inputType.equals(delegate.argTypes().get(0));
+ assert outputType.equals(delegate.returnType());
+ this.delegate = delegate;
+ }
+
+ public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
+ {
+ return delegate.execute(protocolVersion, parameters);
+ }
+ }
+
+ /**
+ * <code>CastFunction</code> that can be used to cast a type into ascii or text types.
+ *
+ * @param <I> the input parameter
+ */
+ private static final class CastAsTextFunction<I> extends CastFunction<I, String>
+ {
+
+ public static <I> CastAsTextFunction<I> create(AbstractType<I> inputType,
+ AbstractType<String> outputType)
+ {
+ return new CastAsTextFunction<I>(inputType, outputType);
+ }
+
+ private CastAsTextFunction(AbstractType<I> inputType,
+ AbstractType<String> outputType)
+ {
+ super(inputType, outputType);
+ }
+
+ public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
+ {
+ ByteBuffer bb = parameters.get(0);
+ if (bb == null)
+ return null;
+
+ StringBuilder builder = new StringBuilder();
+ inputType().getSerializer().toCQLLiteral(bb, builder);
+ return outputType().decompose(builder.toString());
+ }
+ }
+
+ /**
+ * The class must not be instantiated as it contains only static variables.
+ */
+ private CastFcts()
+ {
+ }
+}
diff --git a/src/java/org/apache/cassandra/cql3/functions/Function.java b/src/java/org/apache/cassandra/cql3/functions/Function.java
index ed6e2a7..f50657a 100644
--- a/src/java/org/apache/cassandra/cql3/functions/Function.java
+++ b/src/java/org/apache/cassandra/cql3/functions/Function.java
@@ -47,4 +47,12 @@
public Iterable<Function> getFunctions();
public boolean hasReferenceTo(Function function);
+
+ /**
+ * Returns the name of the function to use within a ResultSet.
+ *
+ * @param columnNames the names of the columns used to call the function
+ * @return the name of the function to use within a ResultSet
+ */
+ public String columnName(List<String> columnNames);
}
diff --git a/src/java/org/apache/cassandra/cql3/functions/FunctionResolver.java b/src/java/org/apache/cassandra/cql3/functions/FunctionResolver.java
index be2daae..9e0b706 100644
--- a/src/java/org/apache/cassandra/cql3/functions/FunctionResolver.java
+++ b/src/java/org/apache/cassandra/cql3/functions/FunctionResolver.java
@@ -126,9 +126,11 @@
}
}
- if (compatibles == null || compatibles.isEmpty())
+ if (compatibles == null)
+ {
throw new InvalidRequestException(String.format("Invalid call to function %s, none of its type signatures match (known type signatures: %s)",
name, format(candidates)));
+ }
if (compatibles.size() > 1)
throw new InvalidRequestException(String.format("Ambiguous call to function %s (can be matched by following signatures: %s): use type casts to disambiguate",
diff --git a/src/java/org/apache/cassandra/cql3/functions/TimeFcts.java b/src/java/org/apache/cassandra/cql3/functions/TimeFcts.java
index 93d6d3b..623feba 100644
--- a/src/java/org/apache/cassandra/cql3/functions/TimeFcts.java
+++ b/src/java/org/apache/cassandra/cql3/functions/TimeFcts.java
@@ -67,7 +67,7 @@
if (bb == null)
return null;
- return ByteBuffer.wrap(UUIDGen.decompose(UUIDGen.minTimeUUID(TimestampType.instance.compose(bb).getTime())));
+ return UUIDGen.toByteBuffer(UUIDGen.minTimeUUID(TimestampType.instance.compose(bb).getTime()));
}
};
@@ -79,7 +79,7 @@
if (bb == null)
return null;
- return ByteBuffer.wrap(UUIDGen.decompose(UUIDGen.maxTimeUUID(TimestampType.instance.compose(bb).getTime())));
+ return UUIDGen.toByteBuffer(UUIDGen.maxTimeUUID(TimestampType.instance.compose(bb).getTime()));
}
};
@@ -87,7 +87,7 @@
* Function that convert a value of <code>TIMEUUID</code> into a value of type <code>TIMESTAMP</code>.
* @deprecated Replaced by the {@link #timeUuidToTimestamp} function
*/
- public static final Function dateOfFct = new NativeScalarFunction("dateof", TimestampType.instance, TimeUUIDType.instance)
+ public static final NativeScalarFunction dateOfFct = new NativeScalarFunction("dateof", TimestampType.instance, TimeUUIDType.instance)
{
private volatile boolean hasLoggedDeprecationWarning;
@@ -113,7 +113,7 @@
* Function that convert a value of type <code>TIMEUUID</code> into an UNIX timestamp.
* @deprecated Replaced by the {@link #timeUuidToUnixTimestamp} function
*/
- public static final Function unixTimestampOfFct = new NativeScalarFunction("unixtimestampof", LongType.instance, TimeUUIDType.instance)
+ public static final NativeScalarFunction unixTimestampOfFct = new NativeScalarFunction("unixtimestampof", LongType.instance, TimeUUIDType.instance)
{
private volatile boolean hasLoggedDeprecationWarning;
@@ -137,7 +137,7 @@
/**
* Function that convert a value of <code>TIMEUUID</code> into a value of type <code>DATE</code>.
*/
- public static final Function timeUuidtoDate = new NativeScalarFunction("todate", SimpleDateType.instance, TimeUUIDType.instance)
+ public static final NativeScalarFunction timeUuidtoDate = new NativeScalarFunction("todate", SimpleDateType.instance, TimeUUIDType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
@@ -153,7 +153,7 @@
/**
* Function that convert a value of type <code>TIMEUUID</code> into a value of type <code>TIMESTAMP</code>.
*/
- public static final Function timeUuidToTimestamp = new NativeScalarFunction("totimestamp", TimestampType.instance, TimeUUIDType.instance)
+ public static final NativeScalarFunction timeUuidToTimestamp = new NativeScalarFunction("totimestamp", TimestampType.instance, TimeUUIDType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
@@ -169,7 +169,7 @@
/**
* Function that convert a value of type <code>TIMEUUID</code> into an UNIX timestamp.
*/
- public static final Function timeUuidToUnixTimestamp = new NativeScalarFunction("tounixtimestamp", LongType.instance, TimeUUIDType.instance)
+ public static final NativeScalarFunction timeUuidToUnixTimestamp = new NativeScalarFunction("tounixtimestamp", LongType.instance, TimeUUIDType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
@@ -184,7 +184,7 @@
/**
* Function that convert a value of type <code>TIMESTAMP</code> into an UNIX timestamp.
*/
- public static final Function timestampToUnixTimestamp = new NativeScalarFunction("tounixtimestamp", LongType.instance, TimestampType.instance)
+ public static final NativeScalarFunction timestampToUnixTimestamp = new NativeScalarFunction("tounixtimestamp", LongType.instance, TimestampType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
@@ -200,7 +200,7 @@
/**
* Function that convert a value of type <code>TIMESTAMP</code> into a <code>DATE</code>.
*/
- public static final Function timestampToDate = new NativeScalarFunction("todate", SimpleDateType.instance, TimestampType.instance)
+ public static final NativeScalarFunction timestampToDate = new NativeScalarFunction("todate", SimpleDateType.instance, TimestampType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
@@ -216,7 +216,7 @@
/**
* Function that convert a value of type <code>TIMESTAMP</code> into a <code>DATE</code>.
*/
- public static final Function dateToTimestamp = new NativeScalarFunction("totimestamp", TimestampType.instance, SimpleDateType.instance)
+ public static final NativeScalarFunction dateToTimestamp = new NativeScalarFunction("totimestamp", TimestampType.instance, SimpleDateType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
@@ -232,7 +232,7 @@
/**
* Function that convert a value of type <code>DATE</code> into an UNIX timestamp.
*/
- public static final Function dateToUnixTimestamp = new NativeScalarFunction("tounixtimestamp", LongType.instance, SimpleDateType.instance)
+ public static final NativeScalarFunction dateToUnixTimestamp = new NativeScalarFunction("tounixtimestamp", LongType.instance, SimpleDateType.instance)
{
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters)
{
diff --git a/src/java/org/apache/cassandra/cql3/restrictions/PrimaryKeyRestrictionSet.java b/src/java/org/apache/cassandra/cql3/restrictions/PrimaryKeyRestrictionSet.java
index 107cbd1..d24799a 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/PrimaryKeyRestrictionSet.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/PrimaryKeyRestrictionSet.java
@@ -138,24 +138,12 @@
}
@Override
- public boolean isOnToken()
- {
- return false;
- }
-
- @Override
public boolean isContains()
{
return contains;
}
@Override
- public boolean isMultiColumn()
- {
- return false;
- }
-
- @Override
public Iterable<Function> getFunctions()
{
return restrictions.getFunctions();
@@ -175,10 +163,24 @@
return new PrimaryKeyRestrictionSet(this, restriction);
}
+ // Whether any of the underlying restriction is an IN
+ private boolean hasIN()
+ {
+ if (isIN())
+ return true;
+
+ for (Restriction restriction : restrictions)
+ {
+ if (restriction.isIN())
+ return true;
+ }
+ return false;
+ }
+
@Override
public NavigableSet<Clustering> valuesAsClustering(QueryOptions options) throws InvalidRequestException
{
- return appendTo(MultiCBuilder.create(comparator), options).build();
+ return appendTo(MultiCBuilder.create(comparator, hasIN()), options).build();
}
@Override
@@ -202,7 +204,7 @@
@Override
public NavigableSet<Slice.Bound> boundsAsClustering(Bound bound, QueryOptions options) throws InvalidRequestException
{
- MultiCBuilder builder = MultiCBuilder.create(comparator);
+ MultiCBuilder builder = MultiCBuilder.create(comparator, hasIN());
int keyPosition = 0;
for (Restriction r : restrictions)
{
diff --git a/src/java/org/apache/cassandra/cql3/restrictions/RestrictionSet.java b/src/java/org/apache/cassandra/cql3/restrictions/RestrictionSet.java
index 700840d..ccab2dc 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/RestrictionSet.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/RestrictionSet.java
@@ -96,13 +96,13 @@
@Override
public final boolean isEmpty()
{
- return getColumnDefs().isEmpty();
+ return restrictions.isEmpty();
}
@Override
public final int size()
{
- return getColumnDefs().size();
+ return restrictions.size();
}
/**
diff --git a/src/java/org/apache/cassandra/cql3/selection/AbstractFunctionSelector.java b/src/java/org/apache/cassandra/cql3/selection/AbstractFunctionSelector.java
index bf1234f..0ccff6b 100644
--- a/src/java/org/apache/cassandra/cql3/selection/AbstractFunctionSelector.java
+++ b/src/java/org/apache/cassandra/cql3/selection/AbstractFunctionSelector.java
@@ -24,9 +24,6 @@
import com.google.common.collect.Iterables;
import org.apache.commons.lang3.text.StrBuilder;
-
-import org.apache.cassandra.cql3.functions.AggregateFcts;
-
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.ColumnSpecification;
import org.apache.cassandra.cql3.functions.Function;
@@ -56,13 +53,7 @@
{
protected String getColumnName()
{
- if (AggregateFcts.isCountRows(fun))
- return "count";
-
- return new StrBuilder(fun.name().toString()).append('(')
- .appendWithSeparators(factories.getColumnNames(), ", ")
- .append(')')
- .toString();
+ return fun.columnName(factories.getColumnNames());
}
protected AbstractType<?> getReturnType()
diff --git a/src/java/org/apache/cassandra/cql3/selection/FieldSelector.java b/src/java/org/apache/cassandra/cql3/selection/FieldSelector.java
index 63b6cc6..965a01a 100644
--- a/src/java/org/apache/cassandra/cql3/selection/FieldSelector.java
+++ b/src/java/org/apache/cassandra/cql3/selection/FieldSelector.java
@@ -65,11 +65,6 @@
};
}
- public boolean isAggregate()
- {
- return false;
- }
-
public void addInput(int protocolVersion, ResultSetBuilder rs) throws InvalidRequestException
{
selected.addInput(protocolVersion, rs);
diff --git a/src/java/org/apache/cassandra/cql3/selection/Selectable.java b/src/java/org/apache/cassandra/cql3/selection/Selectable.java
index 717fe7c..c3e0331 100644
--- a/src/java/org/apache/cassandra/cql3/selection/Selectable.java
+++ b/src/java/org/apache/cassandra/cql3/selection/Selectable.java
@@ -19,12 +19,13 @@
package org.apache.cassandra.cql3.selection;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import org.apache.commons.lang3.text.StrBuilder;
-
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.cql3.CQL3Type;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.cql3.functions.*;
import org.apache.cassandra.db.marshal.AbstractType;
@@ -33,8 +34,7 @@
public abstract class Selectable
{
- public abstract Selector.Factory newSelectorFactory(CFMetaData cfm, List<ColumnDefinition> defs)
- throws InvalidRequestException;
+ public abstract Selector.Factory newSelectorFactory(CFMetaData cfm, List<ColumnDefinition> defs);
protected static int addAndGetIndex(ColumnDefinition def, List<ColumnDefinition> l)
{
@@ -75,7 +75,7 @@
}
public Selector.Factory newSelectorFactory(CFMetaData cfm,
- List<ColumnDefinition> defs) throws InvalidRequestException
+ List<ColumnDefinition> defs)
{
ColumnDefinition def = cfm.getColumnDefinition(id);
if (def == null)
@@ -136,8 +136,7 @@
.toString();
}
- public Selector.Factory newSelectorFactory(CFMetaData cfm,
- List<ColumnDefinition> defs) throws InvalidRequestException
+ public Selector.Factory newSelectorFactory(CFMetaData cfm, List<ColumnDefinition> defs)
{
SelectorFactories factories =
SelectorFactories.createFactoriesAndCollectColumnDefinitions(args, cfm, defs);
@@ -152,6 +151,7 @@
if (fun == null)
throw new InvalidRequestException(String.format("Unknown function '%s'", functionName));
+
if (fun.returnType() == null)
throw new InvalidRequestException(String.format("Unknown function %s called in selection clause",
functionName));
@@ -170,6 +170,12 @@
this.args = args;
}
+ public static Raw newCountRowsFunction()
+ {
+ return new Raw(AggregateFcts.countRowsFunction.name(),
+ Collections.emptyList());
+ }
+
public WithFunction prepare(CFMetaData cfm)
{
List<Selectable> preparedArgs = new ArrayList<>(args.size());
@@ -185,6 +191,71 @@
}
}
+ public static class WithCast extends Selectable
+ {
+ private final CQL3Type type;
+ private final Selectable arg;
+
+ public WithCast(Selectable arg, CQL3Type type)
+ {
+ this.arg = arg;
+ this.type = type;
+ }
+
+ @Override
+ public String toString()
+ {
+ return String.format("cast(%s as %s)", arg, type.toString().toLowerCase());
+ }
+
+ public Selector.Factory newSelectorFactory(CFMetaData cfm, List<ColumnDefinition> defs)
+ {
+ SelectorFactories factories =
+ SelectorFactories.createFactoriesAndCollectColumnDefinitions(Collections.singletonList(arg), cfm, defs);
+
+ Selector.Factory factory = factories.get(0);
+
+ // If the user is trying to cast a type on its own type we simply ignore it.
+ if (type.getType().equals(factory.getReturnType()))
+ {
+ return factory;
+ }
+
+ FunctionName name = FunctionName.nativeFunction(CastFcts.getFunctionName(type));
+ Function fun = FunctionResolver.get(cfm.ksName, name, factories.newInstances(), cfm.ksName, cfm.cfName, null);
+
+ if (fun == null)
+ {
+ throw new InvalidRequestException(String.format("%s cannot be cast to %s",
+ defs.get(0).name,
+ type));
+ }
+ return AbstractFunctionSelector.newFactory(fun, factories);
+ }
+
+ public static class Raw implements Selectable.Raw
+ {
+ private final CQL3Type type;
+ private final Selectable.Raw arg;
+
+ public Raw(Selectable.Raw arg, CQL3Type type)
+ {
+ this.arg = arg;
+ this.type = type;
+ }
+
+ public WithCast prepare(CFMetaData cfm)
+ {
+ return new WithCast(arg.prepare(cfm), type);
+ }
+
+ public boolean processesSelection()
+ {
+ return true;
+ }
+ }
+ }
+
public static class WithFieldSelection extends Selectable
{
public final Selectable selected;
@@ -202,8 +273,7 @@
return String.format("%s.%s", selected, field);
}
- public Selector.Factory newSelectorFactory(CFMetaData cfm,
- List<ColumnDefinition> defs) throws InvalidRequestException
+ public Selector.Factory newSelectorFactory(CFMetaData cfm, List<ColumnDefinition> defs)
{
Selector.Factory factory = selected.newSelectorFactory(cfm, defs);
AbstractType<?> type = factory.newInstance().getType();
diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
index 47396fb..9faf73c 100644
--- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
@@ -260,7 +260,7 @@
/**
* Checks batch size to ensure threshold is met. If not, a warning is logged.
*
- * @param cfs ColumnFamilies that will store the batch's mutations.
+ * @param updates - the batch mutations.
*/
public static void verifyBatchSize(Iterable<PartitionUpdate> updates) throws InvalidRequestException
{
@@ -464,14 +464,7 @@
private ResultMessage executeInternalWithoutCondition(QueryState queryState, QueryOptions options) throws RequestValidationException, RequestExecutionException
{
for (IMutation mutation : getMutations(BatchQueryOptions.withoutPerStatementVariables(options), true, queryState.getTimestamp()))
- {
- assert mutation instanceof Mutation || mutation instanceof CounterMutation;
-
- if (mutation instanceof Mutation)
- ((Mutation) mutation).apply();
- else if (mutation instanceof CounterMutation)
- ((CounterMutation) mutation).apply();
- }
+ mutation.apply();
return null;
}
diff --git a/src/java/org/apache/cassandra/cql3/statements/CreateKeyspaceStatement.java b/src/java/org/apache/cassandra/cql3/statements/CreateKeyspaceStatement.java
index 3eb0ac9..86754b6 100644
--- a/src/java/org/apache/cassandra/cql3/statements/CreateKeyspaceStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateKeyspaceStatement.java
@@ -17,6 +17,7 @@
*/
package org.apache.cassandra.cql3.statements;
+import java.util.regex.Pattern;
import org.apache.cassandra.auth.*;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.config.Schema;
@@ -31,6 +32,8 @@
/** A <code>CREATE KEYSPACE</code> statement parsed from a CQL query. */
public class CreateKeyspaceStatement extends SchemaAlteringStatement
{
+ private static final Pattern PATTERN_WORD_CHARS = Pattern.compile("\\w+");
+
private final String name;
private final KeyspaceAttributes attrs;
private final boolean ifNotExists;
@@ -73,7 +76,7 @@
ThriftValidation.validateKeyspaceNotSystem(name);
// keyspace name
- if (!name.matches("\\w+"))
+ if (!PATTERN_WORD_CHARS.matcher(name).matches())
throw new InvalidRequestException(String.format("\"%s\" is not a valid keyspace name", name));
if (name.length() > Schema.NAME_LENGTH)
throw new InvalidRequestException(String.format("Keyspace names shouldn't be more than %s characters long (got \"%s\")", Schema.NAME_LENGTH, name));
diff --git a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
index c19f970..debb200 100644
--- a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
@@ -19,7 +19,7 @@
import java.nio.ByteBuffer;
import java.util.*;
-
+import java.util.regex.Pattern;
import com.google.common.collect.HashMultiset;
import com.google.common.collect.Multiset;
import org.apache.commons.lang3.StringUtils;
@@ -41,6 +41,8 @@
/** A {@code CREATE TABLE} parsed from a CQL query statement. */
public class CreateTableStatement extends SchemaAlteringStatement
{
+ private static final Pattern PATTERN_WORD_CHARS = Pattern.compile("\\w+");
+
private List<AbstractType<?>> keyTypes;
private List<AbstractType<?>> clusteringTypes;
@@ -202,7 +204,7 @@
public ParsedStatement.Prepared prepare(Types udts) throws RequestValidationException
{
// Column family name
- if (!columnFamily().matches("\\w+"))
+ if (!PATTERN_WORD_CHARS.matcher(columnFamily()).matches())
throw new InvalidRequestException(String.format("\"%s\" is not a valid table name (must be alphanumeric character or underscore only: [a-zA-Z_0-9]+)", columnFamily()));
if (columnFamily().length() > Schema.NAME_LENGTH)
throw new InvalidRequestException(String.format("Table names shouldn't be more than %s characters long (got \"%s\")", Schema.NAME_LENGTH, columnFamily()));
diff --git a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
index 71597f4..ca3b5bc 100644
--- a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
@@ -193,21 +193,21 @@
public void checkAccess(ClientState state) throws InvalidRequestException, UnauthorizedException
{
- state.hasColumnFamilyAccess(keyspace(), columnFamily(), Permission.MODIFY);
+ state.hasColumnFamilyAccess(cfm, Permission.MODIFY);
// CAS updates can be used to simulate a SELECT query, so should require Permission.SELECT as well.
if (hasConditions())
- state.hasColumnFamilyAccess(keyspace(), columnFamily(), Permission.SELECT);
+ state.hasColumnFamilyAccess(cfm, Permission.SELECT);
// MV updates need to get the current state from the table, and might update the views
// Require Permission.SELECT on the base table, and Permission.MODIFY on the views
Iterator<ViewDefinition> views = View.findAll(keyspace(), columnFamily()).iterator();
if (views.hasNext())
{
- state.hasColumnFamilyAccess(keyspace(), columnFamily(), Permission.SELECT);
+ state.hasColumnFamilyAccess(cfm, Permission.SELECT);
do
{
- state.hasColumnFamilyAccess(keyspace(), views.next().viewName, Permission.MODIFY);
+ state.hasColumnFamilyAccess(views.next().metadata, Permission.MODIFY);
} while (views.hasNext());
}
@@ -360,7 +360,8 @@
if (local)
{
- try (ReadOrderGroup orderGroup = group.startOrderGroup(); PartitionIterator iter = group.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = group.executionController();
+ PartitionIterator iter = group.executeInternal(executionController))
{
return asMaterializedMap(iter);
}
@@ -549,14 +550,7 @@
public ResultMessage executeInternalWithoutCondition(QueryState queryState, QueryOptions options) throws RequestValidationException, RequestExecutionException
{
for (IMutation mutation : getMutations(options, true, queryState.getTimestamp()))
- {
- assert mutation instanceof Mutation || mutation instanceof CounterMutation;
-
- if (mutation instanceof Mutation)
- ((Mutation) mutation).apply();
- else if (mutation instanceof CounterMutation)
- ((CounterMutation) mutation).apply();
- }
+ mutation.apply();
return null;
}
@@ -575,7 +569,8 @@
SinglePartitionReadCommand readCommand = request.readCommand(FBUtilities.nowInSeconds());
FilteredPartition current;
- try (ReadOrderGroup orderGroup = readCommand.startOrderGroup(); PartitionIterator iter = readCommand.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = readCommand.executionController();
+ PartitionIterator iter = readCommand.executeInternal(executionController))
{
current = FilteredPartition.create(PartitionIterators.getOnlyElement(iter, readCommand));
}
diff --git a/src/java/org/apache/cassandra/cql3/statements/PropertyDefinitions.java b/src/java/org/apache/cassandra/cql3/statements/PropertyDefinitions.java
index 793285b..590910f 100644
--- a/src/java/org/apache/cassandra/cql3/statements/PropertyDefinitions.java
+++ b/src/java/org/apache/cassandra/cql3/statements/PropertyDefinitions.java
@@ -18,6 +18,7 @@
package org.apache.cassandra.cql3.statements;
import java.util.*;
+import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -26,6 +27,8 @@
public class PropertyDefinitions
{
+ private static final Pattern PATTERN_POSITIVE = Pattern.compile("(1|true|yes)");
+
protected static final Logger logger = LoggerFactory.getLogger(PropertyDefinitions.class);
protected final Map<String, Object> properties = new HashMap<String, Object>();
@@ -91,7 +94,7 @@
public Boolean getBoolean(String key, Boolean defaultValue) throws SyntaxException
{
String value = getSimple(key);
- return (value == null) ? defaultValue : value.toLowerCase().matches("(1|true|yes)");
+ return (value == null) ? defaultValue : PATTERN_POSITIVE.matcher(value.toLowerCase()).matches();
}
// Return a property value, typed as a double
diff --git a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
index e84ab28..4c9c875 100644
--- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
@@ -178,11 +178,11 @@
{
CFMetaData baseTable = View.findBaseTable(keyspace(), columnFamily());
if (baseTable != null)
- state.hasColumnFamilyAccess(keyspace(), baseTable.cfName, Permission.SELECT);
+ state.hasColumnFamilyAccess(baseTable, Permission.SELECT);
}
else
{
- state.hasColumnFamilyAccess(keyspace(), columnFamily(), Permission.SELECT);
+ state.hasColumnFamilyAccess(cfm, Permission.SELECT);
}
for (Function function : getFunctions())
@@ -263,9 +263,9 @@
this.pager = pager;
}
- public static Pager forInternalQuery(QueryPager pager, ReadOrderGroup orderGroup)
+ public static Pager forInternalQuery(QueryPager pager, ReadExecutionController executionController)
{
- return new InternalPager(pager, orderGroup);
+ return new InternalPager(pager, executionController);
}
public static Pager forDistributedQuery(QueryPager pager, ConsistencyLevel consistency, ClientState clientState)
@@ -305,17 +305,17 @@
public static class InternalPager extends Pager
{
- private final ReadOrderGroup orderGroup;
+ private final ReadExecutionController executionController;
- private InternalPager(QueryPager pager, ReadOrderGroup orderGroup)
+ private InternalPager(QueryPager pager, ReadExecutionController executionController)
{
super(pager);
- this.orderGroup = orderGroup;
+ this.executionController = executionController;
}
public PartitionIterator fetchPage(int pageSize)
{
- return pager.fetchPageInternal(pageSize, orderGroup);
+ return pager.fetchPageInternal(pageSize, executionController);
}
}
}
@@ -395,11 +395,11 @@
ReadQuery query = getQuery(options, nowInSec, userLimit);
int pageSize = getPageSize(options);
- try (ReadOrderGroup orderGroup = query.startOrderGroup())
+ try (ReadExecutionController executionController = query.executionController())
{
if (pageSize <= 0 || query.limits().count() <= pageSize)
{
- try (PartitionIterator data = query.executeInternal(orderGroup))
+ try (PartitionIterator data = query.executeInternal(executionController))
{
return processResults(data, options, nowInSec, userLimit);
}
@@ -407,7 +407,7 @@
else
{
QueryPager pager = query.getPager(options.getPagingState(), options.getProtocolVersion());
- return execute(Pager.forInternalQuery(pager, orderGroup), options, pageSize, nowInSec, userLimit);
+ return execute(Pager.forInternalQuery(pager, executionController), options, pageSize, nowInSec, userLimit);
}
}
}
diff --git a/src/java/org/apache/cassandra/db/BlacklistedDirectories.java b/src/java/org/apache/cassandra/db/BlacklistedDirectories.java
index f47fd57..3e6332c 100644
--- a/src/java/org/apache/cassandra/db/BlacklistedDirectories.java
+++ b/src/java/org/apache/cassandra/db/BlacklistedDirectories.java
@@ -66,6 +66,16 @@
return Collections.unmodifiableSet(unwritableDirectories);
}
+ public void markUnreadable(String path)
+ {
+ maybeMarkUnreadable(new File(path));
+ }
+
+ public void markUnwritable(String path)
+ {
+ maybeMarkUnwritable(new File(path));
+ }
+
/**
* Adds parent directory of the file (or the file itself, if it is a directory)
* to the set of unreadable directories.
diff --git a/src/java/org/apache/cassandra/db/BlacklistedDirectoriesMBean.java b/src/java/org/apache/cassandra/db/BlacklistedDirectoriesMBean.java
index 3163b9a..3fb9f39 100644
--- a/src/java/org/apache/cassandra/db/BlacklistedDirectoriesMBean.java
+++ b/src/java/org/apache/cassandra/db/BlacklistedDirectoriesMBean.java
@@ -20,10 +20,13 @@
import java.io.File;
import java.util.Set;
-public interface BlacklistedDirectoriesMBean {
-
+public interface BlacklistedDirectoriesMBean
+{
public Set<File> getUnreadableDirectories();
public Set<File> getUnwritableDirectories();
-
+
+ public void markUnreadable(String path);
+
+ public void markUnwritable(String path);
}
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 3d7be4e..e98a692 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -118,13 +118,27 @@
private static final Logger logger = LoggerFactory.getLogger(ColumnFamilyStore.class);
- private static final ExecutorService flushExecutor = new JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+ private static final ExecutorService flushExecutor = new JMXEnabledThreadPoolExecutor(1,
StageManager.KEEPALIVE,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(),
new NamedThreadFactory("MemtableFlushWriter"),
"internal");
+ private static final ExecutorService [] perDiskflushExecutors = new ExecutorService[DatabaseDescriptor.getAllDataFileLocations().length];
+ static
+ {
+ for (int i = 0; i < DatabaseDescriptor.getAllDataFileLocations().length; i++)
+ {
+ perDiskflushExecutors[i] = new JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+ StageManager.KEEPALIVE,
+ TimeUnit.SECONDS,
+ new LinkedBlockingQueue<Runnable>(),
+ new NamedThreadFactory("PerDiskMemtableFlushWriter_"+i),
+ "internal");
+ }
+ }
+
// post-flush executor is single threaded to provide guarantee that any flush Future on a CF will never return until prior flushes have completed
private static final ExecutorService postFlushExecutor = new JMXEnabledThreadPoolExecutor(1,
StageManager.KEEPALIVE,
@@ -458,7 +472,10 @@
public Directories getDirectories()
{
- return directories;
+ // todo, hack since we need to know the data directories when constructing the compaction strategy
+ if (directories != null)
+ return directories;
+ return new Directories(metadata, initialDirectories);
}
public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, int sstableLevel, SerializationHeader header, LifecycleTransaction txn)
@@ -469,7 +486,7 @@
public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, MetadataCollector metadataCollector, SerializationHeader header, LifecycleTransaction txn)
{
- return getCompactionStrategyManager().createSSTableMultiWriter(descriptor, keyCount, repairedAt, metadataCollector, header, txn);
+ return getCompactionStrategyManager().createSSTableMultiWriter(descriptor, keyCount, repairedAt, metadataCollector, header, indexManager.listIndexes(), txn);
}
/** call when dropping or renaming a CF. Performs mbean housekeeping and invalidates CFS to other operations */
@@ -1033,11 +1050,74 @@
for (Memtable memtable : memtables)
{
- // flush the memtable
- MoreExecutors.sameThreadExecutor().execute(memtable.flushRunnable());
- reclaim(memtable);
- }
+ List<Future<SSTableMultiWriter>> futures = new ArrayList<>();
+ long totalBytesOnDisk = 0;
+ long maxBytesOnDisk = 0;
+ long minBytesOnDisk = Long.MAX_VALUE;
+ List<SSTableReader> sstables = new ArrayList<>();
+ try (LifecycleTransaction txn = LifecycleTransaction.offline(OperationType.FLUSH))
+ {
+ // flush the memtable
+ List<Memtable.FlushRunnable> flushRunnables = memtable.flushRunnables(txn);
+ for (int i = 0; i < flushRunnables.size(); i++)
+ futures.add(perDiskflushExecutors[i].submit(flushRunnables.get(i)));
+
+ List<SSTableMultiWriter> flushResults = Lists.newArrayList(FBUtilities.waitOnFutures(futures));
+
+ try
+ {
+ Iterator<SSTableMultiWriter> writerIterator = flushResults.iterator();
+ while (writerIterator.hasNext())
+ {
+ SSTableMultiWriter writer = writerIterator.next();
+ if (writer.getFilePointer() > 0)
+ {
+ writer.setOpenResult(true).prepareToCommit();
+ }
+ else
+ {
+ maybeFail(writer.abort(null));
+ writerIterator.remove();
+ }
+ }
+ }
+ catch (Throwable t)
+ {
+ for (SSTableMultiWriter writer : flushResults)
+ t = writer.abort(t);
+ t = txn.abort(t);
+ Throwables.propagate(t);
+ }
+
+ txn.prepareToCommit();
+
+ Throwable accumulate = null;
+ for (SSTableMultiWriter writer : flushResults)
+ accumulate = writer.commit(accumulate);
+
+ maybeFail(txn.commit(accumulate));
+
+ for (SSTableMultiWriter writer : flushResults)
+ {
+ Collection<SSTableReader> flushedSSTables = writer.finished();
+ for (SSTableReader sstable : flushedSSTables)
+ {
+ if (sstable != null)
+ {
+ sstables.add(sstable);
+ long size = sstable.bytesOnDisk();
+ totalBytesOnDisk += size;
+ maxBytesOnDisk = Math.max(maxBytesOnDisk, size);
+ minBytesOnDisk = Math.min(minBytesOnDisk, size);
+ }
+ }
+ }
+ }
+ memtable.cfs.replaceFlushed(memtable, sstables);
+ reclaim(memtable);
+ logger.debug("Flushed to {} ({} sstables, {} bytes), biggest {} bytes, smallest {} bytes", sstables, sstables.size(), totalBytesOnDisk, maxBytesOnDisk, minBytesOnDisk);
+ }
// signal the post-flush we've done our work
postFlush.latch.countDown();
}
@@ -1366,6 +1446,11 @@
return CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion);
}
+ public CompactionManager.AllSSTableOpStatus relocateSSTables() throws ExecutionException, InterruptedException
+ {
+ return CompactionManager.instance.relocateSSTables(this);
+ }
+
public void markObsolete(Collection<SSTableReader> sstables, OperationType compactionType)
{
assert !sstables.isEmpty();
@@ -1495,12 +1580,14 @@
TabularDataSupport result = new TabularDataSupport(COUNTER_TYPE);
for (Counter<ByteBuffer> counter : samplerResults.topK)
{
- byte[] key = counter.getItem().array();
+ //Not duplicating the buffer for safety because AbstractSerializer and ByteBufferUtil.bytesToHex
+ //don't modify position or limit
+ ByteBuffer key = counter.getItem();
result.put(new CompositeDataSupport(COUNTER_COMPOSITE_TYPE, COUNTER_NAMES, new Object[] {
- Hex.bytesToHex(key), // raw
+ ByteBufferUtil.bytesToHex(key), // raw
counter.getCount(), // count
counter.getError(), // error
- metadata.getKeyValidator().getString(ByteBuffer.wrap(key)) })); // string
+ metadata.getKeyValidator().getString(key) })); // string
}
return new CompositeDataSupport(SAMPLING_RESULT, SAMPLER_NAMES, new Object[]{
samplerResults.cardinality, result});
@@ -2204,7 +2291,7 @@
public int getUnleveledSSTables()
{
- return this.compactionStrategyManager.getUnleveledSSTables();
+ return compactionStrategyManager.getUnleveledSSTables();
}
public int[] getSSTableCountPerLevel()
diff --git a/src/java/org/apache/cassandra/db/ColumnIndex.java b/src/java/org/apache/cassandra/db/ColumnIndex.java
index ede3f79..749c155 100644
--- a/src/java/org/apache/cassandra/db/ColumnIndex.java
+++ b/src/java/org/apache/cassandra/db/ColumnIndex.java
@@ -25,6 +25,7 @@
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.io.sstable.IndexHelper;
+import org.apache.cassandra.io.sstable.format.SSTableFlushObserver;
import org.apache.cassandra.io.sstable.format.Version;
import org.apache.cassandra.io.util.SequentialWriter;
import org.apache.cassandra.utils.ByteBufferUtil;
@@ -44,11 +45,15 @@
this.columnsIndex = columnsIndex;
}
- public static ColumnIndex writeAndBuildIndex(UnfilteredRowIterator iterator, SequentialWriter output, SerializationHeader header, Version version) throws IOException
+ public static ColumnIndex writeAndBuildIndex(UnfilteredRowIterator iterator,
+ SequentialWriter output,
+ SerializationHeader header,
+ Collection<SSTableFlushObserver> observers,
+ Version version) throws IOException
{
assert !iterator.isEmpty() && version.storeRows();
- Builder builder = new Builder(iterator, output, header, version.correspondingMessagingVersion());
+ Builder builder = new Builder(iterator, output, header, observers, version.correspondingMessagingVersion());
return builder.build();
}
@@ -83,15 +88,19 @@
private DeletionTime openMarker;
+ private final Collection<SSTableFlushObserver> observers;
+
public Builder(UnfilteredRowIterator iterator,
SequentialWriter writer,
SerializationHeader header,
+ Collection<SSTableFlushObserver> observers,
int version)
{
this.iterator = iterator;
this.writer = writer;
this.header = header;
this.version = version;
+ this.observers = observers == null ? Collections.emptyList() : observers;
this.initialPosition = writer.position();
}
@@ -142,6 +151,11 @@
}
UnfilteredSerializer.serializer.serialize(unfiltered, header, writer, pos - previousRowStart, version);
+
+ // notify observers about each new cell added to the row
+ if (!observers.isEmpty() && unfiltered.isRow())
+ ((Row) unfiltered).stream().forEach(cell -> observers.forEach((o) -> o.nextCell(cell)));
+
lastClustering = unfiltered.clustering();
previousRowStart = pos;
++written;
diff --git a/src/java/org/apache/cassandra/db/Columns.java b/src/java/org/apache/cassandra/db/Columns.java
index cad295c..e3c30fa 100644
--- a/src/java/org/apache/cassandra/db/Columns.java
+++ b/src/java/org/apache/cassandra/db/Columns.java
@@ -38,6 +38,7 @@
import org.apache.cassandra.utils.SearchIterator;
import org.apache.cassandra.utils.btree.BTree;
import org.apache.cassandra.utils.btree.BTreeSearchIterator;
+import org.apache.cassandra.utils.btree.BTreeRemoval;
import org.apache.cassandra.utils.btree.UpdateFunction;
/**
@@ -343,7 +344,7 @@
if (!contains(column))
return this;
- Object[] newColumns = BTree.<ColumnDefinition>transformAndFilter(columns, (c) -> c.equals(column) ? null : c);
+ Object[] newColumns = BTreeRemoval.<ColumnDefinition>remove(columns, Comparator.naturalOrder(), column);
return new Columns(newColumns);
}
diff --git a/src/java/org/apache/cassandra/db/CompactTables.java b/src/java/org/apache/cassandra/db/CompactTables.java
index a72e7f2..e3f3b16 100644
--- a/src/java/org/apache/cassandra/db/CompactTables.java
+++ b/src/java/org/apache/cassandra/db/CompactTables.java
@@ -56,6 +56,7 @@
*
* As far as thrift is concerned, one exception to this is super column families, which have a different layout. Namely, a super
* column families is encoded with:
+ * {@code
* CREATE TABLE super (
* key [key_validation_class],
* super_column_name [comparator],
@@ -65,6 +66,7 @@
* "" map<[sub_comparator], [default_validation_class]>
* PRIMARY KEY (key, super_column_name)
* )
+ * }
* In other words, every super column is encoded by a row. That row has one column for each defined "column_metadata", but it also
* has a special map column (whose name is the empty string as this is guaranteed to never conflict with a user-defined
* "column_metadata") which stores the super column "dynamic" sub-columns.
diff --git a/src/java/org/apache/cassandra/db/CounterMutation.java b/src/java/org/apache/cassandra/db/CounterMutation.java
index 6818513..87ca71a 100644
--- a/src/java/org/apache/cassandra/db/CounterMutation.java
+++ b/src/java/org/apache/cassandra/db/CounterMutation.java
@@ -110,7 +110,7 @@
*
* @return the applied resulting Mutation
*/
- public Mutation apply() throws WriteTimeoutException
+ public Mutation applyCounterMutation() throws WriteTimeoutException
{
Mutation result = new Mutation(getKeyspaceName(), key());
Keyspace keyspace = Keyspace.open(getKeyspaceName());
@@ -132,6 +132,11 @@
}
}
+ public void apply()
+ {
+ applyCounterMutation();
+ }
+
private void grabCounterLocks(Keyspace keyspace, List<Lock> locks) throws WriteTimeoutException
{
long startTime = System.nanoTime();
diff --git a/src/java/org/apache/cassandra/db/Directories.java b/src/java/org/apache/cassandra/db/Directories.java
index 8744d43..a572bed 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -49,6 +49,7 @@
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.io.FSError;
import org.apache.cassandra.io.FSWriteError;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.io.sstable.*;
import org.apache.cassandra.utils.ByteBufferUtil;
@@ -289,6 +290,19 @@
return null;
}
+ public DataDirectory getDataDirectoryForFile(File directory)
+ {
+ if (directory != null)
+ {
+ for (DataDirectory dataDirectory : dataDirectories)
+ {
+ if (directory.getAbsolutePath().startsWith(dataDirectory.location.getAbsolutePath()))
+ return dataDirectory;
+ }
+ }
+ return null;
+ }
+
public Descriptor find(String filename)
{
for (File dir : dataPaths)
@@ -403,7 +417,7 @@
for (DataDirectory dataDir : paths)
{
if (BlacklistedDirectories.isUnwritable(getLocationForDisk(dataDir)))
- continue;
+ continue;
DataDirectoryCandidate candidate = new DataDirectoryCandidate(dataDir);
// exclude directory if its total writeSize does not fit to data directory
if (candidate.availableSpace < writeSize)
@@ -413,6 +427,26 @@
return totalAvailable > expectedTotalWriteSize;
}
+ public DataDirectory[] getWriteableLocations()
+ {
+ List<DataDirectory> nonBlacklistedDirs = new ArrayList<>();
+ for (DataDirectory dir : dataDirectories)
+ {
+ if (!BlacklistedDirectories.isUnwritable(dir.location))
+ nonBlacklistedDirs.add(dir);
+ }
+
+ Collections.sort(nonBlacklistedDirs, new Comparator<DataDirectory>()
+ {
+ @Override
+ public int compare(DataDirectory o1, DataDirectory o2)
+ {
+ return o1.location.compareTo(o2.location);
+ }
+ });
+ return nonBlacklistedDirs.toArray(new DataDirectory[nonBlacklistedDirs.size()]);
+ }
+
public static File getSnapshotDirectory(Descriptor desc, String snapshotName)
{
return getSnapshotDirectory(desc.directory, snapshotName);
diff --git a/src/java/org/apache/cassandra/db/IMutation.java b/src/java/org/apache/cassandra/db/IMutation.java
index aad35c3..c734e16 100644
--- a/src/java/org/apache/cassandra/db/IMutation.java
+++ b/src/java/org/apache/cassandra/db/IMutation.java
@@ -17,7 +17,6 @@
*/
package org.apache.cassandra.db;
-import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.UUID;
@@ -25,6 +24,7 @@
public interface IMutation
{
+ public void apply();
public String getKeyspaceName();
public Collection<UUID> getColumnFamilyIds();
public DecoratedKey key();
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java b/src/java/org/apache/cassandra/db/LegacyLayout.java
index b90151e..c032f89 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -695,16 +695,18 @@
protected LegacyCell computeNext()
{
- if (currentRow.hasNext())
- return currentRow.next();
+ while (true)
+ {
+ if (currentRow.hasNext())
+ return currentRow.next();
- if (!iterator.hasNext())
- return endOfData();
+ if (!iterator.hasNext())
+ return endOfData();
- Pair<LegacyRangeTombstoneList, Iterator<LegacyCell>> row = fromRow(metadata, iterator.next());
- deletions.addAll(row.left);
- currentRow = row.right;
- return computeNext();
+ Pair<LegacyRangeTombstoneList, Iterator<LegacyCell>> row = fromRow(metadata, iterator.next());
+ deletions.addAll(row.left);
+ currentRow = row.right;
+ }
}
};
@@ -1770,7 +1772,7 @@
/**
* Almost an entire copy of RangeTombstoneList from C* 2.1. The main difference is that LegacyBoundComparator
- * is used in place of Comparator<Composite> (because Composite doesn't exist any more).
+ * is used in place of {@code Comparator<Composite>} (because Composite doesn't exist any more).
*
* This class is needed to allow us to convert single-row deletions and complex deletions into range tombstones
* and properly merge them into the normal set of range tombstones.
diff --git a/src/java/org/apache/cassandra/db/Memtable.java b/src/java/org/apache/cassandra/db/Memtable.java
index 5d5f7bf..952c045 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -17,7 +17,6 @@
*/
package org.apache.cassandra.db;
-import java.io.File;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;
@@ -45,14 +44,14 @@
import org.apache.cassandra.dht.Murmur3Partitioner.LongToken;
import org.apache.cassandra.index.transactions.UpdateTransaction;
import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.SSTableMultiWriter;
import org.apache.cassandra.io.sstable.SSTableTxnWriter;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
-import org.apache.cassandra.io.util.DiskAwareRunnable;
import org.apache.cassandra.service.ActiveRepairService;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.ObjectSizes;
+import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.MemtableAllocator;
import org.apache.cassandra.utils.memory.MemtablePool;
@@ -254,9 +253,32 @@
return partitions.size();
}
- public FlushRunnable flushRunnable()
+ public List<FlushRunnable> flushRunnables(LifecycleTransaction txn)
{
- return new FlushRunnable(lastReplayPosition.get());
+ List<Range<Token>> localRanges = Range.sort(StorageService.instance.getLocalRanges(cfs.keyspace.getName()));
+
+ if (!cfs.getPartitioner().splitter().isPresent() || localRanges.isEmpty())
+ return Collections.singletonList(new FlushRunnable(lastReplayPosition.get(), txn));
+
+ return createFlushRunnables(localRanges, txn);
+ }
+
+ private List<FlushRunnable> createFlushRunnables(List<Range<Token>> localRanges, LifecycleTransaction txn)
+ {
+ assert cfs.getPartitioner().splitter().isPresent();
+
+ Directories.DataDirectory[] locations = cfs.getDirectories().getWriteableLocations();
+ List<PartitionPosition> boundaries = StorageService.getDiskBoundaries(localRanges, cfs.getPartitioner(), locations);
+ List<FlushRunnable> runnables = new ArrayList<>(boundaries.size());
+ PartitionPosition rangeStart = cfs.getPartitioner().getMinimumToken().minKeyBound();
+ ReplayPosition context = lastReplayPosition.get();
+ for (int i = 0; i < boundaries.size(); i++)
+ {
+ PartitionPosition t = boundaries.get(i);
+ runnables.add(new FlushRunnable(context, rangeStart, t, locations[i], txn));
+ rangeStart = t;
+ }
+ return runnables;
}
public String toString()
@@ -321,23 +343,41 @@
return minTimestamp;
}
- class FlushRunnable extends DiskAwareRunnable
+ class FlushRunnable implements Callable<SSTableMultiWriter>
{
- private final ReplayPosition context;
+ public final ReplayPosition context;
private final long estimatedSize;
+ private final ConcurrentNavigableMap<PartitionPosition, AtomicBTreePartition> toFlush;
private final boolean isBatchLogTable;
+ private final SSTableMultiWriter writer;
- FlushRunnable(ReplayPosition context)
+ // keeping these to be able to log what we are actually flushing
+ private final PartitionPosition from;
+ private final PartitionPosition to;
+
+ FlushRunnable(ReplayPosition context, PartitionPosition from, PartitionPosition to, Directories.DataDirectory flushLocation, LifecycleTransaction txn)
+ {
+ this(context, partitions.subMap(from, to), flushLocation, from, to, txn);
+ }
+
+ FlushRunnable(ReplayPosition context, LifecycleTransaction txn)
+ {
+ this(context, partitions, null, null, null, txn);
+ }
+
+ FlushRunnable(ReplayPosition context, ConcurrentNavigableMap<PartitionPosition, AtomicBTreePartition> toFlush, Directories.DataDirectory flushLocation, PartitionPosition from, PartitionPosition to, LifecycleTransaction txn)
{
this.context = context;
-
+ this.toFlush = toFlush;
+ this.from = from;
+ this.to = to;
long keySize = 0;
- for (PartitionPosition key : partitions.keySet())
+ for (PartitionPosition key : toFlush.keySet())
{
// make sure we don't write non-sensical keys
assert key instanceof DecoratedKey;
- keySize += ((DecoratedKey)key).getKey().remaining();
+ keySize += ((DecoratedKey) key).getKey().remaining();
}
estimatedSize = (long) ((keySize // index entries
+ keySize // keys in data file
@@ -345,21 +385,12 @@
* 1.2); // bloom filter and row index overhead
this.isBatchLogTable = cfs.name.equals(SystemKeyspace.BATCHES) && cfs.keyspace.getName().equals(SystemKeyspace.NAME);
- }
- public long getExpectedWriteSize()
- {
- return estimatedSize;
- }
+ if (flushLocation == null)
+ writer = createFlushWriter(txn, cfs.getSSTablePath(getDirectories().getLocationForDisk(getDirectories().getWriteableLocation(estimatedSize))), columnsCollector.get(), statsCollector.get());
+ else
+ writer = createFlushWriter(txn, cfs.getSSTablePath(getDirectories().getLocationForDisk(flushLocation)), columnsCollector.get(), statsCollector.get());
- protected void runMayThrow() throws Exception
- {
- long writeSize = getExpectedWriteSize();
- Directories.DataDirectory dataDirectory = getWriteDirectory(writeSize);
- File sstableDirectory = cfs.getDirectories().getLocationForDisk(dataDirectory);
- assert sstableDirectory != null : "Flush task is not bound to any disk";
- Collection<SSTableReader> sstables = writeSortedContents(context, sstableDirectory);
- cfs.replaceFlushed(Memtable.this, sstables);
}
protected Directories getDirectories()
@@ -367,90 +398,64 @@
return cfs.getDirectories();
}
- private Collection<SSTableReader> writeSortedContents(ReplayPosition context, File sstableDirectory)
+ private void writeSortedContents(ReplayPosition context)
{
- logger.debug("Writing {}", Memtable.this.toString());
+ logger.debug("Writing {}, flushed range = ({}, {}]", Memtable.this.toString(), from, to);
- Collection<SSTableReader> ssTables;
- try (SSTableTxnWriter writer = createFlushWriter(cfs.getSSTablePath(sstableDirectory), columnsCollector.get(), statsCollector.get()))
+ boolean trackContention = logger.isTraceEnabled();
+ int heavilyContendedRowCount = 0;
+ // (we can't clear out the map as-we-go to free up memory,
+ // since the memtable is being used for queries in the "pending flush" category)
+ for (AtomicBTreePartition partition : toFlush.values())
{
- boolean trackContention = logger.isTraceEnabled();
- int heavilyContendedRowCount = 0;
- // (we can't clear out the map as-we-go to free up memory,
- // since the memtable is being used for queries in the "pending flush" category)
- for (AtomicBTreePartition partition : partitions.values())
+ // Each batchlog partition is a separate entry in the log. And for an entry, we only do 2
+ // operations: 1) we insert the entry and 2) we delete it. Further, BL data is strictly local,
+ // we don't need to preserve tombstones for repair. So if both operation are in this
+ // memtable (which will almost always be the case if there is no ongoing failure), we can
+ // just skip the entry (CASSANDRA-4667).
+ if (isBatchLogTable && !partition.partitionLevelDeletion().isLive() && partition.hasRows())
+ continue;
+
+ if (trackContention && partition.usePessimisticLocking())
+ heavilyContendedRowCount++;
+
+ if (!partition.isEmpty())
{
- // Each batchlog partition is a separate entry in the log. And for an entry, we only do 2
- // operations: 1) we insert the entry and 2) we delete it. Further, BL data is strictly local,
- // we don't need to preserve tombstones for repair. So if both operation are in this
- // memtable (which will almost always be the case if there is no ongoing failure), we can
- // just skip the entry (CASSANDRA-4667).
- if (isBatchLogTable && !partition.partitionLevelDeletion().isLive() && partition.hasRows())
- continue;
-
- if (trackContention && partition.usePessimisticLocking())
- heavilyContendedRowCount++;
-
- if (!partition.isEmpty())
+ try (UnfilteredRowIterator iter = partition.unfilteredIterator())
{
- try (UnfilteredRowIterator iter = partition.unfilteredIterator())
- {
- writer.append(iter);
- }
+ writer.append(iter);
}
}
-
- if (writer.getFilePointer() > 0)
- {
- logger.debug(String.format("Completed flushing %s (%s) for commitlog position %s",
- writer.getFilename(),
- FBUtilities.prettyPrintMemory(writer.getFilePointer()),
- context));
-
- // sstables should contain non-repaired data.
- ssTables = writer.finish(true);
- }
- else
- {
- logger.debug("Completed flushing {}; nothing needed to be retained. Commitlog position was {}",
- writer.getFilename(), context);
- writer.abort();
- ssTables = null;
- }
-
- if (heavilyContendedRowCount > 0)
- logger.trace(String.format("High update contention in %d/%d partitions of %s ", heavilyContendedRowCount, partitions.size(), Memtable.this.toString()));
-
- return ssTables;
}
+
+ logger.debug(String.format("Completed flushing %s (%s) for commitlog position %s",
+ writer.getFilename(),
+ FBUtilities.prettyPrintMemory(writer.getFilePointer()),
+ context));
+
+ if (heavilyContendedRowCount > 0)
+ logger.trace(String.format("High update contention in %d/%d partitions of %s ", heavilyContendedRowCount, toFlush.size(), Memtable.this.toString()));
}
- @SuppressWarnings("resource") // log and writer closed by SSTableTxnWriter
- public SSTableTxnWriter createFlushWriter(String filename,
- PartitionColumns columns,
- EncodingStats stats)
+ public SSTableMultiWriter createFlushWriter(LifecycleTransaction txn,
+ String filename,
+ PartitionColumns columns,
+ EncodingStats stats)
{
- // we operate "offline" here, as we expose the resulting reader consciously when done
- // (although we may want to modify this behaviour in future, to encapsulate full flush behaviour in LifecycleTransaction)
- LifecycleTransaction txn = null;
- try
- {
- txn = LifecycleTransaction.offline(OperationType.FLUSH);
- MetadataCollector sstableMetadataCollector = new MetadataCollector(cfs.metadata.comparator).replayPosition(context);
- return new SSTableTxnWriter(txn,
- cfs.createSSTableMultiWriter(Descriptor.fromFilename(filename),
- (long) partitions.size(),
- ActiveRepairService.UNREPAIRED_SSTABLE,
- sstableMetadataCollector,
- new SerializationHeader(true, cfs.metadata, columns, stats),
- txn));
- }
- catch (Throwable t)
- {
- if (txn != null)
- txn.close();
- throw t;
- }
+ MetadataCollector sstableMetadataCollector = new MetadataCollector(cfs.metadata.comparator).replayPosition(context);
+ return cfs.createSSTableMultiWriter(Descriptor.fromFilename(filename),
+ (long)toFlush.size(),
+ ActiveRepairService.UNREPAIRED_SSTABLE,
+ sstableMetadataCollector,
+ new SerializationHeader(true, cfs.metadata, columns, stats), txn);
+
+ }
+
+ @Override
+ public SSTableMultiWriter call()
+ {
+ writeSortedContents(context);
+ return writer;
}
}
diff --git a/src/java/org/apache/cassandra/db/MultiCBuilder.java b/src/java/org/apache/cassandra/db/MultiCBuilder.java
index be654fa..8353703 100644
--- a/src/java/org/apache/cassandra/db/MultiCBuilder.java
+++ b/src/java/org/apache/cassandra/db/MultiCBuilder.java
@@ -26,44 +26,39 @@
/**
* Builder that allow to build multiple Clustering/Slice.Bound at the same time.
*/
-public class MultiCBuilder
+public abstract class MultiCBuilder
{
/**
* The table comparator.
*/
- private final ClusteringComparator comparator;
+ protected final ClusteringComparator comparator;
/**
- * The elements of the clusterings
+ * The number of clustering elements that have been added.
*/
- private final List<List<ByteBuffer>> elementsList = new ArrayList<>();
-
- /**
- * The number of elements that have been added.
- */
- private int size;
+ protected int size;
/**
* <code>true</code> if the clusterings have been build, <code>false</code> otherwise.
*/
- private boolean built;
+ protected boolean built;
/**
* <code>true</code> if the clusterings contains some <code>null</code> elements.
*/
- private boolean containsNull;
+ protected boolean containsNull;
/**
* <code>true</code> if the composites contains some <code>unset</code> elements.
*/
- private boolean containsUnset;
+ protected boolean containsUnset;
/**
* <code>true</code> if some empty collection have been added.
*/
- private boolean hasMissingElements;
+ protected boolean hasMissingElements;
- private MultiCBuilder(ClusteringComparator comparator)
+ protected MultiCBuilder(ClusteringComparator comparator)
{
this.comparator = comparator;
}
@@ -71,19 +66,11 @@
/**
* Creates a new empty {@code MultiCBuilder}.
*/
- public static MultiCBuilder create(ClusteringComparator comparator)
+ public static MultiCBuilder create(ClusteringComparator comparator, boolean forMultipleValues)
{
- return new MultiCBuilder(comparator);
- }
-
- /**
- * Checks if this builder is empty.
- *
- * @return <code>true</code> if this builder is empty, <code>false</code> otherwise.
- */
- private boolean isEmpty()
- {
- return elementsList.isEmpty();
+ return forMultipleValues
+ ? new MultiClusteringBuilder(comparator)
+ : new OneClusteringBuilder(comparator);
}
/**
@@ -96,25 +83,7 @@
* @param value the value of the next element
* @return this <code>MulitCBuilder</code>
*/
- public MultiCBuilder addElementToAll(ByteBuffer value)
- {
- checkUpdateable();
-
- if (isEmpty())
- elementsList.add(new ArrayList<ByteBuffer>());
-
- for (int i = 0, m = elementsList.size(); i < m; i++)
- {
- if (value == null)
- containsNull = true;
- if (value == ByteBufferUtil.UNSET_BYTE_BUFFER)
- containsUnset = true;
-
- elementsList.get(i).add(value);
- }
- size++;
- return this;
- }
+ public abstract MultiCBuilder addElementToAll(ByteBuffer value);
/**
* Adds individually each of the specified elements to the end of all of the existing clusterings.
@@ -126,42 +95,7 @@
* @param values the elements to add
* @return this <code>CompositeBuilder</code>
*/
- public MultiCBuilder addEachElementToAll(List<ByteBuffer> values)
- {
- checkUpdateable();
-
- if (isEmpty())
- elementsList.add(new ArrayList<ByteBuffer>());
-
- if (values.isEmpty())
- {
- hasMissingElements = true;
- }
- else
- {
- for (int i = 0, m = elementsList.size(); i < m; i++)
- {
- List<ByteBuffer> oldComposite = elementsList.remove(0);
-
- for (int j = 0, n = values.size(); j < n; j++)
- {
- List<ByteBuffer> newComposite = new ArrayList<>(oldComposite);
- elementsList.add(newComposite);
-
- ByteBuffer value = values.get(j);
-
- if (value == null)
- containsNull = true;
- if (value == ByteBufferUtil.UNSET_BYTE_BUFFER)
- containsUnset = true;
-
- newComposite.add(values.get(j));
- }
- }
- }
- size++;
- return this;
- }
+ public abstract MultiCBuilder addEachElementToAll(List<ByteBuffer> values);
/**
* Adds individually each of the specified list of elements to the end of all of the existing composites.
@@ -173,44 +107,12 @@
* @param values the elements to add
* @return this <code>CompositeBuilder</code>
*/
- public MultiCBuilder addAllElementsToAll(List<List<ByteBuffer>> values)
+ public abstract MultiCBuilder addAllElementsToAll(List<List<ByteBuffer>> values);
+
+ protected void checkUpdateable()
{
- checkUpdateable();
-
- if (isEmpty())
- elementsList.add(new ArrayList<ByteBuffer>());
-
- if (values.isEmpty())
- {
- hasMissingElements = true;
- }
- else
- {
- for (int i = 0, m = elementsList.size(); i < m; i++)
- {
- List<ByteBuffer> oldComposite = elementsList.remove(0);
-
- for (int j = 0, n = values.size(); j < n; j++)
- {
- List<ByteBuffer> newComposite = new ArrayList<>(oldComposite);
- elementsList.add(newComposite);
-
- List<ByteBuffer> value = values.get(j);
-
- if (value.isEmpty())
- hasMissingElements = true;
-
- if (value.contains(null))
- containsNull = true;
- if (value.contains(ByteBufferUtil.UNSET_BYTE_BUFFER))
- containsUnset = true;
-
- newComposite.addAll(value);
- }
- }
- size += values.get(0).size();
- }
- return this;
+ if (!hasRemaining() || built)
+ throw new IllegalStateException("this builder cannot be updated anymore");
}
/**
@@ -257,54 +159,14 @@
*
* @return the clusterings
*/
- public NavigableSet<Clustering> build()
- {
- built = true;
-
- if (hasMissingElements)
- return BTreeSet.empty(comparator);
-
- CBuilder builder = CBuilder.create(comparator);
-
- if (elementsList.isEmpty())
- return BTreeSet.of(builder.comparator(), builder.build());
-
- BTreeSet.Builder<Clustering> set = BTreeSet.builder(builder.comparator());
- for (int i = 0, m = elementsList.size(); i < m; i++)
- {
- List<ByteBuffer> elements = elementsList.get(i);
- set.add(builder.buildWith(elements));
- }
- return set.build();
- }
+ public abstract NavigableSet<Clustering> build();
/**
* Builds the <code>clusterings</code> with the specified EOC.
*
* @return the clusterings
*/
- public NavigableSet<Slice.Bound> buildBound(boolean isStart, boolean isInclusive)
- {
- built = true;
-
- if (hasMissingElements)
- return BTreeSet.empty(comparator);
-
- CBuilder builder = CBuilder.create(comparator);
-
- if (elementsList.isEmpty())
- return BTreeSet.of(comparator, builder.buildBound(isStart, isInclusive));
-
- // Use a TreeSet to sort and eliminate duplicates
- BTreeSet.Builder<Slice.Bound> set = BTreeSet.builder(comparator);
-
- for (int i = 0, m = elementsList.size(); i < m; i++)
- {
- List<ByteBuffer> elements = elementsList.get(i);
- set.add(builder.buildBoundWith(elements, isStart, isInclusive));
- }
- return set.build();
- }
+ public abstract NavigableSet<Slice.Bound> buildBound(boolean isStart, boolean isInclusive);
/**
* Checks if some elements can still be added to the clusterings.
@@ -316,9 +178,241 @@
return remainingCount() > 0;
}
- private void checkUpdateable()
+ /**
+ * Specialization of MultiCBuilder when we know only one clustering/bound is created.
+ */
+ private static class OneClusteringBuilder extends MultiCBuilder
{
- if (!hasRemaining() || built)
- throw new IllegalStateException("this builder cannot be updated anymore");
+ /**
+ * The elements of the clusterings
+ */
+ private final ByteBuffer[] elements;
+
+ public OneClusteringBuilder(ClusteringComparator comparator)
+ {
+ super(comparator);
+ this.elements = new ByteBuffer[comparator.size()];
+ }
+
+ public MultiCBuilder addElementToAll(ByteBuffer value)
+ {
+ checkUpdateable();
+
+ if (value == null)
+ containsNull = true;
+ if (value == ByteBufferUtil.UNSET_BYTE_BUFFER)
+ containsUnset = true;
+
+ elements[size++] = value;
+ return this;
+ }
+
+ public MultiCBuilder addEachElementToAll(List<ByteBuffer> values)
+ {
+ if (values.isEmpty())
+ {
+ hasMissingElements = true;
+ return this;
+ }
+
+ assert values.size() == 1;
+
+ return addElementToAll(values.get(0));
+ }
+
+ public MultiCBuilder addAllElementsToAll(List<List<ByteBuffer>> values)
+ {
+ if (values.isEmpty())
+ {
+ hasMissingElements = true;
+ return this;
+ }
+
+ assert values.size() == 1;
+ return addEachElementToAll(values.get(0));
+ }
+
+ public NavigableSet<Clustering> build()
+ {
+ built = true;
+
+ if (hasMissingElements)
+ return BTreeSet.empty(comparator);
+
+ return BTreeSet.of(comparator, size == 0 ? Clustering.EMPTY : new Clustering(elements));
+ }
+
+ public NavigableSet<Slice.Bound> buildBound(boolean isStart, boolean isInclusive)
+ {
+ built = true;
+
+ if (hasMissingElements)
+ return BTreeSet.empty(comparator);
+
+ if (size == 0)
+ return BTreeSet.of(comparator, isStart ? Slice.Bound.BOTTOM : Slice.Bound.TOP);
+
+ ByteBuffer[] newValues = size == elements.length
+ ? elements
+ : Arrays.copyOf(elements, size);
+
+ return BTreeSet.of(comparator, Slice.Bound.create(Slice.Bound.boundKind(isStart, isInclusive), newValues));
+ }
+ }
+
+ /**
+ * MultiCBuilder implementation actually supporting the creation of multiple clustering/bound.
+ */
+ private static class MultiClusteringBuilder extends MultiCBuilder
+ {
+ /**
+ * The elements of the clusterings
+ */
+ private final List<List<ByteBuffer>> elementsList = new ArrayList<>();
+
+ public MultiClusteringBuilder(ClusteringComparator comparator)
+ {
+ super(comparator);
+ }
+
+ public MultiCBuilder addElementToAll(ByteBuffer value)
+ {
+ checkUpdateable();
+
+ if (elementsList.isEmpty())
+ elementsList.add(new ArrayList<ByteBuffer>());
+
+ if (value == null)
+ containsNull = true;
+ else if (value == ByteBufferUtil.UNSET_BYTE_BUFFER)
+ containsUnset = true;
+
+ for (int i = 0, m = elementsList.size(); i < m; i++)
+ elementsList.get(i).add(value);
+
+ size++;
+ return this;
+ }
+
+ public MultiCBuilder addEachElementToAll(List<ByteBuffer> values)
+ {
+ checkUpdateable();
+
+ if (elementsList.isEmpty())
+ elementsList.add(new ArrayList<ByteBuffer>());
+
+ if (values.isEmpty())
+ {
+ hasMissingElements = true;
+ }
+ else
+ {
+ for (int i = 0, m = elementsList.size(); i < m; i++)
+ {
+ List<ByteBuffer> oldComposite = elementsList.remove(0);
+
+ for (int j = 0, n = values.size(); j < n; j++)
+ {
+ List<ByteBuffer> newComposite = new ArrayList<>(oldComposite);
+ elementsList.add(newComposite);
+
+ ByteBuffer value = values.get(j);
+
+ if (value == null)
+ containsNull = true;
+ if (value == ByteBufferUtil.UNSET_BYTE_BUFFER)
+ containsUnset = true;
+
+ newComposite.add(values.get(j));
+ }
+ }
+ }
+ size++;
+ return this;
+ }
+
+ public MultiCBuilder addAllElementsToAll(List<List<ByteBuffer>> values)
+ {
+ checkUpdateable();
+
+ if (elementsList.isEmpty())
+ elementsList.add(new ArrayList<ByteBuffer>());
+
+ if (values.isEmpty())
+ {
+ hasMissingElements = true;
+ }
+ else
+ {
+ for (int i = 0, m = elementsList.size(); i < m; i++)
+ {
+ List<ByteBuffer> oldComposite = elementsList.remove(0);
+
+ for (int j = 0, n = values.size(); j < n; j++)
+ {
+ List<ByteBuffer> newComposite = new ArrayList<>(oldComposite);
+ elementsList.add(newComposite);
+
+ List<ByteBuffer> value = values.get(j);
+
+ if (value.isEmpty())
+ hasMissingElements = true;
+
+ if (value.contains(null))
+ containsNull = true;
+ if (value.contains(ByteBufferUtil.UNSET_BYTE_BUFFER))
+ containsUnset = true;
+
+ newComposite.addAll(value);
+ }
+ }
+ size += values.get(0).size();
+ }
+ return this;
+ }
+
+ public NavigableSet<Clustering> build()
+ {
+ built = true;
+
+ if (hasMissingElements)
+ return BTreeSet.empty(comparator);
+
+ CBuilder builder = CBuilder.create(comparator);
+
+ if (elementsList.isEmpty())
+ return BTreeSet.of(builder.comparator(), builder.build());
+
+ BTreeSet.Builder<Clustering> set = BTreeSet.builder(builder.comparator());
+ for (int i = 0, m = elementsList.size(); i < m; i++)
+ {
+ List<ByteBuffer> elements = elementsList.get(i);
+ set.add(builder.buildWith(elements));
+ }
+ return set.build();
+ }
+
+ public NavigableSet<Slice.Bound> buildBound(boolean isStart, boolean isInclusive)
+ {
+ built = true;
+
+ if (hasMissingElements)
+ return BTreeSet.empty(comparator);
+
+ CBuilder builder = CBuilder.create(comparator);
+
+ if (elementsList.isEmpty())
+ return BTreeSet.of(comparator, builder.buildBound(isStart, isInclusive));
+
+ // Use a TreeSet to sort and eliminate duplicates
+ BTreeSet.Builder<Slice.Bound> set = BTreeSet.builder(comparator);
+
+ for (int i = 0, m = elementsList.size(); i < m; i++)
+ {
+ List<ByteBuffer> elements = elementsList.get(i);
+ set.add(builder.buildBoundWith(elements, isStart, isInclusive));
+ }
+ return set.build();
+ }
}
}
diff --git a/src/java/org/apache/cassandra/db/Mutation.java b/src/java/org/apache/cassandra/db/Mutation.java
index c6ad9b8..4bad781 100644
--- a/src/java/org/apache/cassandra/db/Mutation.java
+++ b/src/java/org/apache/cassandra/db/Mutation.java
@@ -38,15 +38,12 @@
import org.apache.cassandra.net.MessageOut;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.utils.ByteBufferUtil;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
// TODO convert this to a Builder pattern instead of encouraging M.add directly,
// which is less-efficient since we have to keep a mutable HashMap around
public class Mutation implements IMutation
{
public static final MutationSerializer serializer = new MutationSerializer();
- private static final Logger logger = LoggerFactory.getLogger(Mutation.class);
public static final String FORWARD_TO = "FWD_TO";
public static final String FORWARD_FROM = "FWD_FRM";
@@ -272,7 +269,7 @@
buff.append(", modifications=[");
if (shallow)
{
- List<String> cfnames = new ArrayList<String>(modifications.size());
+ List<String> cfnames = new ArrayList<>(modifications.size());
for (UUID cfid : modifications.keySet())
{
CFMetaData cfm = Schema.instance.getCFMetaData(cfid);
@@ -282,7 +279,7 @@
}
else
{
- buff.append("\n ").append(StringUtils.join(modifications.values(), "\n ")).append("\n");
+ buff.append("\n ").append(StringUtils.join(modifications.values(), "\n ")).append('\n');
}
return buff.append("])").toString();
}
diff --git a/src/java/org/apache/cassandra/db/PartitionRangeReadCommand.java b/src/java/org/apache/cassandra/db/PartitionRangeReadCommand.java
index 9fce15e..06ef643 100644
--- a/src/java/org/apache/cassandra/db/PartitionRangeReadCommand.java
+++ b/src/java/org/apache/cassandra/db/PartitionRangeReadCommand.java
@@ -176,7 +176,7 @@
metric.rangeLatency.addNano(latencyNanos);
}
- protected UnfilteredPartitionIterator queryStorage(final ColumnFamilyStore cfs, ReadOrderGroup orderGroup)
+ protected UnfilteredPartitionIterator queryStorage(final ColumnFamilyStore cfs, ReadExecutionController executionController)
{
ColumnFamilyStore.ViewFragment view = cfs.select(View.select(SSTableSet.LIVE, dataRange().keyRange()));
Tracing.trace("Executing seq scan across {} sstables for {}", view.sstables.size(), dataRange().keyRange().getString(metadata().getKeyValidator()));
diff --git a/src/java/org/apache/cassandra/db/RangeTombstoneList.java b/src/java/org/apache/cassandra/db/RangeTombstoneList.java
index c67ea33..baeb227 100644
--- a/src/java/org/apache/cassandra/db/RangeTombstoneList.java
+++ b/src/java/org/apache/cassandra/db/RangeTombstoneList.java
@@ -38,7 +38,7 @@
* A range tombstone has 4 elements: the start and end of the range covered,
* and the deletion infos (markedAt timestamp and local deletion time). The
* markedAt timestamp is what define the priority of 2 overlapping tombstones.
- * That is, given 2 tombstones [0, 10]@t1 and [5, 15]@t2, then if t2 > t1 (and
+ * That is, given 2 tombstones {@code [0, 10]@t1 and [5, 15]@t2, then if t2 > t1} (and
* are the tombstones markedAt values), the 2nd tombstone take precedence over
* the first one on [5, 10]. If such tombstones are added to a RangeTombstoneList,
* the range tombstone list will store them as [[0, 5]@t1, [5, 15]@t2].
@@ -456,7 +456,6 @@
{
if (idx < 0 || idx < finish)
return endOfData();
-
// We want to make sure the range are stricly included within the queried slice as this
// make it easier to combine things when iterator over successive slices. This means that
// for the first and last range we might have to "cut" the range returned.
diff --git a/src/java/org/apache/cassandra/db/ReadCommand.java b/src/java/org/apache/cassandra/db/ReadCommand.java
index 97c3d07..a0b3d55 100644
--- a/src/java/org/apache/cassandra/db/ReadCommand.java
+++ b/src/java/org/apache/cassandra/db/ReadCommand.java
@@ -28,8 +28,10 @@
import org.apache.cassandra.config.*;
import org.apache.cassandra.cql3.Operator;
import org.apache.cassandra.db.filter.*;
+import org.apache.cassandra.db.monitoring.MonitorableImpl;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.db.rows.*;
+import org.apache.cassandra.db.transform.StoppingTransformation;
import org.apache.cassandra.db.transform.Transformation;
import org.apache.cassandra.dht.AbstractBounds;
import org.apache.cassandra.index.Index;
@@ -45,6 +47,7 @@
import org.apache.cassandra.service.ClientWarn;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
/**
@@ -53,10 +56,10 @@
* <p>
* This contains all the informations needed to do a local read.
*/
-public abstract class ReadCommand implements ReadQuery
+public abstract class ReadCommand extends MonitorableImpl implements ReadQuery
{
+ private static final int TEST_ITERATION_DELAY_MILLIS = Integer.valueOf(System.getProperty("cassandra.test.read_iteration_delay_ms", "0"));
protected static final Logger logger = LoggerFactory.getLogger(ReadCommand.class);
-
public static final IVersionedSerializer<ReadCommand> serializer = new Serializer();
// For RANGE_SLICE verb: will either dispatch on 'serializer' for 3.0 or 'legacyRangeSliceCommandSerializer' for earlier version.
// Can be removed (and replaced by 'serializer') once we drop pre-3.0 backward compatibility.
@@ -278,7 +281,7 @@
*/
public abstract ReadCommand copy();
- protected abstract UnfilteredPartitionIterator queryStorage(ColumnFamilyStore cfs, ReadOrderGroup orderGroup);
+ protected abstract UnfilteredPartitionIterator queryStorage(ColumnFamilyStore cfs, ReadExecutionController executionController);
protected abstract int oldestUnrepairedTombstone();
@@ -323,13 +326,13 @@
/**
* Executes this command on the local host.
*
- * @param orderGroup the operation group spanning this command
+ * @param executionController the execution controller spanning this command
*
* @return an iterator over the result of executing this command locally.
*/
@SuppressWarnings("resource") // The result iterator is closed upon exceptions (we know it's fine to potentially not close the intermediary
// iterators created inside the try as long as we do close the original resultIterator), or by closing the result.
- public UnfilteredPartitionIterator executeLocally(ReadOrderGroup orderGroup)
+ public UnfilteredPartitionIterator executeLocally(ReadExecutionController executionController)
{
long startTimeNanos = System.nanoTime();
@@ -347,11 +350,12 @@
}
UnfilteredPartitionIterator resultIterator = searcher == null
- ? queryStorage(cfs, orderGroup)
- : searcher.search(orderGroup);
+ ? queryStorage(cfs, executionController)
+ : searcher.search(executionController);
try
{
+ resultIterator = withStateTracking(resultIterator);
resultIterator = withMetricsRecording(withoutPurgeableTombstones(resultIterator, cfs), cfs.metric, startTimeNanos);
// If we've used a 2ndary index, we know the result already satisfy the primary expression used, so
@@ -375,14 +379,14 @@
protected abstract void recordLatency(TableMetrics metric, long latencyNanos);
- public PartitionIterator executeInternal(ReadOrderGroup orderGroup)
+ public PartitionIterator executeInternal(ReadExecutionController controller)
{
- return UnfilteredPartitionIterators.filter(executeLocally(orderGroup), nowInSec());
+ return UnfilteredPartitionIterators.filter(executeLocally(controller), nowInSec());
}
- public ReadOrderGroup startOrderGroup()
+ public ReadExecutionController executionController()
{
- return ReadOrderGroup.forCommand(this);
+ return ReadExecutionController.forCommand(this);
}
/**
@@ -471,6 +475,55 @@
return Transformation.apply(iter, new MetricRecording());
}
+ protected class CheckForAbort extends StoppingTransformation<BaseRowIterator<?>>
+ {
+ protected BaseRowIterator<?> applyToPartition(BaseRowIterator partition)
+ {
+ if (maybeAbort())
+ {
+ partition.close();
+ return null;
+ }
+
+ return partition;
+ }
+
+ protected Row applyToRow(Row row)
+ {
+ return maybeAbort() ? null : row;
+ }
+
+ private boolean maybeAbort()
+ {
+ if (TEST_ITERATION_DELAY_MILLIS > 0)
+ maybeDelayForTesting();
+
+ if (isAborted())
+ {
+ stop();
+ return true;
+ }
+
+ return false;
+ }
+ }
+
+ protected UnfilteredPartitionIterator withStateTracking(UnfilteredPartitionIterator iter)
+ {
+ return Transformation.apply(iter, new CheckForAbort());
+ }
+
+ protected UnfilteredRowIterator withStateTracking(UnfilteredRowIterator iter)
+ {
+ return Transformation.apply(iter, new CheckForAbort());
+ }
+
+ private void maybeDelayForTesting()
+ {
+ if (!metadata.ksName.startsWith("system"))
+ FBUtilities.sleepQuietly(TEST_ITERATION_DELAY_MILLIS);
+ }
+
/**
* Creates a message for this command.
*/
@@ -488,7 +541,7 @@
{
public WithoutPurgeableTombstones()
{
- super(isForThrift, nowInSec(), cfs.gcBefore(nowInSec()), oldestUnrepairedTombstone(), cfs.getCompactionStrategyManager().onlyPurgeRepairedTombstones());
+ super(isForThrift, cfs.gcBefore(nowInSec()), oldestUnrepairedTombstone(), cfs.getCompactionStrategyManager().onlyPurgeRepairedTombstones());
}
protected long getMaxPurgeableTimestamp()
@@ -520,6 +573,12 @@
return sb.toString();
}
+ // Monitorable interface
+ public String name()
+ {
+ return toCQLString();
+ }
+
private static class Serializer implements IVersionedSerializer<ReadCommand>
{
private static int digestFlag(boolean isDigest)
diff --git a/src/java/org/apache/cassandra/db/ReadCommandVerbHandler.java b/src/java/org/apache/cassandra/db/ReadCommandVerbHandler.java
index 9cde8dc..e2a9678 100644
--- a/src/java/org/apache/cassandra/db/ReadCommandVerbHandler.java
+++ b/src/java/org/apache/cassandra/db/ReadCommandVerbHandler.java
@@ -41,15 +41,24 @@
}
ReadCommand command = message.payload;
+ command.setMonitoringTime(message.constructionTime, message.getTimeout());
+
ReadResponse response;
- try (ReadOrderGroup opGroup = command.startOrderGroup(); UnfilteredPartitionIterator iterator = command.executeLocally(opGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ UnfilteredPartitionIterator iterator = command.executeLocally(executionController))
{
response = command.createResponse(iterator);
}
- MessageOut<ReadResponse> reply = new MessageOut<>(MessagingService.Verb.REQUEST_RESPONSE, response, serializer());
+ if (!command.complete())
+ {
+ Tracing.trace("Discarding partial response to {} (timed out)", message.from);
+ MessagingService.instance().incrementDroppedMessages(message, System.currentTimeMillis() - message.constructionTime.timestamp);
+ return;
+ }
Tracing.trace("Enqueuing response to {}", message.from);
+ MessageOut<ReadResponse> reply = new MessageOut<>(MessagingService.Verb.REQUEST_RESPONSE, response, serializer());
MessagingService.instance().sendReply(reply, id, message.from);
}
}
diff --git a/src/java/org/apache/cassandra/db/ReadOrderGroup.java b/src/java/org/apache/cassandra/db/ReadExecutionController.java
similarity index 82%
rename from src/java/org/apache/cassandra/db/ReadOrderGroup.java
rename to src/java/org/apache/cassandra/db/ReadExecutionController.java
index 416a2b8..4a779e4 100644
--- a/src/java/org/apache/cassandra/db/ReadOrderGroup.java
+++ b/src/java/org/apache/cassandra/db/ReadExecutionController.java
@@ -20,7 +20,7 @@
import org.apache.cassandra.index.Index;
import org.apache.cassandra.utils.concurrent.OpOrder;
-public class ReadOrderGroup implements AutoCloseable
+public class ReadExecutionController implements AutoCloseable
{
// For every reads
private final OpOrder.Group baseOp;
@@ -29,7 +29,7 @@
private final OpOrder.Group indexOp;
private final OpOrder.Group writeOp;
- private ReadOrderGroup(OpOrder.Group baseOp, OpOrder.Group indexOp, OpOrder.Group writeOp)
+ private ReadExecutionController(OpOrder.Group baseOp, OpOrder.Group indexOp, OpOrder.Group writeOp)
{
this.baseOp = baseOp;
this.indexOp = indexOp;
@@ -51,20 +51,25 @@
return writeOp;
}
- public static ReadOrderGroup emptyGroup()
+ public static ReadExecutionController empty()
{
- return new ReadOrderGroup(null, null, null);
+ return new ReadExecutionController(null, null, null);
}
- @SuppressWarnings("resource") // ops closed during group close
- public static ReadOrderGroup forCommand(ReadCommand command)
+ public static ReadExecutionController forReadOp(OpOrder.Group readOp)
+ {
+ return new ReadExecutionController(readOp, null, null);
+ }
+
+ @SuppressWarnings("resource") // ops closed during controller close
+ public static ReadExecutionController forCommand(ReadCommand command)
{
ColumnFamilyStore baseCfs = Keyspace.openAndGetStore(command.metadata());
ColumnFamilyStore indexCfs = maybeGetIndexCfs(baseCfs, command);
if (indexCfs == null)
{
- return new ReadOrderGroup(baseCfs.readOrdering.start(), null, null);
+ return new ReadExecutionController(baseCfs.readOrdering.start(), null, null);
}
else
{
@@ -77,7 +82,7 @@
// TODO: this should perhaps not open and maintain a writeOp for the full duration, but instead only *try* to delete stale entries, without blocking if there's no room
// as it stands, we open a writeOp and keep it open for the duration to ensure that should this CF get flushed to make room we don't block the reclamation of any room being made
writeOp = Keyspace.writeOrder.start();
- return new ReadOrderGroup(baseOp, indexOp, writeOp);
+ return new ReadExecutionController(baseOp, indexOp, writeOp);
}
catch (RuntimeException e)
{
diff --git a/src/java/org/apache/cassandra/db/ReadQuery.java b/src/java/org/apache/cassandra/db/ReadQuery.java
index 178ca7c..ba7b893 100644
--- a/src/java/org/apache/cassandra/db/ReadQuery.java
+++ b/src/java/org/apache/cassandra/db/ReadQuery.java
@@ -35,9 +35,9 @@
{
ReadQuery EMPTY = new ReadQuery()
{
- public ReadOrderGroup startOrderGroup()
+ public ReadExecutionController executionController()
{
- return ReadOrderGroup.emptyGroup();
+ return ReadExecutionController.empty();
}
public PartitionIterator execute(ConsistencyLevel consistency, ClientState clientState) throws RequestExecutionException
@@ -45,7 +45,7 @@
return EmptyIterators.partition();
}
- public PartitionIterator executeInternal(ReadOrderGroup orderGroup)
+ public PartitionIterator executeInternal(ReadExecutionController controller)
{
return EmptyIterators.partition();
}
@@ -86,9 +86,9 @@
* The returned object <b>must</b> be closed on all path and it is thus strongly advised to
* use it in a try-with-ressource construction.
*
- * @return a newly started order group for this {@code ReadQuery}.
+ * @return a newly started execution controller for this {@code ReadQuery}.
*/
- public ReadOrderGroup startOrderGroup();
+ public ReadExecutionController executionController();
/**
* Executes the query at the provided consistency level.
@@ -104,10 +104,10 @@
/**
* Execute the query for internal queries (that is, it basically executes the query locally).
*
- * @param orderGroup the {@code ReadOrderGroup} protecting the read.
+ * @param controller the {@code ReadExecutionController} protecting the read.
* @return the result of the query.
*/
- public PartitionIterator executeInternal(ReadOrderGroup orderGroup);
+ public PartitionIterator executeInternal(ReadExecutionController controller);
/**
* Returns a pager for the query.
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index de4c9c7..4c87d10 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -318,11 +318,11 @@
}
@SuppressWarnings("resource") // we close the created iterator through closing the result of this method (and SingletonUnfilteredPartitionIterator ctor cannot fail)
- protected UnfilteredPartitionIterator queryStorage(final ColumnFamilyStore cfs, ReadOrderGroup orderGroup)
+ protected UnfilteredPartitionIterator queryStorage(final ColumnFamilyStore cfs, ReadExecutionController executionController)
{
UnfilteredRowIterator partition = cfs.isRowCacheEnabled()
- ? getThroughCache(cfs, orderGroup.baseReadOpOrderGroup())
- : queryMemtableAndDisk(cfs, orderGroup.baseReadOpOrderGroup());
+ ? getThroughCache(cfs, executionController)
+ : queryMemtableAndDisk(cfs, executionController);
return new SingletonUnfilteredPartitionIterator(partition, isForThrift());
}
@@ -335,7 +335,7 @@
* If the partition is is not cached, we figure out what filter is "biggest", read
* that from disk, then filter the result and either cache that or return it.
*/
- private UnfilteredRowIterator getThroughCache(ColumnFamilyStore cfs, OpOrder.Group readOp)
+ private UnfilteredRowIterator getThroughCache(ColumnFamilyStore cfs, ReadExecutionController executionController)
{
assert !cfs.isIndex(); // CASSANDRA-5732
assert cfs.isRowCacheEnabled() : String.format("Row cache is not enabled on table [%s]", cfs.name);
@@ -353,7 +353,7 @@
// Some other read is trying to cache the value, just do a normal non-caching read
Tracing.trace("Row cache miss (race)");
cfs.metric.rowCacheMiss.inc();
- return queryMemtableAndDisk(cfs, readOp);
+ return queryMemtableAndDisk(cfs, executionController);
}
CachedPartition cachedPartition = (CachedPartition)cached;
@@ -368,7 +368,7 @@
cfs.metric.rowCacheHitOutOfRange.inc();
Tracing.trace("Ignoring row cache as cached value could not satisfy query");
- return queryMemtableAndDisk(cfs, readOp);
+ return queryMemtableAndDisk(cfs, executionController);
}
cfs.metric.rowCacheMiss.inc();
@@ -393,7 +393,7 @@
{
int rowsToCache = metadata().params.caching.rowsPerPartitionToCache();
@SuppressWarnings("resource") // we close on exception or upon closing the result of this method
- UnfilteredRowIterator iter = SinglePartitionReadCommand.fullPartitionRead(metadata(), nowInSec(), partitionKey()).queryMemtableAndDisk(cfs, readOp);
+ UnfilteredRowIterator iter = SinglePartitionReadCommand.fullPartitionRead(metadata(), nowInSec(), partitionKey()).queryMemtableAndDisk(cfs, executionController);
try
{
// We want to cache only rowsToCache rows
@@ -433,7 +433,7 @@
}
Tracing.trace("Fetching data but not populating cache as query does not query from the start of the partition");
- return queryMemtableAndDisk(cfs, readOp);
+ return queryMemtableAndDisk(cfs, executionController);
}
/**
@@ -453,6 +453,11 @@
*/
public UnfilteredRowIterator queryMemtableAndDisk(ColumnFamilyStore cfs, OpOrder.Group readOp)
{
+ return queryMemtableAndDisk(cfs, ReadExecutionController.forReadOp(readOp));
+ }
+
+ public UnfilteredRowIterator queryMemtableAndDisk(ColumnFamilyStore cfs, ReadExecutionController executionController)
+ {
Tracing.trace("Executing single-partition query on {}", cfs.name);
boolean copyOnHeap = Memtable.MEMORY_POOL.needToCopyOnHeap();
@@ -598,7 +603,7 @@
cfs.metric.samplers.get(TableMetrics.Sampler.READS).addSample(key.getKey(), key.hashCode(), 1);
}
- return merged;
+ return withStateTracking(merged);
}
catch (RuntimeException | Error e)
{
@@ -753,7 +758,7 @@
}
}
- return result.unfilteredIterator(columnFilter(), Slices.ALL, clusteringIndexFilter().isReversed());
+ return withStateTracking(result.unfilteredIterator(columnFilter(), Slices.ALL, clusteringIndexFilter().isReversed()));
}
private ImmutableBTreePartition add(UnfilteredRowIterator iter, ImmutableBTreePartition result, ClusteringIndexNamesFilter filter, boolean isRepaired)
@@ -933,18 +938,18 @@
return commands.get(0).metadata();
}
- public ReadOrderGroup startOrderGroup()
+ public ReadExecutionController executionController()
{
// Note that the only difference between the command in a group must be the partition key on which
// they applied. So as far as ReadOrderGroup is concerned, we can use any of the commands to start one.
- return commands.get(0).startOrderGroup();
+ return commands.get(0).executionController();
}
- public PartitionIterator executeInternal(ReadOrderGroup orderGroup)
+ public PartitionIterator executeInternal(ReadExecutionController controller)
{
List<PartitionIterator> partitions = new ArrayList<>(commands.size());
for (SinglePartitionReadCommand cmd : commands)
- partitions.add(cmd.executeInternal(orderGroup));
+ partitions.add(cmd.executeInternal(controller));
// Because we only have enforce the limit per command, we need to enforce it globally.
return limits.filter(PartitionIterators.concat(partitions), nowInSec);
diff --git a/src/java/org/apache/cassandra/db/Slices.java b/src/java/org/apache/cassandra/db/Slices.java
index 8fa9337..94dea15 100644
--- a/src/java/org/apache/cassandra/db/Slices.java
+++ b/src/java/org/apache/cassandra/db/Slices.java
@@ -738,7 +738,7 @@
public boolean isEQ()
{
- return startValue.equals(endValue);
+ return Objects.equals(startValue, endValue);
}
}
}
diff --git a/src/java/org/apache/cassandra/db/SystemKeyspace.java b/src/java/org/apache/cassandra/db/SystemKeyspace.java
index 205df8b..a51ade0 100644
--- a/src/java/org/apache/cassandra/db/SystemKeyspace.java
+++ b/src/java/org/apache/cassandra/db/SystemKeyspace.java
@@ -450,6 +450,7 @@
.add(TimeFcts.all())
.add(BytesConversionFcts.all())
.add(AggregateFcts.all())
+ .add(CastFcts.all())
.build();
}
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index bf9c2b8..a9b2903 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -432,7 +432,7 @@
return false;
}
}
- return next != null;
+ return true;
}
private Unfiltered readRow(LegacyLayout.LegacyAtom first)
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLog.java b/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
index 851c84e..64e22e0 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
@@ -201,7 +201,7 @@
}
/**
- * @return a ReplayPosition which, if >= one returned from add(), implies add() was started
+ * @return a ReplayPosition which, if {@code >= one} returned from add(), implies add() was started
* (but not necessarily finished) prior to this call
*/
public ReplayPosition getContext()
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java b/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
index 5547d0e..97b26c7 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
@@ -29,6 +29,8 @@
import java.util.Properties;
import java.util.TimeZone;
import java.util.concurrent.*;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor;
import org.apache.cassandra.config.DatabaseDescriptor;
@@ -46,6 +48,10 @@
private static final Logger logger = LoggerFactory.getLogger(CommitLogArchiver.class);
public static final SimpleDateFormat format = new SimpleDateFormat("yyyy:MM:dd HH:mm:ss");
private static final String DELIMITER = ",";
+ private static final Pattern NAME = Pattern.compile("%name");
+ private static final Pattern PATH = Pattern.compile("%path");
+ private static final Pattern FROM = Pattern.compile("%from");
+ private static final Pattern TO = Pattern.compile("%to");
static
{
format.setTimeZone(TimeZone.getTimeZone("GMT"));
@@ -136,8 +142,8 @@
protected void runMayThrow() throws IOException
{
segment.waitForFinalSync();
- String command = archiveCommand.replace("%name", segment.getName());
- command = command.replace("%path", segment.getPath());
+ String command = NAME.matcher(archiveCommand).replaceAll(Matcher.quoteReplacement(segment.getName()));
+ command = PATH.matcher(command).replaceAll(Matcher.quoteReplacement(segment.getPath()));
exec(command);
}
}));
@@ -158,8 +164,8 @@
{
protected void runMayThrow() throws IOException
{
- String command = archiveCommand.replace("%name", name);
- command = command.replace("%path", path);
+ String command = NAME.matcher(archiveCommand).replaceAll(Matcher.quoteReplacement(name));
+ command = PATH.matcher(command).replaceAll(Matcher.quoteReplacement(path));
exec(command);
}
}));
@@ -244,8 +250,8 @@
continue;
}
- String command = restoreCommand.replace("%from", fromFile.getPath());
- command = command.replace("%to", toFile.getPath());
+ String command = FROM.matcher(restoreCommand).replaceAll(Matcher.quoteReplacement(fromFile.getPath()));
+ command = TO.matcher(command).replaceAll(Matcher.quoteReplacement(toFile.getPath()));
try
{
exec(command);
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
index 7169b2f..e0518a9 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
@@ -30,6 +30,7 @@
import java.util.concurrent.atomic.AtomicInteger;
import java.util.zip.CRC32;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Predicate;
import com.google.common.base.Throwables;
import com.google.common.collect.HashMultimap;
@@ -67,13 +68,17 @@
public class CommitLogReplayer
{
+ @VisibleForTesting
+ public static long MAX_OUTSTANDING_REPLAY_BYTES = Long.getLong("cassandra.commitlog_max_outstanding_replay_bytes", 1024 * 1024 * 64);
+ @VisibleForTesting
+ public static MutationInitiator mutationInitiator = new MutationInitiator();
static final String IGNORE_REPLAY_ERRORS_PROPERTY = "cassandra.commitlog.ignorereplayerrors";
private static final Logger logger = LoggerFactory.getLogger(CommitLogReplayer.class);
private static final int MAX_OUTSTANDING_REPLAY_COUNT = Integer.getInteger("cassandra.commitlog_max_outstanding_replay_count", 1024);
private static final int LEGACY_END_OF_SEGMENT_MARKER = 0;
private final Set<Keyspace> keyspacesRecovered;
- private final List<Future<?>> futures;
+ private final Queue<Future<Integer>> futures;
private final Map<UUID, AtomicInteger> invalidMutations;
private final AtomicInteger replayedCount;
private final Map<UUID, ReplayPosition> cfPositions;
@@ -81,14 +86,83 @@
private final CRC32 checksum;
private byte[] buffer;
private byte[] uncompressedBuffer;
+ private long pendingMutationBytes = 0;
private final ReplayFilter replayFilter;
private final CommitLogArchiver archiver;
+ /*
+ * Wrapper around initiating mutations read from the log to make it possible
+ * to spy on initiated mutations for test
+ */
+ @VisibleForTesting
+ public static class MutationInitiator
+ {
+ protected Future<Integer> initiateMutation(final Mutation mutation,
+ final long segmentId,
+ final int serializedSize,
+ final long entryLocation,
+ final CommitLogReplayer clr)
+ {
+ Runnable runnable = new WrappedRunnable()
+ {
+ public void runMayThrow()
+ {
+ if (Schema.instance.getKSMetaData(mutation.getKeyspaceName()) == null)
+ return;
+ if (clr.pointInTimeExceeded(mutation))
+ return;
+
+ final Keyspace keyspace = Keyspace.open(mutation.getKeyspaceName());
+
+ // Rebuild the mutation, omitting column families that
+ // a) the user has requested that we ignore,
+ // b) have already been flushed,
+ // or c) are part of a cf that was dropped.
+ // Keep in mind that the cf.name() is suspect. do every thing based on the cfid instead.
+ Mutation newMutation = null;
+ for (PartitionUpdate update : clr.replayFilter.filter(mutation))
+ {
+ if (Schema.instance.getCF(update.metadata().cfId) == null)
+ continue; // dropped
+
+ ReplayPosition rp = clr.cfPositions.get(update.metadata().cfId);
+
+ // replay if current segment is newer than last flushed one or,
+ // if it is the last known segment, if we are after the replay position
+ if (segmentId > rp.segment || (segmentId == rp.segment && entryLocation > rp.position))
+ {
+ if (newMutation == null)
+ newMutation = new Mutation(mutation.getKeyspaceName(), mutation.key());
+ newMutation.add(update);
+ clr.replayedCount.incrementAndGet();
+ }
+ }
+ if (newMutation != null)
+ {
+ assert !newMutation.isEmpty();
+
+ try
+ {
+ Uninterruptibles.getUninterruptibly(Keyspace.open(newMutation.getKeyspaceName()).applyFromCommitLog(newMutation));
+ }
+ catch (ExecutionException e)
+ {
+ throw Throwables.propagate(e.getCause());
+ }
+
+ clr.keyspacesRecovered.add(keyspace);
+ }
+ }
+ };
+ return StageManager.getStage(Stage.MUTATION).submit(runnable, serializedSize);
+ }
+ }
+
CommitLogReplayer(CommitLog commitLog, ReplayPosition globalPosition, Map<UUID, ReplayPosition> cfPositions, ReplayFilter replayFilter)
{
this.keyspacesRecovered = new NonBlockingHashSet<Keyspace>();
- this.futures = new ArrayList<Future<?>>();
+ this.futures = new ArrayDeque<Future<Integer>>();
this.buffer = new byte[4096];
this.uncompressedBuffer = new byte[4096];
this.invalidMutations = new HashMap<UUID, AtomicInteger>();
@@ -165,6 +239,8 @@
// flush replayed keyspaces
futures.clear();
boolean flushingSystem = false;
+
+ List<Future<?>> futures = new ArrayList<Future<?>>();
for (Keyspace keyspace : keyspacesRecovered)
{
if (keyspace.getName().equals(SystemKeyspace.NAME))
@@ -178,6 +254,7 @@
futures.add(Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.BATCHES).forceFlush());
FBUtilities.waitOnFutures(futures);
+
return replayedCount.get();
}
@@ -230,7 +307,7 @@
Multimap<String, String> toReplay = HashMultimap.create();
for (String rawPair : System.getProperty("cassandra.replayList").split(","))
{
- String[] pair = rawPair.trim().split("\\.");
+ String[] pair = StringUtils.split(rawPair.trim(), '.');
if (pair.length != 2)
throw new IllegalArgumentException("Each table to be replayed must be fully qualified with keyspace name, e.g., 'system.peers'");
@@ -390,7 +467,7 @@
int uncompressedLength = replayEnd - replayPos;
if (uncompressedLength > uncompressedBuffer.length)
uncompressedBuffer = new byte[(int) (1.2 * uncompressedLength)];
- compressedLength = compressor.uncompress(buffer, 0, compressedLength, uncompressedBuffer, 0);
+ compressor.uncompress(buffer, 0, compressedLength, uncompressedBuffer, 0);
sectionReader = new FileSegmentInputStream(ByteBuffer.wrap(uncompressedBuffer), reader.getPath(), replayPos);
errorContext = "compressed section at " + start + " in " + errorContext;
}
@@ -567,62 +644,19 @@
if (logger.isTraceEnabled())
logger.trace("replaying mutation for {}.{}: {}", mutation.getKeyspaceName(), mutation.key(), "{" + StringUtils.join(mutation.getPartitionUpdates().iterator(), ", ") + "}");
- Runnable runnable = new WrappedRunnable()
+ pendingMutationBytes += size;
+ futures.offer(mutationInitiator.initiateMutation(mutation,
+ desc.id,
+ size,
+ entryLocation,
+ this));
+ //If there are finished mutations, or too many outstanding bytes/mutations
+ //drain the futures in the queue
+ while (futures.size() > MAX_OUTSTANDING_REPLAY_COUNT
+ || pendingMutationBytes > MAX_OUTSTANDING_REPLAY_BYTES
+ || (!futures.isEmpty() && futures.peek().isDone()))
{
- public void runMayThrow()
- {
- if (Schema.instance.getKSMetaData(mutation.getKeyspaceName()) == null)
- return;
- if (pointInTimeExceeded(mutation))
- return;
-
- final Keyspace keyspace = Keyspace.open(mutation.getKeyspaceName());
-
- // Rebuild the mutation, omitting column families that
- // a) the user has requested that we ignore,
- // b) have already been flushed,
- // or c) are part of a cf that was dropped.
- // Keep in mind that the cf.name() is suspect. do every thing based on the cfid instead.
- Mutation newMutation = null;
- for (PartitionUpdate update : replayFilter.filter(mutation))
- {
- if (Schema.instance.getCF(update.metadata().cfId) == null)
- continue; // dropped
-
- ReplayPosition rp = cfPositions.get(update.metadata().cfId);
-
- // replay if current segment is newer than last flushed one or,
- // if it is the last known segment, if we are after the replay position
- if (desc.id > rp.segment || (desc.id == rp.segment && entryLocation > rp.position))
- {
- if (newMutation == null)
- newMutation = new Mutation(mutation.getKeyspaceName(), mutation.key());
- newMutation.add(update);
- replayedCount.incrementAndGet();
- }
- }
- if (newMutation != null)
- {
- assert !newMutation.isEmpty();
-
- try
- {
- Uninterruptibles.getUninterruptibly(Keyspace.open(newMutation.getKeyspaceName()).applyFromCommitLog(newMutation));
- }
- catch (ExecutionException e)
- {
- throw Throwables.propagate(e.getCause());
- }
-
- keyspacesRecovered.add(keyspace);
- }
- }
- };
- futures.add(StageManager.getStage(Stage.MUTATION).submit(runnable));
- if (futures.size() > MAX_OUTSTANDING_REPLAY_COUNT)
- {
- FBUtilities.waitOnFutures(futures);
- futures.clear();
+ pendingMutationBytes -= FBUtilities.waitOnFuture(futures.poll());
}
}
diff --git a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index ae8839e..cab56bb 100644
--- a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -28,6 +28,7 @@
import org.apache.cassandra.db.Directories;
import org.apache.cassandra.db.SerializationHeader;
import org.apache.cassandra.db.lifecycle.SSTableSet;
+import org.apache.cassandra.index.Index;
import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
import org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter;
@@ -509,8 +510,14 @@
return groupedSSTables;
}
- public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, MetadataCollector meta, SerializationHeader header, LifecycleTransaction txn)
+ public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor,
+ long keyCount,
+ long repairedAt,
+ MetadataCollector meta,
+ SerializationHeader header,
+ Collection<Index> indexes,
+ LifecycleTransaction txn)
{
- return SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, cfs.metadata, meta, header, txn);
+ return SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, cfs.metadata, meta, header, indexes, txn);
}
}
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionController.java b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 5cb60c5..79c6d53 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -196,9 +196,8 @@
{
// if we don't have bloom filter(bf_fp_chance=1.0 or filter file is missing),
// we check index file instead.
- if (sstable.getBloomFilter() instanceof AlwaysPresentFilter && sstable.getPosition(key, SSTableReader.Operator.EQ, false) != null)
- min = Math.min(min, sstable.getMinTimestamp());
- else if (sstable.getBloomFilter().isPresent(key))
+ if ((sstable.getBloomFilter() instanceof AlwaysPresentFilter && sstable.getPosition(key, SSTableReader.Operator.EQ, false) != null)
+ || sstable.getBloomFilter().isPresent(key))
min = Math.min(min, sstable.getMinTimestamp());
}
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java b/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
index d39da2a..8a3b24b 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
@@ -103,7 +103,7 @@
? EmptyIterators.unfilteredPartition(controller.cfs.metadata, false)
: UnfilteredPartitionIterators.merge(scanners, nowInSec, listener());
boolean isForThrift = merged.isForThrift(); // to stop capture of iterator in Purger, which is confusing for debug
- this.compacted = Transformation.apply(merged, new Purger(isForThrift, controller, nowInSec));
+ this.compacted = Transformation.apply(merged, new Purger(isForThrift, controller));
}
public boolean isForThrift()
@@ -264,9 +264,9 @@
private long compactedUnfiltered;
- private Purger(boolean isForThrift, CompactionController controller, int nowInSec)
+ private Purger(boolean isForThrift, CompactionController controller)
{
- super(isForThrift, nowInSec, controller.gcBefore, controller.compactingRepaired() ? Integer.MIN_VALUE : Integer.MAX_VALUE, controller.cfs.getCompactionStrategyManager().onlyPurgeRepairedTombstones());
+ super(isForThrift, controller.gcBefore, controller.compactingRepaired() ? Integer.MIN_VALUE : Integer.MAX_VALUE, controller.cfs.getCompactionStrategyManager().onlyPurgeRepairedTombstones());
this.controller = controller;
}
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 4304440..57123c3 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -22,6 +22,7 @@
import java.lang.management.ManagementFactory;
import java.util.*;
import java.util.concurrent.*;
+import java.util.stream.Collectors;
import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.management.openmbean.OpenDataException;
@@ -279,7 +280,7 @@
Iterable<SSTableReader> sstables = compacting != null ? Lists.newArrayList(operation.filterSSTables(compacting)) : Collections.<SSTableReader>emptyList();
if (Iterables.isEmpty(sstables))
{
- logger.info("No sstables for {}.{}", cfs.keyspace.getName(), cfs.name);
+ logger.info("No sstables to {} for {}.{}", operationType.name(), cfs.keyspace.getName(), cfs.name);
return AllSSTableOpStatus.SUCCESSFUL;
}
@@ -441,6 +442,77 @@
}, OperationType.CLEANUP);
}
+ public AllSSTableOpStatus relocateSSTables(final ColumnFamilyStore cfs) throws ExecutionException, InterruptedException
+ {
+ if (!cfs.getPartitioner().splitter().isPresent())
+ {
+ logger.info("Partitioner does not support splitting");
+ return AllSSTableOpStatus.ABORTED;
+ }
+ final Collection<Range<Token>> r = StorageService.instance.getLocalRanges(cfs.keyspace.getName());
+
+ if (r.isEmpty())
+ {
+ logger.info("Relocate cannot run before a node has joined the ring");
+ return AllSSTableOpStatus.ABORTED;
+ }
+
+ final List<Range<Token>> localRanges = Range.sort(r);
+ final Directories.DataDirectory[] locations = cfs.getDirectories().getWriteableLocations();
+ final List<PartitionPosition> diskBoundaries = StorageService.getDiskBoundaries(localRanges, cfs.getPartitioner(), locations);
+
+ return parallelAllSSTableOperation(cfs, new OneSSTableOperation()
+ {
+ @Override
+ public Iterable<SSTableReader> filterSSTables(LifecycleTransaction transaction)
+ {
+ Set<SSTableReader> originals = Sets.newHashSet(transaction.originals());
+ Set<SSTableReader> needsRelocation = originals.stream().filter(s -> !inCorrectLocation(s)).collect(Collectors.toSet());
+ transaction.cancel(Sets.difference(originals, needsRelocation));
+
+ Map<Integer, List<SSTableReader>> groupedByDisk = needsRelocation.stream().collect(Collectors.groupingBy((s) ->
+ CompactionStrategyManager.getCompactionStrategyIndex(cfs, cfs.getDirectories(), s)));
+
+ int maxSize = 0;
+ for (List<SSTableReader> diskSSTables : groupedByDisk.values())
+ maxSize = Math.max(maxSize, diskSSTables.size());
+
+ List<SSTableReader> mixedSSTables = new ArrayList<>();
+
+ for (int i = 0; i < maxSize; i++)
+ for (List<SSTableReader> diskSSTables : groupedByDisk.values())
+ if (i < diskSSTables.size())
+ mixedSSTables.add(diskSSTables.get(i));
+
+ return mixedSSTables;
+ }
+
+ private boolean inCorrectLocation(SSTableReader sstable)
+ {
+ if (!cfs.getPartitioner().splitter().isPresent())
+ return true;
+ int directoryIndex = CompactionStrategyManager.getCompactionStrategyIndex(cfs, cfs.getDirectories(), sstable);
+ Directories.DataDirectory[] locations = cfs.getDirectories().getWriteableLocations();
+
+ Directories.DataDirectory location = locations[directoryIndex];
+ PartitionPosition diskLast = diskBoundaries.get(directoryIndex);
+ // the location we get from directoryIndex is based on the first key in the sstable
+ // now we need to make sure the last key is less than the boundary as well:
+ return sstable.descriptor.directory.getAbsolutePath().startsWith(location.location.getAbsolutePath()) && sstable.last.compareTo(diskLast) <= 0;
+ }
+
+ @Override
+ public void execute(LifecycleTransaction txn) throws IOException
+ {
+ logger.debug("Relocating {}", txn.originals());
+ AbstractCompactionTask task = cfs.getCompactionStrategyManager().getCompactionTask(txn, NO_GC, Long.MAX_VALUE);
+ task.setUserDefined(true);
+ task.setCompactionType(OperationType.RELOCATE);
+ task.execute(metrics);
+ }
+ }, OperationType.RELOCATE);
+ }
+
public ListenableFuture<?> submitAntiCompaction(final ColumnFamilyStore cfs,
final Collection<Range<Token>> ranges,
final Refs<SSTableReader> sstables,
@@ -592,7 +664,7 @@
futures.add(executor.submit(runnable));
}
if (nonEmptyTasks > 1)
- logger.info("Cannot perform a full major compaction as repaired and unrepaired sstables cannot be compacted together. These two set of sstables will be compacted separately.");
+ logger.info("Major compaction will not result in a single sstable - repaired and unrepaired data is kept separate and compaction runs per data_file_directory.");
return futures;
}
@@ -622,6 +694,61 @@
FBUtilities.waitOnFutures(futures);
}
+ public void forceUserDefinedCleanup(String dataFiles)
+ {
+ String[] filenames = dataFiles.split(",");
+ HashMap<ColumnFamilyStore, Descriptor> descriptors = Maps.newHashMap();
+
+ for (String filename : filenames)
+ {
+ // extract keyspace and columnfamily name from filename
+ Descriptor desc = Descriptor.fromFilename(filename.trim());
+ if (Schema.instance.getCFMetaData(desc) == null)
+ {
+ logger.warn("Schema does not exist for file {}. Skipping.", filename);
+ continue;
+ }
+ // group by keyspace/columnfamily
+ ColumnFamilyStore cfs = Keyspace.open(desc.ksname).getColumnFamilyStore(desc.cfname);
+ desc = cfs.getDirectories().find(new File(filename.trim()).getName());
+ if (desc != null)
+ descriptors.put(cfs, desc);
+ }
+
+ for (Map.Entry<ColumnFamilyStore,Descriptor> entry : descriptors.entrySet())
+ {
+ ColumnFamilyStore cfs = entry.getKey();
+ Keyspace keyspace = cfs.keyspace;
+ Collection<Range<Token>> ranges = StorageService.instance.getLocalRanges(keyspace.getName());
+ boolean hasIndexes = cfs.indexManager.hasIndexes();
+ SSTableReader sstable = lookupSSTable(cfs, entry.getValue());
+
+ if (ranges.isEmpty())
+ {
+ logger.error("Cleanup cannot run before a node has joined the ring");
+ return;
+ }
+
+ if (sstable == null)
+ {
+ logger.warn("Will not clean {}, it is not an active sstable", entry.getValue());
+ }
+ else
+ {
+ CleanupStrategy cleanupStrategy = CleanupStrategy.get(cfs, ranges, FBUtilities.nowInSeconds());
+ try (LifecycleTransaction txn = cfs.getTracker().tryModify(sstable, OperationType.CLEANUP))
+ {
+ doCleanupOne(cfs, txn, cleanupStrategy, ranges, hasIndexes);
+ }
+ catch (IOException e)
+ {
+ logger.error(String.format("forceUserDefinedCleanup failed: %s", e.getLocalizedMessage()));
+ }
+ }
+ }
+ }
+
+
public Future<?> submitUserDefined(final ColumnFamilyStore cfs, final Collection<Descriptor> dataFiles, final int gcBefore)
{
Runnable runnable = new WrappedRunnable()
@@ -836,9 +963,7 @@
logger.info("Cleaning up {}", sstable);
- File compactionFileLocation = cfs.getDirectories().getWriteableLocationAsFile(cfs.getExpectedCompactedFileSize(txn.originals(), OperationType.CLEANUP));
- if (compactionFileLocation == null)
- throw new IOException("disk full");
+ File compactionFileLocation = sstable.descriptor.directory;
List<SSTableReader> finished;
int nowInSec = FBUtilities.nowInSeconds();
@@ -982,6 +1107,7 @@
repairedAt,
sstable.getSSTableLevel(),
header,
+ cfs.indexManager.listIndexes(),
txn);
}
@@ -1014,6 +1140,7 @@
cfs.metadata,
new MetadataCollector(sstables, cfs.metadata.comparator, minLevel),
SerializationHeader.make(cfs.metadata, sstables),
+ cfs.indexManager.listIndexes(),
txn);
}
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManagerMBean.java b/src/java/org/apache/cassandra/db/compaction/CompactionManagerMBean.java
index d5da0fe..bb67d5f 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManagerMBean.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManagerMBean.java
@@ -45,6 +45,17 @@
public void forceUserDefinedCompaction(String dataFiles);
/**
+ * Triggers the cleanup of user specified sstables.
+ * You can specify files from various keyspaces and columnfamilies.
+ * If you do so, cleanup is performed each file individually
+ *
+ * @param dataFiles a comma separated list of sstable file to cleanup.
+ * must contain keyspace and columnfamily name in path(for 2.1+) or file name itself.
+ */
+ public void forceUserDefinedCleanup(String dataFiles);
+
+
+ /**
* Stop all running compaction-like tasks having the provided {@code type}.
* @param type the type of compaction to stop. Can be one of:
* - COMPACTION
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
index bd72c64..067a0c1 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
@@ -20,8 +20,12 @@
import java.util.*;
import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
import com.google.common.collect.Iterables;
+import org.apache.cassandra.index.Index;
+import com.google.common.primitives.Ints;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -30,6 +34,7 @@
import org.apache.cassandra.db.Directories;
import org.apache.cassandra.db.Memtable;
import org.apache.cassandra.db.SerializationHeader;
+import org.apache.cassandra.db.PartitionPosition;
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.db.lifecycle.SSTableSet;
import org.apache.cassandra.dht.Range;
@@ -42,19 +47,21 @@
import org.apache.cassandra.notifications.*;
import org.apache.cassandra.schema.CompactionParams;
import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.service.StorageService;
/**
* Manages the compaction strategies.
*
- * Currently has two instances of actual compaction strategies - one for repaired data and one for
+ * Currently has two instances of actual compaction strategies per data directory - one for repaired data and one for
* unrepaired data. This is done to be able to totally separate the different sets of sstables.
*/
+
public class CompactionStrategyManager implements INotificationConsumer
{
private static final Logger logger = LoggerFactory.getLogger(CompactionStrategyManager.class);
private final ColumnFamilyStore cfs;
- private volatile AbstractCompactionStrategy repaired;
- private volatile AbstractCompactionStrategy unrepaired;
+ private volatile List<AbstractCompactionStrategy> repaired = new ArrayList<>();
+ private volatile List<AbstractCompactionStrategy> unrepaired = new ArrayList<>();
private volatile boolean enabled = true;
public boolean isActive = true;
private volatile CompactionParams params;
@@ -66,6 +73,7 @@
we will use the new compaction parameters.
*/
private CompactionParams schemaCompactionParams;
+ private Directories.DataDirectory[] locations;
public CompactionStrategyManager(ColumnFamilyStore cfs)
{
@@ -74,6 +82,7 @@
this.cfs = cfs;
reload(cfs.metadata);
params = cfs.metadata.params.compaction;
+ locations = getDirectories().getWriteableLocations();
enabled = params.isEnabled();
}
@@ -90,20 +99,17 @@
maybeReload(cfs.metadata);
- if (repaired.getEstimatedRemainingTasks() > unrepaired.getEstimatedRemainingTasks())
+ List<AbstractCompactionStrategy> strategies = new ArrayList<>(repaired.size() + unrepaired.size());
+ strategies.addAll(repaired);
+ strategies.addAll(unrepaired);
+ Collections.sort(strategies, (o1, o2) -> Ints.compare(o2.getEstimatedRemainingTasks(), o1.getEstimatedRemainingTasks()));
+ for (AbstractCompactionStrategy strategy : strategies)
{
- AbstractCompactionTask repairedTask = repaired.getNextBackgroundTask(gcBefore);
- if (repairedTask != null)
- return repairedTask;
- return unrepaired.getNextBackgroundTask(gcBefore);
+ AbstractCompactionTask task = strategy.getNextBackgroundTask(gcBefore);
+ if (task != null)
+ return task;
}
- else
- {
- AbstractCompactionTask unrepairedTask = unrepaired.getNextBackgroundTask(gcBefore);
- if (unrepairedTask != null)
- return unrepairedTask;
- return repaired.getNextBackgroundTask(gcBefore);
- }
+ return null;
}
public boolean isEnabled()
@@ -134,36 +140,78 @@
if (sstable.openReason != SSTableReader.OpenReason.EARLY)
getCompactionStrategyFor(sstable).addSSTable(sstable);
}
- repaired.startup();
- unrepaired.startup();
+ repaired.forEach(AbstractCompactionStrategy::startup);
+ unrepaired.forEach(AbstractCompactionStrategy::startup);
}
/**
* return the compaction strategy for the given sstable
*
- * returns differently based on the repaired status
+ * returns differently based on the repaired status and which vnode the compaction strategy belongs to
* @param sstable
* @return
*/
private AbstractCompactionStrategy getCompactionStrategyFor(SSTableReader sstable)
{
+ int index = getCompactionStrategyIndex(cfs, getDirectories(), sstable);
if (sstable.isRepaired())
- return repaired;
+ return repaired.get(index);
else
- return unrepaired;
+ return unrepaired.get(index);
+ }
+
+ /**
+ * Get the correct compaction strategy for the given sstable. If the first token starts within a disk boundary, we
+ * will add it to that compaction strategy.
+ *
+ * In the case we are upgrading, the first compaction strategy will get most files - we do not care about which disk
+ * the sstable is on currently (unless we don't know the local tokens yet). Once we start compacting we will write out
+ * sstables in the correct locations and give them to the correct compaction strategy instance.
+ *
+ * @param cfs
+ * @param locations
+ * @param sstable
+ * @return
+ */
+ public static int getCompactionStrategyIndex(ColumnFamilyStore cfs, Directories locations, SSTableReader sstable)
+ {
+ if (!cfs.getPartitioner().splitter().isPresent())
+ return 0;
+
+ Directories.DataDirectory[] directories = locations.getWriteableLocations();
+
+ List<PartitionPosition> boundaries = StorageService.getDiskBoundaries(cfs, locations.getWriteableLocations());
+ if (boundaries == null)
+ {
+ // try to figure out location based on sstable directory:
+ for (int i = 0; i < directories.length; i++)
+ {
+ Directories.DataDirectory directory = directories[i];
+ if (sstable.descriptor.directory.getAbsolutePath().startsWith(directory.location.getAbsolutePath()))
+ return i;
+ }
+ return 0;
+ }
+
+ int pos = Collections.binarySearch(boundaries, sstable.first);
+ assert pos < 0; // boundaries are .minkeybound and .maxkeybound so they should never be equal
+ return -pos - 1;
+
+
}
public void shutdown()
{
isActive = false;
- repaired.shutdown();
- unrepaired.shutdown();
+ repaired.forEach(AbstractCompactionStrategy::shutdown);
+ unrepaired.forEach(AbstractCompactionStrategy::shutdown);
}
public synchronized void maybeReload(CFMetaData metadata)
{
// compare the old schema configuration to the new one, ignore any locally set changes.
- if (metadata.params.compaction.equals(schemaCompactionParams))
+ if (metadata.params.compaction.equals(schemaCompactionParams) &&
+ Arrays.equals(locations, cfs.getDirectories().getWriteableLocations())) // any drives broken?
return;
reload(metadata);
}
@@ -177,6 +225,11 @@
public synchronized void reload(CFMetaData metadata)
{
boolean disabledWithJMX = !enabled && shouldBeEnabled();
+ if (!metadata.params.compaction.equals(schemaCompactionParams))
+ logger.trace("Recreating compaction strategy - compaction parameters changed for {}.{}", cfs.keyspace.getName(), cfs.getTableName());
+ else if (!Arrays.equals(locations, cfs.getDirectories().getWriteableLocations()))
+ logger.trace("Recreating compaction strategy - writeable locations changed for {}.{}", cfs.keyspace.getName(), cfs.getTableName());
+
setStrategy(metadata.params.compaction);
schemaCompactionParams = metadata.params.compaction;
@@ -196,11 +249,13 @@
public int getUnleveledSSTables()
{
- if (repaired instanceof LeveledCompactionStrategy && unrepaired instanceof LeveledCompactionStrategy)
+ if (repaired.get(0) instanceof LeveledCompactionStrategy && unrepaired.get(0) instanceof LeveledCompactionStrategy)
{
int count = 0;
- count += ((LeveledCompactionStrategy)repaired).getLevelSize(0);
- count += ((LeveledCompactionStrategy)unrepaired).getLevelSize(0);
+ for (AbstractCompactionStrategy strategy : repaired)
+ count += ((LeveledCompactionStrategy)strategy).getLevelSize(0);
+ for (AbstractCompactionStrategy strategy : unrepaired)
+ count += ((LeveledCompactionStrategy)strategy).getLevelSize(0);
return count;
}
return 0;
@@ -208,13 +263,19 @@
public synchronized int[] getSSTableCountPerLevel()
{
- if (repaired instanceof LeveledCompactionStrategy && unrepaired instanceof LeveledCompactionStrategy)
+ if (repaired.get(0) instanceof LeveledCompactionStrategy && unrepaired.get(0) instanceof LeveledCompactionStrategy)
{
int [] res = new int[LeveledManifest.MAX_LEVEL_COUNT];
- int[] repairedCountPerLevel = ((LeveledCompactionStrategy) repaired).getAllLevelSize();
- res = sumArrays(res, repairedCountPerLevel);
- int[] unrepairedCountPerLevel = ((LeveledCompactionStrategy) unrepaired).getAllLevelSize();
- res = sumArrays(res, unrepairedCountPerLevel);
+ for (AbstractCompactionStrategy strategy : repaired)
+ {
+ int[] repairedCountPerLevel = ((LeveledCompactionStrategy) strategy).getAllLevelSize();
+ res = sumArrays(res, repairedCountPerLevel);
+ }
+ for (AbstractCompactionStrategy strategy : unrepaired)
+ {
+ int[] unrepairedCountPerLevel = ((LeveledCompactionStrategy) strategy).getAllLevelSize();
+ res = sumArrays(res, unrepairedCountPerLevel);
+ }
return res;
}
return null;
@@ -237,103 +298,112 @@
public boolean shouldDefragment()
{
- assert repaired.getClass().equals(unrepaired.getClass());
- return repaired.shouldDefragment();
+ assert repaired.get(0).getClass().equals(unrepaired.get(0).getClass());
+ return repaired.get(0).shouldDefragment();
}
public Directories getDirectories()
{
- assert repaired.getClass().equals(unrepaired.getClass());
- return repaired.getDirectories();
+ assert repaired.get(0).getClass().equals(unrepaired.get(0).getClass());
+ return repaired.get(0).getDirectories();
}
public synchronized void handleNotification(INotification notification, Object sender)
{
+ maybeReload(cfs.metadata);
if (notification instanceof SSTableAddedNotification)
{
SSTableAddedNotification flushedNotification = (SSTableAddedNotification) notification;
for (SSTableReader sstable : flushedNotification.added)
- {
- if (sstable.isRepaired())
- repaired.addSSTable(sstable);
- else
- unrepaired.addSSTable(sstable);
- }
+ getCompactionStrategyFor(sstable).addSSTable(sstable);
}
else if (notification instanceof SSTableListChangedNotification)
{
+ // a bit of gymnastics to be able to replace sstables in compaction strategies
+ // we use this to know that a compaction finished and where to start the next compaction in LCS
SSTableListChangedNotification listChangedNotification = (SSTableListChangedNotification) notification;
- Set<SSTableReader> repairedRemoved = new HashSet<>();
- Set<SSTableReader> repairedAdded = new HashSet<>();
- Set<SSTableReader> unrepairedRemoved = new HashSet<>();
- Set<SSTableReader> unrepairedAdded = new HashSet<>();
+
+ Directories.DataDirectory [] locations = cfs.getDirectories().getWriteableLocations();
+ int locationSize = cfs.getPartitioner().splitter().isPresent() ? locations.length : 1;
+
+ List<Set<SSTableReader>> repairedRemoved = new ArrayList<>(locationSize);
+ List<Set<SSTableReader>> repairedAdded = new ArrayList<>(locationSize);
+ List<Set<SSTableReader>> unrepairedRemoved = new ArrayList<>(locationSize);
+ List<Set<SSTableReader>> unrepairedAdded = new ArrayList<>(locationSize);
+
+ for (int i = 0; i < locationSize; i++)
+ {
+ repairedRemoved.add(new HashSet<>());
+ repairedAdded.add(new HashSet<>());
+ unrepairedRemoved.add(new HashSet<>());
+ unrepairedAdded.add(new HashSet<>());
+ }
for (SSTableReader sstable : listChangedNotification.removed)
{
+ int i = getCompactionStrategyIndex(cfs, getDirectories(), sstable);
if (sstable.isRepaired())
- repairedRemoved.add(sstable);
+ repairedRemoved.get(i).add(sstable);
else
- unrepairedRemoved.add(sstable);
+ unrepairedRemoved.get(i).add(sstable);
}
for (SSTableReader sstable : listChangedNotification.added)
{
+ int i = getCompactionStrategyIndex(cfs, getDirectories(), sstable);
if (sstable.isRepaired())
- repairedAdded.add(sstable);
+ repairedAdded.get(i).add(sstable);
else
- unrepairedAdded.add(sstable);
- }
- if (!repairedRemoved.isEmpty())
- {
- repaired.replaceSSTables(repairedRemoved, repairedAdded);
- }
- else
- {
- for (SSTableReader sstable : repairedAdded)
- repaired.addSSTable(sstable);
+ unrepairedAdded.get(i).add(sstable);
}
- if (!unrepairedRemoved.isEmpty())
+ for (int i = 0; i < locationSize; i++)
{
- unrepaired.replaceSSTables(unrepairedRemoved, unrepairedAdded);
- }
- else
- {
- for (SSTableReader sstable : unrepairedAdded)
- unrepaired.addSSTable(sstable);
+ if (!repairedRemoved.get(i).isEmpty())
+ repaired.get(i).replaceSSTables(repairedRemoved.get(i), repairedAdded.get(i));
+ else
+ {
+ for (SSTableReader sstable : repairedAdded.get(i))
+ repaired.get(i).addSSTable(sstable);
+ }
+ if (!unrepairedRemoved.get(i).isEmpty())
+ unrepaired.get(i).replaceSSTables(unrepairedRemoved.get(i), unrepairedAdded.get(i));
+ else
+ {
+ for (SSTableReader sstable : unrepairedAdded.get(i))
+ unrepaired.get(i).addSSTable(sstable);
+ }
}
}
else if (notification instanceof SSTableRepairStatusChanged)
{
for (SSTableReader sstable : ((SSTableRepairStatusChanged) notification).sstable)
{
+ int index = getCompactionStrategyIndex(cfs, getDirectories(), sstable);
if (sstable.isRepaired())
{
- unrepaired.removeSSTable(sstable);
- repaired.addSSTable(sstable);
+ unrepaired.get(index).removeSSTable(sstable);
+ repaired.get(index).addSSTable(sstable);
}
else
{
- repaired.removeSSTable(sstable);
- unrepaired.addSSTable(sstable);
+ repaired.get(index).removeSSTable(sstable);
+ unrepaired.get(index).addSSTable(sstable);
}
}
}
else if (notification instanceof SSTableDeletingNotification)
{
- SSTableReader sstable = ((SSTableDeletingNotification)notification).deleting;
- if (sstable.isRepaired())
- repaired.removeSSTable(sstable);
- else
- unrepaired.removeSSTable(sstable);
+ SSTableReader sstable = ((SSTableDeletingNotification) notification).deleting;
+ getCompactionStrategyFor(sstable).removeSSTable(sstable);
}
}
public void enable()
{
if (repaired != null)
- repaired.enable();
+ repaired.forEach(AbstractCompactionStrategy::enable);
if (unrepaired != null)
- unrepaired.enable();
+ unrepaired.forEach(AbstractCompactionStrategy::enable);
// enable this last to make sure the strategies are ready to get calls.
enabled = true;
}
@@ -343,47 +413,64 @@
// disable this first avoid asking disabled strategies for compaction tasks
enabled = false;
if (repaired != null)
- repaired.disable();
+ repaired.forEach(AbstractCompactionStrategy::disable);
if (unrepaired != null)
- unrepaired.disable();
+ unrepaired.forEach(AbstractCompactionStrategy::disable);
}
/**
- * Create ISSTableScanner from the given sstables
+ * Create ISSTableScanners from the given sstables
*
* Delegates the call to the compaction strategies to allow LCS to create a scanner
* @param sstables
- * @param range
+ * @param ranges
* @return
*/
@SuppressWarnings("resource")
public synchronized AbstractCompactionStrategy.ScannerList getScanners(Collection<SSTableReader> sstables, Collection<Range<Token>> ranges)
{
- List<SSTableReader> repairedSSTables = new ArrayList<>();
- List<SSTableReader> unrepairedSSTables = new ArrayList<>();
+ assert repaired.size() == unrepaired.size();
+ List<Set<SSTableReader>> repairedSSTables = new ArrayList<>();
+ List<Set<SSTableReader>> unrepairedSSTables = new ArrayList<>();
+
+ for (int i = 0; i < repaired.size(); i++)
+ {
+ repairedSSTables.add(new HashSet<>());
+ unrepairedSSTables.add(new HashSet<>());
+ }
+
for (SSTableReader sstable : sstables)
{
if (sstable.isRepaired())
- repairedSSTables.add(sstable);
+ repairedSSTables.get(getCompactionStrategyIndex(cfs, getDirectories(), sstable)).add(sstable);
else
- unrepairedSSTables.add(sstable);
+ unrepairedSSTables.get(getCompactionStrategyIndex(cfs, getDirectories(), sstable)).add(sstable);
}
- Set<ISSTableScanner> scanners = new HashSet<>(sstables.size());
+ List<ISSTableScanner> scanners = new ArrayList<>(sstables.size());
for (Range<Token> range : ranges)
{
- AbstractCompactionStrategy.ScannerList repairedScanners = repaired.getScanners(repairedSSTables, range);
- AbstractCompactionStrategy.ScannerList unrepairedScanners = unrepaired.getScanners(unrepairedSSTables, range);
+ List<ISSTableScanner> repairedScanners = new ArrayList<>();
+ List<ISSTableScanner> unrepairedScanners = new ArrayList<>();
- for (ISSTableScanner scanner : Iterables.concat(repairedScanners.scanners, unrepairedScanners.scanners))
+ for (int i = 0; i < repairedSSTables.size(); i++)
+ {
+ if (!repairedSSTables.get(i).isEmpty())
+ repairedScanners.addAll(repaired.get(i).getScanners(repairedSSTables.get(i), range).scanners);
+ }
+ for (int i = 0; i < unrepairedSSTables.size(); i++)
+ {
+ if (!unrepairedSSTables.get(i).isEmpty())
+ scanners.addAll(unrepaired.get(i).getScanners(unrepairedSSTables.get(i), range).scanners);
+ }
+ for (ISSTableScanner scanner : Iterables.concat(repairedScanners, unrepairedScanners))
{
if (!scanners.add(scanner))
scanner.close();
}
}
-
- return new AbstractCompactionStrategy.ScannerList(new ArrayList<>(scanners));
+ return new AbstractCompactionStrategy.ScannerList(scanners);
}
public synchronized AbstractCompactionStrategy.ScannerList getScanners(Collection<SSTableReader> sstables)
@@ -393,21 +480,44 @@
public Collection<Collection<SSTableReader>> groupSSTablesForAntiCompaction(Collection<SSTableReader> sstablesToGroup)
{
- return unrepaired.groupSSTablesForAntiCompaction(sstablesToGroup);
+ Map<Integer, List<SSTableReader>> groups = sstablesToGroup.stream().collect(Collectors.groupingBy((s) -> getCompactionStrategyIndex(cfs, getDirectories(), s)));
+ Collection<Collection<SSTableReader>> anticompactionGroups = new ArrayList<>();
+
+ for (Map.Entry<Integer, List<SSTableReader>> group : groups.entrySet())
+ anticompactionGroups.addAll(unrepaired.get(group.getKey()).groupSSTablesForAntiCompaction(group.getValue()));
+ return anticompactionGroups;
}
public long getMaxSSTableBytes()
{
- return unrepaired.getMaxSSTableBytes();
+ return unrepaired.get(0).getMaxSSTableBytes();
}
public AbstractCompactionTask getCompactionTask(LifecycleTransaction txn, int gcBefore, long maxSSTableBytes)
{
+ maybeReload(cfs.metadata);
+ validateForCompaction(txn.originals());
return getCompactionStrategyFor(txn.originals().iterator().next()).getCompactionTask(txn, gcBefore, maxSSTableBytes);
}
+ private void validateForCompaction(Iterable<SSTableReader> input)
+ {
+ SSTableReader firstSSTable = Iterables.getFirst(input, null);
+ assert firstSSTable != null;
+ boolean repaired = firstSSTable.isRepaired();
+ int firstIndex = getCompactionStrategyIndex(cfs, getDirectories(), firstSSTable);
+ for (SSTableReader sstable : input)
+ {
+ if (sstable.isRepaired() != repaired)
+ throw new UnsupportedOperationException("You can't mix repaired and unrepaired data in a compaction");
+ if (firstIndex != getCompactionStrategyIndex(cfs, getDirectories(), sstable))
+ throw new UnsupportedOperationException("You can't mix sstables from different directories in a compaction");
+ }
+ }
+
public Collection<AbstractCompactionTask> getMaximalTasks(final int gcBefore, final boolean splitOutput)
{
+ maybeReload(cfs.metadata);
// runWithCompactionsDisabled cancels active compactions and disables them, then we are able
// to make the repaired/unrepaired strategies mark their own sstables as compacting. Once the
// sstables are marked the compactions are re-enabled
@@ -418,20 +528,21 @@
{
synchronized (CompactionStrategyManager.this)
{
- Collection<AbstractCompactionTask> repairedTasks = repaired.getMaximalTask(gcBefore, splitOutput);
- Collection<AbstractCompactionTask> unrepairedTasks = unrepaired.getMaximalTask(gcBefore, splitOutput);
-
- if (repairedTasks == null && unrepairedTasks == null)
- return null;
-
- if (repairedTasks == null)
- return unrepairedTasks;
- if (unrepairedTasks == null)
- return repairedTasks;
-
List<AbstractCompactionTask> tasks = new ArrayList<>();
- tasks.addAll(repairedTasks);
- tasks.addAll(unrepairedTasks);
+ for (AbstractCompactionStrategy strategy : repaired)
+ {
+ Collection<AbstractCompactionTask> task = strategy.getMaximalTask(gcBefore, splitOutput);
+ if (task != null)
+ tasks.addAll(task);
+ }
+ for (AbstractCompactionStrategy strategy : unrepaired)
+ {
+ Collection<AbstractCompactionTask> task = strategy.getMaximalTask(gcBefore, splitOutput);
+ if (task != null)
+ tasks.addAll(task);
+ }
+ if (tasks.isEmpty())
+ return null;
return tasks;
}
}
@@ -440,14 +551,18 @@
public AbstractCompactionTask getUserDefinedTask(Collection<SSTableReader> sstables, int gcBefore)
{
+ maybeReload(cfs.metadata);
+ validateForCompaction(sstables);
return getCompactionStrategyFor(sstables.iterator().next()).getUserDefinedTask(sstables, gcBefore);
}
public int getEstimatedRemainingTasks()
{
int tasks = 0;
- tasks += repaired.getEstimatedRemainingTasks();
- tasks += unrepaired.getEstimatedRemainingTasks();
+ for (AbstractCompactionStrategy strategy : repaired)
+ tasks += strategy.getEstimatedRemainingTasks();
+ for (AbstractCompactionStrategy strategy : unrepaired)
+ tasks += strategy.getEstimatedRemainingTasks();
return tasks;
}
@@ -459,10 +574,10 @@
public String getName()
{
- return unrepaired.getName();
+ return unrepaired.get(0).getName();
}
- public List<AbstractCompactionStrategy> getStrategies()
+ public List<List<AbstractCompactionStrategy>> getStrategies()
{
return Arrays.asList(repaired, unrepaired);
}
@@ -480,12 +595,25 @@
private void setStrategy(CompactionParams params)
{
- if (repaired != null)
- repaired.shutdown();
- if (unrepaired != null)
- unrepaired.shutdown();
- repaired = CFMetaData.createCompactionStrategyInstance(cfs, params);
- unrepaired = CFMetaData.createCompactionStrategyInstance(cfs, params);
+ repaired.forEach(AbstractCompactionStrategy::shutdown);
+ unrepaired.forEach(AbstractCompactionStrategy::shutdown);
+ repaired.clear();
+ unrepaired.clear();
+
+ if (cfs.getPartitioner().splitter().isPresent())
+ {
+ locations = cfs.getDirectories().getWriteableLocations();
+ for (int i = 0; i < locations.length; i++)
+ {
+ repaired.add(CFMetaData.createCompactionStrategyInstance(cfs, params));
+ unrepaired.add(CFMetaData.createCompactionStrategyInstance(cfs, params));
+ }
+ }
+ else
+ {
+ repaired.add(CFMetaData.createCompactionStrategyInstance(cfs, params));
+ unrepaired.add(CFMetaData.createCompactionStrategyInstance(cfs, params));
+ }
this.params = params;
}
@@ -499,15 +627,21 @@
return Boolean.parseBoolean(params.options().get(AbstractCompactionStrategy.ONLY_PURGE_REPAIRED_TOMBSTONES));
}
- public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor, long keyCount, long repairedAt, MetadataCollector collector, SerializationHeader header, LifecycleTransaction txn)
+ public SSTableMultiWriter createSSTableMultiWriter(Descriptor descriptor,
+ long keyCount,
+ long repairedAt,
+ MetadataCollector collector,
+ SerializationHeader header,
+ Collection<Index> indexes,
+ LifecycleTransaction txn)
{
if (repairedAt == ActiveRepairService.UNREPAIRED_SSTABLE)
{
- return unrepaired.createSSTableMultiWriter(descriptor, keyCount, repairedAt, collector, header, txn);
+ return unrepaired.get(0).createSSTableMultiWriter(descriptor, keyCount, repairedAt, collector, header, indexes, txn);
}
else
{
- return repaired.createSSTableMultiWriter(descriptor, keyCount, repairedAt, collector, header, txn);
+ return repaired.get(0).createSSTableMultiWriter(descriptor, keyCount, repairedAt, collector, header, indexes, txn);
}
}
}
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index be81c80..6b9fe21 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@ -230,7 +230,7 @@
LifecycleTransaction transaction,
Set<SSTableReader> nonExpiredSSTables)
{
- return new DefaultCompactionWriter(cfs, directories, transaction, nonExpiredSSTables, offline, keepOriginals);
+ return new DefaultCompactionWriter(cfs, directories, transaction, nonExpiredSSTables, offline, keepOriginals, getLevel());
}
public static String updateCompactionHistory(String keyspaceName, String columnFamilyName, long[] mergedRowCounts, long startSize, long endSize)
diff --git a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
index 50f9b71..2dc6ee8 100644
--- a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
@@ -85,7 +85,7 @@
*/
private List<SSTableReader> getNextBackgroundSSTables(final int gcBefore)
{
- if (Iterables.isEmpty(cfs.getSSTables(SSTableSet.LIVE)))
+ if (sstables.isEmpty())
return Collections.emptyList();
Set<SSTableReader> uncompacting = ImmutableSet.copyOf(filter(cfs.getUncompactingSSTables(), sstables::contains));
@@ -212,6 +212,7 @@
{
sstables.remove(sstable);
}
+
/**
* A target time span used for bucketing SSTables based on timestamps.
*/
diff --git a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
index b0cd077..3246f20 100644
--- a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
+++ b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
@@ -354,7 +354,7 @@
// small in L0.
return getSTCSInL0CompactionCandidate();
}
- return new CompactionCandidate(candidates, getNextLevel(candidates), cfs.getCompactionStrategyManager().getMaxSSTableBytes());
+ return new CompactionCandidate(candidates, getNextLevel(candidates), maxSSTableSizeInBytes);
}
private CompactionCandidate getSTCSInL0CompactionCandidate()
diff --git a/src/java/org/apache/cassandra/db/compaction/OperationType.java b/src/java/org/apache/cassandra/db/compaction/OperationType.java
index 20e6df2..84a34c9 100644
--- a/src/java/org/apache/cassandra/db/compaction/OperationType.java
+++ b/src/java/org/apache/cassandra/db/compaction/OperationType.java
@@ -37,7 +37,8 @@
STREAM("Stream"),
WRITE("Write"),
VIEW_BUILD("View build"),
- INDEX_SUMMARY("Index summary redistribution");
+ INDEX_SUMMARY("Index summary redistribution"),
+ RELOCATE("Relocate sstables to correct disk");
public final String type;
public final String fileName;
diff --git a/src/java/org/apache/cassandra/db/compaction/Scrubber.java b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
index 272c2f8..838b0a1 100644
--- a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
+++ b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
@@ -104,11 +104,8 @@
List<SSTableReader> toScrub = Collections.singletonList(sstable);
- // Calculate the expected compacted filesize
- this.destination = cfs.getDirectories().getWriteableLocationAsFile(cfs.getExpectedCompactedFileSize(toScrub, OperationType.SCRUB));
- if (destination == null)
- throw new IOException("disk full");
-
+ int locIndex = CompactionStrategyManager.getCompactionStrategyIndex(cfs, cfs.getDirectories(), sstable);
+ this.destination = cfs.getDirectories().getLocationForDisk(cfs.getDirectories().getWriteableLocations()[locIndex]);
this.isCommutative = cfs.metadata.isCounter();
boolean hasIndexFile = (new File(sstable.descriptor.filenameFor(Component.PRIMARY_INDEX))).exists();
diff --git a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
index f8a8240..e36adf2 100644
--- a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
@@ -21,6 +21,7 @@
import java.util.Map.Entry;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -84,7 +85,7 @@
List<List<SSTableReader>> buckets = getBuckets(createSSTableAndLengthPairs(candidates), sizeTieredOptions.bucketHigh, sizeTieredOptions.bucketLow, sizeTieredOptions.minSSTableSize);
logger.trace("Compaction buckets are {}", buckets);
- updateEstimatedCompactionsByTasks(buckets);
+ estimatedRemainingTasks = getEstimatedCompactionsByTasks(cfs, buckets);
List<SSTableReader> mostInteresting = mostInterestingBucket(buckets, minThreshold, maxThreshold);
if (!mostInteresting.isEmpty())
return mostInteresting;
@@ -282,15 +283,15 @@
return new ArrayList<List<T>>(buckets.values());
}
- private void updateEstimatedCompactionsByTasks(List<List<SSTableReader>> tasks)
+ public static int getEstimatedCompactionsByTasks(ColumnFamilyStore cfs, List<List<SSTableReader>> tasks)
{
int n = 0;
- for (List<SSTableReader> bucket: tasks)
+ for (List<SSTableReader> bucket : tasks)
{
if (bucket.size() >= cfs.getMinimumCompactionThreshold())
n += Math.ceil((double)bucket.size() / cfs.getMaximumCompactionThreshold());
}
- estimatedRemainingTasks = n;
+ return n;
}
public long getMaxSSTableBytes()
diff --git a/src/java/org/apache/cassandra/db/compaction/Upgrader.java b/src/java/org/apache/cassandra/db/compaction/Upgrader.java
index fcd1a3c..3f0f9a3 100644
--- a/src/java/org/apache/cassandra/db/compaction/Upgrader.java
+++ b/src/java/org/apache/cassandra/db/compaction/Upgrader.java
@@ -75,6 +75,7 @@
cfs.metadata,
sstableMetadataCollector,
SerializationHeader.make(cfs.metadata, Sets.newHashSet(sstable)),
+ cfs.indexManager.listIndexes(),
transaction);
}
diff --git a/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java b/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
index 0b3b7d0..46023ce 100644
--- a/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
+++ b/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
@@ -18,18 +18,26 @@
package org.apache.cassandra.db.compaction.writers;
+import java.io.File;
import java.util.Collection;
+import java.util.List;
import java.util.Set;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.Directories;
+import org.apache.cassandra.db.PartitionPosition;
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
import org.apache.cassandra.db.compaction.CompactionTask;
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.io.sstable.SSTableRewriter;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.utils.concurrent.Transactional;
+import org.apache.cassandra.db.compaction.OperationType;
+import org.apache.cassandra.service.StorageService;
/**
@@ -38,6 +46,8 @@
*/
public abstract class CompactionAwareWriter extends Transactional.AbstractTransactional implements Transactional
{
+ protected static final Logger logger = LoggerFactory.getLogger(DefaultCompactionWriter.class);
+
protected final ColumnFamilyStore cfs;
protected final Directories directories;
protected final Set<SSTableReader> nonExpiredSSTables;
@@ -45,9 +55,11 @@
protected final long maxAge;
protected final long minRepairedAt;
- protected final LifecycleTransaction txn;
protected final SSTableRewriter sstableWriter;
- private boolean isInitialized = false;
+ protected final LifecycleTransaction txn;
+ private final Directories.DataDirectory[] locations;
+ private final List<PartitionPosition> diskBoundaries;
+ private int locationIndex;
public CompactionAwareWriter(ColumnFamilyStore cfs,
Directories directories,
@@ -59,12 +71,15 @@
this.cfs = cfs;
this.directories = directories;
this.nonExpiredSSTables = nonExpiredSSTables;
- this.estimatedTotalKeys = SSTableReader.getApproximateKeyCount(nonExpiredSSTables);
- this.maxAge = CompactionTask.getMaxDataAge(nonExpiredSSTables);
- this.minRepairedAt = CompactionTask.getMinRepairedAt(nonExpiredSSTables);
this.txn = txn;
- this.sstableWriter = SSTableRewriter.constructKeepingOriginals(txn, keepOriginals, maxAge, offline);
+ estimatedTotalKeys = SSTableReader.getApproximateKeyCount(nonExpiredSSTables);
+ maxAge = CompactionTask.getMaxDataAge(nonExpiredSSTables);
+ sstableWriter = SSTableRewriter.constructKeepingOriginals(txn, keepOriginals, maxAge, offline);
+ minRepairedAt = CompactionTask.getMinRepairedAt(nonExpiredSSTables);
+ locations = cfs.getDirectories().getWriteableLocations();
+ diskBoundaries = StorageService.getDiskBoundaries(cfs);
+ locationIndex = -1;
}
@Override
@@ -96,6 +111,8 @@
return sstableWriter.finished();
}
+ public abstract List<SSTableReader> finish(long repairedAt);
+
/**
* estimated number of keys we should write
*/
@@ -104,6 +121,11 @@
return estimatedTotalKeys;
}
+ /**
+ * Writes a partition in an implementation specific way
+ * @param partition the partition to append
+ * @return true if the partition was written, false otherwise
+ */
public final boolean append(UnfilteredRowIterator partition)
{
maybeSwitchWriter(partition.partitionKey());
@@ -125,9 +147,26 @@
*/
protected void maybeSwitchWriter(DecoratedKey key)
{
- if (!isInitialized)
- switchCompactionLocation(getDirectories().getWriteableLocation(cfs.getExpectedCompactedFileSize(nonExpiredSSTables, txn.opType())));
- isInitialized = true;
+ if (diskBoundaries == null)
+ {
+ if (locationIndex < 0)
+ {
+ Directories.DataDirectory defaultLocation = getWriteDirectory(nonExpiredSSTables, cfs.getExpectedCompactedFileSize(nonExpiredSSTables, OperationType.UNKNOWN));
+ switchCompactionLocation(defaultLocation);
+ locationIndex = 0;
+ }
+ return;
+ }
+
+ if (locationIndex > -1 && key.compareTo(diskBoundaries.get(locationIndex)) < 0)
+ return;
+
+ int prevIdx = locationIndex;
+ while (locationIndex == -1 || key.compareTo(diskBoundaries.get(locationIndex)) > 0)
+ locationIndex++;
+ if (prevIdx >= 0)
+ logger.debug("Switching write location from {} to {}", locations[prevIdx], locations[locationIndex]);
+ switchCompactionLocation(locations[locationIndex]);
}
/**
@@ -148,13 +187,37 @@
/**
* Return a directory where we can expect expectedWriteSize to fit.
+ *
+ * @param sstables the sstables to compact
+ * @return
*/
- public Directories.DataDirectory getWriteDirectory(long expectedWriteSize)
+ public Directories.DataDirectory getWriteDirectory(Iterable<SSTableReader> sstables, long estimatedWriteSize)
{
- Directories.DataDirectory directory = getDirectories().getWriteableLocation(expectedWriteSize);
- if (directory == null)
- throw new RuntimeException("Insufficient disk space to write " + expectedWriteSize + " bytes");
+ File directory = null;
+ for (SSTableReader sstable : sstables)
+ {
+ if (directory == null)
+ directory = sstable.descriptor.directory;
+ if (!directory.equals(sstable.descriptor.directory))
+ logger.trace("All sstables not from the same disk - putting results in {}", directory);
+ }
+ Directories.DataDirectory d = getDirectories().getDataDirectoryForFile(directory);
+ if (d != null)
+ {
+ if (d.getAvailableSpace() < estimatedWriteSize)
+ throw new RuntimeException(String.format("Not enough space to write %d bytes to %s (%d bytes available)", estimatedWriteSize, d.location, d.getAvailableSpace()));
+ logger.trace("putting compaction results in {}", directory);
+ return d;
+ }
+ d = getDirectories().getWriteableLocation(estimatedWriteSize);
+ if (d == null)
+ throw new RuntimeException("Not enough disk space to store "+estimatedWriteSize+" bytes");
+ return d;
+ }
- return directory;
+ public CompactionAwareWriter setRepairedAt(long repairedAt)
+ {
+ this.sstableWriter.setRepairedAt(repairedAt);
+ return this;
}
}
diff --git a/src/java/org/apache/cassandra/db/compaction/writers/DefaultCompactionWriter.java b/src/java/org/apache/cassandra/db/compaction/writers/DefaultCompactionWriter.java
index 8b90224..5e78834 100644
--- a/src/java/org/apache/cassandra/db/compaction/writers/DefaultCompactionWriter.java
+++ b/src/java/org/apache/cassandra/db/compaction/writers/DefaultCompactionWriter.java
@@ -18,6 +18,7 @@
package org.apache.cassandra.db.compaction.writers;
+import java.util.List;
import java.util.Set;
import org.slf4j.Logger;
@@ -39,16 +40,18 @@
public class DefaultCompactionWriter extends CompactionAwareWriter
{
protected static final Logger logger = LoggerFactory.getLogger(DefaultCompactionWriter.class);
+ private final int sstableLevel;
public DefaultCompactionWriter(ColumnFamilyStore cfs, Directories directories, LifecycleTransaction txn, Set<SSTableReader> nonExpiredSSTables)
{
- this(cfs, directories, txn, nonExpiredSSTables, false, false);
+ this(cfs, directories, txn, nonExpiredSSTables, false, false, 0);
}
@SuppressWarnings("resource")
- public DefaultCompactionWriter(ColumnFamilyStore cfs, Directories directories, LifecycleTransaction txn, Set<SSTableReader> nonExpiredSSTables, boolean offline, boolean keepOriginals)
+ public DefaultCompactionWriter(ColumnFamilyStore cfs, Directories directories, LifecycleTransaction txn, Set<SSTableReader> nonExpiredSSTables, boolean offline, boolean keepOriginals, int sstableLevel)
{
super(cfs, directories, txn, nonExpiredSSTables, offline, keepOriginals);
+ this.sstableLevel = sstableLevel;
}
@Override
@@ -58,20 +61,27 @@
}
@Override
- protected void switchCompactionLocation(Directories.DataDirectory directory)
+ public void switchCompactionLocation(Directories.DataDirectory directory)
{
@SuppressWarnings("resource")
SSTableWriter writer = SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(directory))),
estimatedTotalKeys,
minRepairedAt,
cfs.metadata,
- new MetadataCollector(txn.originals(), cfs.metadata.comparator, 0),
+ new MetadataCollector(txn.originals(), cfs.metadata.comparator, sstableLevel),
SerializationHeader.make(cfs.metadata, nonExpiredSSTables),
+ cfs.indexManager.listIndexes(),
txn);
sstableWriter.switchWriter(writer);
}
@Override
+ public List<SSTableReader> finish(long repairedAt)
+ {
+ return sstableWriter.setRepairedAt(repairedAt).finish();
+ }
+
+ @Override
public long estimatedKeys()
{
return estimatedTotalKeys;
diff --git a/src/java/org/apache/cassandra/db/compaction/writers/MajorLeveledCompactionWriter.java b/src/java/org/apache/cassandra/db/compaction/writers/MajorLeveledCompactionWriter.java
index 6d191f8..0c88ac6 100644
--- a/src/java/org/apache/cassandra/db/compaction/writers/MajorLeveledCompactionWriter.java
+++ b/src/java/org/apache/cassandra/db/compaction/writers/MajorLeveledCompactionWriter.java
@@ -17,12 +17,9 @@
*/
package org.apache.cassandra.db.compaction.writers;
-import java.io.File;
+import java.util.List;
import java.util.Set;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.Directories;
import org.apache.cassandra.db.RowIndexEntry;
@@ -37,15 +34,14 @@
public class MajorLeveledCompactionWriter extends CompactionAwareWriter
{
- private static final Logger logger = LoggerFactory.getLogger(MajorLeveledCompactionWriter.class);
private final long maxSSTableSize;
- private final long expectedWriteSize;
- private final Set<SSTableReader> allSSTables;
private int currentLevel = 1;
private long averageEstimatedKeysPerSSTable;
private long partitionsWritten = 0;
private long totalWrittenInLevel = 0;
private int sstablesWritten = 0;
+ private final long keysPerSSTable;
+ private Directories.DataDirectory sstableDirectory;
public MajorLeveledCompactionWriter(ColumnFamilyStore cfs,
Directories directories,
@@ -67,8 +63,8 @@
{
super(cfs, directories, txn, nonExpiredSSTables, offline, keepOriginals);
this.maxSSTableSize = maxSSTableSize;
- this.allSSTables = txn.originals();
- expectedWriteSize = Math.min(maxSSTableSize, cfs.getExpectedCompactedFileSize(nonExpiredSSTables, txn.opType()));
+ long estimatedSSTables = Math.max(1, SSTableReader.getTotalBytes(nonExpiredSSTables) / maxSSTableSize);
+ keysPerSSTable = estimatedTotalKeys / estimatedSSTables;
}
@Override
@@ -86,27 +82,33 @@
totalWrittenInLevel = 0;
currentLevel++;
}
-
- averageEstimatedKeysPerSSTable = Math.round(((double) averageEstimatedKeysPerSSTable * sstablesWritten + partitionsWritten) / (sstablesWritten + 1));
- switchCompactionLocation(getWriteDirectory(expectedWriteSize));
- partitionsWritten = 0;
- sstablesWritten++;
+ switchCompactionLocation(sstableDirectory);
}
return rie != null;
}
- public void switchCompactionLocation(Directories.DataDirectory directory)
+ @Override
+ public void switchCompactionLocation(Directories.DataDirectory location)
{
- File sstableDirectory = getDirectories().getLocationForDisk(directory);
- @SuppressWarnings("resource")
- SSTableWriter writer = SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(sstableDirectory)),
- averageEstimatedKeysPerSSTable,
- minRepairedAt,
- cfs.metadata,
- new MetadataCollector(allSSTables, cfs.metadata.comparator, currentLevel),
- SerializationHeader.make(cfs.metadata, nonExpiredSSTables),
- txn);
- sstableWriter.switchWriter(writer);
+ this.sstableDirectory = location;
+ averageEstimatedKeysPerSSTable = Math.round(((double) averageEstimatedKeysPerSSTable * sstablesWritten + partitionsWritten) / (sstablesWritten + 1));
+ sstableWriter.switchWriter(SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(sstableDirectory))),
+ keysPerSSTable,
+ minRepairedAt,
+ cfs.metadata,
+ new MetadataCollector(txn.originals(), cfs.metadata.comparator, currentLevel),
+ SerializationHeader.make(cfs.metadata, txn.originals()),
+ cfs.indexManager.listIndexes(),
+ txn));
+ partitionsWritten = 0;
+ sstablesWritten = 0;
+
+ }
+
+ @Override
+ public List<SSTableReader> finish(long repairedAt)
+ {
+ return sstableWriter.setRepairedAt(repairedAt).finish();
}
}
diff --git a/src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java b/src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java
index 142fe87..ac83cc6 100644
--- a/src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java
+++ b/src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java
@@ -17,6 +17,8 @@
*/
package org.apache.cassandra.db.compaction.writers;
+import java.io.File;
+import java.util.List;
import java.util.Set;
import org.apache.cassandra.db.ColumnFamilyStore;
@@ -33,11 +35,11 @@
public class MaxSSTableSizeWriter extends CompactionAwareWriter
{
private final long estimatedTotalKeys;
- private final long expectedWriteSize;
private final long maxSSTableSize;
private final int level;
private final long estimatedSSTables;
private final Set<SSTableReader> allSSTables;
+ private Directories.DataDirectory sstableDirectory;
public MaxSSTableSizeWriter(ColumnFamilyStore cfs,
Directories directories,
@@ -63,34 +65,39 @@
this.allSSTables = txn.originals();
this.level = level;
this.maxSSTableSize = maxSSTableSize;
- long totalSize = cfs.getExpectedCompactedFileSize(nonExpiredSSTables, txn.opType());
- expectedWriteSize = Math.min(maxSSTableSize, totalSize);
estimatedTotalKeys = SSTableReader.getApproximateKeyCount(nonExpiredSSTables);
estimatedSSTables = Math.max(1, estimatedTotalKeys / maxSSTableSize);
}
- @Override
- public boolean realAppend(UnfilteredRowIterator partition)
+ protected boolean realAppend(UnfilteredRowIterator partition)
{
RowIndexEntry rie = sstableWriter.append(partition);
if (sstableWriter.currentWriter().getOnDiskFilePointer() > maxSSTableSize)
- switchCompactionLocation(getWriteDirectory(expectedWriteSize));
+ {
+ switchCompactionLocation(sstableDirectory);
+ }
return rie != null;
}
+ @Override
public void switchCompactionLocation(Directories.DataDirectory location)
{
- @SuppressWarnings("resource")
- SSTableWriter writer = SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(location))),
+ sstableDirectory = location;
+ SSTableWriter writer = SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(sstableDirectory))),
estimatedTotalKeys / estimatedSSTables,
minRepairedAt,
cfs.metadata,
new MetadataCollector(allSSTables, cfs.metadata.comparator, level),
SerializationHeader.make(cfs.metadata, nonExpiredSSTables),
+ cfs.indexManager.listIndexes(),
txn);
sstableWriter.switchWriter(writer);
+ }
+ public List<SSTableReader> finish(long repairedAt)
+ {
+ return sstableWriter.setRepairedAt(repairedAt).finish();
}
@Override
diff --git a/src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java b/src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java
index 796391c..46183dc 100644
--- a/src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java
+++ b/src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java
@@ -17,8 +17,8 @@
*/
package org.apache.cassandra.db.compaction.writers;
-import java.io.File;
import java.util.Arrays;
+import java.util.List;
import java.util.Set;
import org.slf4j.Logger;
@@ -51,6 +51,7 @@
private final Set<SSTableReader> allSSTables;
private long currentBytesToWrite;
private int currentRatioIndex = 0;
+ private Directories.DataDirectory location;
public SplittingSizeTieredCompactionWriter(ColumnFamilyStore cfs, Directories directories, LifecycleTransaction txn, Set<SSTableReader> nonExpiredSSTables)
{
@@ -82,10 +83,7 @@
}
}
ratios = Arrays.copyOfRange(potentialRatios, 0, noPointIndex);
- long currentPartitionsToWrite = Math.round(estimatedTotalKeys * ratios[currentRatioIndex]);
currentBytesToWrite = Math.round(totalSize * ratios[currentRatioIndex]);
- switchCompactionLocation(getWriteDirectory(currentBytesToWrite));
- logger.trace("Ratios={}, expectedKeys = {}, totalSize = {}, currentPartitionsToWrite = {}, currentBytesToWrite = {}", ratios, estimatedTotalKeys, totalSize, currentPartitionsToWrite, currentBytesToWrite);
}
@Override
@@ -96,24 +94,32 @@
{
currentRatioIndex++;
currentBytesToWrite = Math.round(totalSize * ratios[currentRatioIndex]);
- switchCompactionLocation(getWriteDirectory(Math.round(totalSize * ratios[currentRatioIndex])));
+ switchCompactionLocation(location);
+ logger.debug("Switching writer, currentBytesToWrite = {}", currentBytesToWrite);
}
return rie != null;
}
+ @Override
public void switchCompactionLocation(Directories.DataDirectory location)
{
+ this.location = location;
long currentPartitionsToWrite = Math.round(ratios[currentRatioIndex] * estimatedTotalKeys);
- @SuppressWarnings("resource")
SSTableWriter writer = SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(location))),
currentPartitionsToWrite,
minRepairedAt,
cfs.metadata,
new MetadataCollector(allSSTables, cfs.metadata.comparator, 0),
SerializationHeader.make(cfs.metadata, nonExpiredSSTables),
+ cfs.indexManager.listIndexes(),
txn);
logger.trace("Switching writer, currentPartitionsToWrite = {}", currentPartitionsToWrite);
sstableWriter.switchWriter(writer);
+ }
+ @Override
+ public List<SSTableReader> finish(long repairedAt)
+ {
+ return sstableWriter.setRepairedAt(repairedAt).finish();
}
}
diff --git a/src/java/org/apache/cassandra/db/filter/ClusteringIndexNamesFilter.java b/src/java/org/apache/cassandra/db/filter/ClusteringIndexNamesFilter.java
index a81a7a6..17cd802 100644
--- a/src/java/org/apache/cassandra/db/filter/ClusteringIndexNamesFilter.java
+++ b/src/java/org/apache/cassandra/db/filter/ClusteringIndexNamesFilter.java
@@ -229,7 +229,7 @@
public String toCQLString(CFMetaData metadata)
{
- if (clusterings.isEmpty())
+ if (metadata.clusteringColumns().isEmpty() || clusterings.size() <= 1)
return "";
StringBuilder sb = new StringBuilder();
diff --git a/src/java/org/apache/cassandra/db/filter/ColumnFilter.java b/src/java/org/apache/cassandra/db/filter/ColumnFilter.java
index 05eade5..e22c154 100644
--- a/src/java/org/apache/cassandra/db/filter/ColumnFilter.java
+++ b/src/java/org/apache/cassandra/db/filter/ColumnFilter.java
@@ -323,9 +323,12 @@
return "<none>";
StringBuilder sb = new StringBuilder();
- appendColumnDef(sb, defs.next());
while (defs.hasNext())
- appendColumnDef(sb.append(", "), defs.next());
+ {
+ appendColumnDef(sb, defs.next());
+ if (defs.hasNext())
+ sb.append(", ");
+ }
return sb.toString();
}
diff --git a/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java b/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java
index d9d9213..c557bf2 100644
--- a/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java
+++ b/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java
@@ -39,7 +39,8 @@
* A set of log replicas. This class mostly iterates over replicas when writing or reading,
* ensuring consistency among them and hiding replication details from LogFile.
*
- * @see LogReplica, LogFile
+ * @see LogReplica
+ * @see LogFile
*/
public class LogReplicaSet
{
diff --git a/src/java/org/apache/cassandra/db/lifecycle/Tracker.java b/src/java/org/apache/cassandra/db/lifecycle/Tracker.java
index c09d49c..4c73472 100644
--- a/src/java/org/apache/cassandra/db/lifecycle/Tracker.java
+++ b/src/java/org/apache/cassandra/db/lifecycle/Tracker.java
@@ -327,10 +327,10 @@
apply(View.markFlushing(memtable));
}
- public void replaceFlushed(Memtable memtable, Collection<SSTableReader> sstables)
+ public void replaceFlushed(Memtable memtable, Iterable<SSTableReader> sstables)
{
assert !isDummy();
- if (sstables == null || sstables.isEmpty())
+ if (sstables == null || Iterables.isEmpty(sstables))
{
// sstable may be null if we flushed batchlog and nothing needed to be retained
// if it's null, we don't care what state the cfstore is in, we just replace it and continue
diff --git a/src/java/org/apache/cassandra/db/lifecycle/View.java b/src/java/org/apache/cassandra/db/lifecycle/View.java
index b62c7e3..63926ed 100644
--- a/src/java/org/apache/cassandra/db/lifecycle/View.java
+++ b/src/java/org/apache/cassandra/db/lifecycle/View.java
@@ -310,7 +310,7 @@
}
// called after flush: removes memtable from flushingMemtables, and inserts flushed into the live sstable set
- static Function<View, View> replaceFlushed(final Memtable memtable, final Collection<SSTableReader> flushed)
+ static Function<View, View> replaceFlushed(final Memtable memtable, final Iterable<SSTableReader> flushed)
{
return new Function<View, View>()
{
diff --git a/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java b/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
index ad4050d..130ed7b 100644
--- a/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
+++ b/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
@@ -21,6 +21,7 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.regex.Pattern;
import org.apache.cassandra.cql3.Term;
import org.apache.cassandra.serializers.TypeSerializer;
@@ -107,6 +108,10 @@
}
return l.toArray(new ByteBuffer[l.size()]);
}
+ private static final String COLON = ":";
+ private static final Pattern COLON_PAT = Pattern.compile(COLON);
+ private static final String ESCAPED_COLON = "\\\\:";
+ private static final Pattern ESCAPED_COLON_PAT = Pattern.compile(ESCAPED_COLON);
/*
@@ -118,7 +123,7 @@
if (input.isEmpty())
return input;
- String res = input.replaceAll(":", "\\\\:");
+ String res = COLON_PAT.matcher(input).replaceAll(ESCAPED_COLON);
char last = res.charAt(res.length() - 1);
return last == '\\' || last == '!' ? res + '!' : res;
}
@@ -132,7 +137,7 @@
if (input.isEmpty())
return input;
- String res = input.replaceAll("\\\\:", ":");
+ String res = ESCAPED_COLON_PAT.matcher(input).replaceAll(COLON);
char last = res.charAt(res.length() - 1);
return last == '!' ? res.substring(0, res.length() - 1) : res;
}
diff --git a/src/java/org/apache/cassandra/db/marshal/CompositeType.java b/src/java/org/apache/cassandra/db/marshal/CompositeType.java
index 126307b..ca6a617 100644
--- a/src/java/org/apache/cassandra/db/marshal/CompositeType.java
+++ b/src/java/org/apache/cassandra/db/marshal/CompositeType.java
@@ -395,7 +395,9 @@
for (ByteBuffer bb : buffers)
{
ByteBufferUtil.writeShortLength(out, bb.remaining());
- out.put(bb.duplicate());
+ int toCopy = bb.remaining();
+ ByteBufferUtil.arrayCopy(bb, bb.position(), out, out.position(), toCopy);
+ out.position(out.position() + toCopy);
out.put((byte) 0);
}
out.flip();
diff --git a/src/java/org/apache/cassandra/db/marshal/TupleType.java b/src/java/org/apache/cassandra/db/marshal/TupleType.java
index 9480229..362388b 100644
--- a/src/java/org/apache/cassandra/db/marshal/TupleType.java
+++ b/src/java/org/apache/cassandra/db/marshal/TupleType.java
@@ -22,6 +22,7 @@
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
+import java.util.regex.Pattern;
import com.google.common.base.Objects;
@@ -37,6 +38,15 @@
*/
public class TupleType extends AbstractType<ByteBuffer>
{
+ private static final String COLON = ":";
+ private static final Pattern COLON_PAT = Pattern.compile(COLON);
+ private static final String ESCAPED_COLON = "\\\\:";
+ private static final Pattern ESCAPED_COLON_PAT = Pattern.compile(ESCAPED_COLON);
+ private static final String AT = "@";
+ private static final Pattern AT_PAT = Pattern.compile(AT);
+ private static final String ESCAPED_AT = "\\\\@";
+ private static final Pattern ESCAPED_AT_PAT = Pattern.compile(ESCAPED_AT);
+
protected final List<AbstractType<?>> types;
public TupleType(List<AbstractType<?>> types)
@@ -215,7 +225,9 @@
ByteBuffer field = ByteBufferUtil.readBytes(input, size);
// We use ':' as delimiter, and @ to represent null, so escape them in the generated string
- sb.append(type.getString(field).replaceAll(":", "\\\\:").replaceAll("@", "\\\\@"));
+ String fld = COLON_PAT.matcher(type.getString(field)).replaceAll(ESCAPED_COLON);
+ fld = AT_PAT.matcher(fld).replaceAll(ESCAPED_AT);
+ sb.append(fld);
}
return sb.toString();
}
@@ -238,7 +250,9 @@
continue;
AbstractType<?> type = type(i);
- fields[i] = type.fromString(fieldString.replaceAll("\\\\:", ":").replaceAll("\\\\@", "@"));
+ fieldString = ESCAPED_COLON_PAT.matcher(fieldString).replaceAll(COLON);
+ fieldString = ESCAPED_AT_PAT.matcher(fieldString).replaceAll(AT);
+ fields[i] = type.fromString(fieldString);
}
return buildValue(fields);
}
diff --git a/src/java/org/apache/cassandra/db/marshal/UUIDType.java b/src/java/org/apache/cassandra/db/marshal/UUIDType.java
index acaf27c..9722a52 100644
--- a/src/java/org/apache/cassandra/db/marshal/UUIDType.java
+++ b/src/java/org/apache/cassandra/db/marshal/UUIDType.java
@@ -140,7 +140,7 @@
{
try
{
- return ByteBuffer.wrap(UUIDGen.decompose(UUID.fromString(source)));
+ return UUIDGen.toByteBuffer(UUID.fromString(source));
}
catch (IllegalArgumentException e)
{
diff --git a/src/java/org/apache/cassandra/db/monitoring/ApproximateTime.java b/src/java/org/apache/cassandra/db/monitoring/ApproximateTime.java
new file mode 100644
index 0000000..1d57398
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/monitoring/ApproximateTime.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.Config;
+
+/**
+ * This is an approximation of System.currentTimeInMillis(). It updates its
+ * time value at periodic intervals of CHECK_INTERVAL_MS milliseconds
+ * (currently 10 milliseconds by default). It can be used as a faster alternative
+ * to System.currentTimeInMillis() every time an imprecision of a few milliseconds
+ * can be accepted.
+ */
+public class ApproximateTime
+{
+ private static final Logger logger = LoggerFactory.getLogger(ApproximateTime.class);
+ private static final int CHECK_INTERVAL_MS = Math.max(5, Integer.valueOf(System.getProperty(Config.PROPERTY_PREFIX + "approximate_time_precision_ms", "10")));
+
+ private static volatile long time = System.currentTimeMillis();
+ static
+ {
+ logger.info("Scheduling approximate time-check task with a precision of {} milliseconds", CHECK_INTERVAL_MS);
+ ScheduledExecutors.scheduledFastTasks.scheduleWithFixedDelay(() -> time = System.currentTimeMillis(),
+ CHECK_INTERVAL_MS,
+ CHECK_INTERVAL_MS,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public static long currentTimeMillis()
+ {
+ return time;
+ }
+
+ public static long precision()
+ {
+ return 2 * CHECK_INTERVAL_MS;
+ }
+
+}
diff --git a/src/java/org/apache/cassandra/db/monitoring/ConstructionTime.java b/src/java/org/apache/cassandra/db/monitoring/ConstructionTime.java
new file mode 100644
index 0000000..d6b6078
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/monitoring/ConstructionTime.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+public final class ConstructionTime
+{
+ public final long timestamp;
+ public final boolean isCrossNode;
+
+ public ConstructionTime()
+ {
+ this(ApproximateTime.currentTimeMillis());
+ }
+
+ public ConstructionTime(long timestamp)
+ {
+ this(timestamp, false);
+ }
+
+ public ConstructionTime(long timestamp, boolean isCrossNode)
+ {
+ this.timestamp = timestamp;
+ this.isCrossNode = isCrossNode;
+ }
+}
diff --git a/src/java/org/apache/cassandra/db/monitoring/Monitorable.java b/src/java/org/apache/cassandra/db/monitoring/Monitorable.java
new file mode 100644
index 0000000..202ac87
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/monitoring/Monitorable.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+public interface Monitorable
+{
+ String name();
+ ConstructionTime constructionTime();
+ long timeout();
+
+ boolean isInProgress();
+ boolean isAborted();
+ boolean isCompleted();
+
+ boolean abort();
+ boolean complete();
+}
diff --git a/src/java/org/apache/cassandra/db/monitoring/MonitorableImpl.java b/src/java/org/apache/cassandra/db/monitoring/MonitorableImpl.java
new file mode 100644
index 0000000..f89f8ad
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/monitoring/MonitorableImpl.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+public abstract class MonitorableImpl implements Monitorable
+{
+ private MonitoringState state;
+ private ConstructionTime constructionTime;
+ private long timeout;
+
+ protected MonitorableImpl()
+ {
+ this.state = MonitoringState.IN_PROGRESS;
+ }
+
+ /**
+ * This setter is ugly but the construction chain to ReadCommand
+ * is too complex, it would require passing new parameters to all serializers
+ * or specializing the serializers to accept these message properties.
+ */
+ public void setMonitoringTime(ConstructionTime constructionTime, long timeout)
+ {
+ this.constructionTime = constructionTime;
+ this.timeout = timeout;
+ }
+
+ public ConstructionTime constructionTime()
+ {
+ return constructionTime;
+ }
+
+ public long timeout()
+ {
+ return timeout;
+ }
+
+ public boolean isInProgress()
+ {
+ check();
+ return state == MonitoringState.IN_PROGRESS;
+ }
+
+ public boolean isAborted()
+ {
+ check();
+ return state == MonitoringState.ABORTED;
+ }
+
+ public boolean isCompleted()
+ {
+ check();
+ return state == MonitoringState.COMPLETED;
+ }
+
+ public boolean abort()
+ {
+ if (state == MonitoringState.IN_PROGRESS)
+ {
+ if (constructionTime != null)
+ MonitoringTask.addFailedOperation(this, ApproximateTime.currentTimeMillis());
+ state = MonitoringState.ABORTED;
+ return true;
+ }
+
+ return state == MonitoringState.ABORTED;
+ }
+
+ public boolean complete()
+ {
+ if (state == MonitoringState.IN_PROGRESS)
+ {
+ state = MonitoringState.COMPLETED;
+ return true;
+ }
+
+ return state == MonitoringState.COMPLETED;
+ }
+
+ private void check()
+ {
+ if (constructionTime == null || state != MonitoringState.IN_PROGRESS)
+ return;
+
+ long elapsed = ApproximateTime.currentTimeMillis() - constructionTime.timestamp;
+ if (elapsed >= timeout)
+ abort();
+ }
+}
diff --git a/src/java/org/apache/cassandra/db/monitoring/MonitoringState.java b/src/java/org/apache/cassandra/db/monitoring/MonitoringState.java
new file mode 100644
index 0000000..4fe3cf8
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/monitoring/MonitoringState.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+public enum MonitoringState
+{
+ IN_PROGRESS,
+ ABORTED,
+ COMPLETED
+}
diff --git a/src/java/org/apache/cassandra/db/monitoring/MonitoringTask.java b/src/java/org/apache/cassandra/db/monitoring/MonitoringTask.java
new file mode 100644
index 0000000..6d28078
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/monitoring/MonitoringTask.java
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.Config;
+
+import static java.lang.System.getProperty;
+
+/**
+ * A task for monitoring in progress operations, currently only read queries, and aborting them if they time out.
+ * We also log timed out operations, see CASSANDRA-7392.
+ */
+public class MonitoringTask
+{
+ private static final String LINE_SEPARATOR = getProperty("line.separator");
+ private static final Logger logger = LoggerFactory.getLogger(MonitoringTask.class);
+
+ /**
+ * Defines the interval for reporting any operations that have timed out.
+ */
+ private static final int REPORT_INTERVAL_MS = Math.max(0, Integer.valueOf(System.getProperty(Config.PROPERTY_PREFIX + "monitoring_report_interval_ms", "5000")));
+
+ /**
+ * Defines the maximum number of unique timed out queries that will be reported in the logs.
+ * Use a negative number to remove any limit.
+ */
+ private static final int MAX_OPERATIONS = Integer.valueOf(System.getProperty(Config.PROPERTY_PREFIX + "monitoring_max_operations", "50"));
+
+ @VisibleForTesting
+ static MonitoringTask instance = make(REPORT_INTERVAL_MS, MAX_OPERATIONS);
+
+ private final int maxOperations;
+ private final ScheduledFuture<?> reportingTask;
+ private final BlockingQueue<FailedOperation> operationsQueue;
+ private final AtomicLong numDroppedOperations;
+ private long lastLogTime;
+
+ @VisibleForTesting
+ static MonitoringTask make(int reportIntervalMillis, int maxTimedoutOperations)
+ {
+ if (instance != null)
+ {
+ instance.cancel();
+ instance = null;
+ }
+
+ return new MonitoringTask(reportIntervalMillis, maxTimedoutOperations);
+ }
+
+ private MonitoringTask(int reportIntervalMillis, int maxOperations)
+ {
+ this.maxOperations = maxOperations;
+ this.operationsQueue = maxOperations > 0 ? new ArrayBlockingQueue<>(maxOperations) : new LinkedBlockingQueue<>();
+ this.numDroppedOperations = new AtomicLong();
+ this.lastLogTime = ApproximateTime.currentTimeMillis();
+
+ logger.info("Scheduling monitoring task with report interval of {} ms, max operations {}", reportIntervalMillis, maxOperations);
+ this.reportingTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(() -> logFailedOperations(ApproximateTime.currentTimeMillis()),
+ reportIntervalMillis,
+ reportIntervalMillis,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public void cancel()
+ {
+ reportingTask.cancel(false);
+ }
+
+ public static void addFailedOperation(Monitorable operation, long now)
+ {
+ instance.innerAddFailedOperation(operation, now);
+ }
+
+ private void innerAddFailedOperation(Monitorable operation, long now)
+ {
+ if (maxOperations == 0)
+ return; // logging of failed operations disabled
+
+ if (!operationsQueue.offer(new FailedOperation(operation, now)))
+ numDroppedOperations.incrementAndGet();
+ }
+
+ @VisibleForTesting
+ FailedOperations aggregateFailedOperations()
+ {
+ Map<String, FailedOperation> operations = new HashMap<>();
+
+ FailedOperation failedOperation;
+ while((failedOperation = operationsQueue.poll()) != null)
+ {
+ FailedOperation existing = operations.get(failedOperation.name());
+ if (existing != null)
+ existing.addTimeout(failedOperation);
+ else
+ operations.put(failedOperation.name(), failedOperation);
+ }
+
+ return new FailedOperations(operations, numDroppedOperations.getAndSet(0L));
+ }
+
+ @VisibleForTesting
+ List<String> getFailedOperations()
+ {
+ FailedOperations failedOperations = aggregateFailedOperations();
+ String ret = failedOperations.getLogMessage();
+ lastLogTime = ApproximateTime.currentTimeMillis();
+ return ret.isEmpty() ? Collections.emptyList() : Arrays.asList(ret.split("\n"));
+ }
+
+ @VisibleForTesting
+ void logFailedOperations(long now)
+ {
+ FailedOperations failedOperations = aggregateFailedOperations();
+ if (!failedOperations.isEmpty())
+ {
+ long elapsed = now - lastLogTime;
+ logger.warn("{} operations timed out in the last {} msecs, operation list available at debug log level",
+ failedOperations.num(),
+ elapsed);
+
+ if (logger.isDebugEnabled())
+ logger.debug("{} operations timed out in the last {} msecs:{}{}",
+ failedOperations.num(),
+ elapsed,
+ LINE_SEPARATOR,
+ failedOperations.getLogMessage());
+ }
+
+ lastLogTime = now;
+ }
+
+ private static final class FailedOperations
+ {
+ public final Map<String, FailedOperation> operations;
+ public final long numDropped;
+
+ FailedOperations(Map<String, FailedOperation> operations, long numDropped)
+ {
+ this.operations = operations;
+ this.numDropped = numDropped;
+ }
+
+ public boolean isEmpty()
+ {
+ return operations.isEmpty() && numDropped == 0;
+ }
+
+ public long num()
+ {
+ return operations.size() + numDropped;
+ }
+
+ public String getLogMessage()
+ {
+ if (isEmpty())
+ return "";
+
+ final StringBuilder ret = new StringBuilder();
+ operations.values().forEach(o -> addOperation(ret, o));
+
+ if (numDropped > 0)
+ ret.append(LINE_SEPARATOR)
+ .append("... (")
+ .append(numDropped)
+ .append(" were dropped)");
+
+ return ret.toString();
+ }
+
+ private static void addOperation(StringBuilder ret, FailedOperation operation)
+ {
+ if (ret.length() > 0)
+ ret.append(LINE_SEPARATOR);
+
+ ret.append(operation.getLogMessage());
+ }
+ }
+
+ private final static class FailedOperation
+ {
+ public final Monitorable operation;
+ public int numTimeouts;
+ public long totalTime;
+ public long maxTime;
+ public long minTime;
+ private String name;
+
+ FailedOperation(Monitorable operation, long failedAt)
+ {
+ this.operation = operation;
+ numTimeouts = 1;
+ totalTime = failedAt - operation.constructionTime().timestamp;
+ minTime = totalTime;
+ maxTime = totalTime;
+ }
+
+ public String name()
+ {
+ if (name == null)
+ name = operation.name();
+ return name;
+ }
+
+ void addTimeout(FailedOperation operation)
+ {
+ numTimeouts++;
+ totalTime += operation.totalTime;
+ maxTime = Math.max(maxTime, operation.maxTime);
+ minTime = Math.min(minTime, operation.minTime);
+ }
+
+ public String getLogMessage()
+ {
+ if (numTimeouts == 1)
+ return String.format("%s: total time %d msec - timeout %d %s",
+ name(),
+ totalTime,
+ operation.timeout(),
+ operation.constructionTime().isCrossNode ? "msec/cross-node" : "msec");
+ else
+ return String.format("%s (timed out %d times): total time avg/min/max %d/%d/%d msec - timeout %d %s",
+ name(),
+ numTimeouts,
+ totalTime / numTimeouts,
+ minTime,
+ maxTime,
+ operation.timeout(),
+ operation.constructionTime().isCrossNode ? "msec/cross-node" : "msec");
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/db/partitions/CachedPartition.java b/src/java/org/apache/cassandra/db/partitions/CachedPartition.java
index 33e6ecc..0cbaba0 100644
--- a/src/java/org/apache/cassandra/db/partitions/CachedPartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/CachedPartition.java
@@ -45,7 +45,7 @@
/**
* The number of rows that were live at the time the partition was cached.
*
- * See {@link ColumnFamilyStore#isFilterFullyCoveredBy} to see why we need this.
+ * See {@link org.apache.cassandra.db.ColumnFamilyStore#isFilterFullyCoveredBy} to see why we need this.
*
* @return the number of rows in this partition that were live at the time the
* partition was cached (this can be different from the number of live rows now
@@ -58,7 +58,7 @@
* non-deleted cell.
*
* Note that this is generally not a very meaningful number, but this is used by
- * {@link DataLimits#hasEnoughLiveData} as an optimization.
+ * {@link org.apache.cassandra.db.filter.DataLimits#hasEnoughLiveData} as an optimization.
*
* @return the number of row that have at least one non-expiring non-deleted cell.
*/
@@ -86,7 +86,7 @@
* The number of cells in this cached partition that are neither tombstone nor expiring.
*
* Note that this is generally not a very meaningful number, but this is used by
- * {@link DataLimits#hasEnoughLiveData} as an optimization.
+ * {@link org.apache.cassandra.db.filter.DataLimits#hasEnoughLiveData} as an optimization.
*
* @return the number of cells that are neither tombstones nor expiring.
*/
diff --git a/src/java/org/apache/cassandra/db/partitions/PurgeFunction.java b/src/java/org/apache/cassandra/db/partitions/PurgeFunction.java
index 492bab1..b7b01d6 100644
--- a/src/java/org/apache/cassandra/db/partitions/PurgeFunction.java
+++ b/src/java/org/apache/cassandra/db/partitions/PurgeFunction.java
@@ -25,13 +25,13 @@
{
private final boolean isForThrift;
private final DeletionPurger purger;
- private final int nowInSec;
+ private final int gcBefore;
private boolean isReverseOrder;
- public PurgeFunction(boolean isForThrift, int nowInSec, int gcBefore, int oldestUnrepairedTombstone, boolean onlyPurgeRepairedTombstones)
+ public PurgeFunction(boolean isForThrift, int gcBefore, int oldestUnrepairedTombstone, boolean onlyPurgeRepairedTombstones)
{
this.isForThrift = isForThrift;
- this.nowInSec = nowInSec;
+ this.gcBefore = gcBefore;
this.purger = (timestamp, localDeletionTime) ->
!(onlyPurgeRepairedTombstones && localDeletionTime >= oldestUnrepairedTombstone)
&& localDeletionTime < gcBefore
@@ -79,13 +79,13 @@
public Row applyToStatic(Row row)
{
updateProgress();
- return row.purge(purger, nowInSec);
+ return row.purge(purger, gcBefore);
}
public Row applyToRow(Row row)
{
updateProgress();
- return row.purge(purger, nowInSec);
+ return row.purge(purger, gcBefore);
}
public RangeTombstoneMarker applyToMarker(RangeTombstoneMarker marker)
@@ -117,4 +117,4 @@
return purger.shouldPurge(((RangeTombstoneBoundMarker)marker).deletionTime()) ? null : marker;
}
}
-}
+}
\ No newline at end of file
diff --git a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
index 41b1424..b7f6793 100644
--- a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
+++ b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
@@ -177,12 +177,6 @@
{
private final List<UnfilteredRowIterator> toMerge = new ArrayList<>(iterators.size());
- @Override
- public boolean trivialReduceIsTrivial()
- {
- return false;
- }
-
public void reduce(int idx, UnfilteredRowIterator current)
{
toMerge.add(current);
diff --git a/src/java/org/apache/cassandra/db/rows/BufferCell.java b/src/java/org/apache/cassandra/db/rows/BufferCell.java
index 8912f59..4176ba6 100644
--- a/src/java/org/apache/cassandra/db/rows/BufferCell.java
+++ b/src/java/org/apache/cassandra/db/rows/BufferCell.java
@@ -172,7 +172,7 @@
// Note that as long as the expiring column and the tombstone put together live longer than GC grace seconds,
// we'll fulfil our responsibility to repair. See discussion at
// http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/repair-compaction-and-tombstone-rows-td7583481.html
- return BufferCell.tombstone(column, timestamp, localDeletionTime - ttl, path);
+ return BufferCell.tombstone(column, timestamp, localDeletionTime - ttl);
}
}
return this;
diff --git a/src/java/org/apache/cassandra/db/rows/LazilyInitializedUnfilteredRowIterator.java b/src/java/org/apache/cassandra/db/rows/LazilyInitializedUnfilteredRowIterator.java
index 8ba4394..1bf78dd 100644
--- a/src/java/org/apache/cassandra/db/rows/LazilyInitializedUnfilteredRowIterator.java
+++ b/src/java/org/apache/cassandra/db/rows/LazilyInitializedUnfilteredRowIterator.java
@@ -27,7 +27,7 @@
*
* This is used during partition range queries when we know the partition key but want
* to defer the initialization of the rest of the UnfilteredRowIterator until we need those informations.
- * See {@link BigTableScanner#KeyScanningIterator} for instance.
+ * See {@link org.apache.cassandra.io.sstable.format.big.BigTableScanner#KeyScanningIterator} for instance.
*/
public abstract class LazilyInitializedUnfilteredRowIterator extends AbstractIterator<Unfiltered> implements UnfilteredRowIterator
{
diff --git a/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java b/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java
index 5771a86..053b5dc 100644
--- a/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java
+++ b/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java
@@ -27,7 +27,8 @@
/**
* A marker for a range tombstone bound.
* <p>
- * There is 2 types of markers: bounds (see {@link RangeTombstoneBound}) and boundaries (see {@link RangeTombstoneBoundary}).
+ * There is 2 types of markers: bounds (see {@link RangeTombstoneBoundMarker}) and boundaries (see {@link RangeTombstoneBoundaryMarker}).
+ * </p>
*/
public interface RangeTombstoneMarker extends Unfiltered
{
diff --git a/src/java/org/apache/cassandra/db/rows/Row.java b/src/java/org/apache/cassandra/db/rows/Row.java
index 5f79a66..8a67e9b 100644
--- a/src/java/org/apache/cassandra/db/rows/Row.java
+++ b/src/java/org/apache/cassandra/db/rows/Row.java
@@ -394,11 +394,11 @@
public Clustering clustering();
/**
- * Adds the liveness information for the primary key columns of this row.
+ * Adds the liveness information for the partition key columns of this row.
*
* This call is optional (skipping it is equivalent to calling {@code addPartitionKeyLivenessInfo(LivenessInfo.NONE)}).
*
- * @param info the liveness information for the primary key columns of the built row.
+ * @param info the liveness information for the partition key columns of the built row.
*/
public void addPrimaryKeyLivenessInfo(LivenessInfo info);
diff --git a/src/java/org/apache/cassandra/db/rows/Unfiltered.java b/src/java/org/apache/cassandra/db/rows/Unfiltered.java
index ba03741..d34403c 100644
--- a/src/java/org/apache/cassandra/db/rows/Unfiltered.java
+++ b/src/java/org/apache/cassandra/db/rows/Unfiltered.java
@@ -49,7 +49,7 @@
* Validate the data of this atom.
*
* @param metadata the metadata for the table this atom is part of.
- * @throws MarshalException if some of the data in this atom is
+ * @throws org.apache.cassandra.serializers.MarshalException if some of the data in this atom is
* invalid (some value is invalid for its column type, or some field
* is nonsensical).
*/
diff --git a/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorSerializer.java b/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorSerializer.java
index 932ca4c..542f0a2 100644
--- a/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorSerializer.java
+++ b/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorSerializer.java
@@ -36,6 +36,7 @@
* The serialization is composed of a header, follows by the rows and range tombstones of the iterator serialized
* until we read the end of the partition (see UnfilteredSerializer for details). The header itself
* is:
+ * {@code
* <cfid><key><flags><s_header>[<partition_deletion>][<static_row>][<row_estimate>]
* where:
* <cfid> is the table cfid.
@@ -54,6 +55,7 @@
* <static_row> is the static row for this partition as serialized by UnfilteredSerializer.
* <row_estimate> is the (potentially estimated) number of rows serialized. This is only used for
* the purpose of sizing on the receiving end and should not be relied upon too strongly.
+ * }
*
* Please note that the format described above is the on-wire format. On-disk, the format is basically the
* same, but the header is written once per sstable, not once per-partition. Further, the actual row and
diff --git a/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java b/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
index 4efc5eb..0451161 100644
--- a/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
+++ b/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
@@ -29,6 +29,7 @@
/**
* Serialize/deserialize a single Unfiltered (both on-wire and on-disk).
*
+ * {@code
* The encoded format for an unfiltered is <flags>(<row>|<marker>) where:
*
* <flags> is a byte (or two) whose bits are flags used by the rest of the serialization. Each
@@ -66,6 +67,7 @@
* and if it's expiring a delta-encoded int <ttl> (unless it's an expiring cell
* and the ttl and localDeletionTime are indicated by the flags to be the same
* than the row ones, in which case none of those appears).
+ * }
*/
public class UnfilteredSerializer
{
diff --git a/src/java/org/apache/cassandra/db/transform/Stack.java b/src/java/org/apache/cassandra/db/transform/Stack.java
index aac1679..d388f75 100644
--- a/src/java/org/apache/cassandra/db/transform/Stack.java
+++ b/src/java/org/apache/cassandra/db/transform/Stack.java
@@ -4,6 +4,8 @@
class Stack
{
+ public static final Transformation[] EMPTY_TRANSFORMATIONS = new Transformation[0];
+ public static final MoreContentsHolder[] EMPTY_MORE_CONTENTS_HOLDERS = new MoreContentsHolder[0];
static final Stack EMPTY = new Stack();
Transformation[] stack;
@@ -24,8 +26,8 @@
Stack()
{
- stack = new Transformation[0];
- moreContents = new MoreContentsHolder[0];
+ stack = EMPTY_TRANSFORMATIONS;
+ moreContents = EMPTY_MORE_CONTENTS_HOLDERS;
}
Stack(Stack copy)
diff --git a/src/java/org/apache/cassandra/db/view/View.java b/src/java/org/apache/cassandra/db/view/View.java
index 30e9776..1deb055 100644
--- a/src/java/org/apache/cassandra/db/view/View.java
+++ b/src/java/org/apache/cassandra/db/view/View.java
@@ -191,7 +191,8 @@
* If the update contains any range tombstones, there is a possibility that it will not touch a range that is
* currently included in the view.
*
- * @return true if {@param partition} modifies a column included in the view
+ * @param partition the update partition
+ * @return true if partition modifies a column included in the view
*/
public boolean updateAffectsView(AbstractBTreePartition partition)
{
@@ -502,8 +503,8 @@
// Add all of the rows which were recovered from the query to the row set
while (!pager.isExhausted())
{
- try (ReadOrderGroup orderGroup = pager.startOrderGroup();
- PartitionIterator iter = pager.fetchPageInternal(128, orderGroup))
+ try (ReadExecutionController executionController = pager.executionController();
+ PartitionIterator iter = pager.fetchPageInternal(128, executionController))
{
if (!iter.hasNext())
break;
@@ -560,8 +561,8 @@
while (!pager.isExhausted())
{
- try (ReadOrderGroup orderGroup = pager.startOrderGroup();
- PartitionIterator iter = pager.fetchPageInternal(128, orderGroup))
+ try (ReadExecutionController executionController = pager.executionController();
+ PartitionIterator iter = pager.fetchPageInternal(128, executionController))
{
while (iter.hasNext())
{
diff --git a/src/java/org/apache/cassandra/db/view/ViewBuilder.java b/src/java/org/apache/cassandra/db/view/ViewBuilder.java
index 35b023b..8146211 100644
--- a/src/java/org/apache/cassandra/db/view/ViewBuilder.java
+++ b/src/java/org/apache/cassandra/db/view/ViewBuilder.java
@@ -80,8 +80,8 @@
while (!pager.isExhausted())
{
- try (ReadOrderGroup orderGroup = pager.startOrderGroup();
- PartitionIterator partitionIterator = pager.fetchPageInternal(128, orderGroup))
+ try (ReadExecutionController executionController = pager.executionController();
+ PartitionIterator partitionIterator = pager.fetchPageInternal(128, executionController))
{
if (!partitionIterator.hasNext())
return;
diff --git a/src/java/org/apache/cassandra/db/view/ViewUtils.java b/src/java/org/apache/cassandra/db/view/ViewUtils.java
index 4d9517f..4dc1766 100644
--- a/src/java/org/apache/cassandra/db/view/ViewUtils.java
+++ b/src/java/org/apache/cassandra/db/view/ViewUtils.java
@@ -45,7 +45,7 @@
* nodes in the local datacenter when calculating cardinality.
*
* For example, if we have the following ring:
- * A, T1 -> B, T2 -> C, T3 -> A
+ * {@code A, T1 -> B, T2 -> C, T3 -> A}
*
* For the token T1, at RF=1, A would be included, so A's cardinality for T1 is 1. For the token T1, at RF=2, B would
* be included, so B's cardinality for token T1 is 2. For token T3, at RF = 2, A would be included, so A's cardinality
diff --git a/src/java/org/apache/cassandra/dht/BootStrapper.java b/src/java/org/apache/cassandra/dht/BootStrapper.java
index 8d8f5c7..d10aa3b 100644
--- a/src/java/org/apache/cassandra/dht/BootStrapper.java
+++ b/src/java/org/apache/cassandra/dht/BootStrapper.java
@@ -74,7 +74,8 @@
"Bootstrap",
useStrictConsistency,
DatabaseDescriptor.getEndpointSnitch(),
- stateStore);
+ stateStore,
+ true);
streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance));
for (String keyspaceName : Schema.instance.getNonSystemKeyspaces())
diff --git a/src/java/org/apache/cassandra/dht/ExcludingBounds.java b/src/java/org/apache/cassandra/dht/ExcludingBounds.java
index 8fbde28..ed6c2fc 100644
--- a/src/java/org/apache/cassandra/dht/ExcludingBounds.java
+++ b/src/java/org/apache/cassandra/dht/ExcludingBounds.java
@@ -23,7 +23,7 @@
import org.apache.cassandra.utils.Pair;
/**
- * AbstractBounds containing neither of its endpoints: (left, right). Used by CQL key > X AND key < Y range scans.
+ * AbstractBounds containing neither of its endpoints: (left, right). Used by {@code CQL key > X AND key < Y} range scans.
*/
public class ExcludingBounds<T extends RingPosition<T>> extends AbstractBounds<T>
{
diff --git a/src/java/org/apache/cassandra/dht/IPartitioner.java b/src/java/org/apache/cassandra/dht/IPartitioner.java
index e0a08dc..b559a6f 100644
--- a/src/java/org/apache/cassandra/dht/IPartitioner.java
+++ b/src/java/org/apache/cassandra/dht/IPartitioner.java
@@ -20,6 +20,7 @@
import java.nio.ByteBuffer;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.marshal.AbstractType;
@@ -49,6 +50,17 @@
public Token getMinimumToken();
/**
+ * The biggest token for this partitioner, unlike getMinimumToken, this token is actually used and users wanting to
+ * include all tokens need to do getMaximumToken().maxKeyBound()
+ *
+ * Not implemented for the ordered partitioners
+ */
+ default Token getMaximumToken()
+ {
+ throw new UnsupportedOperationException("If you are using a splitting partitioner, getMaximumToken has to be implemented");
+ }
+
+ /**
* @return a Token that can be used to route a given key
* (This is NOT a method to create a Token from its string representation;
* for that, use TokenFactory.fromString.)
@@ -84,4 +96,9 @@
* Used by secondary indices.
*/
public AbstractType<?> partitionOrdering();
+
+ default Optional<Splitter> splitter()
+ {
+ return Optional.empty();
+ }
}
diff --git a/src/java/org/apache/cassandra/dht/IncludingExcludingBounds.java b/src/java/org/apache/cassandra/dht/IncludingExcludingBounds.java
index 19c098e..ac5185f 100644
--- a/src/java/org/apache/cassandra/dht/IncludingExcludingBounds.java
+++ b/src/java/org/apache/cassandra/dht/IncludingExcludingBounds.java
@@ -23,7 +23,7 @@
import org.apache.cassandra.utils.Pair;
/**
- * AbstractBounds containing only its left endpoint: [left, right). Used by CQL key >= X AND key < Y range scans.
+ * AbstractBounds containing only its left endpoint: [left, right). Used by {@code CQL key >= X AND key < Y} range scans.
*/
public class IncludingExcludingBounds<T extends RingPosition<T>> extends AbstractBounds<T>
{
diff --git a/src/java/org/apache/cassandra/dht/Murmur3Partitioner.java b/src/java/org/apache/cassandra/dht/Murmur3Partitioner.java
index d68be3f..f9f6113 100644
--- a/src/java/org/apache/cassandra/dht/Murmur3Partitioner.java
+++ b/src/java/org/apache/cassandra/dht/Murmur3Partitioner.java
@@ -48,6 +48,19 @@
public static final Murmur3Partitioner instance = new Murmur3Partitioner();
public static final AbstractType<?> partitionOrdering = new PartitionerDefinedOrder(instance);
+ private final Splitter splitter = new Splitter(this)
+ {
+ public Token tokenForValue(BigInteger value)
+ {
+ return new LongToken(value.longValue());
+ }
+
+ public BigInteger valueForToken(Token token)
+ {
+ return BigInteger.valueOf(((LongToken) token).token);
+ }
+ };
+
public DecoratedKey decorateKey(ByteBuffer key)
{
long[] hash = getHash(key);
@@ -291,8 +304,18 @@
return LongType.instance;
}
+ public Token getMaximumToken()
+ {
+ return new LongToken(Long.MAX_VALUE);
+ }
+
public AbstractType<?> partitionOrdering()
{
return partitionOrdering;
}
+
+ public Optional<Splitter> splitter()
+ {
+ return Optional.of(splitter);
+ }
}
diff --git a/src/java/org/apache/cassandra/dht/RandomPartitioner.java b/src/java/org/apache/cassandra/dht/RandomPartitioner.java
index b0dea01..96a96ca 100644
--- a/src/java/org/apache/cassandra/dht/RandomPartitioner.java
+++ b/src/java/org/apache/cassandra/dht/RandomPartitioner.java
@@ -50,6 +50,19 @@
public static final RandomPartitioner instance = new RandomPartitioner();
public static final AbstractType<?> partitionOrdering = new PartitionerDefinedOrder(instance);
+ private final Splitter splitter = new Splitter(this)
+ {
+ public Token tokenForValue(BigInteger value)
+ {
+ return new BigIntegerToken(value);
+ }
+
+ public BigInteger valueForToken(Token token)
+ {
+ return ((BigIntegerToken)token).getTokenValue();
+ }
+ };
+
public DecoratedKey decorateKey(ByteBuffer key)
{
return new CachedHashDecoratedKey(getToken(key), key);
@@ -194,6 +207,11 @@
return ownerships;
}
+ public Token getMaximumToken()
+ {
+ return new BigIntegerToken(MAXIMUM);
+ }
+
public AbstractType<?> getTokenValidator()
{
return IntegerType.instance;
@@ -203,4 +221,10 @@
{
return partitionOrdering;
}
+
+ public Optional<Splitter> splitter()
+ {
+ return Optional.of(splitter);
+ }
+
}
diff --git a/src/java/org/apache/cassandra/dht/Range.java b/src/java/org/apache/cassandra/dht/Range.java
index b4fed65..48df70b 100644
--- a/src/java/org/apache/cassandra/dht/Range.java
+++ b/src/java/org/apache/cassandra/dht/Range.java
@@ -164,7 +164,7 @@
boolean thatwraps = isWrapAround(that.left, that.right);
if (!thiswraps && !thatwraps)
{
- // neither wraps. the straightforward case.
+ // neither wraps: the straightforward case.
if (!(left.compareTo(that.right) < 0 && that.left.compareTo(right) < 0))
return Collections.emptySet();
return rangeSet(new Range<T>(ObjectUtils.max(this.left, that.left),
@@ -172,7 +172,7 @@
}
if (thiswraps && thatwraps)
{
- // if the starts are the same, one contains the other, which we have already ruled out.
+ //both wrap: if the starts are the same, one contains the other, which we have already ruled out.
assert !this.left.equals(that.left);
// two wrapping ranges always intersect.
// since we have already determined that neither this nor that contains the other, we have 2 cases,
@@ -186,9 +186,9 @@
? intersectionBothWrapping(this, that)
: intersectionBothWrapping(that, this);
}
- if (thiswraps && !thatwraps)
+ if (thiswraps) // this wraps, that does not wrap
return intersectionOneWrapping(this, that);
- assert (!thiswraps && thatwraps);
+ // the last case: this does not wrap, that wraps
return intersectionOneWrapping(that, this);
}
@@ -494,6 +494,23 @@
return new Range<T>(left, newRight);
}
+ public static <T extends RingPosition<T>> List<Range<T>> sort(Collection<Range<T>> ranges)
+ {
+ List<Range<T>> output = new ArrayList<>(ranges.size());
+ for (Range<T> r : ranges)
+ output.addAll(r.unwrap());
+ // sort by left
+ Collections.sort(output, new Comparator<Range<T>>()
+ {
+ public int compare(Range<T> b1, Range<T> b2)
+ {
+ return b1.left.compareTo(b2.left);
+ }
+ });
+ return output;
+ }
+
+
/**
* Compute a range of keys corresponding to a given range of token.
*/
diff --git a/src/java/org/apache/cassandra/dht/RangeStreamer.java b/src/java/org/apache/cassandra/dht/RangeStreamer.java
index 3da6bc8..47e0c15 100644
--- a/src/java/org/apache/cassandra/dht/RangeStreamer.java
+++ b/src/java/org/apache/cassandra/dht/RangeStreamer.java
@@ -117,13 +117,14 @@
String description,
boolean useStrictConsistency,
IEndpointSnitch snitch,
- StreamStateStore stateStore)
+ StreamStateStore stateStore,
+ boolean connectSequentially)
{
this.metadata = metadata;
this.tokens = tokens;
this.address = address;
this.description = description;
- this.streamPlan = new StreamPlan(description, true);
+ this.streamPlan = new StreamPlan(description, true, connectSequentially);
this.useStrictConsistency = useStrictConsistency;
this.snitch = snitch;
this.stateStore = stateStore;
diff --git a/src/java/org/apache/cassandra/dht/Splitter.java b/src/java/org/apache/cassandra/dht/Splitter.java
new file mode 100644
index 0000000..4268e83
--- /dev/null
+++ b/src/java/org/apache/cassandra/dht/Splitter.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.dht;
+
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Partition splitter.
+ */
+public abstract class Splitter
+{
+ private final IPartitioner partitioner;
+
+ protected Splitter(IPartitioner partitioner)
+ {
+ this.partitioner = partitioner;
+ }
+
+ protected abstract Token tokenForValue(BigInteger value);
+
+ protected abstract BigInteger valueForToken(Token token);
+
+ public List<Token> splitOwnedRanges(int parts, List<Range<Token>> localRanges, boolean dontSplitRanges)
+ {
+ if (localRanges.isEmpty() || parts == 1)
+ return Collections.singletonList(partitioner.getMaximumToken());
+
+ BigInteger totalTokens = BigInteger.ZERO;
+ for (Range<Token> r : localRanges)
+ {
+ BigInteger right = valueForToken(token(r.right));
+ totalTokens = totalTokens.add(right.subtract(valueForToken(r.left)));
+ }
+ BigInteger perPart = totalTokens.divide(BigInteger.valueOf(parts));
+ // the range owned is so tiny we can't split it:
+ if (perPart.equals(BigInteger.ZERO))
+ return Collections.singletonList(partitioner.getMaximumToken());
+
+ if (dontSplitRanges)
+ return splitOwnedRangesNoPartialRanges(localRanges, perPart, parts);
+
+ List<Token> boundaries = new ArrayList<>();
+ BigInteger sum = BigInteger.ZERO;
+ for (Range<Token> r : localRanges)
+ {
+ Token right = token(r.right);
+ BigInteger currentRangeWidth = valueForToken(right).subtract(valueForToken(r.left)).abs();
+ BigInteger left = valueForToken(r.left);
+ while (sum.add(currentRangeWidth).compareTo(perPart) >= 0)
+ {
+ BigInteger withinRangeBoundary = perPart.subtract(sum);
+ left = left.add(withinRangeBoundary);
+ boundaries.add(tokenForValue(left));
+ currentRangeWidth = currentRangeWidth.subtract(withinRangeBoundary);
+ sum = BigInteger.ZERO;
+ }
+ sum = sum.add(currentRangeWidth);
+ }
+ boundaries.set(boundaries.size() - 1, partitioner.getMaximumToken());
+
+ assert boundaries.size() == parts : boundaries.size() +"!="+parts+" "+boundaries+":"+localRanges;
+ return boundaries;
+ }
+
+ private List<Token> splitOwnedRangesNoPartialRanges(List<Range<Token>> localRanges, BigInteger perPart, int parts)
+ {
+ List<Token> boundaries = new ArrayList<>(parts);
+ BigInteger sum = BigInteger.ZERO;
+ int i = 0;
+ while (boundaries.size() < parts - 1)
+ {
+ Range<Token> r = localRanges.get(i);
+ Range<Token> nextRange = localRanges.get(i + 1);
+ Token right = token(r.right);
+ Token nextRight = token(nextRange.right);
+
+ BigInteger currentRangeWidth = valueForToken(right).subtract(valueForToken(r.left));
+ BigInteger nextRangeWidth = valueForToken(nextRight).subtract(valueForToken(nextRange.left));
+ sum = sum.add(currentRangeWidth);
+ // does this or next range take us beyond the per part limit?
+ if (sum.compareTo(perPart) > 0 || sum.add(nextRangeWidth).compareTo(perPart) > 0)
+ {
+ // Either this or the next range will take us beyond the perPart limit. Will stopping now or
+ // adding the next range create the smallest difference to perPart?
+ BigInteger diffCurrent = sum.subtract(perPart).abs();
+ BigInteger diffNext = sum.add(nextRangeWidth).subtract(perPart).abs();
+ if (diffNext.compareTo(diffCurrent) >= 0)
+ {
+ sum = BigInteger.ZERO;
+ boundaries.add(right);
+ }
+ }
+ i++;
+ }
+ boundaries.add(partitioner.getMaximumToken());
+ return boundaries;
+ }
+
+ /**
+ * We avoid calculating for wrap around ranges, instead we use the actual max token, and then, when translating
+ * to PartitionPositions, we include tokens from .minKeyBound to .maxKeyBound to make sure we include all tokens.
+ */
+ private Token token(Token t)
+ {
+ return t.equals(partitioner.getMinimumToken()) ? partitioner.getMaximumToken() : t;
+ }
+
+}
diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java
index a0754b1..5a1179f 100644
--- a/src/java/org/apache/cassandra/gms/FailureDetector.java
+++ b/src/java/org/apache/cassandra/gms/FailureDetector.java
@@ -246,7 +246,7 @@
// it's worth being defensive here so minor bugs don't cause disproportionate
// badness. (See CASSANDRA-1463 for an example).
if (epState == null)
- logger.error("unknown endpoint {}", ep);
+ logger.error("Unknown endpoint: " + ep, new IllegalArgumentException(""));
return epState != null && epState.isAlive();
}
diff --git a/src/java/org/apache/cassandra/gms/TokenSerializer.java b/src/java/org/apache/cassandra/gms/TokenSerializer.java
index 1404258..da4ee7d 100644
--- a/src/java/org/apache/cassandra/gms/TokenSerializer.java
+++ b/src/java/org/apache/cassandra/gms/TokenSerializer.java
@@ -19,7 +19,7 @@
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.dht.Token;
-
+import org.apache.cassandra.utils.ByteBufferUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -37,9 +37,9 @@
{
for (Token token : tokens)
{
- byte[] bintoken = partitioner.getTokenFactory().toByteArray(token).array();
- out.writeInt(bintoken.length);
- out.write(bintoken);
+ ByteBuffer tokenBuffer = partitioner.getTokenFactory().toByteArray(token);
+ assert tokenBuffer.arrayOffset() == 0;
+ ByteBufferUtil.writeWithLength(tokenBuffer.array(), out);
}
out.writeInt(0);
}
diff --git a/src/java/org/apache/cassandra/index/Index.java b/src/java/org/apache/cassandra/index/Index.java
index 71dcfc9..47d8ff0 100644
--- a/src/java/org/apache/cassandra/index/Index.java
+++ b/src/java/org/apache/cassandra/index/Index.java
@@ -1,12 +1,15 @@
package org.apache.cassandra.index;
+import java.util.Collection;
import java.util.Optional;
+import java.util.Set;
import java.util.concurrent.Callable;
import java.util.function.BiFunction;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.Operator;
import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.compaction.OperationType;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.partitions.PartitionIterator;
@@ -14,7 +17,12 @@
import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
import org.apache.cassandra.db.rows.Row;
import org.apache.cassandra.exceptions.InvalidRequestException;
+import org.apache.cassandra.index.internal.CollatedViewIndexBuilder;
import org.apache.cassandra.index.transactions.IndexTransaction;
+import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.ReducingKeyIterator;
+import org.apache.cassandra.io.sstable.format.SSTableFlushObserver;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.schema.IndexMetadata;
import org.apache.cassandra.utils.concurrent.OpOrder;
@@ -110,10 +118,56 @@
{
/*
+ * Helpers for building indexes from SSTable data
+ */
+
+ /**
+ * Provider of {@code SecondaryIndexBuilder} instances. See {@code getBuildTaskSupport} and
+ * {@code SecondaryIndexManager.buildIndexesBlocking} for more detail.
+ */
+ interface IndexBuildingSupport
+ {
+ SecondaryIndexBuilder getIndexBuildTask(ColumnFamilyStore cfs, Set<Index> indexes, Collection<SSTableReader> sstables);
+ }
+
+ /**
+ * Default implementation of {@code IndexBuildingSupport} which uses a {@code ReducingKeyIterator} to obtain a
+ * collated view of the data in the SSTables.
+ */
+ public static class CollatedViewIndexBuildingSupport implements IndexBuildingSupport
+ {
+ public SecondaryIndexBuilder getIndexBuildTask(ColumnFamilyStore cfs, Set<Index> indexes, Collection<SSTableReader> sstables)
+ {
+ return new CollatedViewIndexBuilder(cfs, indexes, new ReducingKeyIterator(sstables));
+ }
+ }
+
+ /**
+ * Singleton instance of {@code CollatedViewIndexBuildingSupport}, which may be used by any {@code Index}
+ * implementation.
+ */
+ public static final CollatedViewIndexBuildingSupport INDEX_BUILDER_SUPPORT = new CollatedViewIndexBuildingSupport();
+
+ /*
* Management functions
*/
/**
+ * Get an instance of a helper to provide tasks for building the index from a set of SSTable data.
+ * When processing a number of indexes to be rebuilt, {@code SecondaryIndexManager.buildIndexesBlocking} groups
+ * those with the same {@code IndexBuildingSupport} instance, allowing multiple indexes to be built with a
+ * single pass through the data. The singleton instance returned from the default method implementation builds
+ * indexes using a {@code ReducingKeyIterator} to provide a collated view of the SSTable data.
+ *
+ * @return an instance of the index build taski helper. Index implementations which return <b>the same instance</b>
+ * will be built using a single task.
+ */
+ default IndexBuildingSupport getBuildTaskSupport()
+ {
+ return INDEX_BUILDER_SUPPORT;
+ }
+
+ /**
* Return a task to perform any initialization work when a new index instance is created.
* This may involve costly operations such as (re)building the index, and is performed asynchronously
* by SecondaryIndexManager
@@ -183,13 +237,25 @@
* false enables the index implementation (or some other component) to control if and when SSTable data is
* incorporated into the index.
*
- * This is called by SecondaryIndexManager in buildIndexBlocking, buildAllIndexesBlocking & rebuildIndexesBlocking
+ * This is called by SecondaryIndexManager in buildIndexBlocking, buildAllIndexesBlocking and rebuildIndexesBlocking
* where a return value of false causes the index to be exluded from the set of those which will process the
* SSTable data.
* @return if the index should be included in the set which processes SSTable data, false otherwise.
*/
public boolean shouldBuildBlocking();
+ /**
+ * Get flush observer to observe partition/cell events generated by flushing SSTable (memtable flush or compaction).
+ *
+ * @param descriptor The descriptor of the sstable observer is requested for.
+ * @param opType The type of the operation which requests observer e.g. memtable flush or compaction.
+ *
+ * @return SSTable flush observer.
+ */
+ default SSTableFlushObserver getFlushObserver(Descriptor descriptor, OperationType opType)
+ {
+ return null;
+ }
/*
* Index selection
@@ -405,7 +471,7 @@
* See CASSANDRA-8717 for further discussion.
*
* The function takes a PartitionIterator of the results from the replicas which has already been collated
- * & reconciled, along with the command being executed. It returns another PartitionIterator containing the results
+ * and reconciled, along with the command being executed. It returns another PartitionIterator containing the results
* of the transformation (which may be the same as the input if the transformation is a no-op).
*/
public BiFunction<PartitionIterator, ReadCommand, PartitionIterator> postProcessorFor(ReadCommand command);
@@ -430,9 +496,9 @@
public interface Searcher
{
/**
- * @param orderGroup the collection of OpOrder.Groups which the ReadCommand is being performed under.
+ * @param executionController the collection of OpOrder.Groups which the ReadCommand is being performed under.
* @return partitions from the base table matching the criteria of the search.
*/
- public UnfilteredPartitionIterator search(ReadOrderGroup orderGroup);
+ public UnfilteredPartitionIterator search(ReadExecutionController executionController);
}
}
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexBuilder.java b/src/java/org/apache/cassandra/index/SecondaryIndexBuilder.java
index e66f0a3..9ec8a4e 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexBuilder.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexBuilder.java
@@ -17,61 +17,12 @@
*/
package org.apache.cassandra.index;
-import java.io.IOException;
-import java.util.Set;
-import java.util.UUID;
-
-import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.DecoratedKey;
-import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.compaction.CompactionInfo;
-import org.apache.cassandra.db.compaction.CompactionInterruptedException;
-import org.apache.cassandra.db.compaction.OperationType;
-import org.apache.cassandra.io.sstable.ReducingKeyIterator;
-import org.apache.cassandra.utils.UUIDGen;
/**
* Manages building an entire index from column family data. Runs on to compaction manager.
*/
-public class SecondaryIndexBuilder extends CompactionInfo.Holder
+public abstract class SecondaryIndexBuilder extends CompactionInfo.Holder
{
- private final ColumnFamilyStore cfs;
- private final Set<Index> indexers;
- private final ReducingKeyIterator iter;
- private final UUID compactionId;
-
- public SecondaryIndexBuilder(ColumnFamilyStore cfs, Set<Index> indexers, ReducingKeyIterator iter)
- {
- this.cfs = cfs;
- this.indexers = indexers;
- this.iter = iter;
- this.compactionId = UUIDGen.getTimeUUID();
- }
-
- public CompactionInfo getCompactionInfo()
- {
- return new CompactionInfo(cfs.metadata,
- OperationType.INDEX_BUILD,
- iter.getBytesRead(),
- iter.getTotalBytes(),
- compactionId);
- }
-
- public void build()
- {
- try
- {
- while (iter.hasNext())
- {
- if (isStopRequested())
- throw new CompactionInterruptedException(getCompactionInfo());
- DecoratedKey key = iter.next();
- Keyspace.indexPartition(key, cfs, indexers);
- }
- }
- finally
- {
- iter.close();
- }
- }
+ public abstract void build();
}
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
index 16cb9c4..7349167 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
@@ -52,7 +52,6 @@
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.index.internal.CassandraIndex;
import org.apache.cassandra.index.transactions.*;
-import org.apache.cassandra.io.sstable.ReducingKeyIterator;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.schema.IndexMetadata;
import org.apache.cassandra.schema.Indexes;
@@ -360,11 +359,20 @@
indexes.stream().map(i -> i.getIndexMetadata().name).collect(Collectors.joining(",")),
sstables.stream().map(SSTableReader::toString).collect(Collectors.joining(",")));
- SecondaryIndexBuilder builder = new SecondaryIndexBuilder(baseCfs,
- indexes,
- new ReducingKeyIterator(sstables));
- Future<?> future = CompactionManager.instance.submitIndexBuild(builder);
- FBUtilities.waitOnFuture(future);
+ Map<Index.IndexBuildingSupport, Set<Index>> byType = new HashMap<>();
+ for (Index index : indexes)
+ {
+ Set<Index> stored = byType.computeIfAbsent(index.getBuildTaskSupport(), i -> new HashSet<>());
+ stored.add(index);
+ }
+
+ List<Future<?>> futures = byType.entrySet()
+ .stream()
+ .map((e) -> e.getKey().getIndexBuildTask(baseCfs, e.getValue(), sstables))
+ .map(CompactionManager.instance::submitIndexBuild)
+ .collect(Collectors.toList());
+
+ FBUtilities.waitOnFutures(futures);
flushIndexesBlocking(indexes);
logger.info("Index build of {} complete",
diff --git a/src/java/org/apache/cassandra/index/internal/CassandraIndex.java b/src/java/org/apache/cassandra/index/internal/CassandraIndex.java
index 7cb4471..b884895 100644
--- a/src/java/org/apache/cassandra/index/internal/CassandraIndex.java
+++ b/src/java/org/apache/cassandra/index/internal/CassandraIndex.java
@@ -681,9 +681,9 @@
metadata.name,
getSSTableNames(sstables));
- SecondaryIndexBuilder builder = new SecondaryIndexBuilder(baseCfs,
- Collections.singleton(this),
- new ReducingKeyIterator(sstables));
+ SecondaryIndexBuilder builder = new CollatedViewIndexBuilder(baseCfs,
+ Collections.singleton(this),
+ new ReducingKeyIterator(sstables));
Future<?> future = CompactionManager.instance.submitIndexBuild(builder);
FBUtilities.waitOnFuture(future);
indexCfs.forceBlockingFlush();
@@ -755,6 +755,9 @@
return getFunctions(indexMetadata, parseTarget(baseCfs.metadata, indexMetadata)).newIndexInstance(baseCfs, indexMetadata);
}
+ private static final Pattern TWO_QUOTES = Pattern.compile("\"\"");
+ private static final String QUOTE = "\"";
+
// Public because it's also used to convert index metadata into a thrift-compatible format
public static Pair<ColumnDefinition, IndexTarget.Type> parseTarget(CFMetaData cfm,
IndexMetadata indexDef)
@@ -784,10 +787,10 @@
// abc"def -> abc""def.
// Because the target string is stored in a CQL compatible form, we
// need to un-escape any such quotes to get the actual column name
- if (columnName.startsWith("\""))
+ if (columnName.startsWith(QUOTE))
{
columnName = StringUtils.substring(StringUtils.substring(columnName, 1), 0, -1);
- columnName = columnName.replaceAll("\"\"", "\"");
+ columnName = TWO_QUOTES.matcher(columnName).replaceAll(QUOTE);
}
// if it's not a CQL table, we can't assume that the column name is utf8, so
diff --git a/src/java/org/apache/cassandra/index/internal/CassandraIndexFunctions.java b/src/java/org/apache/cassandra/index/internal/CassandraIndexFunctions.java
index b7cb3a2..89eebdf 100644
--- a/src/java/org/apache/cassandra/index/internal/CassandraIndexFunctions.java
+++ b/src/java/org/apache/cassandra/index/internal/CassandraIndexFunctions.java
@@ -54,7 +54,7 @@
/**
* Add the clustering columns for a specific type of index table to the a CFMetaData.Builder (which is being
* used to construct the index table's CFMetadata. In the default implementation, the clustering columns of the
- * index table hold the partition key & clustering columns of the base table. This is overridden in several cases:
+ * index table hold the partition key and clustering columns of the base table. This is overridden in several cases:
* * When the indexed value is itself a clustering column, in which case, we only need store the base table's
* *other* clustering values in the index - the indexed value being the index table's partition key
* * When the indexed value is a collection value, in which case we also need to capture the cell path from the base
diff --git a/src/java/org/apache/cassandra/index/internal/CassandraIndexSearcher.java b/src/java/org/apache/cassandra/index/internal/CassandraIndexSearcher.java
index 72d2528..1e28579 100644
--- a/src/java/org/apache/cassandra/index/internal/CassandraIndexSearcher.java
+++ b/src/java/org/apache/cassandra/index/internal/CassandraIndexSearcher.java
@@ -36,14 +36,14 @@
@SuppressWarnings("resource") // Both the OpOrder and 'indexIter' are closed on exception, or through the closing of the result
// of this method.
- public UnfilteredPartitionIterator search(ReadOrderGroup orderGroup)
+ public UnfilteredPartitionIterator search(ReadExecutionController executionController)
{
// the value of the index expression is the partition key in the index table
DecoratedKey indexKey = index.getBackingTable().get().decorateKey(expression.getIndexValue());
- UnfilteredRowIterator indexIter = queryIndex(indexKey, command, orderGroup);
+ UnfilteredRowIterator indexIter = queryIndex(indexKey, command, executionController);
try
{
- return queryDataFromIndex(indexKey, UnfilteredRowIterators.filter(indexIter, command.nowInSec()), command, orderGroup);
+ return queryDataFromIndex(indexKey, UnfilteredRowIterators.filter(indexIter, command.nowInSec()), command, executionController);
}
catch (RuntimeException | Error e)
{
@@ -52,13 +52,13 @@
}
}
- private UnfilteredRowIterator queryIndex(DecoratedKey indexKey, ReadCommand command, ReadOrderGroup orderGroup)
+ private UnfilteredRowIterator queryIndex(DecoratedKey indexKey, ReadCommand command, ReadExecutionController executionController)
{
ClusteringIndexFilter filter = makeIndexFilter(command);
ColumnFamilyStore indexCfs = index.getBackingTable().get();
CFMetaData indexCfm = indexCfs.metadata;
return SinglePartitionReadCommand.create(indexCfm, command.nowInSec(), indexKey, ColumnFilter.all(indexCfm), filter)
- .queryMemtableAndDisk(indexCfs, orderGroup.indexReadOpOrderGroup());
+ .queryMemtableAndDisk(indexCfs, executionController.indexReadOpOrderGroup());
}
private ClusteringIndexFilter makeIndexFilter(ReadCommand command)
@@ -168,5 +168,5 @@
protected abstract UnfilteredPartitionIterator queryDataFromIndex(DecoratedKey indexKey,
RowIterator indexHits,
ReadCommand command,
- ReadOrderGroup orderGroup);
+ ReadExecutionController executionController);
}
diff --git a/src/java/org/apache/cassandra/index/internal/CollatedViewIndexBuilder.java b/src/java/org/apache/cassandra/index/internal/CollatedViewIndexBuilder.java
new file mode 100644
index 0000000..8ea7a68
--- /dev/null
+++ b/src/java/org/apache/cassandra/index/internal/CollatedViewIndexBuilder.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.index.internal;
+
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.compaction.CompactionInfo;
+import org.apache.cassandra.db.compaction.CompactionInterruptedException;
+import org.apache.cassandra.db.compaction.OperationType;
+import org.apache.cassandra.index.Index;
+import org.apache.cassandra.index.SecondaryIndexBuilder;
+import org.apache.cassandra.io.sstable.ReducingKeyIterator;
+import org.apache.cassandra.utils.UUIDGen;
+
+/**
+ * Manages building an entire index from column family data. Runs on to compaction manager.
+ */
+public class CollatedViewIndexBuilder extends SecondaryIndexBuilder
+{
+ private final ColumnFamilyStore cfs;
+ private final Set<Index> indexers;
+ private final ReducingKeyIterator iter;
+ private final UUID compactionId;
+
+ public CollatedViewIndexBuilder(ColumnFamilyStore cfs, Set<Index> indexers, ReducingKeyIterator iter)
+ {
+ this.cfs = cfs;
+ this.indexers = indexers;
+ this.iter = iter;
+ this.compactionId = UUIDGen.getTimeUUID();
+ }
+
+ public CompactionInfo getCompactionInfo()
+ {
+ return new CompactionInfo(cfs.metadata,
+ OperationType.INDEX_BUILD,
+ iter.getBytesRead(),
+ iter.getTotalBytes(),
+ compactionId);
+ }
+
+ public void build()
+ {
+ try
+ {
+ while (iter.hasNext())
+ {
+ if (isStopRequested())
+ throw new CompactionInterruptedException(getCompactionInfo());
+ DecoratedKey key = iter.next();
+ Keyspace.indexPartition(key, cfs, indexers);
+ }
+ }
+ finally
+ {
+ iter.close();
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/index/internal/composites/ClusteringColumnIndex.java b/src/java/org/apache/cassandra/index/internal/composites/ClusteringColumnIndex.java
index b932602..f207e9b 100644
--- a/src/java/org/apache/cassandra/index/internal/composites/ClusteringColumnIndex.java
+++ b/src/java/org/apache/cassandra/index/internal/composites/ClusteringColumnIndex.java
@@ -36,12 +36,14 @@
* has no impact) and v the cell value.
*
* Such a cell is always indexed by this index (or rather, it is indexed if
+ * {@code
* n >= columnDef.componentIndex, which will always be the case in practice)
* and it will generate (makeIndexColumnName()) an index entry whose:
* - row key will be ck_i (getIndexedValue()) where i == columnDef.componentIndex.
* - cell name will
* rk ck_0 ... ck_{i-1} ck_{i+1} ck_n
* where rk is the row key of the initial cell and i == columnDef.componentIndex.
+ * }
*/
public class ClusteringColumnIndex extends CassandraIndex
{
diff --git a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
index 765ae4d..ddaa653 100644
--- a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
+++ b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
@@ -21,9 +21,6 @@
import java.util.ArrayList;
import java.util.List;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.ClusteringIndexNamesFilter;
@@ -41,8 +38,6 @@
public class CompositesSearcher extends CassandraIndexSearcher
{
- private static final Logger logger = LoggerFactory.getLogger(CompositesSearcher.class);
-
public CompositesSearcher(ReadCommand command,
RowFilter.Expression expression,
CassandraIndex index)
@@ -58,7 +53,7 @@
protected UnfilteredPartitionIterator queryDataFromIndex(final DecoratedKey indexKey,
final RowIterator indexHits,
final ReadCommand command,
- final ReadOrderGroup orderGroup)
+ final ReadExecutionController executionController)
{
assert indexHits.staticRow() == Rows.EMPTY_STATIC_ROW;
@@ -95,70 +90,75 @@
private boolean prepareNext()
{
- if (next != null)
- return true;
-
- if (nextEntry == null)
+ while (true)
{
- if (!indexHits.hasNext())
- return false;
+ if (next != null)
+ return true;
- nextEntry = index.decodeEntry(indexKey, indexHits.next());
- }
-
- // Gather all index hits belonging to the same partition and query the data for those hits.
- // TODO: it's much more efficient to do 1 read for all hits to the same partition than doing
- // 1 read per index hit. However, this basically mean materializing all hits for a partition
- // in memory so we should consider adding some paging mechanism. However, index hits should
- // be relatively small so it's much better than the previous code that was materializing all
- // *data* for a given partition.
- BTreeSet.Builder<Clustering> clusterings = BTreeSet.builder(index.baseCfs.getComparator());
- List<IndexEntry> entries = new ArrayList<>();
- DecoratedKey partitionKey = index.baseCfs.decorateKey(nextEntry.indexedKey);
-
- while (nextEntry != null && partitionKey.getKey().equals(nextEntry.indexedKey))
- {
- // We're queried a slice of the index, but some hits may not match some of the clustering column constraints
- if (isMatchingEntry(partitionKey, nextEntry, command))
+ if (nextEntry == null)
{
- clusterings.add(nextEntry.indexedEntryClustering);
- entries.add(nextEntry);
+ if (!indexHits.hasNext())
+ return false;
+
+ nextEntry = index.decodeEntry(indexKey, indexHits.next());
}
- nextEntry = indexHits.hasNext() ? index.decodeEntry(indexKey, indexHits.next()) : null;
+ // Gather all index hits belonging to the same partition and query the data for those hits.
+ // TODO: it's much more efficient to do 1 read for all hits to the same partition than doing
+ // 1 read per index hit. However, this basically mean materializing all hits for a partition
+ // in memory so we should consider adding some paging mechanism. However, index hits should
+ // be relatively small so it's much better than the previous code that was materializing all
+ // *data* for a given partition.
+ BTreeSet.Builder<Clustering> clusterings = BTreeSet.builder(index.baseCfs.getComparator());
+ List<IndexEntry> entries = new ArrayList<>();
+ DecoratedKey partitionKey = index.baseCfs.decorateKey(nextEntry.indexedKey);
+
+ while (nextEntry != null && partitionKey.getKey().equals(nextEntry.indexedKey))
+ {
+ // We're queried a slice of the index, but some hits may not match some of the clustering column constraints
+ if (isMatchingEntry(partitionKey, nextEntry, command))
+ {
+ clusterings.add(nextEntry.indexedEntryClustering);
+ entries.add(nextEntry);
+ }
+
+ nextEntry = indexHits.hasNext() ? index.decodeEntry(indexKey, indexHits.next()) : null;
+ }
+
+ // Because we've eliminated entries that don't match the clustering columns, it's possible we added nothing
+ if (clusterings.isEmpty())
+ {
+ continue;
+ }
+
+ // Query the gathered index hits. We still need to filter stale hits from the resulting query.
+ ClusteringIndexNamesFilter filter = new ClusteringIndexNamesFilter(clusterings.build(), false);
+ SinglePartitionReadCommand dataCmd = SinglePartitionReadCommand.create(index.baseCfs.metadata,
+ command.nowInSec(),
+ command.columnFilter(),
+ command.rowFilter(),
+ DataLimits.NONE,
+ partitionKey,
+ filter);
+ @SuppressWarnings("resource") // We close right away if empty, and if it's assign to next it will be called either
+ // by the next caller of next, or through closing this iterator is this come before.
+ UnfilteredRowIterator dataIter = filterStaleEntries(dataCmd.queryMemtableAndDisk(index.baseCfs,
+ executionController.baseReadOpOrderGroup()),
+ indexKey.getKey(),
+ entries,
+ executionController.writeOpOrderGroup(),
+ command.nowInSec());
+
+
+ if (dataIter.isEmpty())
+ {
+ dataIter.close();
+ continue;
+ }
+
+ next = dataIter;
+ return true;
}
-
- // Because we've eliminated entries that don't match the clustering columns, it's possible we added nothing
- if (clusterings.isEmpty())
- return prepareNext();
-
- // Query the gathered index hits. We still need to filter stale hits from the resulting query.
- ClusteringIndexNamesFilter filter = new ClusteringIndexNamesFilter(clusterings.build(), false);
- SinglePartitionReadCommand dataCmd = SinglePartitionReadCommand.create(index.baseCfs.metadata,
- command.nowInSec(),
- command.columnFilter(),
- command.rowFilter(),
- DataLimits.NONE,
- partitionKey,
- filter);
- @SuppressWarnings("resource") // We close right away if empty, and if it's assign to next it will be called either
- // by the next caller of next, or through closing this iterator is this come before.
- UnfilteredRowIterator dataIter = filterStaleEntries(dataCmd.queryMemtableAndDisk(index.baseCfs,
- orderGroup.baseReadOpOrderGroup()),
- indexKey.getKey(),
- entries,
- orderGroup.writeOpOrderGroup(),
- command.nowInSec());
-
-
- if (dataIter.isEmpty())
- {
- dataIter.close();
- return prepareNext();
- }
-
- next = dataIter;
- return true;
}
public void remove()
diff --git a/src/java/org/apache/cassandra/index/internal/keys/KeysSearcher.java b/src/java/org/apache/cassandra/index/internal/keys/KeysSearcher.java
index f00bb27..3400f43 100644
--- a/src/java/org/apache/cassandra/index/internal/keys/KeysSearcher.java
+++ b/src/java/org/apache/cassandra/index/internal/keys/KeysSearcher.java
@@ -47,7 +47,7 @@
protected UnfilteredPartitionIterator queryDataFromIndex(final DecoratedKey indexKey,
final RowIterator indexHits,
final ReadCommand command,
- final ReadOrderGroup orderGroup)
+ final ReadExecutionController executionController)
{
assert indexHits.staticRow() == Rows.EMPTY_STATIC_ROW;
@@ -102,10 +102,10 @@
// Otherwise, we close right away if empty, and if it's assigned to next it will be called either
// by the next caller of next, or through closing this iterator is this come before.
UnfilteredRowIterator dataIter = filterIfStale(dataCmd.queryMemtableAndDisk(index.baseCfs,
- orderGroup.baseReadOpOrderGroup()),
+ executionController.baseReadOpOrderGroup()),
hit,
indexKey.getKey(),
- orderGroup.writeOpOrderGroup(),
+ executionController.writeOpOrderGroup(),
isForThrift(),
command.nowInSec());
diff --git a/src/java/org/apache/cassandra/index/transactions/CleanupTransaction.java b/src/java/org/apache/cassandra/index/transactions/CleanupTransaction.java
index 1d6ba56..29ee14c 100644
--- a/src/java/org/apache/cassandra/index/transactions/CleanupTransaction.java
+++ b/src/java/org/apache/cassandra/index/transactions/CleanupTransaction.java
@@ -26,7 +26,7 @@
*
* Notifies registered indexers of each partition being removed and
*
- * Compaction & Cleanup are somewhat simpler than dealing with incoming writes,
+ * Compaction and Cleanup are somewhat simpler than dealing with incoming writes,
* being only concerned with cleaning up stale index entries.
*
* When multiple versions of a row are compacted, the CleanupTransaction is
diff --git a/src/java/org/apache/cassandra/index/transactions/IndexTransaction.java b/src/java/org/apache/cassandra/index/transactions/IndexTransaction.java
index 3fb8235..3d4b7e2 100644
--- a/src/java/org/apache/cassandra/index/transactions/IndexTransaction.java
+++ b/src/java/org/apache/cassandra/index/transactions/IndexTransaction.java
@@ -26,7 +26,7 @@
* Used on the regular write path and when indexing newly acquired SSTables from streaming or sideloading. This type
* of transaction may include both row inserts and updates to rows previously existing in the base Memtable. Instances
* are scoped to a single partition update and are obtained from the factory method
- * @{code SecondaryIndexManager#newUpdateTransaction}
+ * {@code SecondaryIndexManager#newUpdateTransaction}
*
* * {@code CompactionTransaction}
* Used during compaction when stale entries which have been superceded are cleaned up from the index. As rows in a
diff --git a/src/java/org/apache/cassandra/index/transactions/UpdateTransaction.java b/src/java/org/apache/cassandra/index/transactions/UpdateTransaction.java
index c78304a..51533c2 100644
--- a/src/java/org/apache/cassandra/index/transactions/UpdateTransaction.java
+++ b/src/java/org/apache/cassandra/index/transactions/UpdateTransaction.java
@@ -53,7 +53,7 @@
* onInserted(row)* -- called for each Row not already present in the Memtable
* onUpdated(existing, updated)* -- called for any Row in the update for where a version was already present
* in the Memtable. It's important to note here that existing is the previous
- * row from the Memtable & updated is the final version replacing it. It is
+ * row from the Memtable and updated is the final version replacing it. It is
* *not* the incoming row, but the result of merging the incoming and existing
* rows.
* commit() -- finally, finish is called when the new Partition is swapped into the Memtable
diff --git a/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java b/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
index 62348ec..0213fd5 100644
--- a/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
@@ -22,6 +22,7 @@
import java.io.IOException;
import java.io.Closeable;
import java.nio.ByteBuffer;
+import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
@@ -65,7 +66,8 @@
0,
ActiveRepairService.UNREPAIRED_SSTABLE,
0,
- new SerializationHeader(true, metadata, columns, EncodingStats.NO_STATS));
+ new SerializationHeader(true, metadata, columns, EncodingStats.NO_STATS),
+ Collections.emptySet());
}
private static Descriptor createDescriptor(File directory, final String keyspace, final String columnFamily, final SSTableFormat.Type fmt)
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java b/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
index 3286522..043f6fa 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
@@ -159,7 +159,7 @@
client.init(keyspace);
outputHandler.output("Established connection to initial hosts");
- StreamPlan plan = new StreamPlan("Bulk Load", 0, connectionsPerHost, false, false).connectionFactory(client.getConnectionFactory());
+ StreamPlan plan = new StreamPlan("Bulk Load", 0, connectionsPerHost, false, false, false).connectionFactory(client.getConnectionFactory());
Map<InetAddress, Collection<Range<Token>>> endpointToRanges = client.getEndpointToRangesMap();
openSSTables(endpointToRanges);
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableSimpleIterator.java b/src/java/org/apache/cassandra/io/sstable/SSTableSimpleIterator.java
index 365d469..2f4df1a 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableSimpleIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableSimpleIterator.java
@@ -139,24 +139,27 @@
protected Unfiltered computeNext()
{
- try
+ while (true)
{
- if (!deserializer.hasNext())
- return endOfData();
-
- Unfiltered unfiltered = deserializer.readNext();
- if (metadata.isStaticCompactTable() && unfiltered.kind() == Unfiltered.Kind.ROW)
+ try
{
- Row row = (Row) unfiltered;
- ColumnDefinition def = metadata.getColumnDefinition(LegacyLayout.encodeClustering(metadata, row.clustering()));
- if (def != null && def.isStatic())
- return computeNext();
+ if (!deserializer.hasNext())
+ return endOfData();
+
+ Unfiltered unfiltered = deserializer.readNext();
+ if (metadata.isStaticCompactTable() && unfiltered.kind() == Unfiltered.Kind.ROW)
+ {
+ Row row = (Row) unfiltered;
+ ColumnDefinition def = metadata.getColumnDefinition(LegacyLayout.encodeClustering(metadata, row.clustering()));
+ if (def != null && def.isStatic())
+ continue;
+ }
+ return unfiltered;
}
- return unfiltered;
- }
- catch (IOException e)
- {
- throw new IOError(e);
+ catch (IOException e)
+ {
+ throw new IOError(e);
+ }
}
}
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableTxnWriter.java b/src/java/org/apache/cassandra/io/sstable/SSTableTxnWriter.java
index e889d85..5286ac5 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableTxnWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableTxnWriter.java
@@ -27,6 +27,7 @@
import org.apache.cassandra.db.compaction.OperationType;
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
+import org.apache.cassandra.index.Index;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
import org.apache.cassandra.utils.concurrent.Transactional;
@@ -102,12 +103,18 @@
}
@SuppressWarnings("resource") // log and writer closed during postCleanup
- public static SSTableTxnWriter create(CFMetaData cfm, Descriptor descriptor, long keyCount, long repairedAt, int sstableLevel, SerializationHeader header)
+ public static SSTableTxnWriter create(CFMetaData cfm,
+ Descriptor descriptor,
+ long keyCount,
+ long repairedAt,
+ int sstableLevel,
+ SerializationHeader header,
+ Collection<Index> indexes)
{
// if the column family store does not exist, we create a new default SSTableMultiWriter to use:
LifecycleTransaction txn = LifecycleTransaction.offline(OperationType.WRITE);
MetadataCollector collector = new MetadataCollector(cfm.comparator).sstableLevel(sstableLevel);
- SSTableMultiWriter writer = SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, cfm, collector, header, txn);
+ SSTableMultiWriter writer = SimpleSSTableMultiWriter.create(descriptor, keyCount, repairedAt, cfm, collector, header, indexes, txn);
return new SSTableTxnWriter(txn, writer);
}
diff --git a/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java b/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
index fd1b9a7..2217ae2 100644
--- a/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
@@ -27,6 +27,7 @@
import org.apache.cassandra.db.SerializationHeader;
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
+import org.apache.cassandra.index.Index;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.sstable.format.SSTableWriter;
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -34,9 +35,11 @@
public class SimpleSSTableMultiWriter implements SSTableMultiWriter
{
private final SSTableWriter writer;
+ private final LifecycleTransaction txn;
- protected SimpleSSTableMultiWriter(SSTableWriter writer)
+ protected SimpleSSTableMultiWriter(SSTableWriter writer, LifecycleTransaction txn)
{
+ this.txn = txn;
this.writer = writer;
}
@@ -89,6 +92,7 @@
public Throwable abort(Throwable accumulate)
{
+ txn.untrackNew(writer);
return writer.abort(accumulate);
}
@@ -109,9 +113,10 @@
CFMetaData cfm,
MetadataCollector metadataCollector,
SerializationHeader header,
+ Collection<Index> indexes,
LifecycleTransaction txn)
{
- SSTableWriter writer = SSTableWriter.create(descriptor, keyCount, repairedAt, cfm, metadataCollector, header, txn);
- return new SimpleSSTableMultiWriter(writer);
+ SSTableWriter writer = SSTableWriter.create(descriptor, keyCount, repairedAt, cfm, metadataCollector, header, indexes, txn);
+ return new SimpleSSTableMultiWriter(writer, txn);
}
}
diff --git a/src/java/org/apache/cassandra/io/sstable/format/RangeAwareSSTableWriter.java b/src/java/org/apache/cassandra/io/sstable/format/RangeAwareSSTableWriter.java
new file mode 100644
index 0000000..674ed7f
--- /dev/null
+++ b/src/java/org/apache/cassandra/io/sstable/format/RangeAwareSSTableWriter.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable.format;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.UUID;
+
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Directories;
+import org.apache.cassandra.db.PartitionPosition;
+import org.apache.cassandra.db.SerializationHeader;
+import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
+import org.apache.cassandra.db.rows.UnfilteredRowIterator;
+import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.SSTableMultiWriter;
+import org.apache.cassandra.service.StorageService;
+
+public class RangeAwareSSTableWriter implements SSTableMultiWriter
+{
+ private final List<PartitionPosition> boundaries;
+ private final Directories.DataDirectory[] directories;
+ private final int sstableLevel;
+ private final long estimatedKeys;
+ private final long repairedAt;
+ private final SSTableFormat.Type format;
+ private final SerializationHeader.Component header;
+ private final LifecycleTransaction txn;
+ private int currentIndex = -1;
+ public final ColumnFamilyStore cfs;
+ private final List<SSTableMultiWriter> finishedWriters = new ArrayList<>();
+ private final List<SSTableReader> finishedReaders = new ArrayList<>();
+ private SSTableMultiWriter currentWriter = null;
+
+ public RangeAwareSSTableWriter(ColumnFamilyStore cfs, long estimatedKeys, long repairedAt, SSTableFormat.Type format, int sstableLevel, long totalSize, LifecycleTransaction txn, SerializationHeader.Component header) throws IOException
+ {
+ directories = cfs.getDirectories().getWriteableLocations();
+ this.sstableLevel = sstableLevel;
+ this.cfs = cfs;
+ this.estimatedKeys = estimatedKeys / directories.length;
+ this.repairedAt = repairedAt;
+ this.format = format;
+ this.txn = txn;
+ this.header = header;
+ boundaries = StorageService.getDiskBoundaries(cfs, directories);
+ if (boundaries == null)
+ {
+ Directories.DataDirectory localDir = cfs.getDirectories().getWriteableLocation(totalSize);
+ if (localDir == null)
+ throw new IOException("Insufficient disk space to store " + totalSize + " bytes");
+ Descriptor desc = Descriptor.fromFilename(cfs.getSSTablePath(cfs.getDirectories().getLocationForDisk(localDir), format));
+ currentWriter = cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, sstableLevel, header.toHeader(cfs.metadata), txn);
+ }
+ }
+
+ private void maybeSwitchWriter(DecoratedKey key)
+ {
+ if (boundaries == null)
+ return;
+
+ boolean switched = false;
+ while (currentIndex < 0 || key.compareTo(boundaries.get(currentIndex)) > 0)
+ {
+ switched = true;
+ currentIndex++;
+ }
+
+ if (switched)
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+
+ Descriptor desc = Descriptor.fromFilename(cfs.getSSTablePath(cfs.getDirectories().getLocationForDisk(directories[currentIndex])), format);
+ currentWriter = cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, sstableLevel, header.toHeader(cfs.metadata), txn);
+ }
+ }
+
+ public boolean append(UnfilteredRowIterator partition)
+ {
+ maybeSwitchWriter(partition.partitionKey());
+ return currentWriter.append(partition);
+ }
+
+ @Override
+ public Collection<SSTableReader> finish(long repairedAt, long maxDataAge, boolean openResult)
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+ currentWriter = null;
+ for (SSTableMultiWriter writer : finishedWriters)
+ {
+ if (writer.getFilePointer() > 0)
+ finishedReaders.addAll(writer.finish(repairedAt, maxDataAge, openResult));
+ else
+ SSTableMultiWriter.abortOrDie(writer);
+ }
+ return finishedReaders;
+ }
+
+ @Override
+ public Collection<SSTableReader> finish(boolean openResult)
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+ currentWriter = null;
+ for (SSTableMultiWriter writer : finishedWriters)
+ {
+ if (writer.getFilePointer() > 0)
+ finishedReaders.addAll(writer.finish(openResult));
+ else
+ SSTableMultiWriter.abortOrDie(writer);
+ }
+ return finishedReaders;
+ }
+
+ @Override
+ public Collection<SSTableReader> finished()
+ {
+ return finishedReaders;
+ }
+
+ @Override
+ public SSTableMultiWriter setOpenResult(boolean openResult)
+ {
+ finishedWriters.forEach((w) -> w.setOpenResult(openResult));
+ currentWriter.setOpenResult(openResult);
+ return this;
+ }
+
+ public String getFilename()
+ {
+ return String.join("/", cfs.keyspace.getName(), cfs.getTableName());
+ }
+
+ @Override
+ public long getFilePointer()
+ {
+ return currentWriter.getFilePointer();
+ }
+
+ @Override
+ public UUID getCfId()
+ {
+ return currentWriter.getCfId();
+ }
+
+ @Override
+ public Throwable commit(Throwable accumulate)
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+ currentWriter = null;
+ for (SSTableMultiWriter writer : finishedWriters)
+ accumulate = writer.commit(accumulate);
+ return accumulate;
+ }
+
+ @Override
+ public Throwable abort(Throwable accumulate)
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+ currentWriter = null;
+ for (SSTableMultiWriter finishedWriter : finishedWriters)
+ accumulate = finishedWriter.abort(accumulate);
+
+ return accumulate;
+ }
+
+ @Override
+ public void prepareToCommit()
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+ currentWriter = null;
+ finishedWriters.forEach(SSTableMultiWriter::prepareToCommit);
+ }
+
+ @Override
+ public void close()
+ {
+ if (currentWriter != null)
+ finishedWriters.add(currentWriter);
+ currentWriter = null;
+ finishedWriters.forEach(SSTableMultiWriter::close);
+ }
+}
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableFlushObserver.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableFlushObserver.java
new file mode 100644
index 0000000..d6f54e2
--- /dev/null
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableFlushObserver.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable.format;
+
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.rows.ColumnData;
+
+/**
+ * Observer for events in the lifecycle of writing out an sstable.
+ */
+public interface SSTableFlushObserver
+{
+ /**
+ * Called before writing any data to the sstable.
+ */
+ void begin();
+
+ /**
+ * Called when a new partition in being written to the sstable,
+ * but before any cells are processed (see {@link #nextCell(ColumnData)}).
+ *
+ * @param key The key being appended to SSTable.
+ * @param indexPosition The position of the key in the SSTable PRIMARY_INDEX file.
+ */
+ void startPartition(DecoratedKey key, long indexPosition);
+
+ /**
+ * Called after the cell is written to the sstable.
+ * Will be preceded by a call to {@code startPartition(DecoratedKey, long)},
+ * and the cell should be assumed to belong to that row.
+ *
+ * @param cell The cell being added to the row.
+ */
+ void nextCell(ColumnData cell);
+
+ /**
+ * Called when all data is written to the file and it's ready to be finished up.
+ */
+ void complete();
+}
\ No newline at end of file
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 8788766..8206092 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -435,7 +435,7 @@
: new BufferedSegmentedFile.Builder())
{
if (!sstable.loadSummary(ibuilder, dbuilder))
- sstable.buildSummary(false, ibuilder, dbuilder, false, Downsampling.BASE_SAMPLING_LEVEL);
+ sstable.buildSummary(false, false, Downsampling.BASE_SAMPLING_LEVEL);
sstable.ifile = ibuilder.buildIndex(sstable.descriptor, sstable.indexSummary);
sstable.dfile = dbuilder.buildData(sstable.descriptor, statsMetadata);
sstable.bf = FilterFactory.AlwaysPresent;
@@ -734,7 +734,7 @@
boolean builtSummary = false;
if (recreateBloomFilter || !summaryLoaded)
{
- buildSummary(recreateBloomFilter, ibuilder, dbuilder, summaryLoaded, Downsampling.BASE_SAMPLING_LEVEL);
+ buildSummary(recreateBloomFilter, summaryLoaded, Downsampling.BASE_SAMPLING_LEVEL);
builtSummary = true;
}
@@ -745,7 +745,7 @@
// Check for an index summary that was downsampled even though the serialization format doesn't support
// that. If it was downsampled, rebuild it. See CASSANDRA-8993 for details.
- if (!descriptor.version.hasSamplingLevel() && !builtSummary && !validateSummarySamplingLevel() && ifile != null)
+ if (ifile != null && !descriptor.version.hasSamplingLevel() && !builtSummary && !validateSummarySamplingLevel())
{
indexSummary.close();
ifile.close();
@@ -757,7 +757,7 @@
try(SegmentedFile.Builder ibuilderRebuild = SegmentedFile.getBuilder(DatabaseDescriptor.getIndexAccessMode(), false);
SegmentedFile.Builder dbuilderRebuild = SegmentedFile.getBuilder(DatabaseDescriptor.getDiskAccessMode(), compression))
{
- buildSummary(false, ibuilderRebuild, dbuilderRebuild, false, Downsampling.BASE_SAMPLING_LEVEL);
+ buildSummary(false, false, Downsampling.BASE_SAMPLING_LEVEL);
ifile = ibuilderRebuild.buildIndex(descriptor, indexSummary);
dfile = dbuilderRebuild.buildData(descriptor, sstableMetadata);
saveSummary(ibuilderRebuild, dbuilderRebuild);
@@ -796,12 +796,10 @@
* Build index summary(and optionally bloom filter) by reading through Index.db file.
*
* @param recreateBloomFilter true if recreate bloom filter
- * @param ibuilder
- * @param dbuilder
* @param summaryLoaded true if index summary is already loaded and not need to build again
* @throws IOException
*/
- private void buildSummary(boolean recreateBloomFilter, SegmentedFile.Builder ibuilder, SegmentedFile.Builder dbuilder, boolean summaryLoaded, int samplingLevel) throws IOException
+ private void buildSummary(boolean recreateBloomFilter, boolean summaryLoaded, int samplingLevel) throws IOException
{
if (!components.contains(Component.PRIMARY_INDEX))
return;
@@ -826,7 +824,7 @@
while ((indexPosition = primaryIndex.getFilePointer()) != indexSize)
{
ByteBuffer key = ByteBufferUtil.readWithShortLength(primaryIndex);
- RowIndexEntry indexEntry = rowIndexSerializer.deserialize(primaryIndex);
+ /*RowIndexEntry indexEntry = */rowIndexSerializer.deserialize(primaryIndex);
DecoratedKey decoratedKey = decorateKey(key);
if (first == null)
first = decoratedKey;
@@ -1263,7 +1261,7 @@
/**
* Gets the position in the index file to start scanning to find the given key (at most indexInterval keys away,
- * modulo downsampling of the index summary). Always returns a value >= 0
+ * modulo downsampling of the index summary). Always returns a {@code value >= 0}
*/
public long getIndexScanPosition(PartitionPosition key)
{
@@ -1674,7 +1672,6 @@
* When calling this function, the caller must ensure that the SSTableReader is not referenced anywhere
* except for threads holding a reference.
*
- * @return true if the this is the first time the file was marked obsolete. Calling this
* multiple times is usually buggy (see exceptions in Tracker.unmarkCompacting and removeOldSSTablesSize).
*/
public void markObsolete(Runnable tidier)
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableWriter.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableWriter.java
index 4cbbd70..3203964 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableWriter.java
@@ -18,20 +18,21 @@
package org.apache.cassandra.io.sstable.format;
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.Sets;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.RowIndexEntry;
import org.apache.cassandra.db.SerializationHeader;
+import org.apache.cassandra.db.compaction.OperationType;
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
+import org.apache.cassandra.index.Index;
+import org.apache.cassandra.io.FSWriteError;
import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.SSTable;
@@ -58,6 +59,7 @@
protected final RowIndexEntry.IndexSerializer rowIndexEntrySerializer;
protected final SerializationHeader header;
protected final TransactionalProxy txnProxy = txnProxy();
+ protected final Collection<SSTableFlushObserver> observers;
protected abstract TransactionalProxy txnProxy();
@@ -69,12 +71,13 @@
protected boolean openResult;
}
- protected SSTableWriter(Descriptor descriptor,
- long keyCount,
- long repairedAt,
- CFMetaData metadata,
- MetadataCollector metadataCollector,
- SerializationHeader header)
+ protected SSTableWriter(Descriptor descriptor,
+ long keyCount,
+ long repairedAt,
+ CFMetaData metadata,
+ MetadataCollector metadataCollector,
+ SerializationHeader header,
+ Collection<SSTableFlushObserver> observers)
{
super(descriptor, components(metadata), metadata);
this.keyCount = keyCount;
@@ -82,6 +85,7 @@
this.metadataCollector = metadataCollector;
this.header = header;
this.rowIndexEntrySerializer = descriptor.version.getSSTableFormat().getIndexSerializer(metadata, descriptor.version, header);
+ this.observers = observers == null ? Collections.emptySet() : observers;
}
public static SSTableWriter create(Descriptor descriptor,
@@ -90,16 +94,23 @@
CFMetaData metadata,
MetadataCollector metadataCollector,
SerializationHeader header,
+ Collection<Index> indexes,
LifecycleTransaction txn)
{
Factory writerFactory = descriptor.getFormat().getWriterFactory();
- return writerFactory.open(descriptor, keyCount, repairedAt, metadata, metadataCollector, header, txn);
+ return writerFactory.open(descriptor, keyCount, repairedAt, metadata, metadataCollector, header, observers(descriptor, indexes, txn.opType()), txn);
}
- public static SSTableWriter create(Descriptor descriptor, long keyCount, long repairedAt, int sstableLevel, SerializationHeader header, LifecycleTransaction txn)
+ public static SSTableWriter create(Descriptor descriptor,
+ long keyCount,
+ long repairedAt,
+ int sstableLevel,
+ SerializationHeader header,
+ Collection<Index> indexes,
+ LifecycleTransaction txn)
{
CFMetaData metadata = Schema.instance.getCFMetaData(descriptor);
- return create(metadata, descriptor, keyCount, repairedAt, sstableLevel, header, txn);
+ return create(metadata, descriptor, keyCount, repairedAt, sstableLevel, header, indexes, txn);
}
public static SSTableWriter create(CFMetaData metadata,
@@ -108,21 +119,34 @@
long repairedAt,
int sstableLevel,
SerializationHeader header,
+ Collection<Index> indexes,
LifecycleTransaction txn)
{
MetadataCollector collector = new MetadataCollector(metadata.comparator).sstableLevel(sstableLevel);
- return create(descriptor, keyCount, repairedAt, metadata, collector, header, txn);
+ return create(descriptor, keyCount, repairedAt, metadata, collector, header, indexes, txn);
}
- public static SSTableWriter create(String filename, long keyCount, long repairedAt, int sstableLevel, SerializationHeader header,LifecycleTransaction txn)
+ public static SSTableWriter create(String filename,
+ long keyCount,
+ long repairedAt,
+ int sstableLevel,
+ SerializationHeader header,
+ Collection<Index> indexes,
+ LifecycleTransaction txn)
{
- return create(Descriptor.fromFilename(filename), keyCount, repairedAt, sstableLevel, header, txn);
+ return create(Descriptor.fromFilename(filename), keyCount, repairedAt, sstableLevel, header, indexes, txn);
}
@VisibleForTesting
- public static SSTableWriter create(String filename, long keyCount, long repairedAt, SerializationHeader header, LifecycleTransaction txn)
+ public static SSTableWriter create(String filename,
+ long keyCount,
+ long repairedAt,
+ SerializationHeader header,
+ Collection<Index> indexes,
+ LifecycleTransaction txn)
{
- return create(Descriptor.fromFilename(filename), keyCount, repairedAt, 0, header, txn);
+ Descriptor descriptor = Descriptor.fromFilename(filename);
+ return create(descriptor, keyCount, repairedAt, 0, header, indexes, txn);
}
private static Set<Component> components(CFMetaData metadata)
@@ -150,6 +174,27 @@
return components;
}
+ private static Collection<SSTableFlushObserver> observers(Descriptor descriptor,
+ Collection<Index> indexes,
+ OperationType operationType)
+ {
+ if (indexes == null)
+ return Collections.emptyList();
+
+ List<SSTableFlushObserver> observers = new ArrayList<>(indexes.size());
+ for (Index index : indexes)
+ {
+ SSTableFlushObserver observer = index.getFlushObserver(descriptor, operationType);
+ if (observer != null)
+ {
+ observer.begin();
+ observers.add(observer);
+ }
+ }
+
+ return ImmutableList.copyOf(observers);
+ }
+
public abstract void mark();
/**
@@ -211,6 +256,7 @@
{
setOpenResult(openResult);
txnProxy.finish();
+ observers.forEach(SSTableFlushObserver::complete);
return finished();
}
@@ -285,6 +331,7 @@
CFMetaData metadata,
MetadataCollector metadataCollector,
SerializationHeader header,
+ Collection<SSTableFlushObserver> observers,
LifecycleTransaction txn);
}
}
diff --git a/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java b/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java
index 1f2a98f..e030b5b 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java
@@ -17,6 +17,7 @@
*/
package org.apache.cassandra.io.sstable.format.big;
+import java.util.Collection;
import java.util.Set;
import org.apache.cassandra.config.CFMetaData;
@@ -25,10 +26,7 @@
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.Descriptor;
-import org.apache.cassandra.io.sstable.format.SSTableFormat;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
-import org.apache.cassandra.io.sstable.format.SSTableWriter;
-import org.apache.cassandra.io.sstable.format.Version;
+import org.apache.cassandra.io.sstable.format.*;
import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
import org.apache.cassandra.net.MessagingService;
@@ -88,9 +86,10 @@
CFMetaData metadata,
MetadataCollector metadataCollector,
SerializationHeader header,
+ Collection<SSTableFlushObserver> observers,
LifecycleTransaction txn)
{
- return new BigTableWriter(descriptor, keyCount, repairedAt, metadata, metadataCollector, header, txn);
+ return new BigTableWriter(descriptor, keyCount, repairedAt, metadata, metadataCollector, header, observers, txn);
}
}
diff --git a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index 2335e47..c1e2037 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@ -18,12 +18,14 @@
package org.apache.cassandra.io.sstable.format.big;
import java.io.*;
+import java.util.Collection;
import java.util.Map;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
import org.apache.cassandra.db.transform.Transformation;
import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.format.SSTableFlushObserver;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.sstable.format.SSTableWriter;
@@ -63,9 +65,10 @@
CFMetaData metadata,
MetadataCollector metadataCollector,
SerializationHeader header,
+ Collection<SSTableFlushObserver> observers,
LifecycleTransaction txn)
{
- super(descriptor, keyCount, repairedAt, metadata, metadataCollector, header);
+ super(descriptor, keyCount, repairedAt, metadata, metadataCollector, header, observers);
txn.trackNew(this); // must track before any files are created
if (compression)
@@ -143,10 +146,11 @@
return null;
long startPosition = beforeAppend(key);
+ observers.forEach((o) -> o.startPartition(key, iwriter.indexFile.position()));
try (UnfilteredRowIterator collecting = Transformation.apply(iterator, new StatsCollector(metadataCollector)))
{
- ColumnIndex index = ColumnIndex.writeAndBuildIndex(collecting, dataFile, header, descriptor.version);
+ ColumnIndex index = ColumnIndex.writeAndBuildIndex(collecting, dataFile, header, observers, descriptor.version);
RowIndexEntry entry = RowIndexEntry.create(startPosition, collecting.partitionLevelDeletion(), index);
diff --git a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataSerializer.java b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataSerializer.java
index 635adcd..ae1787a 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataSerializer.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataSerializer.java
@@ -37,7 +37,7 @@
import org.apache.cassandra.utils.FBUtilities;
/**
- * Metadata serializer for SSTables version >= 'k'.
+ * Metadata serializer for SSTables {@code version >= 'k'}.
*
* <pre>
* File format := | number of components (4 bytes) | toc | component1 | component2 | ... |
diff --git a/src/java/org/apache/cassandra/io/util/DataIntegrityMetadata.java b/src/java/org/apache/cassandra/io/util/DataIntegrityMetadata.java
index 0a89d74..eb84a89 100644
--- a/src/java/org/apache/cassandra/io/util/DataIntegrityMetadata.java
+++ b/src/java/org/apache/cassandra/io/util/DataIntegrityMetadata.java
@@ -193,6 +193,7 @@
{
ByteBuffer byteBuffer = ByteBuffer.allocate(4);
byteBuffer.putInt(incrementalChecksumValue);
+ assert byteBuffer.arrayOffset() == 0;
fullChecksum.update(byteBuffer.array(), 0, byteBuffer.array().length);
}
incrementalChecksum.reset();
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 80a7fe2..91242b8 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@ -170,6 +170,7 @@
public byte[] getData()
{
+ assert buffer.arrayOffset() == 0;
return buffer.array();
}
diff --git a/src/java/org/apache/cassandra/io/util/DiskAwareRunnable.java b/src/java/org/apache/cassandra/io/util/DiskAwareRunnable.java
index 925efd6..a083218 100644
--- a/src/java/org/apache/cassandra/io/util/DiskAwareRunnable.java
+++ b/src/java/org/apache/cassandra/io/util/DiskAwareRunnable.java
@@ -24,7 +24,12 @@
{
protected Directories.DataDirectory getWriteDirectory(long writeSize)
{
- Directories.DataDirectory directory = getDirectories().getWriteableLocation(writeSize);
+ Directories.DataDirectory directory;
+ directory = getDirectory();
+
+ if (directory == null) // ok panic - write anywhere
+ directory = getDirectories().getWriteableLocation(writeSize);
+
if (directory == null)
throw new RuntimeException("Insufficient disk space to write " + writeSize + " bytes");
@@ -36,4 +41,14 @@
* @return Directories instance for the CF.
*/
protected abstract Directories getDirectories();
+ protected abstract Directories.DataDirectory getDirectory();
+
+ /**
+ * Called if no disk is available with free space for the full write size.
+ * @return true if the scope of the task was successfully reduced.
+ */
+ public boolean reduceScopeForLimitedSpace()
+ {
+ return false;
+ }
}
diff --git a/src/java/org/apache/cassandra/io/util/FileSegmentInputStream.java b/src/java/org/apache/cassandra/io/util/FileSegmentInputStream.java
index 425c7d6..b6cf94a 100644
--- a/src/java/org/apache/cassandra/io/util/FileSegmentInputStream.java
+++ b/src/java/org/apache/cassandra/io/util/FileSegmentInputStream.java
@@ -68,12 +68,6 @@
buffer.position((int) (pos - offset));
}
- @Override
- public boolean markSupported()
- {
- return false;
- }
-
public FileMark mark()
{
throw new UnsupportedOperationException();
diff --git a/src/java/org/apache/cassandra/io/util/FileUtils.java b/src/java/org/apache/cassandra/io/util/FileUtils.java
index 46f2de5..d982e15 100644
--- a/src/java/org/apache/cassandra/io/util/FileUtils.java
+++ b/src/java/org/apache/cassandra/io/util/FileUtils.java
@@ -23,6 +23,7 @@
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
+import java.nio.file.attribute.BasicFileAttributes;
import java.text.DecimalFormat;
import java.util.Arrays;
import java.util.Collections;
@@ -518,25 +519,34 @@
break;
}
}
+
/**
* Get the size of a directory in bytes
- * @param directory The directory for which we need size.
+ * @param folder The directory for which we need size.
* @return The size of the directory
*/
- public static long folderSize(File directory)
+ public static long folderSize(File folder)
{
- long length = 0;
- for (File file : directory.listFiles())
+ final long [] sizeArr = {0L};
+ try
{
- if (file.isFile())
- length += file.length();
- else
- length += folderSize(file);
+ Files.walkFileTree(folder.toPath(), new SimpleFileVisitor<Path>()
+ {
+ @Override
+ public FileVisitResult visitFile(Path file, BasicFileAttributes attrs)
+ {
+ sizeArr[0] += attrs.size();
+ return FileVisitResult.CONTINUE;
+ }
+ });
}
- return length;
+ catch (IOException e)
+ {
+ logger.error("Error while getting {} folder size. {}", folder, e);
+ }
+ return sizeArr[0];
}
-
public static void copyTo(DataInput in, OutputStream out, int length) throws IOException
{
byte[] buffer = new byte[64 * 1024];
diff --git a/src/java/org/apache/cassandra/io/util/RandomAccessReader.java b/src/java/org/apache/cassandra/io/util/RandomAccessReader.java
index b495bf0..e6da1a7 100644
--- a/src/java/org/apache/cassandra/io/util/RandomAccessReader.java
+++ b/src/java/org/apache/cassandra/io/util/RandomAccessReader.java
@@ -308,10 +308,10 @@
* represented by zero or more characters followed by {@code '\n'}, {@code
* '\r'}, {@code "\r\n"} or the end of file marker. The string does not
* include the line terminating sequence.
- * <p/>
+ * <p>
* Blocks until a line terminating sequence has been read, the end of the
* file is reached or an exception is thrown.
- *
+ * </p>
* @return the contents of the line or {@code null} if no characters have
* been read before the end of the file has been reached.
* @throws IOException if this file is closed or another I/O error occurs.
diff --git a/src/java/org/apache/cassandra/io/util/RebufferingInputStream.java b/src/java/org/apache/cassandra/io/util/RebufferingInputStream.java
index 3068746..d85f1be 100644
--- a/src/java/org/apache/cassandra/io/util/RebufferingInputStream.java
+++ b/src/java/org/apache/cassandra/io/util/RebufferingInputStream.java
@@ -275,16 +275,4 @@
return -1;
}
}
-
- @Override
- public void reset() throws IOException
- {
- throw new IOException("mark/reset not supported");
- }
-
- @Override
- public boolean markSupported()
- {
- return false;
- }
}
diff --git a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
index 307a07f..f0342d4 100644
--- a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
+++ b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
@@ -28,6 +28,7 @@
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.locator.TokenMetadata.Topology;
import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.Pair;
import com.google.common.collect.Multimap;
@@ -48,14 +49,12 @@
*/
public class NetworkTopologyStrategy extends AbstractReplicationStrategy
{
- private final IEndpointSnitch snitch;
private final Map<String, Integer> datacenters;
private static final Logger logger = LoggerFactory.getLogger(NetworkTopologyStrategy.class);
public NetworkTopologyStrategy(String keyspaceName, TokenMetadata tokenMetadata, IEndpointSnitch snitch, Map<String, String> configOptions) throws ConfigurationException
{
super(keyspaceName, tokenMetadata, snitch, configOptions);
- this.snitch = snitch;
Map<String, Integer> newDatacenters = new HashMap<String, Integer>();
if (configOptions != null)
@@ -75,17 +74,78 @@
}
/**
- * calculate endpoints in one pass through the tokens by tracking our progress in each DC, rack etc.
+ * Endpoint adder applying the replication rules for a given DC.
*/
- @SuppressWarnings("serial")
+ private static final class DatacenterEndpoints
+ {
+ /** List accepted endpoints get pushed into. */
+ Set<InetAddress> endpoints;
+ /**
+ * Racks encountered so far. Replicas are put into separate racks while possible.
+ * For efficiency the set is shared between the instances, using the location pair (dc, rack) to make sure
+ * clashing names aren't a problem.
+ */
+ Set<Pair<String, String>> racks;
+
+ /** Number of replicas left to fill from this DC. */
+ int rfLeft;
+ int acceptableRackRepeats;
+
+ DatacenterEndpoints(int rf, int rackCount, int nodeCount, Set<InetAddress> endpoints, Set<Pair<String, String>> racks)
+ {
+ this.endpoints = endpoints;
+ this.racks = racks;
+ // If there aren't enough nodes in this DC to fill the RF, the number of nodes is the effective RF.
+ this.rfLeft = Math.min(rf, nodeCount);
+ // If there aren't enough racks in this DC to fill the RF, we'll still use at least one node from each rack,
+ // and the difference is to be filled by the first encountered nodes.
+ acceptableRackRepeats = rf - rackCount;
+ }
+
+ /**
+ * Attempts to add an endpoint to the replicas for this datacenter, adding to the endpoints set if successful.
+ * Returns true if the endpoint was added, and this datacenter does not require further replicas.
+ */
+ boolean addEndpointAndCheckIfDone(InetAddress ep, Pair<String,String> location)
+ {
+ if (done())
+ return false;
+
+ if (racks.add(location))
+ {
+ // New rack.
+ --rfLeft;
+ boolean added = endpoints.add(ep);
+ assert added;
+ return done();
+ }
+ if (acceptableRackRepeats <= 0)
+ // There must be rfLeft distinct racks left, do not add any more rack repeats.
+ return false;
+ if (!endpoints.add(ep))
+ // Cannot repeat a node.
+ return false;
+ // Added a node that is from an already met rack to match RF when there aren't enough racks.
+ --acceptableRackRepeats;
+ --rfLeft;
+ return done();
+ }
+
+ boolean done()
+ {
+ assert rfLeft >= 0;
+ return rfLeft == 0;
+ }
+ }
+
+ /**
+ * calculate endpoints in one pass through the tokens by tracking our progress in each DC.
+ */
public List<InetAddress> calculateNaturalEndpoints(Token searchToken, TokenMetadata tokenMetadata)
{
// we want to preserve insertion order so that the first added endpoint becomes primary
Set<InetAddress> replicas = new LinkedHashSet<>();
- // replicas we have found in each DC
- Map<String, Set<InetAddress>> dcReplicas = new HashMap<>(datacenters.size());
- for (Map.Entry<String, Integer> dc : datacenters.entrySet())
- dcReplicas.put(dc.getKey(), new HashSet<InetAddress>(dc.getValue()));
+ Set<Pair<String, String>> seenRacks = new HashSet<>();
Topology topology = tokenMetadata.getTopology();
// all endpoints in each DC, so we can check when we have exhausted all the members of a DC
@@ -94,74 +154,45 @@
Map<String, Multimap<String, InetAddress>> racks = topology.getDatacenterRacks();
assert !allEndpoints.isEmpty() && !racks.isEmpty() : "not aware of any cluster members";
- // tracks the racks we have already placed replicas in
- Map<String, Set<String>> seenRacks = new HashMap<>(datacenters.size());
- for (Map.Entry<String, Integer> dc : datacenters.entrySet())
- seenRacks.put(dc.getKey(), new HashSet<String>());
+ int dcsToFill = 0;
+ Map<String, DatacenterEndpoints> dcs = new HashMap<>(datacenters.size() * 2);
- // tracks the endpoints that we skipped over while looking for unique racks
- // when we relax the rack uniqueness we can append this to the current result so we don't have to wind back the iterator
- Map<String, Set<InetAddress>> skippedDcEndpoints = new HashMap<>(datacenters.size());
- for (Map.Entry<String, Integer> dc : datacenters.entrySet())
- skippedDcEndpoints.put(dc.getKey(), new LinkedHashSet<InetAddress>());
+ // Create a DatacenterEndpoints object for each non-empty DC.
+ for (Map.Entry<String, Integer> en : datacenters.entrySet())
+ {
+ String dc = en.getKey();
+ int rf = en.getValue();
+ int nodeCount = sizeOrZero(allEndpoints.get(dc));
+
+ if (rf <= 0 || nodeCount <= 0)
+ continue;
+
+ DatacenterEndpoints dcEndpoints = new DatacenterEndpoints(rf, sizeOrZero(racks.get(dc)), nodeCount, replicas, seenRacks);
+ dcs.put(dc, dcEndpoints);
+ ++dcsToFill;
+ }
Iterator<Token> tokenIter = TokenMetadata.ringIterator(tokenMetadata.sortedTokens(), searchToken, false);
- while (tokenIter.hasNext() && !hasSufficientReplicas(dcReplicas, allEndpoints))
+ while (dcsToFill > 0 && tokenIter.hasNext())
{
Token next = tokenIter.next();
InetAddress ep = tokenMetadata.getEndpoint(next);
- String dc = snitch.getDatacenter(ep);
- // have we already found all replicas for this dc?
- if (!datacenters.containsKey(dc) || hasSufficientReplicas(dc, dcReplicas, allEndpoints))
- continue;
- // can we skip checking the rack?
- if (seenRacks.get(dc).size() == racks.get(dc).keySet().size())
- {
- dcReplicas.get(dc).add(ep);
- replicas.add(ep);
- }
- else
- {
- String rack = snitch.getRack(ep);
- // is this a new rack?
- if (seenRacks.get(dc).contains(rack))
- {
- skippedDcEndpoints.get(dc).add(ep);
- }
- else
- {
- dcReplicas.get(dc).add(ep);
- replicas.add(ep);
- seenRacks.get(dc).add(rack);
- // if we've run out of distinct racks, add the hosts we skipped past already (up to RF)
- if (seenRacks.get(dc).size() == racks.get(dc).keySet().size())
- {
- Iterator<InetAddress> skippedIt = skippedDcEndpoints.get(dc).iterator();
- while (skippedIt.hasNext() && !hasSufficientReplicas(dc, dcReplicas, allEndpoints))
- {
- InetAddress nextSkipped = skippedIt.next();
- dcReplicas.get(dc).add(nextSkipped);
- replicas.add(nextSkipped);
- }
- }
- }
- }
+ Pair<String, String> location = topology.getLocation(ep);
+ DatacenterEndpoints dcEndpoints = dcs.get(location.left);
+ if (dcEndpoints != null && dcEndpoints.addEndpointAndCheckIfDone(ep, location))
+ --dcsToFill;
}
-
- return new ArrayList<InetAddress>(replicas);
+ return new ArrayList<>(replicas);
}
- private boolean hasSufficientReplicas(String dc, Map<String, Set<InetAddress>> dcReplicas, Multimap<String, InetAddress> allEndpoints)
+ private int sizeOrZero(Multimap<?, ?> collection)
{
- return dcReplicas.get(dc).size() >= Math.min(allEndpoints.get(dc).size(), getReplicationFactor(dc));
+ return collection != null ? collection.asMap().size() : 0;
}
- private boolean hasSufficientReplicas(Map<String, Set<InetAddress>> dcReplicas, Multimap<String, InetAddress> allEndpoints)
+ private int sizeOrZero(Collection<?> collection)
{
- for (String dc : datacenters.keySet())
- if (!hasSufficientReplicas(dc, dcReplicas, allEndpoints))
- return false;
- return true;
+ return collection != null ? collection.size() : 0;
}
public int getReplicationFactor()
@@ -192,10 +223,4 @@
validateReplicationFactor(e.getValue());
}
}
-
- public Collection<String> recognizedOptions()
- {
- // We explicitely allow all options
- return null;
- }
}
diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java b/src/java/org/apache/cassandra/locator/TokenMetadata.java
index f6e9cf7..d47b681 100644
--- a/src/java/org/apache/cassandra/locator/TokenMetadata.java
+++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java
@@ -868,20 +868,20 @@
public Token getPredecessor(Token token)
{
- List tokens = sortedTokens();
+ List<Token> tokens = sortedTokens();
int index = Collections.binarySearch(tokens, token);
// assert index >= 0 : token + " not found in " + StringUtils.join(tokenToEndpointMap.keySet(), ", ");
if (index < 0) index = -index-1;
- return (Token) (index == 0 ? tokens.get(tokens.size() - 1) : tokens.get(index - 1));
+ return index == 0 ? tokens.get(tokens.size() - 1) : tokens.get(index - 1);
}
public Token getSuccessor(Token token)
{
- List tokens = sortedTokens();
+ List<Token> tokens = sortedTokens();
int index = Collections.binarySearch(tokens, token);
// assert index >= 0 : token + " not found in " + StringUtils.join(tokenToEndpointMap.keySet(), ", ");
if (index < 0) return (Token) tokens.get(-index-1);
- return (Token) ((index == (tokens.size() - 1)) ? tokens.get(0) : tokens.get(index + 1));
+ return (index == (tokens.size() - 1)) ? tokens.get(0) : tokens.get(index + 1);
}
/** @return a copy of the bootstrapping tokens map */
@@ -942,7 +942,7 @@
}
}
- public static int firstTokenIndex(final ArrayList ring, Token start, boolean insertMin)
+ public static int firstTokenIndex(final ArrayList<Token> ring, Token start, boolean insertMin)
{
assert ring.size() > 0;
// insert the minimum token (at index == -1) if we were asked to include it and it isn't a member of the ring
@@ -970,7 +970,7 @@
{
if (ring.isEmpty())
return includeMin ? Iterators.singletonIterator(start.getPartitioner().getMinimumToken())
- : Iterators.<Token>emptyIterator();
+ : Collections.emptyIterator();
final boolean insertMin = includeMin && !ring.get(0).isMinimum();
final int startIndex = firstTokenIndex(ring, start, insertMin);
@@ -1308,5 +1308,14 @@
{
return dcRacks;
}
+
+ /**
+ * @return The DC and rack of the given endpoint.
+ */
+ public Pair<String, String> getLocation(InetAddress addr)
+ {
+ return currentLocations.get(addr);
+ }
+
}
}
diff --git a/src/java/org/apache/cassandra/metrics/CassandraMetricsRegistry.java b/src/java/org/apache/cassandra/metrics/CassandraMetricsRegistry.java
index 1cd3b6c..0d9e6d6 100644
--- a/src/java/org/apache/cassandra/metrics/CassandraMetricsRegistry.java
+++ b/src/java/org/apache/cassandra/metrics/CassandraMetricsRegistry.java
@@ -191,6 +191,18 @@
mBeanServer.unregisterMBean(name.getMBeanName());
} catch (Exception ignore) {}
}
+
+ /**
+ * Strips a single final '$' from input
+ *
+ * @param s String to strip
+ * @return a string with one less '$' at end
+ */
+ private static String withoutFinalDollar(String s)
+ {
+ int l = s.length();
+ return (l!=0 && '$' == s.charAt(l-1))?s.substring(0,l-1):s;
+ }
public interface MetricMBean
{
@@ -601,7 +613,7 @@
public MetricName(Class<?> klass, String name, String scope)
{
this(klass.getPackage() == null ? "" : klass.getPackage().getName(),
- klass.getSimpleName().replaceAll("\\$$", ""),
+ withoutFinalDollar(klass.getSimpleName()),
name,
scope);
}
@@ -811,7 +823,7 @@
{
if (type == null || type.isEmpty())
{
- type = klass.getSimpleName().replaceAll("\\$$", "");
+ type = withoutFinalDollar(klass.getSimpleName());
}
return type;
}
diff --git a/src/java/org/apache/cassandra/metrics/CompactionMetrics.java b/src/java/org/apache/cassandra/metrics/CompactionMetrics.java
index 19eadc8..f3a9d00 100644
--- a/src/java/org/apache/cassandra/metrics/CompactionMetrics.java
+++ b/src/java/org/apache/cassandra/metrics/CompactionMetrics.java
@@ -24,6 +24,7 @@
import com.codahale.metrics.Gauge;
import com.codahale.metrics.Meter;
+import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.Keyspace;
@@ -44,6 +45,9 @@
/** Estimated number of compactions remaining to perform */
public final Gauge<Integer> pendingTasks;
+ /** Estimated number of compactions remaining to perform, group by keyspace and then table name */
+ public final Gauge<Map<String, Map<String, Integer>>> pendingTasksByTableName;
+
/** Number of completed compactions since server [re]start */
public final Gauge<Long> completedTasks;
/** Total number of compactions since server [re]start */
@@ -68,6 +72,54 @@
return n + compactions.size();
}
});
+
+ pendingTasksByTableName = Metrics.register(factory.createMetricName("PendingTasksByTableName"),
+ new Gauge<Map<String, Map<String, Integer>>>()
+ {
+ @Override
+ public Map<String, Map<String, Integer>> getValue() {
+ Map<String, Map<String, Integer>> resultMap = new HashMap<>();
+ // estimation of compactions need to be done
+ for (String keyspaceName : Schema.instance.getKeyspaces())
+ {
+ for (ColumnFamilyStore cfs : Keyspace.open(keyspaceName).getColumnFamilyStores())
+ {
+ int taskNumber = cfs.getCompactionStrategyManager().getEstimatedRemainingTasks();
+ if (taskNumber > 0)
+ {
+ if (!resultMap.containsKey(keyspaceName))
+ {
+ resultMap.put(keyspaceName, new HashMap<>());
+ }
+ resultMap.get(keyspaceName).put(cfs.getTableName(), taskNumber);
+ }
+ }
+ }
+
+ // currently running compactions
+ for (CompactionInfo.Holder compaction : compactions)
+ {
+ CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();
+ if (!resultMap.containsKey(metaData.ksName))
+ {
+ resultMap.put(metaData.ksName, new HashMap<>());
+ }
+
+ Map<String, Integer> tableNameToCountMap = resultMap.get(metaData.ksName);
+ if (tableNameToCountMap.containsKey(metaData.cfName))
+ {
+ tableNameToCountMap.put(metaData.cfName,
+ tableNameToCountMap.get(metaData.cfName) + 1);
+ }
+ else
+ {
+ tableNameToCountMap.put(metaData.cfName, 1);
+ }
+ }
+ return resultMap;
+ }
+ });
+
completedTasks = Metrics.register(factory.createMetricName("CompletedTasks"), new Gauge<Long>()
{
public Long getValue()
diff --git a/src/java/org/apache/cassandra/metrics/DroppedMessageMetrics.java b/src/java/org/apache/cassandra/metrics/DroppedMessageMetrics.java
index 58c80fb..2a94c9f 100644
--- a/src/java/org/apache/cassandra/metrics/DroppedMessageMetrics.java
+++ b/src/java/org/apache/cassandra/metrics/DroppedMessageMetrics.java
@@ -18,6 +18,8 @@
package org.apache.cassandra.metrics;
import com.codahale.metrics.Meter;
+import com.codahale.metrics.Timer;
+
import org.apache.cassandra.net.MessagingService;
import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
@@ -30,9 +32,17 @@
/** Number of dropped messages */
public final Meter dropped;
+ /** The dropped latency within node */
+ public final Timer internalDroppedLatency;
+
+ /** The cross node dropped latency */
+ public final Timer crossNodeDroppedLatency;
+
public DroppedMessageMetrics(MessagingService.Verb verb)
{
MetricNameFactory factory = new DefaultNameFactory("DroppedMessage", verb.toString());
dropped = Metrics.meter(factory.createMetricName("Dropped"));
+ internalDroppedLatency = Metrics.timer(factory.createMetricName("InternalDroppedLatency"));
+ crossNodeDroppedLatency = Metrics.timer(factory.createMetricName("CrossNodeDroppedLatency"));
}
}
diff --git a/src/java/org/apache/cassandra/metrics/TableMetrics.java b/src/java/org/apache/cassandra/metrics/TableMetrics.java
index 85bf7f6..c8c214e 100644
--- a/src/java/org/apache/cassandra/metrics/TableMetrics.java
+++ b/src/java/org/apache/cassandra/metrics/TableMetrics.java
@@ -24,13 +24,14 @@
import com.codahale.metrics.*;
import com.codahale.metrics.Timer;
+import com.google.common.collect.Iterables;
import com.google.common.collect.Maps;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.Memtable;
import org.apache.cassandra.db.lifecycle.SSTableSet;
+import org.apache.cassandra.io.compress.CompressionMetadata;
import org.apache.cassandra.io.sstable.format.SSTableReader;
-import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
import org.apache.cassandra.utils.EstimatedHistogram;
import org.apache.cassandra.utils.TopKSampler;
@@ -42,6 +43,8 @@
public class TableMetrics
{
+ public static final long[] EMPTY = new long[0];
+
/** Total amount of data stored in the memtable that resides on-heap, including column related overhead and partitions overwritten. */
public final Gauge<Long> memtableOnHeapSize;
/** Total amount of data stored in the memtable that resides off-heap, including column related overhead and partitions overwritten. */
@@ -171,7 +174,7 @@
Iterator<SSTableReader> iterator = sstables.iterator();
if (!iterator.hasNext())
{
- return new long[0];
+ return EMPTY;
}
long[] firstBucket = getHistogram.getHistogram(iterator.next()).getBuckets(false);
long[] values = new long[firstBucket.length];
@@ -323,36 +326,14 @@
{
public Double getValue()
{
- double sum = 0;
- int total = 0;
- for (SSTableReader sstable : cfs.getSSTables(SSTableSet.CANONICAL))
- {
- if (sstable.getCompressionRatio() != MetadataCollector.NO_COMPRESSION_RATIO)
- {
- sum += sstable.getCompressionRatio();
- total++;
- }
- }
- return total != 0 ? sum / total : 0;
+ return computeCompressionRatio(cfs.getSSTables(SSTableSet.CANONICAL));
}
}, new Gauge<Double>() // global gauge
{
public Double getValue()
{
- double sum = 0;
- int total = 0;
- for (Keyspace keyspace : Keyspace.all())
- {
- for (SSTableReader sstable : keyspace.getAllSSTables(SSTableSet.CANONICAL))
- {
- if (sstable.getCompressionRatio() != MetadataCollector.NO_COMPRESSION_RATIO)
- {
- sum += sstable.getCompressionRatio();
- total++;
- }
- }
- }
- return total != 0 ? sum / total : 0;
+ return computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
+ p -> p.getAllSSTables(SSTableSet.CANONICAL))));
}
});
readLatency = new LatencyMetrics(factory, "Read", cfs.keyspace.metric.readLatency, globalReadLatency);
@@ -748,6 +729,32 @@
}
/**
+ * Computes the compression ratio for the specified SSTables
+ *
+ * @param sstables the SSTables
+ * @return the compression ratio for the specified SSTables
+ */
+ private static Double computeCompressionRatio(Iterable<SSTableReader> sstables)
+ {
+ double compressedLengthSum = 0;
+ double dataLengthSum = 0;
+ for (SSTableReader sstable : sstables)
+ {
+ if (sstable.compression)
+ {
+ // We should not have any sstable which are in an open early mode as the sstable were selected
+ // using SSTableSet.CANONICAL.
+ assert sstable.openReason != SSTableReader.OpenReason.EARLY;
+
+ CompressionMetadata compressionMetadata = sstable.getCompressionMetadata();
+ compressedLengthSum += compressionMetadata.compressedFileLength;
+ dataLengthSum += compressionMetadata.dataLength;
+ }
+ }
+ return dataLengthSum != 0 ? compressedLengthSum / dataLengthSum : 0;
+ }
+
+ /**
* Create a histogram-like interface that will register both a CF, keyspace and global level
* histogram and forward any updates to both
*/
diff --git a/src/java/org/apache/cassandra/net/IAsyncCallback.java b/src/java/org/apache/cassandra/net/IAsyncCallback.java
index a29260c..d159e0c 100644
--- a/src/java/org/apache/cassandra/net/IAsyncCallback.java
+++ b/src/java/org/apache/cassandra/net/IAsyncCallback.java
@@ -31,7 +31,7 @@
*/
public interface IAsyncCallback<T>
{
- public static Predicate<InetAddress> isAlive = new Predicate<InetAddress>()
+ Predicate<InetAddress> isAlive = new Predicate<InetAddress>()
{
public boolean apply(InetAddress endpoint)
{
@@ -42,7 +42,7 @@
/**
* @param msg response received.
*/
- public void response(MessageIn<T> msg);
+ void response(MessageIn<T> msg);
/**
* @return true if this callback is on the read path and its latency should be
diff --git a/src/java/org/apache/cassandra/net/IAsyncCallbackWithFailure.java b/src/java/org/apache/cassandra/net/IAsyncCallbackWithFailure.java
index 744bb62..546a416 100644
--- a/src/java/org/apache/cassandra/net/IAsyncCallbackWithFailure.java
+++ b/src/java/org/apache/cassandra/net/IAsyncCallbackWithFailure.java
@@ -25,5 +25,5 @@
/**
* Called when there is an exception on the remote node or timeout happens
*/
- public void onFailure(InetAddress from);
+ void onFailure(InetAddress from);
}
diff --git a/src/java/org/apache/cassandra/net/IVerbHandler.java b/src/java/org/apache/cassandra/net/IVerbHandler.java
index 574f30f..b9f1a54 100644
--- a/src/java/org/apache/cassandra/net/IVerbHandler.java
+++ b/src/java/org/apache/cassandra/net/IVerbHandler.java
@@ -19,6 +19,8 @@
import java.io.IOException;
+import org.apache.cassandra.db.ReadCommand;
+
/**
* IVerbHandler provides the method that all verb handlers need to implement.
* The concrete implementation of this interface would provide the functionality
@@ -36,5 +38,5 @@
* @param message - incoming message that needs handling.
* @param id
*/
- public void doVerb(MessageIn<T> message, int id) throws IOException;
+ void doVerb(MessageIn<T> message, int id) throws IOException;
}
diff --git a/src/java/org/apache/cassandra/net/IncomingTcpConnection.java b/src/java/org/apache/cassandra/net/IncomingTcpConnection.java
index de64444..2a09bf4 100644
--- a/src/java/org/apache/cassandra/net/IncomingTcpConnection.java
+++ b/src/java/org/apache/cassandra/net/IncomingTcpConnection.java
@@ -187,18 +187,7 @@
else
id = input.readInt();
- long timestamp = System.currentTimeMillis();
- boolean isCrossNodeTimestamp = false;
- // make sure to readInt, even if cross_node_to is not enabled
- int partial = input.readInt();
- if (DatabaseDescriptor.hasCrossNodeTimeout())
- {
- long crossNodeTimestamp = (timestamp & 0xFFFFFFFF00000000L) | (((partial & 0xFFFFFFFFL) << 2) >> 2);
- isCrossNodeTimestamp = (timestamp != crossNodeTimestamp);
- timestamp = crossNodeTimestamp;
- }
-
- MessageIn message = MessageIn.read(input, version, id);
+ MessageIn message = MessageIn.read(input, version, id, MessageIn.readTimestamp(input));
if (message == null)
{
// callback expired; nothing to do
@@ -206,7 +195,7 @@
}
if (version <= MessagingService.current_version)
{
- MessagingService.instance().receive(message, id, timestamp, isCrossNodeTimestamp);
+ MessagingService.instance().receive(message, id);
}
else
{
diff --git a/src/java/org/apache/cassandra/net/MessageDeliveryTask.java b/src/java/org/apache/cassandra/net/MessageDeliveryTask.java
index ce6eebc..d9f8b7c 100644
--- a/src/java/org/apache/cassandra/net/MessageDeliveryTask.java
+++ b/src/java/org/apache/cassandra/net/MessageDeliveryTask.java
@@ -33,25 +33,22 @@
private final MessageIn message;
private final int id;
- private final long constructionTime;
- private final boolean isCrossNodeTimestamp;
- public MessageDeliveryTask(MessageIn message, int id, long timestamp, boolean isCrossNodeTimestamp)
+ public MessageDeliveryTask(MessageIn message, int id)
{
assert message != null;
this.message = message;
this.id = id;
- this.constructionTime = timestamp;
- this.isCrossNodeTimestamp = isCrossNodeTimestamp;
}
public void run()
{
MessagingService.Verb verb = message.verb;
+ long timeTaken = System.currentTimeMillis() - message.constructionTime.timestamp;
if (MessagingService.DROPPABLE_VERBS.contains(verb)
- && System.currentTimeMillis() > constructionTime + message.getTimeout())
+ && timeTaken > message.getTimeout())
{
- MessagingService.instance().incrementDroppedMessages(verb, isCrossNodeTimestamp);
+ MessagingService.instance().incrementDroppedMessages(message, timeTaken);
return;
}
@@ -83,7 +80,7 @@
}
if (GOSSIP_VERBS.contains(message.verb))
- Gossiper.instance.setLastProcessedMessageAt(constructionTime);
+ Gossiper.instance.setLastProcessedMessageAt(message.constructionTime.timestamp);
}
private void handleFailure(Throwable t)
diff --git a/src/java/org/apache/cassandra/net/MessageIn.java b/src/java/org/apache/cassandra/net/MessageIn.java
index 64b8e81..c6e4d89 100644
--- a/src/java/org/apache/cassandra/net/MessageIn.java
+++ b/src/java/org/apache/cassandra/net/MessageIn.java
@@ -24,40 +24,55 @@
import com.google.common.collect.ImmutableMap;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import org.apache.cassandra.concurrent.Stage;
import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.monitoring.ConstructionTime;
+import org.apache.cassandra.db.monitoring.MonitorableImpl;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.util.DataInputPlus;
import org.apache.cassandra.io.util.FileUtils;
public class MessageIn<T>
{
- private static final Logger logger = LoggerFactory.getLogger(MessageIn.class);
-
public final InetAddress from;
public final T payload;
public final Map<String, byte[]> parameters;
public final MessagingService.Verb verb;
public final int version;
+ public final ConstructionTime constructionTime;
- private MessageIn(InetAddress from, T payload, Map<String, byte[]> parameters, MessagingService.Verb verb, int version)
+ private MessageIn(InetAddress from,
+ T payload,
+ Map<String, byte[]> parameters,
+ MessagingService.Verb verb,
+ int version,
+ ConstructionTime constructionTime)
{
this.from = from;
this.payload = payload;
this.parameters = parameters;
this.verb = verb;
this.version = version;
+ this.constructionTime = constructionTime;
}
- public static <T> MessageIn<T> create(InetAddress from, T payload, Map<String, byte[]> parameters, MessagingService.Verb verb, int version)
+ public static <T> MessageIn<T> create(InetAddress from,
+ T payload,
+ Map<String, byte[]> parameters,
+ MessagingService.Verb verb,
+ int version,
+ ConstructionTime constructionTime)
{
- return new MessageIn<T>(from, payload, parameters, verb, version);
+ return new MessageIn<>(from, payload, parameters, verb, version, constructionTime);
}
public static <T2> MessageIn<T2> read(DataInputPlus in, int version, int id) throws IOException
{
+ return read(in, version, id, new ConstructionTime());
+ }
+
+ public static <T2> MessageIn<T2> read(DataInputPlus in, int version, int id, ConstructionTime constructionTime) throws IOException
+ {
InetAddress from = CompactEndpointSerializationHelper.deserialize(in);
MessagingService.Verb verb = MessagingService.Verb.values()[in.readInt()];
@@ -94,9 +109,31 @@
serializer = (IVersionedSerializer<T2>) callback.serializer;
}
if (payloadSize == 0 || serializer == null)
- return create(from, null, parameters, verb, version);
+ return create(from, null, parameters, verb, version, constructionTime);
+
T2 payload = serializer.deserialize(in, version);
- return MessageIn.create(from, payload, parameters, verb, version);
+ return MessageIn.create(from, payload, parameters, verb, version, constructionTime);
+ }
+
+ public static ConstructionTime createTimestamp()
+ {
+ return new ConstructionTime();
+ }
+
+ public static ConstructionTime readTimestamp(DataInputPlus input) throws IOException
+ {
+ // make sure to readInt, even if cross_node_to is not enabled
+ int partial = input.readInt();
+ if(DatabaseDescriptor.hasCrossNodeTimeout())
+ {
+ long timestamp = System.currentTimeMillis();
+ long crossNodeTimestamp = (timestamp & 0xFFFFFFFF00000000L) | (((partial & 0xFFFFFFFFL) << 2) >> 2);
+ return new ConstructionTime(crossNodeTimestamp, timestamp != crossNodeTimestamp);
+ }
+ else
+ {
+ return new ConstructionTime();
+ }
}
public Stage getMessageType()
diff --git a/src/java/org/apache/cassandra/net/MessagingService.java b/src/java/org/apache/cassandra/net/MessagingService.java
index d416dca..76f4967 100644
--- a/src/java/org/apache/cassandra/net/MessagingService.java
+++ b/src/java/org/apache/cassandra/net/MessagingService.java
@@ -810,7 +810,7 @@
}
}
- public void receive(MessageIn message, int id, long timestamp, boolean isCrossNodeTimestamp)
+ public void receive(MessageIn message, int id)
{
TraceState state = Tracing.instance.initializeFromMessage(message);
if (state != null)
@@ -821,7 +821,7 @@
if (!ms.allowIncomingMessage(message, id))
return;
- Runnable runnable = new MessageDeliveryTask(message, id, timestamp, isCrossNodeTimestamp);
+ Runnable runnable = new MessageDeliveryTask(message, id);
LocalAwareExecutorService stage = StageManager.getStage(message.getMessageType());
assert stage != null : "No stage for message type " + message.verb;
@@ -958,12 +958,37 @@
incrementDroppedMessages(verb, false);
}
+ public void incrementDroppedMessages(Verb verb, long timeTaken)
+ {
+ incrementDroppedMessages(verb, timeTaken, false);
+ }
+
+ public void incrementDroppedMessages(MessageIn message, long timeTaken)
+ {
+ incrementDroppedMessages(message.verb, timeTaken, message.constructionTime.isCrossNode);
+ }
+
+ public void incrementDroppedMessages(Verb verb, long timeTaken, boolean isCrossNodeTimeout)
+ {
+ assert DROPPABLE_VERBS.contains(verb) : "Verb " + verb + " should not legally be dropped";
+ incrementDroppedMessages(droppedMessagesMap.get(verb), timeTaken, isCrossNodeTimeout);
+ }
+
public void incrementDroppedMessages(Verb verb, boolean isCrossNodeTimeout)
{
assert DROPPABLE_VERBS.contains(verb) : "Verb " + verb + " should not legally be dropped";
incrementDroppedMessages(droppedMessagesMap.get(verb), isCrossNodeTimeout);
}
+ private void incrementDroppedMessages(DroppedMessages droppedMessages, long timeTaken, boolean isCrossNodeTimeout)
+ {
+ if (isCrossNodeTimeout)
+ droppedMessages.metrics.crossNodeDroppedLatency.update(timeTaken, TimeUnit.MILLISECONDS);
+ else
+ droppedMessages.metrics.internalDroppedLatency.update(timeTaken, TimeUnit.MILLISECONDS);
+ incrementDroppedMessages(droppedMessages, isCrossNodeTimeout);
+ }
+
private void incrementDroppedMessages(DroppedMessages droppedMessages, boolean isCrossNodeTimeout)
{
droppedMessages.metrics.dropped.mark();
@@ -996,11 +1021,14 @@
int droppedCrossNodeTimeout = droppedMessages.droppedCrossNodeTimeout.getAndSet(0);
if (droppedInternalTimeout > 0 || droppedCrossNodeTimeout > 0)
{
- ret.add(String.format("%s messages were dropped in last %d ms: %d for internal timeout and %d for cross node timeout",
- verb,
- LOG_DROPPED_INTERVAL_IN_MS,
- droppedInternalTimeout,
- droppedCrossNodeTimeout));
+ ret.add(String.format("%s messages were dropped in last %d ms: %d for internal timeout and %d for cross node timeout."
+ + " Mean internal dropped latency: %d ms and Mean cross-node dropped latency: %d ms",
+ verb,
+ LOG_DROPPED_INTERVAL_IN_MS,
+ droppedInternalTimeout,
+ droppedCrossNodeTimeout,
+ TimeUnit.NANOSECONDS.toMillis((long)droppedMessages.metrics.internalDroppedLatency.getSnapshot().getMean()),
+ TimeUnit.NANOSECONDS.toMillis((long)droppedMessages.metrics.crossNodeDroppedLatency.getSnapshot().getMean())));
}
}
return ret;
diff --git a/src/java/org/apache/cassandra/repair/LocalSyncTask.java b/src/java/org/apache/cassandra/repair/LocalSyncTask.java
index daace01..e1da497 100644
--- a/src/java/org/apache/cassandra/repair/LocalSyncTask.java
+++ b/src/java/org/apache/cassandra/repair/LocalSyncTask.java
@@ -73,7 +73,7 @@
isIncremental = prs.isIncremental;
}
Tracing.traceRepair(message);
- new StreamPlan("Repair", repairedAt, 1, false, isIncremental).listeners(this)
+ new StreamPlan("Repair", repairedAt, 1, false, isIncremental, false).listeners(this)
.flushBeforeTransfer(true)
// request ranges from the remote node
.requestRanges(dst, preferred, desc.keyspace, differences, desc.columnFamily)
diff --git a/src/java/org/apache/cassandra/repair/StreamingRepairTask.java b/src/java/org/apache/cassandra/repair/StreamingRepairTask.java
index 25ef06e..b6936b6 100644
--- a/src/java/org/apache/cassandra/repair/StreamingRepairTask.java
+++ b/src/java/org/apache/cassandra/repair/StreamingRepairTask.java
@@ -62,7 +62,7 @@
ActiveRepairService.ParentRepairSession prs = ActiveRepairService.instance.getParentRepairSession(desc.parentSessionId);
isIncremental = prs.isIncremental;
}
- new StreamPlan("Repair", repairedAt, 1, false, isIncremental).listeners(this)
+ new StreamPlan("Repair", repairedAt, 1, false, isIncremental, false).listeners(this)
.flushBeforeTransfer(true)
// request ranges from the remote node
.requestRanges(dest, preferred, desc.keyspace, request.ranges, desc.columnFamily)
diff --git a/src/java/org/apache/cassandra/repair/Validator.java b/src/java/org/apache/cassandra/repair/Validator.java
index 217c9de..e51dc0e 100644
--- a/src/java/org/apache/cassandra/repair/Validator.java
+++ b/src/java/org/apache/cassandra/repair/Validator.java
@@ -128,7 +128,7 @@
* Called (in order) for every row present in the CF.
* Hashes the row, and adds it to the tree being built.
*
- * @param row Row to add hash
+ * @param partition Partition to add hash
*/
public void add(UnfilteredRowIterator partition)
{
diff --git a/src/java/org/apache/cassandra/schema/IndexMetadata.java b/src/java/org/apache/cassandra/schema/IndexMetadata.java
index 7c60a64..04e06ab 100644
--- a/src/java/org/apache/cassandra/schema/IndexMetadata.java
+++ b/src/java/org/apache/cassandra/schema/IndexMetadata.java
@@ -21,6 +21,7 @@
import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.util.*;
+import java.util.regex.Pattern;
import java.util.stream.Collectors;
import com.google.common.base.Objects;
@@ -46,6 +47,10 @@
public final class IndexMetadata
{
private static final Logger logger = LoggerFactory.getLogger(IndexMetadata.class);
+
+ private static final Pattern PATTERN_NON_WORD_CHAR = Pattern.compile("\\W");
+ private static final Pattern PATTERN_WORD_CHARS = Pattern.compile("\\w+");
+
public static final Serializer serializer = new Serializer();
@@ -127,15 +132,15 @@
public static boolean isNameValid(String name)
{
- return name != null && !name.isEmpty() && name.matches("\\w+");
+ return name != null && !name.isEmpty() && PATTERN_WORD_CHARS.matcher(name).matches();
}
public static String getDefaultIndexName(String cfName, String root)
{
if (root == null)
- return (cfName + "_" + "idx").replaceAll("\\W", "");
+ return PATTERN_NON_WORD_CHAR.matcher(cfName + "_" + "idx").replaceAll("");
else
- return (cfName + "_" + root + "_idx").replaceAll("\\W", "");
+ return PATTERN_NON_WORD_CHAR.matcher(cfName + "_" + root + "_idx").replaceAll("");
}
public void validate(CFMetaData cfm)
diff --git a/src/java/org/apache/cassandra/schema/Indexes.java b/src/java/org/apache/cassandra/schema/Indexes.java
index 49a1d3b..eb49d39 100644
--- a/src/java/org/apache/cassandra/schema/Indexes.java
+++ b/src/java/org/apache/cassandra/schema/Indexes.java
@@ -95,7 +95,7 @@
/**
* Get the index with the specified id
*
- * @param name a UUID which identifies an index
+ * @param id a UUID which identifies an index
* @return an empty {@link Optional} if no index with the specified id is found; a non-empty optional of
* {@link IndexMetadata} otherwise
*/
@@ -107,7 +107,7 @@
/**
* Answer true if contains an index with the specified id.
- * @param name a UUID which identifies an index.
+ * @param id a UUID which identifies an index.
* @return true if an index with the specified id is found; false otherwise
*/
public boolean has(UUID id)
diff --git a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
index f1ea6cf..a28423d 100644
--- a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
+++ b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
@@ -298,8 +298,8 @@
for (String table : ALL)
{
ReadCommand cmd = getReadCommandForTableSchema(table);
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup();
- PartitionIterator schema = cmd.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ PartitionIterator schema = cmd.executeInternal(executionController))
{
while (schema.hasNext())
{
@@ -346,7 +346,8 @@
private static void convertSchemaToMutations(Map<DecoratedKey, Mutation> mutationMap, String schemaTableName)
{
ReadCommand cmd = getReadCommandForTableSchema(schemaTableName);
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iter = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iter = cmd.executeLocally(executionController))
{
while (iter.hasNext())
{
diff --git a/src/java/org/apache/cassandra/security/CipherFactory.java b/src/java/org/apache/cassandra/security/CipherFactory.java
new file mode 100644
index 0000000..7c1495a
--- /dev/null
+++ b/src/java/org/apache/cassandra/security/CipherFactory.java
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.security;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.Key;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.util.Arrays;
+import java.util.concurrent.ExecutionException;
+import javax.crypto.Cipher;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.spec.IvParameterSpec;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.TransparentDataEncryptionOptions;
+
+/**
+ * A factory for loading encryption keys from {@link KeyProvider} instances.
+ * Maintains a cache of loaded keys to avoid invoking the key provider on every call.
+ */
+public class CipherFactory
+{
+ private final Logger logger = LoggerFactory.getLogger(CipherFactory.class);
+
+ /**
+ * Keep around thread local instances of Cipher as they are quite expensive to instantiate (@code Cipher#getInstance).
+ * Bonus points if you can avoid calling (@code Cipher#init); hence, the point of the supporting struct
+ * for caching Cipher instances.
+ */
+ private static final ThreadLocal<CachedCipher> cipherThreadLocal = new ThreadLocal<>();
+
+ private final SecureRandom secureRandom;
+ private final LoadingCache<String, Key> cache;
+ private final int ivLength;
+ private final KeyProvider keyProvider;
+
+ public CipherFactory(TransparentDataEncryptionOptions options)
+ {
+ logger.info("initializing CipherFactory");
+ ivLength = options.iv_length;
+
+ try
+ {
+ secureRandom = SecureRandom.getInstance("SHA1PRNG");
+ Class<KeyProvider> keyProviderClass = (Class<KeyProvider>)Class.forName(options.key_provider.class_name);
+ Constructor ctor = keyProviderClass.getConstructor(TransparentDataEncryptionOptions.class);
+ keyProvider = (KeyProvider)ctor.newInstance(options);
+ }
+ catch (Exception e)
+ {
+ throw new RuntimeException("couldn't load cipher factory", e);
+ }
+
+ cache = CacheBuilder.newBuilder() // by default cache is unbounded
+ .maximumSize(64) // a value large enough that we should never even get close (so nothing gets evicted)
+ .concurrencyLevel(Runtime.getRuntime().availableProcessors())
+ .removalListener(new RemovalListener<String, Key>()
+ {
+ public void onRemoval(RemovalNotification<String, Key> notice)
+ {
+ // maybe reload the key? (to avoid the reload being on the user's dime)
+ logger.info("key {} removed from cipher key cache", notice.getKey());
+ }
+ })
+ .build(new CacheLoader<String, Key>()
+ {
+ @Override
+ public Key load(String alias) throws Exception
+ {
+ logger.info("loading secret key for alias {}", alias);
+ return keyProvider.getSecretKey(alias);
+ }
+ });
+ }
+
+ public Cipher getEncryptor(String transformation, String keyAlias) throws IOException
+ {
+ byte[] iv = new byte[ivLength];
+ secureRandom.nextBytes(iv);
+ return buildCipher(transformation, keyAlias, iv, Cipher.ENCRYPT_MODE);
+ }
+
+ public Cipher getDecryptor(String transformation, String keyAlias, byte[] iv) throws IOException
+ {
+ assert iv != null && iv.length > 0 : "trying to decrypt, but the initialization vector is empty";
+ return buildCipher(transformation, keyAlias, iv, Cipher.DECRYPT_MODE);
+ }
+
+ @VisibleForTesting
+ Cipher buildCipher(String transformation, String keyAlias, byte[] iv, int cipherMode) throws IOException
+ {
+ try
+ {
+ CachedCipher cachedCipher = cipherThreadLocal.get();
+ if (cachedCipher != null)
+ {
+ Cipher cipher = cachedCipher.cipher;
+ // rigorous checks to make sure we've absolutely got the correct instance (with correct alg/key/iv/...)
+ if (cachedCipher.mode == cipherMode && cipher.getAlgorithm().equals(transformation)
+ && cachedCipher.keyAlias.equals(keyAlias) && Arrays.equals(cipher.getIV(), iv))
+ return cipher;
+ }
+
+ Key key = retrieveKey(keyAlias);
+ Cipher cipher = Cipher.getInstance(transformation);
+ cipher.init(cipherMode, key, new IvParameterSpec(iv));
+ cipherThreadLocal.set(new CachedCipher(cipherMode, keyAlias, cipher));
+ return cipher;
+ }
+ catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidAlgorithmParameterException | InvalidKeyException e)
+ {
+ logger.error("could not build cipher", e);
+ throw new IOException("cannot load cipher", e);
+ }
+ }
+
+ private Key retrieveKey(String keyAlias) throws IOException
+ {
+ try
+ {
+ return cache.get(keyAlias);
+ }
+ catch (ExecutionException e)
+ {
+ if (e.getCause() instanceof IOException)
+ throw (IOException)e.getCause();
+ throw new IOException("failed to load key from cache: " + keyAlias, e);
+ }
+ }
+
+ /**
+ * A simple struct to use with the thread local caching of Cipher as we can't get the mode (encrypt/decrypt) nor
+ * key_alias (or key!) from the Cipher itself to use for comparisons
+ */
+ private static class CachedCipher
+ {
+ public final int mode;
+ public final String keyAlias;
+ public final Cipher cipher;
+
+ private CachedCipher(int mode, String keyAlias, Cipher cipher)
+ {
+ this.mode = mode;
+ this.keyAlias = keyAlias;
+ this.cipher = cipher;
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/security/EncryptionContext.java b/src/java/org/apache/cassandra/security/EncryptionContext.java
new file mode 100644
index 0000000..dff6894
--- /dev/null
+++ b/src/java/org/apache/cassandra/security/EncryptionContext.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.security;
+
+import java.io.IOException;
+import java.util.Collections;
+import javax.crypto.Cipher;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Objects;
+
+import org.apache.cassandra.config.TransparentDataEncryptionOptions;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.io.compress.ICompressor;
+import org.apache.cassandra.io.compress.LZ4Compressor;
+
+/**
+ * A (largely) immutable wrapper for the application-wide file-level encryption settings.
+ */
+public class EncryptionContext
+{
+ public static final String ENCRYPTION_CIPHER = "encCipher";
+ public static final String ENCRYPTION_KEY_ALIAS = "encKeyAlias";
+ public static final String ENCRYPTION_IV = "encIV";
+
+ private final TransparentDataEncryptionOptions tdeOptions;
+ private final ICompressor compressor;
+ private final CipherFactory cipherFactory;
+
+ private final int chunkLength;
+
+ public EncryptionContext()
+ {
+ this(new TransparentDataEncryptionOptions());
+ }
+
+ public EncryptionContext(TransparentDataEncryptionOptions tdeOptions)
+ {
+ this(tdeOptions, true);
+ }
+
+ @VisibleForTesting
+ public EncryptionContext(TransparentDataEncryptionOptions tdeOptions, boolean init)
+ {
+ this.tdeOptions = tdeOptions;
+ compressor = LZ4Compressor.create(Collections.<String, String>emptyMap());
+ chunkLength = tdeOptions.chunk_length_kb * 1024;
+
+ // always attempt to load the cipher factory, as we could be in the situation where the user has disabled encryption,
+ // but has existing commitlogs and sstables on disk that are still git addencrypted (and still need to be read)
+ CipherFactory factory = null;
+
+ if (tdeOptions.enabled && init)
+ {
+ try
+ {
+ factory = new CipherFactory(tdeOptions);
+ }
+ catch (Exception e)
+ {
+ throw new ConfigurationException("failed to load key provider for transparent data encryption", e);
+ }
+ }
+
+ cipherFactory = factory;
+ }
+
+ public ICompressor getCompressor()
+ {
+ return compressor;
+ }
+
+ public Cipher getEncryptor() throws IOException
+ {
+ return cipherFactory.getEncryptor(tdeOptions.cipher, tdeOptions.key_alias);
+ }
+
+ public Cipher getDecryptor(byte[] IV) throws IOException
+ {
+ return cipherFactory.getDecryptor(tdeOptions.cipher, tdeOptions.key_alias, IV);
+ }
+
+ public boolean isEnabled()
+ {
+ return tdeOptions.enabled;
+ }
+
+ public int getChunkLength()
+ {
+ return chunkLength;
+ }
+
+ public TransparentDataEncryptionOptions getTransparentDataEncryptionOptions()
+ {
+ return tdeOptions;
+ }
+
+ public boolean equals(Object o)
+ {
+ return o instanceof EncryptionContext && equals((EncryptionContext) o);
+ }
+
+ public boolean equals(EncryptionContext other)
+ {
+ return Objects.equal(tdeOptions, other.tdeOptions) && Objects.equal(compressor, other.compressor);
+ }
+}
diff --git a/src/java/org/apache/cassandra/security/JKSKeyProvider.java b/src/java/org/apache/cassandra/security/JKSKeyProvider.java
new file mode 100644
index 0000000..af9d515
--- /dev/null
+++ b/src/java/org/apache/cassandra/security/JKSKeyProvider.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.security;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.TransparentDataEncryptionOptions;
+
+/**
+ * A {@code KeyProvider} that retrieves keys from a java keystore.
+ */
+public class JKSKeyProvider implements KeyProvider
+{
+ private final Logger logger = LoggerFactory.getLogger(JKSKeyProvider.class);
+ static final String PROP_KEYSTORE = "keystore";
+ static final String PROP_KEYSTORE_PW = "keystore_password";
+ static final String PROP_KEYSTORE_TYPE = "store_type";
+ static final String PROP_KEY_PW = "key_password";
+
+ private final KeyStore store;
+ private final boolean isJceks;
+ private final TransparentDataEncryptionOptions options;
+
+ public JKSKeyProvider(TransparentDataEncryptionOptions options)
+ {
+ this.options = options;
+ logger.info("initializing keystore from file {}", options.get(PROP_KEYSTORE));
+ try (FileInputStream inputStream = new FileInputStream(options.get(PROP_KEYSTORE)))
+ {
+ store = KeyStore.getInstance(options.get(PROP_KEYSTORE_TYPE));
+ store.load(inputStream, options.get(PROP_KEYSTORE_PW).toCharArray());
+ isJceks = store.getType().equalsIgnoreCase("jceks");
+ }
+ catch (Exception e)
+ {
+ throw new RuntimeException("couldn't load keystore", e);
+ }
+ }
+
+ public Key getSecretKey(String keyAlias) throws IOException
+ {
+ // there's a lovely behavior with jceks files that all aliases are lower-cased
+ if (isJceks)
+ keyAlias = keyAlias.toLowerCase();
+
+ Key key;
+ try
+ {
+ String password = options.get(PROP_KEY_PW);
+ if (password == null || password.isEmpty())
+ password = options.get(PROP_KEYSTORE_PW);
+ key = store.getKey(keyAlias, password.toCharArray());
+ }
+ catch (Exception e)
+ {
+ throw new IOException("unable to load key from keystore");
+ }
+ if (key == null)
+ throw new IOException(String.format("key %s was not found in keystore", keyAlias));
+ return key;
+ }
+}
diff --git a/src/java/org/apache/cassandra/security/KeyProvider.java b/src/java/org/apache/cassandra/security/KeyProvider.java
new file mode 100644
index 0000000..f380aed
--- /dev/null
+++ b/src/java/org/apache/cassandra/security/KeyProvider.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.security;
+
+import java.io.IOException;
+import java.security.Key;
+
+/**
+ * Customizable key retrieval mechanism. Implementations should expect that retrieved keys will be cached.
+ * Further, each key will be requested non-concurrently (that is, no stampeding herds for the same key), although
+ * unique keys may be requested concurrently (unless you mark {@code getSecretKey} synchronized).
+ *
+ * Implementations must provide a constructor that accepts {@code TransparentDataEncryptionOptions} as the sole parameter.
+ */
+public interface KeyProvider
+{
+ Key getSecretKey(String alias) throws IOException;
+}
diff --git a/src/java/org/apache/cassandra/serializers/AsciiSerializer.java b/src/java/org/apache/cassandra/serializers/AsciiSerializer.java
index b013b23..e265cb2 100644
--- a/src/java/org/apache/cassandra/serializers/AsciiSerializer.java
+++ b/src/java/org/apache/cassandra/serializers/AsciiSerializer.java
@@ -35,7 +35,7 @@
for (int i = bytes.position(); i < bytes.limit(); i++)
{
byte b = bytes.get(i);
- if (b < 0 || b > 127)
+ if (b < 0)
throw new MarshalException("Invalid byte for ascii: " + Byte.toString(b));
}
}
diff --git a/src/java/org/apache/cassandra/serializers/ByteSerializer.java b/src/java/org/apache/cassandra/serializers/ByteSerializer.java
index 8c736cb..9d34fbc 100644
--- a/src/java/org/apache/cassandra/serializers/ByteSerializer.java
+++ b/src/java/org/apache/cassandra/serializers/ByteSerializer.java
@@ -28,7 +28,7 @@
public Byte deserialize(ByteBuffer bytes)
{
- return bytes.remaining() == 0 ? null : bytes.get(bytes.position());
+ return bytes == null || bytes.remaining() == 0 ? null : bytes.get(bytes.position());
}
public ByteBuffer serialize(Byte value)
diff --git a/src/java/org/apache/cassandra/serializers/TypeSerializer.java b/src/java/org/apache/cassandra/serializers/TypeSerializer.java
index e7bb830..fff06f8 100644
--- a/src/java/org/apache/cassandra/serializers/TypeSerializer.java
+++ b/src/java/org/apache/cassandra/serializers/TypeSerializer.java
@@ -23,11 +23,17 @@
public interface TypeSerializer<T>
{
public ByteBuffer serialize(T value);
+
+ /*
+ * Does not modify the position or limit of the buffer even temporarily.
+ */
public T deserialize(ByteBuffer bytes);
/*
* Validate that the byte array is a valid sequence for the type this represents.
* This guarantees deserialize() can be called without errors.
+ *
+ * Does not modify the position or limit of the buffer even temporarily
*/
public void validate(ByteBuffer bytes) throws MarshalException;
diff --git a/src/java/org/apache/cassandra/serializers/UTF8Serializer.java b/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
index be5d2d6..bb708e0 100644
--- a/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
+++ b/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
@@ -93,10 +93,8 @@
if (b == (byte)0xf0)
// 0xf0, 0x90-0xbf, 0x80-0xbf, 0x80-0xbf
state = State.FOUR_90bf;
- else if (b == (byte)0xf4)
- // 0xf4, 0x80-0xbf, 0x80-0xbf, 0x80-0xbf
- state = State.FOUR_80bf_3;
else
+ // 0xf4, 0x80-0xbf, 0x80-0xbf, 0x80-0xbf
// 0xf1-0xf3, 0x80-0xbf, 0x80-0xbf, 0x80-0xbf
state = State.FOUR_80bf_3;
break;
diff --git a/src/java/org/apache/cassandra/serializers/UUIDSerializer.java b/src/java/org/apache/cassandra/serializers/UUIDSerializer.java
index f8e2582..4501f34 100644
--- a/src/java/org/apache/cassandra/serializers/UUIDSerializer.java
+++ b/src/java/org/apache/cassandra/serializers/UUIDSerializer.java
@@ -34,7 +34,7 @@
public ByteBuffer serialize(UUID value)
{
- return value == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : ByteBuffer.wrap(UUIDGen.decompose(value));
+ return value == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : UUIDGen.toByteBuffer(value);
}
public void validate(ByteBuffer bytes) throws MarshalException
diff --git a/src/java/org/apache/cassandra/service/CacheService.java b/src/java/org/apache/cassandra/service/CacheService.java
index c51a5d1..03d7cd4 100644
--- a/src/java/org/apache/cassandra/service/CacheService.java
+++ b/src/java/org/apache/cassandra/service/CacheService.java
@@ -64,7 +64,7 @@
public static final String MBEAN_NAME = "org.apache.cassandra.db:type=Caches";
- public static enum CacheType
+ public enum CacheType
{
KEY_CACHE("KeyCache"),
ROW_CACHE("RowCache"),
@@ -72,7 +72,7 @@
private final String name;
- private CacheType(String typeName)
+ CacheType(String typeName)
{
name = typeName;
}
@@ -430,9 +430,9 @@
//Keyspace and CF name are deserialized by AutoSaving cache and used to fetch the CFS provided as a
//parameter so they aren't deserialized here, even though they are serialized by this serializer
final ByteBuffer buffer = ByteBufferUtil.readWithLength(in);
- final int rowsToCache = cfs.metadata.params.caching.rowsPerPartitionToCache();
if (cfs == null || !cfs.isRowCacheEnabled())
return null;
+ final int rowsToCache = cfs.metadata.params.caching.rowsPerPartitionToCache();
assert(!cfs.isIndex());//Shouldn't have row cache entries for indexes
return StageManager.getStage(Stage.READ).submit(new Callable<Pair<RowCacheKey, IRowCacheEntry>>()
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index f9183a7..183abf8 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -326,20 +326,6 @@
SystemKeyspace.finishStartup();
- // start server internals
- StorageService.instance.registerDaemon(this);
- try
- {
- StorageService.instance.initServer();
- }
- catch (ConfigurationException e)
- {
- System.err.println(e.getMessage() + "\nFatal configuration error; unable to start server. See log for stacktrace.");
- exitOrFail(1, "Fatal configuration error", e);
- }
-
- Mx4jTool.maybeLoad();
-
// Metrics
String metricsReporterConfigFile = System.getProperty("cassandra.metricsReporterConfigFile");
if (metricsReporterConfigFile != null)
@@ -356,6 +342,20 @@
}
}
+ // start server internals
+ StorageService.instance.registerDaemon(this);
+ try
+ {
+ StorageService.instance.initServer();
+ }
+ catch (ConfigurationException e)
+ {
+ System.err.println(e.getMessage() + "\nFatal configuration error; unable to start server. See log for stacktrace.");
+ exitOrFail(1, "Fatal configuration error", e);
+ }
+
+ Mx4jTool.maybeLoad();
+
if (!FBUtilities.getBroadcastAddress().equals(InetAddress.getLoopbackAddress()))
waitForGossipToSettle();
@@ -619,7 +619,8 @@
stop();
destroy();
// completely shut down cassandra
- if(!runManaged) {
+ if(!runManaged)
+ {
System.exit(0);
}
}
@@ -679,21 +680,24 @@
instance.activate();
}
- private void exitOrFail(int code, String message) {
+ private void exitOrFail(int code, String message)
+ {
exitOrFail(code, message, null);
}
- private void exitOrFail(int code, String message, Throwable cause) {
- if(runManaged) {
- RuntimeException t = cause!=null ? new RuntimeException(message, cause) : new RuntimeException(message);
- throw t;
- }
- else {
- logger.error(message, cause);
- System.exit(code);
- }
-
+ private void exitOrFail(int code, String message, Throwable cause)
+ {
+ if (runManaged)
+ {
+ RuntimeException t = cause!=null ? new RuntimeException(message, cause) : new RuntimeException(message);
+ throw t;
}
+ else
+ {
+ logger.error(message, cause);
+ System.exit(code);
+ }
+ }
static class NativeAccess implements NativeAccessMBean
{
diff --git a/src/java/org/apache/cassandra/service/ClientState.java b/src/java/org/apache/cassandra/service/ClientState.java
index f2ed984..78bcf8a 100644
--- a/src/java/org/apache/cassandra/service/ClientState.java
+++ b/src/java/org/apache/cassandra/service/ClientState.java
@@ -29,6 +29,7 @@
import org.apache.cassandra.auth.*;
import org.apache.cassandra.config.Config;
+import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.cql3.QueryHandler;
@@ -249,6 +250,12 @@
hasAccess(keyspace, perm, DataResource.table(keyspace, columnFamily));
}
+ public void hasColumnFamilyAccess(CFMetaData cfm, Permission perm)
+ throws UnauthorizedException, InvalidRequestException
+ {
+ hasAccess(cfm.ksName, perm, cfm.resource);
+ }
+
private void hasAccess(String keyspace, Permission perm, DataResource resource)
throws UnauthorizedException, InvalidRequestException
{
@@ -267,7 +274,7 @@
public void ensureHasPermission(Permission perm, IResource resource) throws UnauthorizedException
{
- if (DatabaseDescriptor.getAuthorizer() instanceof AllowAllAuthorizer)
+ if (!DatabaseDescriptor.getAuthorizer().requireAuthorization())
return;
// Access to built in functions is unrestricted
@@ -283,7 +290,7 @@
public void ensureHasPermission(Permission permission, Function function)
{
// Save creating a FunctionResource is we don't need to
- if (DatabaseDescriptor.getAuthorizer() instanceof AllowAllAuthorizer)
+ if (!DatabaseDescriptor.getAuthorizer().requireAuthorization())
return;
// built in functions are always available to all
diff --git a/src/java/org/apache/cassandra/service/ReadCallback.java b/src/java/org/apache/cassandra/service/ReadCallback.java
index 8747004..47eacdf 100644
--- a/src/java/org/apache/cassandra/service/ReadCallback.java
+++ b/src/java/org/apache/cassandra/service/ReadCallback.java
@@ -192,7 +192,8 @@
result,
Collections.<String, byte[]>emptyMap(),
MessagingService.Verb.INTERNAL_RESPONSE,
- MessagingService.current_version);
+ MessagingService.current_version,
+ MessageIn.createTimestamp());
response(message);
}
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java b/src/java/org/apache/cassandra/service/StartupChecks.java
index ad6a104..e903721 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -36,7 +36,6 @@
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.exceptions.StartupException;
import org.apache.cassandra.io.sstable.Descriptor;
-import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.utils.*;
/**
@@ -231,11 +230,6 @@
public void execute() throws StartupException
{
final Set<String> invalid = new HashSet<>();
- final Set<String> nonSSTablePaths = new HashSet<>();
- nonSSTablePaths.add(FileUtils.getCanonicalPath(DatabaseDescriptor.getCommitLogLocation()));
- nonSSTablePaths.add(FileUtils.getCanonicalPath(DatabaseDescriptor.getSavedCachesLocation()));
- nonSSTablePaths.add(FileUtils.getCanonicalPath(DatabaseDescriptor.getHintsDirectory()));
-
FileVisitor<Path> sstableVisitor = new SimpleFileVisitor<Path>()
{
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException
@@ -259,8 +253,7 @@
{
String name = dir.getFileName().toString();
return (name.equals(Directories.SNAPSHOT_SUBDIR)
- || name.equals(Directories.BACKUPS_SUBDIR)
- || nonSSTablePaths.contains(dir.toFile().getCanonicalPath()))
+ || name.equals(Directories.BACKUPS_SUBDIR))
? FileVisitResult.SKIP_SUBTREE
: FileVisitResult.CONTINUE;
}
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java b/src/java/org/apache/cassandra/service/StorageProxy.java
index 8fa2082..b84128e 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -47,6 +47,7 @@
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.DataLimits;
import org.apache.cassandra.db.filter.TombstoneOverwhelmingException;
+import org.apache.cassandra.db.monitoring.ConstructionTime;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.db.rows.RowIterator;
import org.apache.cassandra.db.view.ViewUtils;
@@ -512,6 +513,8 @@
MessageOut<Commit> message = new MessageOut<Commit>(MessagingService.Verb.PAXOS_COMMIT, proposal, Commit.serializer);
for (InetAddress destination : Iterables.concat(naturalEndpoints, pendingEndpoints))
{
+ checkHintOverload(destination);
+
if (FailureDetector.instance.isAlive(destination))
{
if (shouldBlock)
@@ -1453,7 +1456,7 @@
{
assert mutation instanceof CounterMutation;
- Mutation result = ((CounterMutation) mutation).apply();
+ Mutation result = ((CounterMutation) mutation).applyCounterMutation();
responseHandler.response(null);
Set<InetAddress> remotes = Sets.difference(ImmutableSet.copyOf(targets),
@@ -1777,10 +1780,25 @@
{
try
{
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); UnfilteredPartitionIterator iterator = command.executeLocally(orderGroup))
+ command.setMonitoringTime(new ConstructionTime(constructionTime), timeout);
+
+ ReadResponse response;
+ try (ReadExecutionController executionController = command.executionController();
+ UnfilteredPartitionIterator iterator = command.executeLocally(executionController))
{
- handler.response(command.createResponse(iterator));
+ response = command.createResponse(iterator);
}
+
+ if (command.complete())
+ {
+ handler.response(response);
+ }
+ else
+ {
+ MessagingService.instance().incrementDroppedMessages(verb, System.currentTimeMillis() - constructionTime);
+ handler.onFailure(FBUtilities.getBroadcastAddress());
+ }
+
MessagingService.instance().addLatency(FBUtilities.getBroadcastAddress(), TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start));
}
catch (Throwable t)
@@ -2415,20 +2433,23 @@
*/
private static abstract class DroppableRunnable implements Runnable
{
- private final long constructionTime = System.nanoTime();
- private final MessagingService.Verb verb;
+ final long constructionTime;
+ final MessagingService.Verb verb;
+ final long timeout;
public DroppableRunnable(MessagingService.Verb verb)
{
+ this.constructionTime = System.currentTimeMillis();
this.verb = verb;
+ this.timeout = DatabaseDescriptor.getTimeout(verb);
}
public final void run()
{
-
- if (TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - constructionTime) > DatabaseDescriptor.getTimeout(verb))
+ long timeTaken = System.currentTimeMillis() - constructionTime;
+ if (timeTaken > timeout)
{
- MessagingService.instance().incrementDroppedMessages(verb);
+ MessagingService.instance().incrementDroppedMessages(verb, timeTaken);
return;
}
try
@@ -2441,11 +2462,6 @@
}
}
- protected MessagingService.Verb verb()
- {
- return verb;
- }
-
abstract protected void runMayThrow() throws Exception;
}
@@ -2460,10 +2476,12 @@
public final void run()
{
final MessagingService.Verb verb = verb();
- if (System.currentTimeMillis() > constructionTime + DatabaseDescriptor.getTimeout(verb))
+ long mutationTimeout = DatabaseDescriptor.getTimeout(verb);
+ long timeTaken = System.currentTimeMillis() - constructionTime;
+ if (timeTaken > mutationTimeout)
{
- if (MessagingService.DROPPABLE_VERBS.contains(verb()))
- MessagingService.instance().incrementDroppedMessages(verb);
+ if (MessagingService.DROPPABLE_VERBS.contains(verb))
+ MessagingService.instance().incrementDroppedMessages(verb, timeTaken);
HintRunnable runnable = new HintRunnable(Collections.singleton(FBUtilities.getBroadcastAddress()))
{
protected void runMayThrow() throws Exception
diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java
index 288a82e..580b75b 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -98,9 +98,11 @@
import org.apache.cassandra.db.CounterMutationVerbHandler;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.DefinitionsUpdateVerbHandler;
+import org.apache.cassandra.db.Directories;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.MigrationRequestVerbHandler;
import org.apache.cassandra.db.MutationVerbHandler;
+import org.apache.cassandra.db.PartitionPosition;
import org.apache.cassandra.db.RangeSliceVerbHandler;
import org.apache.cassandra.db.ReadCommandVerbHandler;
import org.apache.cassandra.db.ReadRepairVerbHandler;
@@ -117,6 +119,7 @@
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.RangeStreamer;
import org.apache.cassandra.dht.RingPosition;
+import org.apache.cassandra.dht.Splitter;
import org.apache.cassandra.dht.StreamStateStore;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.dht.Token.TokenFactory;
@@ -193,7 +196,7 @@
public class StorageService extends NotificationBroadcasterSupport implements IEndpointStateChangeSubscriber, StorageServiceMBean
{
private static final Logger logger = LoggerFactory.getLogger(StorageService.class);
-
+
public static final int RING_DELAY = getRingDelay(); // delay after which we assume ring has stablized
private final JMXProgressSupport progressSupport = new JMXProgressSupport(this);
@@ -1139,7 +1142,8 @@
"Rebuild",
!replacing && useStrictConsistency,
DatabaseDescriptor.getEndpointSnitch(),
- streamStateStore);
+ streamStateStore,
+ false);
streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance));
if (sourceDc != null)
streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc));
@@ -2670,6 +2674,18 @@
}
}
+ public int relocateSSTables(String keyspaceName, String ... columnFamilies) throws IOException, ExecutionException, InterruptedException
+ {
+ CompactionManager.AllSSTableOpStatus status = CompactionManager.AllSSTableOpStatus.SUCCESSFUL;
+ for (ColumnFamilyStore cfs : getValidColumnFamilies(false, false, keyspaceName, columnFamilies))
+ {
+ CompactionManager.AllSSTableOpStatus oneStatus = cfs.relocateSSTables();
+ if (oneStatus != CompactionManager.AllSSTableOpStatus.SUCCESSFUL)
+ status = oneStatus;
+ }
+ return status.statusCode;
+ }
+
/**
* Takes the snapshot for the given keyspaces. A snapshot name must be specified.
*
@@ -2752,7 +2768,7 @@
Map<Keyspace, List<String>> keyspaceColumnfamily = new HashMap<Keyspace, List<String>>();
for (String table : tableList)
{
- String splittedString[] = table.split("\\.");
+ String splittedString[] = StringUtils.split(table, '.');
if (splittedString.length == 2)
{
String keyspaceName = splittedString[0];
@@ -4504,4 +4520,61 @@
logger.info(String.format("Updated hinted_handoff_throttle_in_kb to %d", throttleInKB));
}
+ public static List<PartitionPosition> getDiskBoundaries(ColumnFamilyStore cfs, Directories.DataDirectory[] directories)
+ {
+ if (!cfs.getPartitioner().splitter().isPresent())
+ return null;
+
+ Collection<Range<Token>> lr;
+
+ if (StorageService.instance.isBootstrapMode())
+ {
+ lr = StorageService.instance.getTokenMetadata().getPendingRanges(cfs.keyspace.getName(), FBUtilities.getBroadcastAddress());
+ }
+ else
+ {
+ // Reason we use use the future settled TMD is that if we decommission a node, we want to stream
+ // from that node to the correct location on disk, if we didn't, we would put new files in the wrong places.
+ // We do this to minimize the amount of data we need to move in rebalancedisks once everything settled
+ TokenMetadata tmd = StorageService.instance.getTokenMetadata().cloneAfterAllSettled();
+ lr = cfs.keyspace.getReplicationStrategy().getAddressRanges(tmd).get(FBUtilities.getBroadcastAddress());
+ }
+
+ if (lr == null || lr.isEmpty())
+ return null;
+ List<Range<Token>> localRanges = Range.sort(lr);
+
+ return getDiskBoundaries(localRanges, cfs.getPartitioner(), directories);
+ }
+
+ public static List<PartitionPosition> getDiskBoundaries(ColumnFamilyStore cfs)
+ {
+ return getDiskBoundaries(cfs, cfs.getDirectories().getWriteableLocations());
+ }
+
+ /**
+ * Returns a list of disk boundaries, the result will differ depending on whether vnodes are enabled or not.
+ *
+ * What is returned are upper bounds for the disks, meaning everything from partitioner.minToken up to
+ * getDiskBoundaries(..).get(0) should be on the first disk, everything between 0 to 1 should be on the second disk
+ * etc.
+ *
+ * The final entry in the returned list will always be the partitioner maximum tokens upper key bound
+ *
+ * @param localRanges
+ * @param partitioner
+ * @param dataDirectories
+ * @return
+ */
+ public static List<PartitionPosition> getDiskBoundaries(List<Range<Token>> localRanges, IPartitioner partitioner, Directories.DataDirectory[] dataDirectories)
+ {
+ assert partitioner.splitter().isPresent();
+ Splitter splitter = partitioner.splitter().get();
+ List<Token> boundaries = splitter.splitOwnedRanges(dataDirectories.length, localRanges, DatabaseDescriptor.getNumTokens() > 1);
+ List<PartitionPosition> diskBoundaries = new ArrayList<>();
+ for (int i = 0; i < boundaries.size() - 1; i++)
+ diskBoundaries.add(boundaries.get(i).maxKeyBound());
+ diskBoundaries.add(partitioner.getMaximumToken().maxKeyBound());
+ return diskBoundaries;
+ }
}
diff --git a/src/java/org/apache/cassandra/service/StorageServiceMBean.java b/src/java/org/apache/cassandra/service/StorageServiceMBean.java
index 049b4f9..4e2f632 100644
--- a/src/java/org/apache/cassandra/service/StorageServiceMBean.java
+++ b/src/java/org/apache/cassandra/service/StorageServiceMBean.java
@@ -248,6 +248,7 @@
*/
public void forceKeyspaceCompaction(boolean splitOutput, String keyspaceName, String... tableNames) throws IOException, ExecutionException, InterruptedException;
+ public int relocateSSTables(String keyspace, String ... cfnames) throws IOException, ExecutionException, InterruptedException;
/**
* Trigger a cleanup of keys on a single keyspace
*/
@@ -384,7 +385,7 @@
* If classQualifer is not empty but level is empty/null, it will set the level to null for the defined classQualifer<br>
* If level cannot be parsed, then the level will be defaulted to DEBUG<br>
* <br>
- * The logback configuration should have < jmxConfigurator /> set
+ * The logback configuration should have {@code < jmxConfigurator />} set
*
* @param classQualifier The logger's classQualifer
* @param level The log level
@@ -423,7 +424,7 @@
/**
* given a list of tokens (representing the nodes in the cluster), returns
- * a mapping from "token -> %age of cluster owned by that token"
+ * a mapping from {@code "token -> %age of cluster owned by that token"}
*/
public Map<InetAddress, Float> getOwnership();
@@ -518,7 +519,7 @@
* Load new SSTables to the given keyspace/table
*
* @param ksName The parent keyspace name
- * @param cfName The ColumnFamily name where SSTables belong
+ * @param tableName The ColumnFamily name where SSTables belong
*/
public void loadNewSSTables(String ksName, String tableName);
diff --git a/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java b/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
index 2599b8d..386d7ae 100644
--- a/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
+++ b/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
@@ -52,9 +52,9 @@
this.remainingInPartition = limits.perPartitionCount();
}
- public ReadOrderGroup startOrderGroup()
+ public ReadExecutionController executionController()
{
- return command.startOrderGroup();
+ return command.executionController();
}
public PartitionIterator fetchPage(int pageSize, ConsistencyLevel consistency, ClientState clientState) throws RequestValidationException, RequestExecutionException
@@ -67,14 +67,14 @@
return Transformation.apply(nextPageReadCommand(pageSize).execute(consistency, clientState), pager);
}
- public PartitionIterator fetchPageInternal(int pageSize, ReadOrderGroup orderGroup) throws RequestValidationException, RequestExecutionException
+ public PartitionIterator fetchPageInternal(int pageSize, ReadExecutionController executionController) throws RequestValidationException, RequestExecutionException
{
if (isExhausted())
return EmptyIterators.partition();
pageSize = Math.min(pageSize, remaining);
Pager pager = new Pager(limits.forPaging(pageSize), command.nowInSec());
- return Transformation.apply(nextPageReadCommand(pageSize).executeInternal(orderGroup), pager);
+ return Transformation.apply(nextPageReadCommand(pageSize).executeInternal(executionController), pager);
}
private class Pager extends Transformation<RowIterator>
diff --git a/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java b/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
index 8caa14d..922df2e 100644
--- a/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
+++ b/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
@@ -106,14 +106,14 @@
return true;
}
- public ReadOrderGroup startOrderGroup()
+ public ReadExecutionController executionController()
{
// Note that for all pagers, the only difference is the partition key to which it applies, so in practice we
// can use any of the sub-pager ReadOrderGroup group to protect the whole pager
for (int i = current; i < pagers.length; i++)
{
if (pagers[i] != null)
- return pagers[i].startOrderGroup();
+ return pagers[i].executionController();
}
throw new AssertionError("Shouldn't be called on an exhausted pager");
}
@@ -129,10 +129,10 @@
}
@SuppressWarnings("resource") // iter closed via countingIter
- public PartitionIterator fetchPageInternal(int pageSize, ReadOrderGroup orderGroup) throws RequestValidationException, RequestExecutionException
+ public PartitionIterator fetchPageInternal(int pageSize, ReadExecutionController executionController) throws RequestValidationException, RequestExecutionException
{
int toQuery = Math.min(remaining, pageSize);
- PagersIterator iter = new PagersIterator(toQuery, null, null, orderGroup);
+ PagersIterator iter = new PagersIterator(toQuery, null, null, executionController);
DataLimits.Counter counter = limit.forPaging(toQuery).newCounter(nowInSec, true);
iter.setCounter(counter);
return counter.applyTo(iter);
@@ -149,14 +149,14 @@
private final ClientState clientState;
// For internal queries
- private final ReadOrderGroup orderGroup;
+ private final ReadExecutionController executionController;
- public PagersIterator(int pageSize, ConsistencyLevel consistency, ClientState clientState, ReadOrderGroup orderGroup)
+ public PagersIterator(int pageSize, ConsistencyLevel consistency, ClientState clientState, ReadExecutionController executionController)
{
this.pageSize = pageSize;
this.consistency = consistency;
this.clientState = clientState;
- this.orderGroup = orderGroup;
+ this.executionController = executionController;
}
public void setCounter(DataLimits.Counter counter)
@@ -177,7 +177,7 @@
int toQuery = pageSize - counter.counted();
result = consistency == null
- ? pagers[current].fetchPageInternal(toQuery, orderGroup)
+ ? pagers[current].fetchPageInternal(toQuery, executionController)
: pagers[current].fetchPage(toQuery, consistency, clientState);
}
return result.next();
diff --git a/src/java/org/apache/cassandra/service/pager/QueryPager.java b/src/java/org/apache/cassandra/service/pager/QueryPager.java
index cdf2b97..e2d7f5e 100644
--- a/src/java/org/apache/cassandra/service/pager/QueryPager.java
+++ b/src/java/org/apache/cassandra/service/pager/QueryPager.java
@@ -18,8 +18,8 @@
package org.apache.cassandra.service.pager;
import org.apache.cassandra.db.ConsistencyLevel;
+import org.apache.cassandra.db.ReadExecutionController;
import org.apache.cassandra.db.EmptyIterators;
-import org.apache.cassandra.db.ReadOrderGroup;
import org.apache.cassandra.db.partitions.PartitionIterator;
import org.apache.cassandra.exceptions.RequestExecutionException;
import org.apache.cassandra.exceptions.RequestValidationException;
@@ -46,11 +46,11 @@
*/
public interface QueryPager
{
- public static final QueryPager EMPTY = new QueryPager()
+ QueryPager EMPTY = new QueryPager()
{
- public ReadOrderGroup startOrderGroup()
+ public ReadExecutionController executionController()
{
- return ReadOrderGroup.emptyGroup();
+ return ReadExecutionController.empty();
}
public PartitionIterator fetchPage(int pageSize, ConsistencyLevel consistency, ClientState clientState) throws RequestValidationException, RequestExecutionException
@@ -58,7 +58,7 @@
return EmptyIterators.partition();
}
- public PartitionIterator fetchPageInternal(int pageSize, ReadOrderGroup orderGroup) throws RequestValidationException, RequestExecutionException
+ public PartitionIterator fetchPageInternal(int pageSize, ReadExecutionController executionController) throws RequestValidationException, RequestExecutionException
{
return EmptyIterators.partition();
}
@@ -99,16 +99,16 @@
*
* @return a newly started order group for this {@code QueryPager}.
*/
- public ReadOrderGroup startOrderGroup();
+ public ReadExecutionController executionController();
/**
* Fetches the next page internally (in other, this does a local query).
*
* @param pageSize the maximum number of elements to return in the next page.
- * @param orderGroup the {@code ReadOrderGroup} protecting the read.
+ * @param executionController the {@code ReadExecutionController} protecting the read.
* @return the page of result.
*/
- public PartitionIterator fetchPageInternal(int pageSize, ReadOrderGroup orderGroup) throws RequestValidationException, RequestExecutionException;
+ public PartitionIterator fetchPageInternal(int pageSize, ReadExecutionController executionController) throws RequestValidationException, RequestExecutionException;
/**
* Whether or not this pager is exhausted, i.e. whether or not a call to
diff --git a/src/java/org/apache/cassandra/streaming/StreamCoordinator.java b/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
index 603366d..2cb75f7 100644
--- a/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
+++ b/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
@@ -41,19 +41,23 @@
// streaming is handled directly by the ConnectionHandler's incoming and outgoing threads.
private static final DebuggableThreadPoolExecutor streamExecutor = DebuggableThreadPoolExecutor.createWithFixedPoolSize("StreamConnectionEstablisher",
FBUtilities.getAvailableProcessors());
+ private final boolean connectSequentially;
private Map<InetAddress, HostStreamingData> peerSessions = new HashMap<>();
private final int connectionsPerHost;
private StreamConnectionFactory factory;
private final boolean keepSSTableLevel;
private final boolean isIncremental;
+ private Iterator<StreamSession> sessionsToConnect = null;
- public StreamCoordinator(int connectionsPerHost, boolean keepSSTableLevel, boolean isIncremental, StreamConnectionFactory factory)
+ public StreamCoordinator(int connectionsPerHost, boolean keepSSTableLevel, boolean isIncremental,
+ StreamConnectionFactory factory, boolean connectSequentially)
{
this.connectionsPerHost = connectionsPerHost;
this.factory = factory;
this.keepSSTableLevel = keepSSTableLevel;
this.isIncremental = isIncremental;
+ this.connectSequentially = connectSequentially;
}
public void setConnectionFactory(StreamConnectionFactory factory)
@@ -89,12 +93,59 @@
return connectionsPerHost == 0;
}
- public void connectAllStreamSessions()
+ public void connect(StreamResultFuture future)
+ {
+ if (this.connectSequentially)
+ connectSequentially(future);
+ else
+ connectAllStreamSessions();
+ }
+
+ private void connectAllStreamSessions()
{
for (HostStreamingData data : peerSessions.values())
data.connectAllStreamSessions();
}
+ private void connectSequentially(StreamResultFuture future)
+ {
+ sessionsToConnect = getAllStreamSessions().iterator();
+ future.addEventListener(new StreamEventHandler()
+ {
+ public void handleStreamEvent(StreamEvent event)
+ {
+ if (event.eventType == StreamEvent.Type.STREAM_PREPARED || event.eventType == StreamEvent.Type.STREAM_COMPLETE)
+ connectNext();
+ }
+
+ public void onSuccess(StreamState result)
+ {
+
+ }
+
+ public void onFailure(Throwable t)
+ {
+
+ }
+ });
+ connectNext();
+ }
+
+ private void connectNext()
+ {
+ if (sessionsToConnect == null)
+ return;
+
+ if (sessionsToConnect.hasNext())
+ {
+ StreamSession next = sessionsToConnect.next();
+ logger.debug("Connecting next session {} with {}.", next.planId(), next.peer.getHostAddress());
+ streamExecutor.execute(new StreamSessionConnector(next));
+ }
+ else
+ logger.debug("Finished connecting all sessions");
+ }
+
public synchronized Set<InetAddress> getPeers()
{
return new HashSet<>(peerSessions.keySet());
diff --git a/src/java/org/apache/cassandra/streaming/StreamPlan.java b/src/java/org/apache/cassandra/streaming/StreamPlan.java
index 0d963ed..e9d43cb 100644
--- a/src/java/org/apache/cassandra/streaming/StreamPlan.java
+++ b/src/java/org/apache/cassandra/streaming/StreamPlan.java
@@ -32,6 +32,7 @@
*/
public class StreamPlan
{
+ public static final String[] EMPTY_COLUMN_FAMILIES = new String[0];
private final UUID planId = UUIDGen.getTimeUUID();
private final String description;
private final List<StreamEventHandler> handlers = new ArrayList<>();
@@ -47,19 +48,21 @@
*/
public StreamPlan(String description)
{
- this(description, ActiveRepairService.UNREPAIRED_SSTABLE, 1, false, false);
+ this(description, ActiveRepairService.UNREPAIRED_SSTABLE, 1, false, false, false);
}
- public StreamPlan(String description, boolean keepSSTableLevels)
+ public StreamPlan(String description, boolean keepSSTableLevels, boolean connectSequentially)
{
- this(description, ActiveRepairService.UNREPAIRED_SSTABLE, 1, keepSSTableLevels, false);
+ this(description, ActiveRepairService.UNREPAIRED_SSTABLE, 1, keepSSTableLevels, false, connectSequentially);
}
- public StreamPlan(String description, long repairedAt, int connectionsPerHost, boolean keepSSTableLevels, boolean isIncremental)
+ public StreamPlan(String description, long repairedAt, int connectionsPerHost, boolean keepSSTableLevels,
+ boolean isIncremental, boolean connectSequentially)
{
this.description = description;
this.repairedAt = repairedAt;
- this.coordinator = new StreamCoordinator(connectionsPerHost, keepSSTableLevels, isIncremental, new DefaultConnectionFactory());
+ this.coordinator = new StreamCoordinator(connectionsPerHost, keepSSTableLevels, isIncremental, new DefaultConnectionFactory(),
+ connectSequentially);
}
/**
@@ -73,7 +76,7 @@
*/
public StreamPlan requestRanges(InetAddress from, InetAddress connecting, String keyspace, Collection<Range<Token>> ranges)
{
- return requestRanges(from, connecting, keyspace, ranges, new String[0]);
+ return requestRanges(from, connecting, keyspace, ranges, EMPTY_COLUMN_FAMILIES);
}
/**
@@ -114,7 +117,7 @@
*/
public StreamPlan transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection<Range<Token>> ranges)
{
- return transferRanges(to, connecting, keyspace, ranges, new String[0]);
+ return transferRanges(to, connecting, keyspace, ranges, EMPTY_COLUMN_FAMILIES);
}
/**
diff --git a/src/java/org/apache/cassandra/streaming/StreamReader.java b/src/java/org/apache/cassandra/streaming/StreamReader.java
index 268f974..838664d 100644
--- a/src/java/org/apache/cassandra/streaming/StreamReader.java
+++ b/src/java/org/apache/cassandra/streaming/StreamReader.java
@@ -35,9 +35,9 @@
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.rows.*;
-import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
import org.apache.cassandra.io.sstable.SSTableSimpleIterator;
+import org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter;
import org.apache.cassandra.io.sstable.format.SSTableFormat;
import org.apache.cassandra.io.sstable.format.Version;
import org.apache.cassandra.io.util.DataInputPlus;
@@ -64,8 +64,6 @@
protected final SerializationHeader.Component header;
protected final int fileSeqNum;
- protected Descriptor desc;
-
public StreamReader(FileMessageHeader header, StreamSession session)
{
this.session = session;
@@ -116,7 +114,7 @@
{
writePartition(deserializer, writer);
// TODO move this to BytesReadTracker
- session.progress(desc, ProgressInfo.Direction.IN, in.getBytesRead(), totalSize);
+ session.progress(writer.getFilename(), ProgressInfo.Direction.IN, in.getBytesRead(), totalSize);
}
logger.debug("[Stream #{}] Finished receiving file #{} from {} readBytes = {}, totalSize = {}",
session.planId(), fileSeqNum, session.peer, in.getBytesRead(), totalSize);
@@ -144,10 +142,8 @@
Directories.DataDirectory localDir = cfs.getDirectories().getWriteableLocation(totalSize);
if (localDir == null)
throw new IOException("Insufficient disk space to store " + totalSize + " bytes");
- desc = Descriptor.fromFilename(cfs.getSSTablePath(cfs.getDirectories().getLocationForDisk(localDir), format));
-
- return cfs.createSSTableMultiWriter(desc, estimatedKeys, repairedAt, sstableLevel, header.toHeader(cfs.metadata), session.getTransaction(cfId));
+ return new RangeAwareSSTableWriter(cfs, estimatedKeys, repairedAt, format, sstableLevel, totalSize, session.getTransaction(cfId), header);
}
protected void drain(InputStream dis, long bytesRead) throws IOException
diff --git a/src/java/org/apache/cassandra/streaming/StreamResultFuture.java b/src/java/org/apache/cassandra/streaming/StreamResultFuture.java
index 99adab0..2297c83 100644
--- a/src/java/org/apache/cassandra/streaming/StreamResultFuture.java
+++ b/src/java/org/apache/cassandra/streaming/StreamResultFuture.java
@@ -71,10 +71,12 @@
private StreamResultFuture(UUID planId, String description, boolean keepSSTableLevels, boolean isIncremental)
{
- this(planId, description, new StreamCoordinator(0, keepSSTableLevels, isIncremental, new DefaultConnectionFactory()));
+ this(planId, description, new StreamCoordinator(0, keepSSTableLevels, isIncremental,
+ new DefaultConnectionFactory(), false));
}
- static StreamResultFuture init(UUID planId, String description, Collection<StreamEventHandler> listeners, StreamCoordinator coordinator)
+ static StreamResultFuture init(UUID planId, String description, Collection<StreamEventHandler> listeners,
+ StreamCoordinator coordinator)
{
StreamResultFuture future = createAndRegister(planId, description, coordinator);
if (listeners != null)
@@ -90,7 +92,8 @@
{
session.init(future);
}
- coordinator.connectAllStreamSessions();
+
+ coordinator.connect(future);
return future;
}
diff --git a/src/java/org/apache/cassandra/streaming/StreamSession.java b/src/java/org/apache/cassandra/streaming/StreamSession.java
index 5355c3e..9078acc 100644
--- a/src/java/org/apache/cassandra/streaming/StreamSession.java
+++ b/src/java/org/apache/cassandra/streaming/StreamSession.java
@@ -584,9 +584,9 @@
receivers.get(message.header.cfId).received(message.sstable);
}
- public void progress(Descriptor desc, ProgressInfo.Direction direction, long bytes, long total)
+ public void progress(String filename, ProgressInfo.Direction direction, long bytes, long total)
{
- ProgressInfo progress = new ProgressInfo(peer, index, desc.filenameFor(Component.DATA), direction, bytes, total);
+ ProgressInfo progress = new ProgressInfo(peer, index, filename, direction, bytes, total);
streamResult.handleProgress(progress);
}
diff --git a/src/java/org/apache/cassandra/streaming/StreamWriter.java b/src/java/org/apache/cassandra/streaming/StreamWriter.java
index 721ae1e..0cedade 100644
--- a/src/java/org/apache/cassandra/streaming/StreamWriter.java
+++ b/src/java/org/apache/cassandra/streaming/StreamWriter.java
@@ -108,7 +108,7 @@
long lastBytesRead = write(file, validator, readOffset, length, bytesRead);
bytesRead += lastBytesRead;
progress += (lastBytesRead - readOffset);
- session.progress(sstable.descriptor, ProgressInfo.Direction.OUT, progress, totalSize);
+ session.progress(sstable.descriptor.filenameFor(Component.DATA), ProgressInfo.Direction.OUT, progress, totalSize);
readOffset = 0;
}
diff --git a/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java b/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java
index 5210d5b..5a47787 100644
--- a/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java
+++ b/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java
@@ -26,6 +26,7 @@
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.io.sstable.SSTableMultiWriter;
+import org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -104,7 +105,7 @@
{
writePartition(deserializer, writer);
// when compressed, report total bytes of compressed chunks read since remoteFile.size is the sum of chunks transferred
- session.progress(desc, ProgressInfo.Direction.IN, cis.getTotalCompressedBytesRead(), totalSize);
+ session.progress(writer.getFilename(), ProgressInfo.Direction.IN, cis.getTotalCompressedBytesRead(), totalSize);
}
}
logger.debug("[Stream #{}] Finished receiving file #{} from {} readBytes = {}, totalSize = {}", session.planId(), fileSeqNum,
diff --git a/src/java/org/apache/cassandra/streaming/compress/CompressedStreamWriter.java b/src/java/org/apache/cassandra/streaming/compress/CompressedStreamWriter.java
index f37af29..657da88 100644
--- a/src/java/org/apache/cassandra/streaming/compress/CompressedStreamWriter.java
+++ b/src/java/org/apache/cassandra/streaming/compress/CompressedStreamWriter.java
@@ -29,6 +29,7 @@
import org.slf4j.LoggerFactory;
import org.apache.cassandra.io.compress.CompressionMetadata;
+import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.util.ChannelProxy;
import org.apache.cassandra.io.util.DataOutputStreamPlus;
@@ -88,7 +89,7 @@
long lastWrite = out.applyToChannel((wbc) -> fc.transferTo(section.left + bytesTransferredFinal, toTransfer, wbc));
bytesTransferred += lastWrite;
progress += lastWrite;
- session.progress(sstable.descriptor, ProgressInfo.Direction.OUT, progress, totalSize);
+ session.progress(sstable.descriptor.filenameFor(Component.DATA), ProgressInfo.Direction.OUT, progress, totalSize);
}
}
logger.debug("[Stream #{}] Finished streaming file {} to {}, bytesTransferred = {}, totalSize = {}",
diff --git a/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java b/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java
index bde5310..87c6409 100644
--- a/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java
+++ b/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java
@@ -56,8 +56,8 @@
* Slightly modified version of the Apache Thrift TThreadPoolServer.
* <p>
* This allows passing an executor so you have more control over the actual
- * behaviour of the tasks being run.
- * <p/>
+ * behavior of the tasks being run.
+ * </p>
* Newer version of Thrift should make this obsolete.
*/
public class CustomTThreadPoolServer extends TServer
diff --git a/src/java/org/apache/cassandra/tools/NodeProbe.java b/src/java/org/apache/cassandra/tools/NodeProbe.java
index cdce497..291eeed 100644
--- a/src/java/org/apache/cassandra/tools/NodeProbe.java
+++ b/src/java/org/apache/cassandra/tools/NodeProbe.java
@@ -299,6 +299,11 @@
ssProxy.forceKeyspaceCompaction(splitOutput, keyspaceName, tableNames);
}
+ public void relocateSSTables(String keyspace, String[] cfnames) throws IOException, ExecutionException, InterruptedException
+ {
+ ssProxy.relocateSSTables(keyspace, cfnames);
+ }
+
public void forceKeyspaceFlush(String keyspaceName, String... tableNames) throws IOException, ExecutionException, InterruptedException
{
ssProxy.forceKeyspaceFlush(keyspaceName, tableNames);
@@ -1208,6 +1213,7 @@
CassandraMetricsRegistry.JmxCounterMBean.class);
case "CompletedTasks":
case "PendingTasks":
+ case "PendingTasksByTableName":
return JMX.newMBeanProxy(mbeanServerConn,
new ObjectName("org.apache.cassandra.metrics:type=Compaction,name=" + metricName),
CassandraMetricsRegistry.JmxGaugeMBean.class).getValue();
diff --git a/src/java/org/apache/cassandra/tools/NodeTool.java b/src/java/org/apache/cassandra/tools/NodeTool.java
index 5067a13..8e93450 100644
--- a/src/java/org/apache/cassandra/tools/NodeTool.java
+++ b/src/java/org/apache/cassandra/tools/NodeTool.java
@@ -136,7 +136,8 @@
DisableHintsForDC.class,
EnableHintsForDC.class,
FailureDetectorInfo.class,
- RefreshSizeEstimates.class
+ RefreshSizeEstimates.class,
+ RelocateSSTables.class
);
Cli.CliBuilder<Runnable> builder = Cli.builder("nodetool");
diff --git a/src/java/org/apache/cassandra/tools/SSTableMetadataViewer.java b/src/java/org/apache/cassandra/tools/SSTableMetadataViewer.java
index 3728d34..5b47004 100644
--- a/src/java/org/apache/cassandra/tools/SSTableMetadataViewer.java
+++ b/src/java/org/apache/cassandra/tools/SSTableMetadataViewer.java
@@ -23,6 +23,8 @@
import java.util.EnumSet;
import java.util.Map;
+import org.apache.cassandra.io.compress.CompressionMetadata;
+import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.metadata.*;
@@ -54,6 +56,10 @@
ValidationMetadata validation = (ValidationMetadata) metadata.get(MetadataType.VALIDATION);
StatsMetadata stats = (StatsMetadata) metadata.get(MetadataType.STATS);
CompactionMetadata compaction = (CompactionMetadata) metadata.get(MetadataType.COMPACTION);
+ CompressionMetadata compression = null;
+ File compressionFile = new File(descriptor.filenameFor(Component.COMPRESSION_INFO));
+ if (compressionFile.exists())
+ compression = CompressionMetadata.create(fname);
out.printf("SSTable: %s%n", descriptor);
if (validation != null)
@@ -66,7 +72,9 @@
out.printf("Minimum timestamp: %s%n", stats.minTimestamp);
out.printf("Maximum timestamp: %s%n", stats.maxTimestamp);
out.printf("SSTable max local deletion time: %s%n", stats.maxLocalDeletionTime);
- out.printf("Compression ratio: %s%n", stats.compressionRatio);
+ out.printf("Compressor: %s%n", compression != null ? compression.compressor().getClass().getName() : "-");
+ if (compression != null)
+ out.printf("Compression ratio: %s%n", stats.compressionRatio);
out.printf("Estimated droppable tombstones: %s%n", stats.getEstimatedDroppableTombstoneRatio((int) (System.currentTimeMillis() / 1000)));
out.printf("SSTable Level: %d%n", stats.sstableLevel);
out.printf("Repaired at: %d%n", stats.repairedAt);
diff --git a/src/java/org/apache/cassandra/tools/SSTableOfflineRelevel.java b/src/java/org/apache/cassandra/tools/SSTableOfflineRelevel.java
index b27b07a..b62512a 100644
--- a/src/java/org/apache/cassandra/tools/SSTableOfflineRelevel.java
+++ b/src/java/org/apache/cassandra/tools/SSTableOfflineRelevel.java
@@ -17,18 +17,20 @@
*/
package org.apache.cassandra.tools;
+import java.io.File;
import java.io.IOException;
import java.io.PrintStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
-import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.google.common.base.Throwables;
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.SetMultimap;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.ColumnFamilyStore;
@@ -96,7 +98,7 @@
Keyspace ks = Keyspace.openWithoutSSTables(keyspace);
ColumnFamilyStore cfs = ks.getColumnFamilyStore(columnfamily);
Directories.SSTableLister lister = cfs.getDirectories().sstableLister(Directories.OnTxnErr.THROW).skipTemporary(true);
- Set<SSTableReader> sstables = new HashSet<>();
+ SetMultimap<File, SSTableReader> sstableMultimap = HashMultimap.create();
for (Map.Entry<Descriptor, Set<Component>> sstable : lister.list().entrySet())
{
if (sstable.getKey() != null)
@@ -104,7 +106,7 @@
try
{
SSTableReader reader = SSTableReader.open(sstable.getKey());
- sstables.add(reader);
+ sstableMultimap.put(reader.descriptor.directory, reader);
}
catch (Throwable t)
{
@@ -113,13 +115,20 @@
}
}
}
- if (sstables.isEmpty())
+ if (sstableMultimap.isEmpty())
{
out.println("No sstables to relevel for "+keyspace+"."+columnfamily);
System.exit(1);
}
- Relevel rl = new Relevel(sstables);
- rl.relevel(dryRun);
+ for (File directory : sstableMultimap.keySet())
+ {
+ if (!sstableMultimap.get(directory).isEmpty())
+ {
+ Relevel rl = new Relevel(sstableMultimap.get(directory));
+ out.println("For sstables in " + directory + ":");
+ rl.relevel(dryRun);
+ }
+ }
System.exit(0);
}
diff --git a/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java b/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
index cbb054a..1348d05 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
@@ -17,17 +17,22 @@
*/
package org.apache.cassandra.tools.nodetool;
-import static com.google.common.collect.Iterables.toArray;
-import io.airlift.command.Command;
-
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import java.util.Set;
-
import javax.management.openmbean.TabularData;
+import io.airlift.command.Command;
+
import org.apache.cassandra.tools.NodeProbe;
import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
+import static com.google.common.collect.Iterables.toArray;
+
@Command(name = "compactionhistory", description = "Print history of compaction")
public class CompactionHistory extends NodeToolCmd
{
@@ -48,10 +53,69 @@
System.out.printf(format, toArray(indexNames, Object.class));
Set<?> values = tabularData.keySet();
+ List<CompactionHistoryRow> chr = new ArrayList<>();
for (Object eachValue : values)
{
List<?> value = (List<?>) eachValue;
- System.out.printf(format, toArray(value, Object.class));
+ CompactionHistoryRow chc = new CompactionHistoryRow((String)value.get(0),
+ (String)value.get(1),
+ (String)value.get(2),
+ (Long)value.get(3),
+ (Long)value.get(4),
+ (Long)value.get(5),
+ (String)value.get(6));
+ chr.add(chc);
}
+ Collections.sort(chr);
+ for (CompactionHistoryRow eachChc : chr)
+ {
+ System.out.printf(format, eachChc.getAllAsArray());
+ }
+ }
+}
+
+/**
+ * Allows the Compaction History output to be ordered by 'compactedAt' - that is the
+ * time at which compaction finished.
+ */
+class CompactionHistoryRow implements Comparable<CompactionHistoryRow>
+{
+ private final String id;
+ private final String ksName;
+ private final String cfName;
+ private final long compactedAt;
+ private final long bytesIn;
+ private final long bytesOut;
+ private final String rowMerged;
+
+ CompactionHistoryRow(String id, String ksName, String cfName, long compactedAt, long bytesIn, long bytesOut, String rowMerged)
+ {
+ this.id = id;
+ this.ksName = ksName;
+ this.cfName = cfName;
+ this.compactedAt = compactedAt;
+ this.bytesIn = bytesIn;
+ this.bytesOut = bytesOut;
+ this.rowMerged = rowMerged;
+ }
+
+ public int compareTo(CompactionHistoryRow chc)
+ {
+ return Long.signum(chc.compactedAt - this.compactedAt);
+ }
+
+ public Object[] getAllAsArray()
+ {
+ Object[] obj = new Object[7];
+ obj[0] = this.id;
+ obj[1] = this.ksName;
+ obj[2] = this.cfName;
+ Instant instant = Instant.ofEpochMilli(this.compactedAt);
+ LocalDateTime ldt = LocalDateTime.ofInstant(instant, ZoneId.systemDefault());
+ obj[3] = ldt.toString();
+ obj[4] = this.bytesIn;
+ obj[5] = this.bytesOut;
+ obj[6] = this.rowMerged;
+ return obj;
}
}
\ No newline at end of file
diff --git a/src/java/org/apache/cassandra/tools/nodetool/CompactionStats.java b/src/java/org/apache/cassandra/tools/nodetool/CompactionStats.java
index e57d2ee..b950a1c 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/CompactionStats.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/CompactionStats.java
@@ -25,6 +25,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
import org.apache.cassandra.db.compaction.CompactionManagerMBean;
import org.apache.cassandra.db.compaction.OperationType;
@@ -45,7 +46,34 @@
public void execute(NodeProbe probe)
{
CompactionManagerMBean cm = probe.getCompactionManagerProxy();
- System.out.println("pending tasks: " + probe.getCompactionMetric("PendingTasks"));
+ Map<String, Map<String, Integer>> pendingTaskNumberByTable =
+ (Map<String, Map<String, Integer>>) probe.getCompactionMetric("PendingTasksByTableName");
+ int numTotalPendingTask = 0;
+ for (Entry<String, Map<String, Integer>> ksEntry : pendingTaskNumberByTable.entrySet())
+ {
+ for (Entry<String, Integer> tableEntry : ksEntry.getValue().entrySet())
+ numTotalPendingTask += tableEntry.getValue();
+ }
+ System.out.println("pending tasks: "+numTotalPendingTask);
+ for (Entry<String, Map<String, Integer>> ksEntry : pendingTaskNumberByTable.entrySet())
+ {
+ String ksName = ksEntry.getKey();
+ for (Entry<String, Integer> tableEntry : ksEntry.getValue().entrySet())
+ {
+ String tableName = tableEntry.getKey();
+ int pendingTaskCount = tableEntry.getValue();
+
+ StringBuilder builder = new StringBuilder();
+ builder.append("- ");
+ builder.append(ksName);
+ builder.append('.');
+ builder.append(tableName);
+ builder.append(": ");
+ builder.append(pendingTaskCount);
+ System.out.println(builder.toString());
+ }
+ }
+ System.out.println();
long remainingBytes = 0;
List<Map<String, String>> compactions = cm.getCompactions();
if (!compactions.isEmpty())
diff --git a/src/java/org/apache/cassandra/tools/nodetool/RelocateSSTables.java b/src/java/org/apache/cassandra/tools/nodetool/RelocateSSTables.java
new file mode 100644
index 0000000..8522bc4
--- /dev/null
+++ b/src/java/org/apache/cassandra/tools/nodetool/RelocateSSTables.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.tools.nodetool;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import io.airlift.command.Arguments;
+import io.airlift.command.Command;
+import org.apache.cassandra.tools.NodeProbe;
+import org.apache.cassandra.tools.NodeTool;
+
+@Command(name = "relocatesstables", description = "Relocates sstables to the correct disk")
+public class RelocateSSTables extends NodeTool.NodeToolCmd
+{
+ @Arguments(usage = "<keyspace> <table>", description = "The keyspace and table name")
+ private List<String> args = new ArrayList<>();
+
+ @Override
+ public void execute(NodeProbe probe)
+ {
+ List<String> keyspaces = parseOptionalKeyspace(args, probe);
+ String[] cfnames = parseOptionalTables(args);
+ try
+ {
+ for (String keyspace : keyspaces)
+ probe.relocateSSTables(keyspace, cfnames);
+ }
+ catch (Exception e)
+ {
+ throw new RuntimeException("Got error while relocating", e);
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/tools/nodetool/Ring.java b/src/java/org/apache/cassandra/tools/nodetool/Ring.java
index 03d9449..55220a1 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/Ring.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/Ring.java
@@ -88,7 +88,7 @@
}
catch (IllegalArgumentException ex)
{
- System.out.printf("%nError: " + ex.getMessage() + "%n");
+ System.out.printf("%nError: %s%n", ex.getMessage());
return;
}
@@ -174,4 +174,4 @@
}
System.out.println();
}
-}
\ No newline at end of file
+}
diff --git a/src/java/org/apache/cassandra/tools/nodetool/Status.java b/src/java/org/apache/cassandra/tools/nodetool/Status.java
index 99f745d..a43b703 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/Status.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/Status.java
@@ -81,7 +81,7 @@
}
catch (IllegalArgumentException ex)
{
- System.out.printf("%nError: " + ex.getMessage() + "%n");
+ System.out.printf("%nError: %s%n", ex.getMessage());
System.exit(1);
}
@@ -204,4 +204,4 @@
return format;
}
-}
\ No newline at end of file
+}
diff --git a/src/java/org/apache/cassandra/tools/nodetool/TableHistograms.java b/src/java/org/apache/cassandra/tools/nodetool/TableHistograms.java
index 23f1c43..8f4ffa6 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/TableHistograms.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/TableHistograms.java
@@ -34,16 +34,29 @@
@Command(name = "tablehistograms", description = "Print statistic histograms for a given table")
public class TableHistograms extends NodeToolCmd
{
- @Arguments(usage = "<keyspace> <table>", description = "The keyspace and table name")
+ @Arguments(usage = "<keyspace> <table> | <keyspace.table>", description = "The keyspace and table name")
private List<String> args = new ArrayList<>();
@Override
public void execute(NodeProbe probe)
{
- checkArgument(args.size() == 2, "tablehistograms requires keyspace and table name arguments");
-
- String keyspace = args.get(0);
- String table = args.get(1);
+ String keyspace = null, table = null;
+ if (args.size() == 2)
+ {
+ keyspace = args.get(0);
+ table = args.get(1);
+ }
+ else if (args.size() == 1)
+ {
+ String[] input = args.get(0).split("\\.");
+ checkArgument(input.length == 2, "tablehistograms requires keyspace and table name arguments");
+ keyspace = input[0];
+ table = input[1];
+ }
+ else
+ {
+ checkArgument(false, "tablehistograms requires keyspace and table name arguments");
+ }
// calculate percentile of row size and column count
long[] estimatedPartitionSize = (long[]) probe.getColumnFamilyMetric(keyspace, table, "EstimatedPartitionSizeHistogram");
@@ -66,14 +79,12 @@
}
else
{
- long[] partitionSizeBucketOffsets = new EstimatedHistogram(estimatedPartitionSize.length).getBucketOffsets();
- long[] columnCountBucketOffsets = new EstimatedHistogram(estimatedColumnCount.length).getBucketOffsets();
- EstimatedHistogram partitionSizeHist = new EstimatedHistogram(partitionSizeBucketOffsets, estimatedPartitionSize);
- EstimatedHistogram columnCountHist = new EstimatedHistogram(columnCountBucketOffsets, estimatedColumnCount);
+ EstimatedHistogram partitionSizeHist = new EstimatedHistogram(estimatedPartitionSize);
+ EstimatedHistogram columnCountHist = new EstimatedHistogram(estimatedColumnCount);
if (partitionSizeHist.isOverflowed())
{
- System.err.println(String.format("Row sizes are larger than %s, unable to calculate percentiles", partitionSizeBucketOffsets[partitionSizeBucketOffsets.length - 1]));
+ System.err.println(String.format("Row sizes are larger than %s, unable to calculate percentiles", partitionSizeHist.getLargestBucketOffset()));
for (int i = 0; i < offsetPercentiles.length; i++)
estimatedRowSizePercentiles[i] = Double.NaN;
}
@@ -85,7 +96,7 @@
if (columnCountHist.isOverflowed())
{
- System.err.println(String.format("Column counts are larger than %s, unable to calculate percentiles", columnCountBucketOffsets[columnCountBucketOffsets.length - 1]));
+ System.err.println(String.format("Column counts are larger than %s, unable to calculate percentiles", columnCountHist.getLargestBucketOffset()));
for (int i = 0; i < estimatedColumnCountPercentiles.length; i++)
estimatedColumnCountPercentiles[i] = Double.NaN;
}
diff --git a/src/java/org/apache/cassandra/utils/ByteBufferUtil.java b/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
index 27f46b6..6bcec96 100644
--- a/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
+++ b/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
@@ -162,7 +162,6 @@
public static byte[] getArray(ByteBuffer buffer)
{
int length = buffer.remaining();
-
if (buffer.hasArray())
{
int boff = buffer.arrayOffset() + buffer.position();
@@ -384,7 +383,6 @@
/**
* @param in data input
- * @return null
* @throws IOException if an I/O error occurs.
*/
public static void skipShortLength(DataInputPlus in) throws IOException
@@ -508,8 +506,16 @@
};
}
+ /*
+ * Does not modify position or limit of buffer even temporarily
+ * so this is safe even without duplication.
+ */
public static String bytesToHex(ByteBuffer bytes)
{
+ if (bytes.hasArray()) {
+ return Hex.bytesToHex(bytes.array(), bytes.arrayOffset() + bytes.position(), bytes.remaining());
+ }
+
final int offset = bytes.position();
final int size = bytes.remaining();
final char[] c = new char[size * 2];
diff --git a/src/java/org/apache/cassandra/utils/CassandraVersion.java b/src/java/org/apache/cassandra/utils/CassandraVersion.java
index 62d68be..759ca97 100644
--- a/src/java/org/apache/cassandra/utils/CassandraVersion.java
+++ b/src/java/org/apache/cassandra/utils/CassandraVersion.java
@@ -26,12 +26,19 @@
/**
* Implements versioning used in Cassandra and CQL.
- * <p/>
+ * <p>
* Note: The following code uses a slight variation from the semver document (http://semver.org).
+ * </p>
*/
public class CassandraVersion implements Comparable<CassandraVersion>
{
- private static final String VERSION_REGEXP = "(\\d+)\\.(\\d+)\\.(\\d+)(\\-[.\\w]+)?([.+][.\\w]+)?";
+ /**
+ * note: 3rd group matches to words but only allows number and checked after regexp test.
+ * this is because 3rd and the last can be identical.
+ **/
+ private static final String VERSION_REGEXP = "(\\d+)\\.(\\d+)(?:\\.(\\w+))?(\\-[.\\w]+)?([.+][.\\w]+)?";
+ private static final Pattern PATTERN_WHITESPACE = Pattern.compile("\\w+");
+
private static final Pattern pattern = Pattern.compile(VERSION_REGEXP);
private static final Pattern SNAPSHOT = Pattern.compile("-SNAPSHOT");
@@ -42,15 +49,6 @@
private final String[] preRelease;
private final String[] build;
- private CassandraVersion(int major, int minor, int patch, String[] preRelease, String[] build)
- {
- this.major = major;
- this.minor = minor;
- this.patch = patch;
- this.preRelease = preRelease;
- this.build = build;
- }
-
/**
* Parse a version from a string.
*
@@ -69,7 +67,7 @@
{
this.major = Integer.parseInt(matcher.group(1));
this.minor = Integer.parseInt(matcher.group(2));
- this.patch = Integer.parseInt(matcher.group(3));
+ this.patch = matcher.group(3) != null ? Integer.parseInt(matcher.group(3)) : 0;
String pr = matcher.group(4);
String bld = matcher.group(5);
@@ -79,7 +77,7 @@
}
catch (NumberFormatException e)
{
- throw new IllegalArgumentException("Invalid version value: " + version);
+ throw new IllegalArgumentException("Invalid version value: " + version, e);
}
}
@@ -87,10 +85,10 @@
{
// Drop initial - or +
str = str.substring(1);
- String[] parts = str.split("\\.");
+ String[] parts = StringUtils.split(str, '.');
for (String part : parts)
{
- if (!part.matches("\\w+"))
+ if (!PATTERN_WHITESPACE.matcher(part).matches())
throw new IllegalArgumentException("Invalid version value: " + version);
}
return parts;
@@ -123,13 +121,14 @@
/**
* Returns a version that is backward compatible with this version amongst a list
* of provided version, or null if none can be found.
- * <p/>
+ * <p>
* For instance:
* "2.0.0".findSupportingVersion("2.0.0", "3.0.0") == "2.0.0"
* "2.0.0".findSupportingVersion("2.1.3", "3.0.0") == "2.1.3"
* "2.0.0".findSupportingVersion("3.0.0") == null
* "2.0.3".findSupportingVersion("2.0.0") == "2.0.0"
* "2.1.0".findSupportingVersion("2.0.0") == null
+ * </p>
*/
public CassandraVersion findSupportingVersion(CassandraVersion... versions)
{
diff --git a/src/java/org/apache/cassandra/utils/EstimatedHistogram.java b/src/java/org/apache/cassandra/utils/EstimatedHistogram.java
index 0249980..aa141f7 100644
--- a/src/java/org/apache/cassandra/utils/EstimatedHistogram.java
+++ b/src/java/org/apache/cassandra/utils/EstimatedHistogram.java
@@ -65,6 +65,18 @@
buckets = new AtomicLongArray(bucketOffsets.length + 1);
}
+ /**
+ * Create EstimatedHistogram from only bucket data.
+ *
+ * @param bucketData bucket data
+ */
+ public EstimatedHistogram(long[] bucketData)
+ {
+ assert bucketData != null && bucketData.length > 0 : "Bucket data must be an array of size more than 0";
+ bucketOffsets = newOffsets(bucketData.length - 1, false);
+ buckets = new AtomicLongArray(bucketData);
+ }
+
public EstimatedHistogram(long[] offsets, long[] bucketData)
{
assert bucketData.length == offsets.length +1;
@@ -242,6 +254,14 @@
}
/**
+ * @return the largest bucket offset
+ */
+ public long getLargestBucketOffset()
+ {
+ return bucketOffsets[bucketOffsets.length - 1];
+ }
+
+ /**
* @return true if this histogram has overflowed -- that is, a value larger than our largest bucket could bound was added
*/
public boolean isOverflowed()
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java b/src/java/org/apache/cassandra/utils/FBUtilities.java
index 5f0e0a0..0bd2a51 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -832,4 +832,16 @@
throw new RuntimeException(e);
}
}
+
+ public static void sleepQuietly(long millis)
+ {
+ try
+ {
+ Thread.sleep(millis);
+ }
+ catch (InterruptedException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
}
diff --git a/src/java/org/apache/cassandra/utils/FastByteOperations.java b/src/java/org/apache/cassandra/utils/FastByteOperations.java
index 68e395c..cf8d305 100644
--- a/src/java/org/apache/cassandra/utils/FastByteOperations.java
+++ b/src/java/org/apache/cassandra/utils/FastByteOperations.java
@@ -333,7 +333,7 @@
* @param memoryOffset2 Where to start comparing in the right buffer (pure memory address if buffer1 is null, or relative otherwise)
* @param length1 How much to compare from the left buffer
* @param length2 How much to compare from the right buffer
- * @return 0 if equal, < 0 if left is less than right, etc.
+ * @return 0 if equal, {@code < 0} if left is less than right, etc.
*/
@Inline
public static int compareTo(Object buffer1, long memoryOffset1, int length1,
diff --git a/src/java/org/apache/cassandra/utils/Hex.java b/src/java/org/apache/cassandra/utils/Hex.java
index 0883c34..c4b4586 100644
--- a/src/java/org/apache/cassandra/utils/Hex.java
+++ b/src/java/org/apache/cassandra/utils/Hex.java
@@ -70,10 +70,15 @@
public static String bytesToHex(byte... bytes)
{
- char[] c = new char[bytes.length * 2];
- for (int i = 0; i < bytes.length; i++)
+ return bytesToHex(bytes, 0, bytes.length);
+ }
+
+ public static String bytesToHex(byte bytes[], int offset, int length)
+ {
+ char[] c = new char[length * 2];
+ for (int i = 0; i < length; i++)
{
- int bint = bytes[i];
+ int bint = bytes[i + offset];
c[i * 2] = byteToChar[(bint & 0xf0) >> 4];
c[1 + i * 2] = byteToChar[bint & 0x0f];
}
@@ -96,7 +101,7 @@
try
{
s = stringConstructor.newInstance(0, c.length, c);
- }
+ }
catch (InvocationTargetException ite) {
// The underlying constructor failed. Unwrapping the exception.
Throwable cause = ite.getCause();
diff --git a/src/java/org/apache/cassandra/utils/HistogramBuilder.java b/src/java/org/apache/cassandra/utils/HistogramBuilder.java
index 5d22352..093c52c 100644
--- a/src/java/org/apache/cassandra/utils/HistogramBuilder.java
+++ b/src/java/org/apache/cassandra/utils/HistogramBuilder.java
@@ -25,6 +25,9 @@
public class HistogramBuilder
{
+ public static final long[] EMPTY_LONG_ARRAY = new long[]{};
+ public static final long[] ZERO = new long[]{ 0 };
+
public HistogramBuilder() {}
public HistogramBuilder(long[] values)
{
@@ -73,7 +76,7 @@
final long[] values = this.values;
if (count == 0)
- return new EstimatedHistogram(new long[] { }, new long[] { 0 });
+ return new EstimatedHistogram(EMPTY_LONG_ARRAY, ZERO);
long min = Long.MAX_VALUE, max = Long.MIN_VALUE;
double sum = 0, sumsq = 0;
@@ -114,7 +117,7 @@
// minormax == mean we have no range to produce, but given the exclusive starts
// that begin at zero by default (or -Inf) in EstimatedHistogram we have to generate a min range
// to indicate where we start from
- return ismin ? new long[] { mean - 1 } : new long[0];
+ return ismin ? new long[] { mean - 1 } : EMPTY_LONG_ARRAY;
if (stdev < 1)
{
diff --git a/src/java/org/apache/cassandra/utils/IntervalTree.java b/src/java/org/apache/cassandra/utils/IntervalTree.java
index b92112e..4cf1222 100644
--- a/src/java/org/apache/cassandra/utils/IntervalTree.java
+++ b/src/java/org/apache/cassandra/utils/IntervalTree.java
@@ -272,20 +272,21 @@
protected I computeNext()
{
- if (current != null && current.hasNext())
- return current.next();
+ while (true)
+ {
+ if (current != null && current.hasNext())
+ return current.next();
- IntervalNode node = stack.pollFirst();
- if (node == null)
- return endOfData();
+ IntervalNode node = stack.pollFirst();
+ if (node == null)
+ return endOfData();
- current = node.intersectsLeft.iterator();
+ current = node.intersectsLeft.iterator();
- // We know this is the smaller not returned yet, but before doing
- // its parent, we must do everyone on it's right.
- gotoMinOf(node.right);
-
- return computeNext();
+ // We know this is the smaller not returned yet, but before doing
+ // its parent, we must do everyone on it's right.
+ gotoMinOf(node.right);
+ }
}
private void gotoMinOf(IntervalNode node)
diff --git a/src/java/org/apache/cassandra/utils/MergeIterator.java b/src/java/org/apache/cassandra/utils/MergeIterator.java
index 0cc5306..70daad9 100644
--- a/src/java/org/apache/cassandra/utils/MergeIterator.java
+++ b/src/java/org/apache/cassandra/utils/MergeIterator.java
@@ -17,11 +17,8 @@
*/
package org.apache.cassandra.utils;
-import java.io.Closeable;
import java.util.*;
-import org.apache.cassandra.utils.AbstractIterator;
-
/** Merges sorted input iterators which individually contain unique items. */
public abstract class MergeIterator<In,Out> extends AbstractIterator<Out> implements IMergeIterator<In, Out>
{
diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java b/src/java/org/apache/cassandra/utils/MerkleTree.java
index b3bccac..bc39b91 100644
--- a/src/java/org/apache/cassandra/utils/MerkleTree.java
+++ b/src/java/org/apache/cassandra/utils/MerkleTree.java
@@ -326,19 +326,28 @@
TreeRange getHelper(Hashable hashable, Token pleft, Token pright, byte depth, Token t)
{
- if (hashable instanceof Leaf)
+ while (true)
{
- // we've reached a hash: wrap it up and deliver it
- return new TreeRange(this, pleft, pright, depth, hashable);
- }
- // else: node.
+ if (hashable instanceof Leaf)
+ {
+ // we've reached a hash: wrap it up and deliver it
+ return new TreeRange(this, pleft, pright, depth, hashable);
+ }
+ // else: node.
- Inner node = (Inner)hashable;
- if (Range.contains(pleft, node.token, t))
- // left child contains token
- return getHelper(node.lchild, pleft, node.token, inc(depth), t);
- // else: right child contains token
- return getHelper(node.rchild, node.token, pright, inc(depth), t);
+ Inner node = (Inner) hashable;
+ depth = inc(depth);
+ if (Range.contains(pleft, node.token, t))
+ { // left child contains token
+ hashable = node.lchild;
+ pright = node.token;
+ }
+ else
+ { // else: right child contains token
+ hashable = node.rchild;
+ pleft = node.token;
+ }
+ }
}
/**
@@ -404,33 +413,42 @@
*/
private Hashable findHelper(Hashable current, Range<Token> activeRange, Range<Token> find) throws StopRecursion
{
- if (current instanceof Leaf)
+ while (true)
{
- if (!find.contains(activeRange))
- // we are not fully contained in this range!
+ if (current instanceof Leaf)
+ {
+ if (!find.contains(activeRange))
+ // we are not fully contained in this range!
+ throw new StopRecursion.BadRange();
+ return current;
+ }
+ // else: node.
+
+ Inner node = (Inner) current;
+ Range<Token> leftRange = new Range<>(activeRange.left, node.token);
+ Range<Token> rightRange = new Range<>(node.token, activeRange.right);
+
+ if (find.contains(activeRange))
+ // this node is fully contained in the range
+ return node.calc();
+
+ // else: one of our children contains the range
+
+ if (leftRange.contains(find))
+ { // left child contains/matches the range
+ current = node.lchild;
+ activeRange = leftRange;
+ }
+ else if (rightRange.contains(find))
+ { // right child contains/matches the range
+ current = node.rchild;
+ activeRange = rightRange;
+ }
+ else
+ {
throw new StopRecursion.BadRange();
- return current;
+ }
}
- // else: node.
-
- Inner node = (Inner)current;
- Range<Token> leftRange = new Range<Token>(activeRange.left, node.token);
- Range<Token> rightRange = new Range<Token>(node.token, activeRange.right);
-
- if (find.contains(activeRange))
- // this node is fully contained in the range
- return node.calc();
-
- // else: one of our children contains the range
-
- if (leftRange.contains(find))
- // left child contains/matches the range
- return findHelper(node.lchild, leftRange, find);
- else if (rightRange.contains(find))
- // right child contains/matches the range
- return findHelper(node.rchild, rightRange, find);
- else
- throw new StopRecursion.BadRange();
}
/**
diff --git a/src/java/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillis.java b/src/java/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillis.java
index f124383..69b7a47 100644
--- a/src/java/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillis.java
+++ b/src/java/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillis.java
@@ -19,6 +19,7 @@
import java.util.concurrent.TimeUnit;
+import org.apache.cassandra.concurrent.ScheduledExecutors;
import org.apache.cassandra.config.Config;
import com.google.common.annotations.VisibleForTesting;
@@ -36,9 +37,6 @@
private static volatile long TIMESTAMP_BASE[] = new long[] { System.currentTimeMillis(), System.nanoTime() };
- @VisibleForTesting
- public static final Object TIMESTAMP_UPDATE = new Object();
-
/*
* System.currentTimeMillis() is 25 nanoseconds. This is 2 nanoseconds (maybe) according to JMH.
* Faster than calling both currentTimeMillis() and nanoTime().
@@ -48,41 +46,29 @@
* These timestamps don't order with System.currentTimeMillis() because currentTimeMillis() can tick over
* before this one does. I have seen it behind by as much as 2ms on Linux and 25ms on Windows.
*/
- public static final long convert(long nanoTime)
+ public static long convert(long nanoTime)
{
final long timestampBase[] = TIMESTAMP_BASE;
return timestampBase[0] + TimeUnit.NANOSECONDS.toMillis(nanoTime - timestampBase[1]);
}
+ public static void updateNow()
+ {
+ ScheduledExecutors.scheduledFastTasks.submit(NanoTimeToCurrentTimeMillis::updateTimestampBase);
+ }
+
static
{
- //Pick up updates from NTP periodically
- Thread t = new Thread("NanoTimeToCurrentTimeMillis updater")
- {
- @Override
- public void run()
- {
- while (true)
- {
- try
- {
- synchronized (TIMESTAMP_UPDATE)
- {
- TIMESTAMP_UPDATE.wait(TIMESTAMP_UPDATE_INTERVAL);
- }
- }
- catch (InterruptedException e)
- {
- return;
- }
+ ScheduledExecutors.scheduledFastTasks.scheduleWithFixedDelay(NanoTimeToCurrentTimeMillis::updateTimestampBase,
+ TIMESTAMP_UPDATE_INTERVAL,
+ TIMESTAMP_UPDATE_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
- TIMESTAMP_BASE = new long[] {
- Math.max(TIMESTAMP_BASE[0], System.currentTimeMillis()),
- Math.max(TIMESTAMP_BASE[1], System.nanoTime()) };
- }
- }
- };
- t.setDaemon(true);
- t.start();
+ private static void updateTimestampBase()
+ {
+ TIMESTAMP_BASE = new long[] {
+ Math.max(TIMESTAMP_BASE[0], System.currentTimeMillis()),
+ Math.max(TIMESTAMP_BASE[1], System.nanoTime()) };
}
}
diff --git a/src/java/org/apache/cassandra/utils/SearchIterator.java b/src/java/org/apache/cassandra/utils/SearchIterator.java
index 5309f4a..95cb33c 100644
--- a/src/java/org/apache/cassandra/utils/SearchIterator.java
+++ b/src/java/org/apache/cassandra/utils/SearchIterator.java
@@ -26,7 +26,7 @@
* if this or any key greater has already been returned by the iterator, the method may
* choose to return null, the correct or incorrect output, or fail an assertion.
*
- * it is permitted to search past the end of the iterator, i.e. !hasNext() => next(?) == null
+ * it is permitted to search past the end of the iterator, i.e. {@code !hasNext() => next(?) == null}
*
* @param key to search for
* @return value associated with key, if present in direction of travel
diff --git a/src/java/org/apache/cassandra/utils/UUIDGen.java b/src/java/org/apache/cassandra/utils/UUIDGen.java
index df07e1f..2adbbe6 100644
--- a/src/java/org/apache/cassandra/utils/UUIDGen.java
+++ b/src/java/org/apache/cassandra/utils/UUIDGen.java
@@ -107,6 +107,15 @@
return new UUID(raw.getLong(raw.position()), raw.getLong(raw.position() + 8));
}
+ public static ByteBuffer toByteBuffer(UUID uuid)
+ {
+ ByteBuffer buffer = ByteBuffer.allocate(16);
+ buffer.putLong(uuid.getMostSignificantBits());
+ buffer.putLong(uuid.getLeastSignificantBits());
+ buffer.flip();
+ return buffer;
+ }
+
/** decomposes a uuid into raw bytes. */
public static byte[] decompose(UUID uuid)
{
@@ -195,7 +204,7 @@
* of a type 1 UUID (a time-based UUID).
*
* To specify a 100-nanoseconds precision timestamp, one should provide a milliseconds timestamp and
- * a number 0 <= n < 10000 such that n*100 is the number of nanoseconds within that millisecond.
+ * a number {@code 0 <= n < 10000} such that n*100 is the number of nanoseconds within that millisecond.
*
* <p><i><b>Warning:</b> This method is not guaranteed to return unique UUIDs; Multiple
* invocations using identical timestamps will result in identical UUIDs.</i></p>
diff --git a/src/java/org/apache/cassandra/utils/btree/BTree.java b/src/java/org/apache/cassandra/utils/btree/BTree.java
index fe08011..1c3d2e2 100644
--- a/src/java/org/apache/cassandra/utils/btree/BTree.java
+++ b/src/java/org/apache/cassandra/utils/btree/BTree.java
@@ -67,6 +67,8 @@
// NB we encode Path indexes as Bytes, so this needs to be less than Byte.MAX_VALUE / 2
static final int FAN_FACTOR = 1 << FAN_SHIFT;
+ static final int MINIMAL_NODE_SIZE = FAN_FACTOR >> 1;
+
// An empty BTree Leaf - which is the same as an empty BTree
static final Object[] EMPTY_LEAF = new Object[1];
@@ -296,6 +298,40 @@
/**
* Modifies the provided btree directly. THIS SHOULD NOT BE USED WITHOUT EXTREME CARE as BTrees are meant to be immutable.
+ * Finds and replaces the item provided by index in the tree.
+ */
+ public static <V> void replaceInSitu(Object[] tree, int index, V replace)
+ {
+ // WARNING: if semantics change, see also InternalCursor.seekTo, which mirrors this implementation
+ if ((index < 0) | (index >= size(tree)))
+ throw new IndexOutOfBoundsException(index + " not in range [0.." + size(tree) + ")");
+
+ while (!isLeaf(tree))
+ {
+ final int[] sizeMap = getSizeMap(tree);
+ int boundary = Arrays.binarySearch(sizeMap, index);
+ if (boundary >= 0)
+ {
+ // exact match, in this branch node
+ assert boundary < sizeMap.length - 1;
+ tree[boundary] = replace;
+ return;
+ }
+
+ boundary = -1 -boundary;
+ if (boundary > 0)
+ {
+ assert boundary < sizeMap.length;
+ index -= (1 + sizeMap[boundary - 1]);
+ }
+ tree = (Object[]) tree[getChildStart(tree) + boundary];
+ }
+ assert index < getLeafKeyEnd(tree);
+ tree[index] = replace;
+ }
+
+ /**
+ * Modifies the provided btree directly. THIS SHOULD NOT BE USED WITHOUT EXTREME CARE as BTrees are meant to be immutable.
* Finds and replaces the provided item in the tree. Both should sort as equal to each other (although this is not enforced)
*/
public static <V> void replaceInSitu(Object[] node, Comparator<? super V> comparator, V find, V replace)
@@ -1073,11 +1109,20 @@
return node.length >= FAN_FACTOR / 2 && node.length <= FAN_FACTOR + 1;
}
+ final int keyCount = getBranchKeyEnd(node);
+ if ((!isRoot && keyCount < FAN_FACTOR / 2) || keyCount > FAN_FACTOR + 1)
+ return false;
+
int type = 0;
+ int size = -1;
+ int[] sizeMap = getSizeMap(node);
// compare each child node with the branch element at the head of this node it corresponds with
for (int i = getChildStart(node); i < getChildEnd(node) ; i++)
{
Object[] child = (Object[]) node[i];
+ size += size(child) + 1;
+ if (sizeMap[i - getChildStart(node)] != size)
+ return false;
Object localmax = i < node.length - 2 ? node[i - getChildStart(node)] : max;
if (!isWellFormed(cmp, child, false, min, localmax))
return false;
diff --git a/src/java/org/apache/cassandra/utils/btree/BTreeRemoval.java b/src/java/org/apache/cassandra/utils/btree/BTreeRemoval.java
new file mode 100644
index 0000000..a4402b8
--- /dev/null
+++ b/src/java/org/apache/cassandra/utils/btree/BTreeRemoval.java
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cassandra.utils.btree;
+
+import java.util.Arrays;
+import java.util.Comparator;
+
+public class BTreeRemoval
+{
+ /**
+ * Remove |elem| from |btree|. If it's not present then return |btree| itself.
+ */
+ public static <V> Object[] remove(final Object[] btree, final Comparator<? super V> comparator, final V elem)
+ {
+ if (BTree.isEmpty(btree))
+ return btree;
+ int index = -1;
+ V elemToSwap = null;
+ int lb = 0;
+ Object[] node = btree;
+ while (true)
+ {
+ int keyEnd = BTree.getKeyEnd(node);
+ int i = Arrays.binarySearch((V[]) node, 0, keyEnd, elem, comparator);
+
+ if (i >= 0)
+ {
+ if (BTree.isLeaf(node))
+ index = lb + i;
+ else
+ {
+ final int indexInNode = BTree.getSizeMap(node)[i];
+ index = lb + indexInNode - 1;
+ elemToSwap = BTree.findByIndex(node, indexInNode - 1);
+ }
+ break;
+ }
+ if (BTree.isLeaf(node))
+ return btree;
+
+ i = -1 - i;
+ if (i > 0)
+ lb += BTree.getSizeMap(node)[i - 1] + 1;
+
+ node = (Object[]) node[keyEnd + i];
+ }
+ if (BTree.size(btree) == 1)
+ return BTree.empty();
+ Object[] result = removeFromLeaf(btree, index);
+ if (elemToSwap != null)
+ BTree.replaceInSitu(result, index, elemToSwap);
+ return result;
+ }
+
+ /**
+ * Remove |elem| from |btree|. It has to be present and it has to reside in a leaf node.
+ */
+ private static Object[] removeFromLeaf(Object[] node, int index)
+ {
+ Object[] result = null;
+ Object[] prevNode = null;
+ int prevI = -1;
+ boolean needsCopy = true;
+ while (!BTree.isLeaf(node))
+ {
+ final int keyEnd = BTree.getBranchKeyEnd(node);
+ int i = -1 - Arrays.binarySearch(BTree.getSizeMap(node), index);
+ if (i > 0)
+ index -= (1 + BTree.getSizeMap(node)[i - 1]);
+ Object[] nextNode = (Object[]) node[keyEnd + i];
+ boolean nextNodeNeedsCopy = true;
+ if (BTree.getKeyEnd(nextNode) > BTree.MINIMAL_NODE_SIZE)
+ node = copyIfNeeded(node, needsCopy);
+ else if (i > 0 && BTree.getKeyEnd((Object[]) node[keyEnd + i - 1]) > BTree.MINIMAL_NODE_SIZE)
+ {
+ node = copyIfNeeded(node, needsCopy);
+ final Object[] leftNeighbour = (Object[]) node[keyEnd + i - 1];
+ index++;
+ if (!BTree.isLeaf(leftNeighbour))
+ index += BTree.size((Object[])leftNeighbour[BTree.getChildEnd(leftNeighbour) - 1]);
+ nextNode = rotateLeft(node, i);
+ }
+ else if (i < keyEnd && BTree.getKeyEnd((Object[]) node[keyEnd + i + 1]) > BTree.MINIMAL_NODE_SIZE)
+ {
+ node = copyIfNeeded(node, needsCopy);
+ nextNode = rotateRight(node, i);
+ }
+ else
+ {
+ nextNodeNeedsCopy = false;
+ if (i > 0)
+ {
+ final Object[] leftNeighbour = (Object[]) node[keyEnd + i - 1];
+ final Object nodeKey = node[i - 1];
+ node = keyEnd == 1 ? null : copyWithKeyAndChildRemoved(node, i - 1, i - 1, false);
+ nextNode = merge(leftNeighbour, nextNode, nodeKey);
+ i = i - 1;
+ index += BTree.size(leftNeighbour) + 1;
+ }
+ else
+ {
+ final Object[] rightNeighbour = (Object[]) node[keyEnd + i + 1];
+ final Object nodeKey = node[i];
+ node = keyEnd == 1 ? null : copyWithKeyAndChildRemoved(node, i, i, false);
+ nextNode = merge(nextNode, rightNeighbour, nodeKey);
+ }
+ }
+
+ if (node != null)
+ {
+ final int[] sizeMap = BTree.getSizeMap(node);
+ for (int j = i; j < sizeMap.length; ++j)
+ sizeMap[j] -= 1;
+ if (prevNode != null)
+ prevNode[prevI] = node;
+ else
+ result = node;
+ prevNode = node;
+ prevI = BTree.getChildStart(node) + i;
+ }
+
+ node = nextNode;
+ needsCopy = nextNodeNeedsCopy;
+ }
+ final int keyEnd = BTree.getLeafKeyEnd(node);
+ final Object[] newLeaf = new Object[(keyEnd & 1) == 1 ? keyEnd : keyEnd - 1];
+ copyKeys(node, newLeaf, 0, index);
+ if (prevNode != null)
+ prevNode[prevI] = newLeaf;
+ else
+ result = newLeaf;
+ return result;
+ }
+
+ private static Object[] rotateRight(final Object[] node, final int i) {
+ final int keyEnd = BTree.getBranchKeyEnd(node);
+ final Object[] nextNode = (Object[]) node[keyEnd + i];
+ final Object[] rightNeighbour = (Object[]) node[keyEnd + i + 1];
+ final boolean leaves = BTree.isLeaf(nextNode);
+ final int nextKeyEnd = BTree.getKeyEnd(nextNode);
+ final Object[] newChild = leaves ? null : (Object[]) rightNeighbour[BTree.getChildStart(rightNeighbour)];
+ final Object[] newNextNode =
+ copyWithKeyAndChildInserted(nextNode, nextKeyEnd, node[i], BTree.getChildCount(nextNode), newChild);
+ node[i] = rightNeighbour[0];
+ node[keyEnd + i + 1] = copyWithKeyAndChildRemoved(rightNeighbour, 0, 0, true);
+ BTree.getSizeMap(node)[i] +=
+ leaves ? 1 : 1 + BTree.size((Object[]) newNextNode[BTree.getChildEnd(newNextNode) - 1]);
+ return newNextNode;
+ }
+
+ private static Object[] rotateLeft(final Object[] node, final int i) {
+ final int keyEnd = BTree.getBranchKeyEnd(node);
+ final Object[] nextNode = (Object[]) node[keyEnd + i];
+ final Object[] leftNeighbour = (Object[]) node[keyEnd + i - 1];
+ final int leftNeighbourEndKey = BTree.getKeyEnd(leftNeighbour);
+ final boolean leaves = BTree.isLeaf(nextNode);
+ final Object[] newChild = leaves ? null : (Object[]) leftNeighbour[BTree.getChildEnd(leftNeighbour) - 1];
+ final Object[] newNextNode = copyWithKeyAndChildInserted(nextNode, 0, node[i - 1], 0, newChild);
+ node[i - 1] = leftNeighbour[leftNeighbourEndKey - 1];
+ node[keyEnd + i - 1] = copyWithKeyAndChildRemoved(leftNeighbour, leftNeighbourEndKey - 1, leftNeighbourEndKey, true);
+ BTree.getSizeMap(node)[i - 1] -= leaves ? 1 : 1 + BTree.getSizeMap(newNextNode)[0];
+ return newNextNode;
+ }
+
+ private static <V> Object[] copyWithKeyAndChildInserted(final Object[] node, final int keyIndex, final V key, final int childIndex, final Object[] child)
+ {
+ final boolean leaf = BTree.isLeaf(node);
+ final int keyEnd = BTree.getKeyEnd(node);
+ final Object[] copy;
+ if (leaf)
+ copy = new Object[keyEnd + ((keyEnd & 1) == 1 ? 2 : 1)];
+ else
+ copy = new Object[node.length + 2];
+
+ if (keyIndex > 0)
+ System.arraycopy(node, 0, copy, 0, keyIndex);
+ copy[keyIndex] = key;
+ if (keyIndex < keyEnd)
+ System.arraycopy(node, keyIndex, copy, keyIndex + 1, keyEnd - keyIndex);
+
+ if (!leaf)
+ {
+ if (childIndex > 0)
+ System.arraycopy(node,
+ BTree.getChildStart(node),
+ copy,
+ keyEnd + 1,
+ childIndex);
+ copy[keyEnd + 1 + childIndex] = child;
+ if (childIndex <= keyEnd)
+ System.arraycopy(node,
+ BTree.getChildStart(node) + childIndex,
+ copy,
+ keyEnd + childIndex + 2,
+ keyEnd - childIndex + 1);
+ final int[] sizeMap = BTree.getSizeMap(node);
+ final int[] newSizeMap = new int[sizeMap.length + 1];
+ if (childIndex > 0)
+ System.arraycopy(sizeMap, 0, newSizeMap, 0, childIndex);
+ final int childSize = BTree.size(child);
+ newSizeMap[childIndex] = childSize + ((childIndex == 0) ? 0 : newSizeMap[childIndex - 1] + 1);
+ for (int i = childIndex + 1; i < newSizeMap.length; ++i)
+ newSizeMap[i] = sizeMap[i - 1] + childSize + 1;
+ copy[copy.length - 1] = newSizeMap;
+ }
+ return copy;
+ }
+
+ private static Object[] copyWithKeyAndChildRemoved(final Object[] node, final int keyIndex, final int childIndex, final boolean substractSize)
+ {
+ final boolean leaf = BTree.isLeaf(node);
+ final int keyEnd = BTree.getKeyEnd(node);
+ final Object[] newNode;
+ if (leaf)
+ newNode = new Object[keyEnd - ((keyEnd & 1) == 1 ? 0 : 1)];
+ else
+ newNode = new Object[node.length - 2];
+ int offset = copyKeys(node, newNode, 0, keyIndex);
+ if (!leaf)
+ {
+ offset = copyChildren(node, newNode, offset, childIndex);
+ final int[] nodeSizeMap = BTree.getSizeMap(node);
+ final int[] newNodeSizeMap = new int[nodeSizeMap.length - 1];
+ int pos = 0;
+ final int sizeToRemove = BTree.size((Object[])node[BTree.getChildStart(node) + childIndex]) + 1;
+ for (int i = 0; i < nodeSizeMap.length; ++i)
+ if (i != childIndex)
+ newNodeSizeMap[pos++] = nodeSizeMap[i] -
+ ((substractSize && i > childIndex) ? sizeToRemove : 0);
+ newNode[offset] = newNodeSizeMap;
+ }
+ return newNode;
+ }
+
+ private static <V> Object[] merge(final Object[] left, final Object[] right, final V nodeKey) {
+ assert BTree.getKeyEnd(left) == BTree.MINIMAL_NODE_SIZE;
+ assert BTree.getKeyEnd(right) == BTree.MINIMAL_NODE_SIZE;
+ final boolean leaves = BTree.isLeaf(left);
+ final Object[] result;
+ if (leaves)
+ result = new Object[BTree.MINIMAL_NODE_SIZE * 2 + 1];
+ else
+ result = new Object[left.length + right.length];
+ int offset = 0;
+ offset = copyKeys(left, result, offset);
+ result[offset++] = nodeKey;
+ offset = copyKeys(right, result, offset);
+ if (!leaves)
+ {
+ offset = copyChildren(left, result, offset);
+ offset = copyChildren(right, result, offset);
+ final int[] leftSizeMap = BTree.getSizeMap(left);
+ final int[] rightSizeMap = BTree.getSizeMap(right);
+ final int[] newSizeMap = new int[leftSizeMap.length + rightSizeMap.length];
+ offset = 0;
+ offset = copySizeMap(leftSizeMap, newSizeMap, offset, 0);
+ offset = copySizeMap(rightSizeMap, newSizeMap, offset, leftSizeMap[leftSizeMap.length - 1] + 1);
+ result[result.length - 1] = newSizeMap;
+ }
+ return result;
+ }
+
+ private static int copyKeys(final Object[] from, final Object[] to, final int offset)
+ {
+ final int keysCount = BTree.getKeyEnd(from);
+ System.arraycopy(from, 0, to, offset, keysCount);
+ return offset + keysCount;
+ }
+
+ private static int copyKeys(final Object[] from, final Object[] to, final int offset, final int skipIndex)
+ {
+ final int keysCount = BTree.getKeyEnd(from);
+ if (skipIndex > 0)
+ System.arraycopy(from, 0, to, offset, skipIndex);
+ if (skipIndex + 1 < keysCount)
+ System.arraycopy(from, skipIndex + 1, to, offset + skipIndex, keysCount - skipIndex - 1);
+ return offset + keysCount - 1;
+ }
+
+ private static int copyChildren(final Object[] from, final Object[] to, final int offset)
+ {
+ assert !BTree.isLeaf(from);
+ final int start = BTree.getChildStart(from);
+ final int childCount = BTree.getChildCount(from);
+ System.arraycopy(from, start, to, offset, childCount);
+ return offset + childCount;
+ }
+
+ private static int copyChildren(final Object[] from, final Object[] to, final int offset, final int skipIndex)
+ {
+ assert !BTree.isLeaf(from);
+ final int start = BTree.getChildStart(from);
+ final int childCount = BTree.getChildCount(from);
+ if (skipIndex > 0)
+ System.arraycopy(from, start, to, offset, skipIndex);
+ if (skipIndex + 1 <= childCount)
+ System.arraycopy(from, start + skipIndex + 1, to, offset + skipIndex, childCount - skipIndex - 1);
+ return offset + childCount - 1;
+ }
+
+ private static int copySizeMap(final int[] from, final int[] to, final int offset, final int extra)
+ {
+ for (int i = 0; i < from.length; ++i)
+ to[offset + i] = from[i] + extra;
+ return offset + from.length;
+ }
+
+ private static Object[] copyIfNeeded(final Object[] node, boolean needCopy)
+ {
+ if (!needCopy) return node;
+ final Object[] copy = new Object[node.length];
+ System.arraycopy(node, 0, copy, 0, node.length);
+ if (!BTree.isLeaf(node))
+ {
+ final int[] sizeMap = BTree.getSizeMap(node);
+ final int[] copySizeMap = new int[sizeMap.length];
+ System.arraycopy(sizeMap, 0, copySizeMap, 0, sizeMap.length);
+ copy[copy.length - 1] = copySizeMap;
+ }
+ return copy;
+ }
+}
diff --git a/src/java/org/apache/cassandra/utils/btree/BTreeSet.java b/src/java/org/apache/cassandra/utils/btree/BTreeSet.java
index 03fa1ec..9517009 100644
--- a/src/java/org/apache/cassandra/utils/btree/BTreeSet.java
+++ b/src/java/org/apache/cassandra/utils/btree/BTreeSet.java
@@ -639,6 +639,6 @@
public static <V> BTreeSet<V> of(Comparator<? super V> comparator, V value)
{
- return new BTreeSet<>(BTree.build(ImmutableList.of(value), UpdateFunction.<V>noOp()), comparator);
+ return new BTreeSet<>(BTree.singleton(value), comparator);
}
}
diff --git a/src/java/org/apache/cassandra/utils/btree/TreeCursor.java b/src/java/org/apache/cassandra/utils/btree/TreeCursor.java
index 5e55698..60c0eb9 100644
--- a/src/java/org/apache/cassandra/utils/btree/TreeCursor.java
+++ b/src/java/org/apache/cassandra/utils/btree/TreeCursor.java
@@ -219,8 +219,7 @@
return;
}
- NodeCursor<K> cur = this.cur;
- cur = root();
+ NodeCursor<K> cur = root();
assert cur.nodeOffset == 0;
while (true)
{
diff --git a/test/conf/cassandra.keystore b/test/conf/cassandra.keystore
new file mode 100644
index 0000000..9a704ca
--- /dev/null
+++ b/test/conf/cassandra.keystore
Binary files differ
diff --git a/test/conf/cassandra_encryption.yaml b/test/conf/cassandra_encryption.yaml
new file mode 100644
index 0000000..47e1312
--- /dev/null
+++ b/test/conf/cassandra_encryption.yaml
@@ -0,0 +1,14 @@
+transparent_data_encryption_options:
+ enabled: true
+ chunk_length_kb: 2
+ cipher: AES/CBC/PKCS5Padding
+ key_alias: testing:1
+ # CBC requires iv length to be 16 bytes
+ # iv_length: 16
+ key_provider:
+ - class_name: org.apache.cassandra.security.JKSKeyProvider
+ parameters:
+ - keystore: test/conf/cassandra.keystore
+ keystore_password: cassandra
+ store_type: JCEKS
+ key_password: cassandra
diff --git a/test/conf/logback-test.xml b/test/conf/logback-test.xml
index 0e1bb76..2591890 100644
--- a/test/conf/logback-test.xml
+++ b/test/conf/logback-test.xml
@@ -42,13 +42,13 @@
<immediateFlush>false</immediateFlush>
</encoder>
</appender>
-
+
<appender name="STDOUT" target="System.out" class="org.apache.cassandra.ConsoleAppender">
<encoder>
<pattern>%-5level %date{HH:mm:ss,SSS} %msg%n</pattern>
</encoder>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
- <level>INFO</level>
+ <level>DEBUG</level>
</filter>
</appender>
@@ -59,6 +59,8 @@
<logger name="org.apache.hadoop" level="WARN"/>
+ <logger name="org.apache.cassandra.db.monitoring" level="DEBUG"/>
+
<!-- Do not change the name of this appender. LogbackStatusListener uses the thread name
tied to the appender name to know when to write to real stdout/stderr vs forwarding to logback -->
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
diff --git a/test/long/org/apache/cassandra/db/compaction/LongLeveledCompactionStrategyTest.java b/test/long/org/apache/cassandra/db/compaction/LongLeveledCompactionStrategyTest.java
index 96ee072..0f05524 100644
--- a/test/long/org/apache/cassandra/db/compaction/LongLeveledCompactionStrategyTest.java
+++ b/test/long/org/apache/cassandra/db/compaction/LongLeveledCompactionStrategyTest.java
@@ -59,8 +59,8 @@
Keyspace keyspace = Keyspace.open(ksname);
ColumnFamilyStore store = keyspace.getColumnFamilyStore(cfname);
store.disableAutoCompaction();
-
- LeveledCompactionStrategy lcs = (LeveledCompactionStrategy)store.getCompactionStrategyManager().getStrategies().get(1);
+ CompactionStrategyManager mgr = store.getCompactionStrategyManager();
+ LeveledCompactionStrategy lcs = (LeveledCompactionStrategy) mgr.getStrategies().get(1).get(0);
ByteBuffer value = ByteBuffer.wrap(new byte[100 * 1024]); // 100 KB value, make it easy to have multiple files
diff --git a/test/unit/org/apache/cassandra/Util.java b/test/unit/org/apache/cassandra/Util.java
index 7ce8f04..87a07b0 100644
--- a/test/unit/org/apache/cassandra/Util.java
+++ b/test/unit/org/apache/cassandra/Util.java
@@ -275,7 +275,8 @@
public static void assertEmptyUnfiltered(ReadCommand command)
{
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); UnfilteredPartitionIterator iterator = command.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ UnfilteredPartitionIterator iterator = command.executeLocally(executionController))
{
if (iterator.hasNext())
{
@@ -289,7 +290,8 @@
public static void assertEmpty(ReadCommand command)
{
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); PartitionIterator iterator = command.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ PartitionIterator iterator = command.executeInternal(executionController))
{
if (iterator.hasNext())
{
@@ -304,7 +306,8 @@
public static List<ImmutableBTreePartition> getAllUnfiltered(ReadCommand command)
{
List<ImmutableBTreePartition> results = new ArrayList<>();
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); UnfilteredPartitionIterator iterator = command.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ UnfilteredPartitionIterator iterator = command.executeLocally(executionController))
{
while (iterator.hasNext())
{
@@ -320,7 +323,8 @@
public static List<FilteredPartition> getAll(ReadCommand command)
{
List<FilteredPartition> results = new ArrayList<>();
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); PartitionIterator iterator = command.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ PartitionIterator iterator = command.executeInternal(executionController))
{
while (iterator.hasNext())
{
@@ -335,7 +339,8 @@
public static Row getOnlyRowUnfiltered(ReadCommand cmd)
{
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iterator = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iterator = cmd.executeLocally(executionController))
{
assert iterator.hasNext() : "Expecting one row in one partition but got nothing";
try (UnfilteredRowIterator partition = iterator.next())
@@ -352,7 +357,8 @@
public static Row getOnlyRow(ReadCommand cmd)
{
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); PartitionIterator iterator = cmd.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ PartitionIterator iterator = cmd.executeInternal(executionController))
{
assert iterator.hasNext() : "Expecting one row in one partition but got nothing";
try (RowIterator partition = iterator.next())
@@ -368,7 +374,8 @@
public static ImmutableBTreePartition getOnlyPartitionUnfiltered(ReadCommand cmd)
{
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iterator = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iterator = cmd.executeLocally(executionController))
{
assert iterator.hasNext() : "Expecting a single partition but got nothing";
try (UnfilteredRowIterator partition = iterator.next())
@@ -381,7 +388,8 @@
public static FilteredPartition getOnlyPartition(ReadCommand cmd)
{
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); PartitionIterator iterator = cmd.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ PartitionIterator iterator = cmd.executeInternal(executionController))
{
assert iterator.hasNext() : "Expecting a single partition but got nothing";
try (RowIterator partition = iterator.next())
@@ -567,9 +575,9 @@
public static UnfilteredPartitionIterator executeLocally(PartitionRangeReadCommand command,
ColumnFamilyStore cfs,
- ReadOrderGroup orderGroup)
+ ReadExecutionController controller)
{
- return new InternalPartitionRangeReadCommand(command).queryStorageInternal(cfs, orderGroup);
+ return new InternalPartitionRangeReadCommand(command).queryStorageInternal(cfs, controller);
}
private static final class InternalPartitionRangeReadCommand extends PartitionRangeReadCommand
@@ -590,9 +598,9 @@
}
private UnfilteredPartitionIterator queryStorageInternal(ColumnFamilyStore cfs,
- ReadOrderGroup orderGroup)
+ ReadExecutionController controller)
{
- return queryStorage(cfs, orderGroup);
+ return queryStorage(cfs, controller);
}
}
}
diff --git a/test/unit/org/apache/cassandra/config/CFMetaDataTest.java b/test/unit/org/apache/cassandra/config/CFMetaDataTest.java
index 9d91df3..188f72f 100644
--- a/test/unit/org/apache/cassandra/config/CFMetaDataTest.java
+++ b/test/unit/org/apache/cassandra/config/CFMetaDataTest.java
@@ -49,6 +49,8 @@
import org.junit.Test;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
public class CFMetaDataTest
{
@@ -173,4 +175,22 @@
assertEquals(cfm.params, params);
assertEquals(new HashSet<>(cfm.allColumns()), columns);
}
+
+ @Test
+ public void testIsNameValidPositive()
+ {
+ assertTrue(CFMetaData.isNameValid("abcdefghijklmnopqrstuvwxyz"));
+ assertTrue(CFMetaData.isNameValid("ABCDEFGHIJKLMNOPQRSTUVWXYZ"));
+ assertTrue(CFMetaData.isNameValid("_01234567890"));
+ }
+
+ @Test
+ public void testIsNameValidNegative()
+ {
+ assertFalse(CFMetaData.isNameValid(null));
+ assertFalse(CFMetaData.isNameValid(""));
+ assertFalse(CFMetaData.isNameValid(" "));
+ assertFalse(CFMetaData.isNameValid("@"));
+ assertFalse(CFMetaData.isNameValid("!"));
+ }
}
diff --git a/test/unit/org/apache/cassandra/config/DatabaseDescriptorTest.java b/test/unit/org/apache/cassandra/config/DatabaseDescriptorTest.java
index 3a3b6ee..84f0235 100644
--- a/test/unit/org/apache/cassandra/config/DatabaseDescriptorTest.java
+++ b/test/unit/org/apache/cassandra/config/DatabaseDescriptorTest.java
@@ -23,6 +23,8 @@
import java.net.Inet6Address;
import java.net.InetAddress;
import java.net.NetworkInterface;
+import java.util.Arrays;
+import java.util.Collection;
import java.util.Enumeration;
import org.junit.BeforeClass;
@@ -43,6 +45,8 @@
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
@RunWith(OrderedJUnit4ClassRunner.class)
public class DatabaseDescriptorTest
{
@@ -265,4 +269,15 @@
DatabaseDescriptor.applyAddressConfig(testConfig);
}
+
+ @Test
+ public void testTokensFromString()
+ {
+ assertTrue(DatabaseDescriptor.tokensFromString(null).isEmpty());
+ Collection<String> tokens = DatabaseDescriptor.tokensFromString(" a,b ,c , d, f,g,h");
+ assertEquals(7, tokens.size());
+ assertTrue(tokens.containsAll(Arrays.asList(new String[]{"a", "b", "c", "d", "f", "g", "h"})));
+
+
+ }
}
diff --git a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
index c287883..158110c 100644
--- a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
+++ b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
@@ -26,6 +26,7 @@
import junit.framework.Assert;
import org.apache.cassandra.db.marshal.BytesType;
import org.apache.cassandra.utils.ByteBufferUtil;
+import static org.junit.Assert.assertEquals;
public class ColumnIdentifierTest
{
@@ -57,5 +58,23 @@
{
return v < 0 ? -1 : v > 0 ? 1 : 0;
}
+
+ @Test
+ public void testMaybeQuote()
+ {
+ String unquotable = "a";
+ assertEquals(unquotable, ColumnIdentifier.maybeQuote(unquotable));
+ unquotable = "z4";
+ assertEquals(unquotable, ColumnIdentifier.maybeQuote(unquotable));
+ unquotable = "m_4_";
+ assertEquals(unquotable, ColumnIdentifier.maybeQuote(unquotable));
+ unquotable = "f__";
+ assertEquals(unquotable, ColumnIdentifier.maybeQuote(unquotable));
+
+ assertEquals("\"A\"", ColumnIdentifier.maybeQuote("A"));
+ assertEquals("\"4b\"", ColumnIdentifier.maybeQuote("4b"));
+ assertEquals("\"\"\"\"", ColumnIdentifier.maybeQuote("\""));
+ assertEquals("\"\"\"a\"\"b\"\"\"", ColumnIdentifier.maybeQuote("\"a\"b\""));
+ }
}
diff --git a/test/unit/org/apache/cassandra/cql3/functions/CastFctsTest.java b/test/unit/org/apache/cassandra/cql3/functions/CastFctsTest.java
new file mode 100644
index 0000000..9b4b570
--- /dev/null
+++ b/test/unit/org/apache/cassandra/cql3/functions/CastFctsTest.java
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.cql3.functions;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Date;
+
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.serializers.SimpleDateSerializer;
+import org.apache.cassandra.utils.UUIDGen;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormat;
+import org.junit.Test;
+
+public class CastFctsTest extends CQLTester
+{
+ @Test
+ public void testInvalidQueries() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a int primary key, b text, c double)");
+
+ assertInvalidSyntaxMessage("no viable alternative at input '(' (... b, c) VALUES ([CAST](...)",
+ "INSERT INTO %s (a, b, c) VALUES (CAST(? AS int), ?, ?)", 1.6, "test", 6.3);
+
+ assertInvalidSyntaxMessage("no viable alternative at input '(' (..." + KEYSPACE + "." + currentTable()
+ + " SET c = [cast](...)",
+ "UPDATE %s SET c = cast(? as double) WHERE a = ?", 1, 1);
+
+ assertInvalidSyntaxMessage("no viable alternative at input '(' (...= ? WHERE a = [CAST] (...)",
+ "UPDATE %s SET c = ? WHERE a = CAST (? AS INT)", 1, 2.0);
+
+ assertInvalidSyntaxMessage("no viable alternative at input '(' (..." + KEYSPACE + "." + currentTable()
+ + " WHERE a = [CAST] (...)",
+ "DELETE FROM %s WHERE a = CAST (? AS INT)", 1, 2.0);
+
+ assertInvalidSyntaxMessage("no viable alternative at input '(' (..." + KEYSPACE + "." + currentTable()
+ + " WHERE a = [CAST] (...)",
+ "SELECT * FROM %s WHERE a = CAST (? AS INT)", 1, 2.0);
+
+ assertInvalidMessage("a cannot be cast to boolean", "SELECT CAST(a AS boolean) FROM %s");
+ }
+
+ @Test
+ public void testNumericCastsInSelectionClause() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a tinyint primary key,"
+ + " b smallint,"
+ + " c int,"
+ + " d bigint,"
+ + " e float,"
+ + " f double,"
+ + " g decimal,"
+ + " h varint,"
+ + " i int)");
+
+ execute("INSERT INTO %s (a, b, c, d, e, f, g, h) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
+ (byte) 1, (short) 2, 3, 4L, 5.2F, 6.3, BigDecimal.valueOf(6.3), BigInteger.valueOf(4));
+
+ assertColumnNames(execute("SELECT CAST(b AS int), CAST(c AS int), CAST(d AS double) FROM %s"),
+ "cast(b as int)",
+ "c",
+ "cast(d as double)");
+
+ assertRows(execute("SELECT CAST(a AS tinyint), " +
+ "CAST(b AS tinyint), " +
+ "CAST(c AS tinyint), " +
+ "CAST(d AS tinyint), " +
+ "CAST(e AS tinyint), " +
+ "CAST(f AS tinyint), " +
+ "CAST(g AS tinyint), " +
+ "CAST(h AS tinyint), " +
+ "CAST(i AS tinyint) FROM %s"),
+ row((byte) 1, (byte) 2, (byte) 3, (byte) 4L, (byte) 5, (byte) 6, (byte) 6, (byte) 4, null));
+
+ assertRows(execute("SELECT CAST(a AS smallint), " +
+ "CAST(b AS smallint), " +
+ "CAST(c AS smallint), " +
+ "CAST(d AS smallint), " +
+ "CAST(e AS smallint), " +
+ "CAST(f AS smallint), " +
+ "CAST(g AS smallint), " +
+ "CAST(h AS smallint), " +
+ "CAST(i AS smallint) FROM %s"),
+ row((short) 1, (short) 2, (short) 3, (short) 4L, (short) 5, (short) 6, (short) 6, (short) 4, null));
+
+ assertRows(execute("SELECT CAST(a AS int), " +
+ "CAST(b AS int), " +
+ "CAST(c AS int), " +
+ "CAST(d AS int), " +
+ "CAST(e AS int), " +
+ "CAST(f AS int), " +
+ "CAST(g AS int), " +
+ "CAST(h AS int), " +
+ "CAST(i AS int) FROM %s"),
+ row(1, 2, 3, 4, 5, 6, 6, 4, null));
+
+ assertRows(execute("SELECT CAST(a AS bigint), " +
+ "CAST(b AS bigint), " +
+ "CAST(c AS bigint), " +
+ "CAST(d AS bigint), " +
+ "CAST(e AS bigint), " +
+ "CAST(f AS bigint), " +
+ "CAST(g AS bigint), " +
+ "CAST(h AS bigint), " +
+ "CAST(i AS bigint) FROM %s"),
+ row(1L, 2L, 3L, 4L, 5L, 6L, 6L, 4L, null));
+
+ assertRows(execute("SELECT CAST(a AS float), " +
+ "CAST(b AS float), " +
+ "CAST(c AS float), " +
+ "CAST(d AS float), " +
+ "CAST(e AS float), " +
+ "CAST(f AS float), " +
+ "CAST(g AS float), " +
+ "CAST(h AS float), " +
+ "CAST(i AS float) FROM %s"),
+ row(1.0F, 2.0F, 3.0F, 4.0F, 5.2F, 6.3F, 6.3F, 4.0F, null));
+
+ assertRows(execute("SELECT CAST(a AS double), " +
+ "CAST(b AS double), " +
+ "CAST(c AS double), " +
+ "CAST(d AS double), " +
+ "CAST(e AS double), " +
+ "CAST(f AS double), " +
+ "CAST(g AS double), " +
+ "CAST(h AS double), " +
+ "CAST(i AS double) FROM %s"),
+ row(1.0, 2.0, 3.0, 4.0, (double) 5.2F, 6.3, 6.3, 4.0, null));
+
+ assertRows(execute("SELECT CAST(a AS decimal), " +
+ "CAST(b AS decimal), " +
+ "CAST(c AS decimal), " +
+ "CAST(d AS decimal), " +
+ "CAST(e AS decimal), " +
+ "CAST(f AS decimal), " +
+ "CAST(g AS decimal), " +
+ "CAST(h AS decimal), " +
+ "CAST(i AS decimal) FROM %s"),
+ row(BigDecimal.valueOf(1.0),
+ BigDecimal.valueOf(2.0),
+ BigDecimal.valueOf(3.0),
+ BigDecimal.valueOf(4.0),
+ BigDecimal.valueOf(5.2F),
+ BigDecimal.valueOf(6.3),
+ BigDecimal.valueOf(6.3),
+ BigDecimal.valueOf(4.0),
+ null));
+
+ assertRows(execute("SELECT CAST(a AS ascii), " +
+ "CAST(b AS ascii), " +
+ "CAST(c AS ascii), " +
+ "CAST(d AS ascii), " +
+ "CAST(e AS ascii), " +
+ "CAST(f AS ascii), " +
+ "CAST(g AS ascii), " +
+ "CAST(h AS ascii), " +
+ "CAST(i AS ascii) FROM %s"),
+ row("1",
+ "2",
+ "3",
+ "4",
+ "5.2",
+ "6.3",
+ "6.3",
+ "4",
+ null));
+
+ assertRows(execute("SELECT CAST(a AS text), " +
+ "CAST(b AS text), " +
+ "CAST(c AS text), " +
+ "CAST(d AS text), " +
+ "CAST(e AS text), " +
+ "CAST(f AS text), " +
+ "CAST(g AS text), " +
+ "CAST(h AS text), " +
+ "CAST(i AS text) FROM %s"),
+ row("1",
+ "2",
+ "3",
+ "4",
+ "5.2",
+ "6.3",
+ "6.3",
+ "4",
+ null));
+ }
+
+ @Test
+ public void testTimeCastsInSelectionClause() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a timeuuid primary key, b timestamp, c date, d time)");
+
+ DateTime dateTime = DateTimeFormat.forPattern("yyyy-MM-dd hh:mm:ss")
+ .withZone(DateTimeZone.UTC)
+ .parseDateTime("2015-05-21 11:03:02");
+
+ DateTime date = DateTimeFormat.forPattern("yyyy-MM-dd")
+ .withZone(DateTimeZone.UTC)
+ .parseDateTime("2015-05-21");
+
+ long timeInMillis = dateTime.getMillis();
+
+ execute("INSERT INTO %s (a, b, c, d) VALUES (?, '2015-05-21 11:03:02+00', '2015-05-21', '11:03:02')",
+ UUIDGen.getTimeUUID(timeInMillis));
+
+ assertRows(execute("SELECT CAST(a AS timestamp), " +
+ "CAST(b AS timestamp), " +
+ "CAST(c AS timestamp) FROM %s"),
+ row(new Date(dateTime.getMillis()), new Date(dateTime.getMillis()), new Date(date.getMillis())));
+
+ int timeInMillisToDay = SimpleDateSerializer.timeInMillisToDay(date.getMillis());
+ assertRows(execute("SELECT CAST(a AS date), " +
+ "CAST(b AS date), " +
+ "CAST(c AS date) FROM %s"),
+ row(timeInMillisToDay, timeInMillisToDay, timeInMillisToDay));
+
+ assertRows(execute("SELECT CAST(b AS text), " +
+ "CAST(c AS text), " +
+ "CAST(d AS text) FROM %s"),
+ row("2015-05-21T11:03:02.000Z", "2015-05-21", "11:03:02.000000000"));
+ }
+
+ @Test
+ public void testOtherTypeCastsInSelectionClause() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a ascii primary key,"
+ + " b inet,"
+ + " c boolean)");
+
+ execute("INSERT INTO %s (a, b, c) VALUES (?, '127.0.0.1', ?)",
+ "test", true);
+
+ assertRows(execute("SELECT CAST(a AS text), " +
+ "CAST(b AS text), " +
+ "CAST(c AS text) FROM %s"),
+ row("test", "127.0.0.1", "true"));
+ }
+
+ @Test
+ public void testCastsWithReverseOrder() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a int,"
+ + " b smallint,"
+ + " c double,"
+ + " primary key (a, b)) WITH CLUSTERING ORDER BY (b DESC);");
+
+ execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)",
+ 1, (short) 2, 6.3);
+
+ assertRows(execute("SELECT CAST(a AS tinyint), " +
+ "CAST(b AS tinyint), " +
+ "CAST(c AS tinyint) FROM %s"),
+ row((byte) 1, (byte) 2, (byte) 6));
+
+ assertRows(execute("SELECT CAST(CAST(a AS tinyint) AS smallint), " +
+ "CAST(CAST(b AS tinyint) AS smallint), " +
+ "CAST(CAST(c AS tinyint) AS smallint) FROM %s"),
+ row((short) 1, (short) 2, (short) 6));
+
+ assertRows(execute("SELECT CAST(CAST(CAST(a AS tinyint) AS double) AS text), " +
+ "CAST(CAST(CAST(b AS tinyint) AS double) AS text), " +
+ "CAST(CAST(CAST(c AS tinyint) AS double) AS text) FROM %s"),
+ row("1.0", "2.0", "6.0"));
+
+ String f = createFunction(KEYSPACE, "int",
+ "CREATE FUNCTION %s(val int) " +
+ "RETURNS NULL ON NULL INPUT " +
+ "RETURNS double " +
+ "LANGUAGE java " +
+ "AS 'return (double)val;'");
+
+ assertRows(execute("SELECT " + f + "(CAST(b AS int)) FROM %s"),
+ row((double) 2));
+
+ assertRows(execute("SELECT CAST(" + f + "(CAST(b AS int)) AS text) FROM %s"),
+ row("2.0"));
+ }
+
+ @Test
+ public void testCounterCastsInSelectionClause() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a int primary key, b counter)");
+
+ execute("UPDATE %s SET b = b + 2 WHERE a = 1");
+
+ assertRows(execute("SELECT CAST(b AS tinyint), " +
+ "CAST(b AS smallint), " +
+ "CAST(b AS int), " +
+ "CAST(b AS bigint), " +
+ "CAST(b AS float), " +
+ "CAST(b AS double), " +
+ "CAST(b AS decimal), " +
+ "CAST(b AS ascii), " +
+ "CAST(b AS text) FROM %s"),
+ row((byte) 2, (short) 2, 2, 2L, 2.0F, 2.0, BigDecimal.valueOf(2.0), "2", "2"));
+ }
+}
diff --git a/test/unit/org/apache/cassandra/cql3/statements/PropertyDefinitionsTest.java b/test/unit/org/apache/cassandra/cql3/statements/PropertyDefinitionsTest.java
new file mode 100644
index 0000000..417fcdc
--- /dev/null
+++ b/test/unit/org/apache/cassandra/cql3/statements/PropertyDefinitionsTest.java
@@ -0,0 +1,61 @@
+package org.apache.cassandra.cql3.statements;
+
+import org.junit.After;
+import org.junit.Test;
+import org.junit.Before;
+
+import static org.junit.Assert.assertEquals;
+
+public class PropertyDefinitionsTest {
+
+ PropertyDefinitions pd;
+
+ @Before
+ public void setUp()
+ {
+ pd = new PropertyDefinitions();
+ }
+
+ @After
+ public void clear()
+ {
+ pd = null;
+ }
+
+
+ @Test
+ public void testGetBooleanExistant()
+ {
+ String key = "one";
+ pd.addProperty(key, "1");
+ assertEquals(Boolean.TRUE, pd.getBoolean(key, null));
+
+ key = "TRUE";
+ pd.addProperty(key, "TrUe");
+ assertEquals(Boolean.TRUE, pd.getBoolean(key, null));
+
+ key = "YES";
+ pd.addProperty(key, "YeS");
+ assertEquals(Boolean.TRUE, pd.getBoolean(key, null));
+
+ key = "BAD_ONE";
+ pd.addProperty(key, " 1");
+ assertEquals(Boolean.FALSE, pd.getBoolean(key, null));
+
+ key = "BAD_TRUE";
+ pd.addProperty(key, "true ");
+ assertEquals(Boolean.FALSE, pd.getBoolean(key, null));
+
+ key = "BAD_YES";
+ pd.addProperty(key, "ye s");
+ assertEquals(Boolean.FALSE, pd.getBoolean(key, null));
+ }
+
+ @Test
+ public void testGetBooleanNonexistant()
+ {
+ assertEquals(Boolean.FALSE, pd.getBoolean("nonexistant", Boolean.FALSE));
+ assertEquals(Boolean.TRUE, pd.getBoolean("nonexistant", Boolean.TRUE));
+ }
+
+}
diff --git a/test/unit/org/apache/cassandra/cql3/validation/operations/AggregationTest.java b/test/unit/org/apache/cassandra/cql3/validation/operations/AggregationTest.java
index 221f48e..4667d5a 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/AggregationTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/AggregationTest.java
@@ -110,6 +110,7 @@
assertRows(execute("SELECT COUNT(b), count(c), count(e), count(f) FROM %s LIMIT 2"), row(4L, 3L, 3L, 3L));
assertRows(execute("SELECT COUNT(b), count(c), count(e), count(f) FROM %s WHERE a = 1 LIMIT 2"),
row(4L, 3L, 3L, 3L));
+ assertRows(execute("SELECT AVG(CAST(b AS double)) FROM %s"), row(11.0/4));
}
@Test
diff --git a/test/unit/org/apache/cassandra/db/CleanupTest.java b/test/unit/org/apache/cassandra/db/CleanupTest.java
index 167f3b0..c15bdb4 100644
--- a/test/unit/org/apache/cassandra/db/CleanupTest.java
+++ b/test/unit/org/apache/cassandra/db/CleanupTest.java
@@ -174,6 +174,32 @@
}
@Test
+ public void testuserDefinedCleanupWithNewToken() throws ExecutionException, InterruptedException, UnknownHostException
+ {
+ StorageService.instance.getTokenMetadata().clearUnsafe();
+
+ Keyspace keyspace = Keyspace.open(KEYSPACE1);
+ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_STANDARD1);
+
+ // insert data and verify we get it back w/ range query
+ fillCF(cfs, "val", LOOPS);
+
+ assertEquals(LOOPS, Util.getAll(Util.cmd(cfs).build()).size());
+ TokenMetadata tmd = StorageService.instance.getTokenMetadata();
+
+ byte[] tk1 = new byte[1], tk2 = new byte[1];
+ tk1[0] = 2;
+ tk2[0] = 1;
+ tmd.updateNormalToken(new BytesToken(tk1), InetAddress.getByName("127.0.0.1"));
+ tmd.updateNormalToken(new BytesToken(tk2), InetAddress.getByName("127.0.0.2"));
+
+ for(SSTableReader r: cfs.getLiveSSTables())
+ CompactionManager.instance.forceUserDefinedCleanup(r.getFilename());
+
+ assertEquals(0, Util.getAll(Util.cmd(cfs).build()).size());
+ }
+
+ @Test
public void testNeedsCleanup() throws Exception
{
// setup
diff --git a/test/unit/org/apache/cassandra/db/KeyspaceTest.java b/test/unit/org/apache/cassandra/db/KeyspaceTest.java
index d864fa3..c0c2753 100644
--- a/test/unit/org/apache/cassandra/db/KeyspaceTest.java
+++ b/test/unit/org/apache/cassandra/db/KeyspaceTest.java
@@ -135,7 +135,8 @@
ClusteringIndexSliceFilter filter = new ClusteringIndexSliceFilter(slices, reversed);
SinglePartitionReadCommand command = singlePartitionSlice(cfs, key, filter, limit);
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); PartitionIterator iterator = command.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ PartitionIterator iterator = command.executeInternal(executionController))
{
try (RowIterator rowIterator = iterator.next())
{
@@ -208,7 +209,8 @@
PartitionColumns columns = PartitionColumns.of(cfs.metadata.getColumnDefinition(new ColumnIdentifier("c", false)));
ClusteringIndexSliceFilter filter = new ClusteringIndexSliceFilter(Slices.ALL, false);
SinglePartitionReadCommand command = singlePartitionSlice(cfs, "0", filter, null);
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); PartitionIterator iterator = command.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ PartitionIterator iterator = command.executeInternal(executionController))
{
try (RowIterator rowIterator = iterator.next())
{
@@ -222,7 +224,8 @@
private static void assertRowsInResult(ColumnFamilyStore cfs, SinglePartitionReadCommand command, int ... columnValues)
{
- try (ReadOrderGroup orderGroup = command.startOrderGroup(); PartitionIterator iterator = command.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = command.executionController();
+ PartitionIterator iterator = command.executeInternal(executionController))
{
if (columnValues.length == 0)
{
diff --git a/test/unit/org/apache/cassandra/db/ReadCommandTest.java b/test/unit/org/apache/cassandra/db/ReadCommandTest.java
new file mode 100644
index 0000000..663080b
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/ReadCommandTest.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db;
+
+import java.util.List;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.db.partitions.FilteredPartition;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.utils.ByteBufferUtil;
+
+import static org.junit.Assert.assertEquals;
+
+public class ReadCommandTest
+{
+ private static final String KEYSPACE = "ReadCommandTest";
+ private static final String CF1 = "Standard1";
+ private static final String CF2 = "Standard2";
+
+ @BeforeClass
+ public static void defineSchema() throws ConfigurationException
+ {
+ CFMetaData metadata1 = SchemaLoader.standardCFMD(KEYSPACE, CF1);
+
+ CFMetaData metadata2 = CFMetaData.Builder.create(KEYSPACE, CF2)
+ .addPartitionKey("key", BytesType.instance)
+ .addClusteringColumn("col", AsciiType.instance)
+ .addRegularColumn("a", AsciiType.instance)
+ .addRegularColumn("b", AsciiType.instance).build();
+
+ SchemaLoader.prepareServer();
+ SchemaLoader.createKeyspace(KEYSPACE,
+ KeyspaceParams.simple(1),
+ metadata1,
+ metadata2);
+ }
+
+ @Test
+ public void testPartitionRangeAbort() throws Exception
+ {
+ ColumnFamilyStore cfs = Keyspace.open(KEYSPACE).getColumnFamilyStore(CF1);
+
+ new RowUpdateBuilder(cfs.metadata, 0, ByteBufferUtil.bytes("key1"))
+ .clustering("Column1")
+ .add("val", ByteBufferUtil.bytes("abcd"))
+ .build()
+ .apply();
+
+ cfs.forceBlockingFlush();
+
+ new RowUpdateBuilder(cfs.metadata, 0, ByteBufferUtil.bytes("key2"))
+ .clustering("Column1")
+ .add("val", ByteBufferUtil.bytes("abcd"))
+ .build()
+ .apply();
+
+ ReadCommand readCommand = Util.cmd(cfs).build();
+ assertEquals(2, Util.getAll(readCommand).size());
+
+ readCommand.abort();
+ assertEquals(0, Util.getAll(readCommand).size());
+ }
+
+ @Test
+ public void testSinglePartitionSliceAbort() throws Exception
+ {
+ ColumnFamilyStore cfs = Keyspace.open(KEYSPACE).getColumnFamilyStore(CF2);
+
+ cfs.truncateBlocking();
+
+ new RowUpdateBuilder(cfs.metadata, 0, ByteBufferUtil.bytes("key"))
+ .clustering("cc")
+ .add("a", ByteBufferUtil.bytes("abcd"))
+ .build()
+ .apply();
+
+ cfs.forceBlockingFlush();
+
+ new RowUpdateBuilder(cfs.metadata, 0, ByteBufferUtil.bytes("key"))
+ .clustering("dd")
+ .add("a", ByteBufferUtil.bytes("abcd"))
+ .build()
+ .apply();
+
+ ReadCommand readCommand = Util.cmd(cfs, Util.dk("key")).build();
+
+ List<FilteredPartition> partitions = Util.getAll(readCommand);
+ assertEquals(1, partitions.size());
+ assertEquals(2, partitions.get(0).rowCount());
+
+ readCommand.abort();
+ assertEquals(0, Util.getAll(readCommand).size());
+ }
+
+ @Test
+ public void testSinglePartitionNamesAbort() throws Exception
+ {
+ ColumnFamilyStore cfs = Keyspace.open(KEYSPACE).getColumnFamilyStore(CF2);
+
+ cfs.truncateBlocking();
+
+ new RowUpdateBuilder(cfs.metadata, 0, ByteBufferUtil.bytes("key"))
+ .clustering("cc")
+ .add("a", ByteBufferUtil.bytes("abcd"))
+ .build()
+ .apply();
+
+ cfs.forceBlockingFlush();
+
+ new RowUpdateBuilder(cfs.metadata, 0, ByteBufferUtil.bytes("key"))
+ .clustering("dd")
+ .add("a", ByteBufferUtil.bytes("abcd"))
+ .build()
+ .apply();
+
+ ReadCommand readCommand = Util.cmd(cfs, Util.dk("key")).includeRow("cc").includeRow("dd").build();
+
+ List<FilteredPartition> partitions = Util.getAll(readCommand);
+ assertEquals(1, partitions.size());
+ assertEquals(2, partitions.get(0).rowCount());
+
+ readCommand.abort();
+ assertEquals(0, Util.getAll(readCommand).size());
+ }
+}
diff --git a/test/unit/org/apache/cassandra/db/RecoveryManagerTest.java b/test/unit/org/apache/cassandra/db/RecoveryManagerTest.java
index baf9466..788757c 100644
--- a/test/unit/org/apache/cassandra/db/RecoveryManagerTest.java
+++ b/test/unit/org/apache/cassandra/db/RecoveryManagerTest.java
@@ -20,7 +20,12 @@
import java.io.IOException;
import java.util.Date;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -45,11 +50,71 @@
import org.apache.cassandra.db.commitlog.CommitLogArchiver;
import org.apache.cassandra.schema.KeyspaceParams;
import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.db.commitlog.CommitLogReplayer;
@RunWith(OrderedJUnit4ClassRunner.class)
public class RecoveryManagerTest
{
private static Logger logger = LoggerFactory.getLogger(RecoveryManagerTest.class);
+ static final Semaphore blocker = new Semaphore(0);
+ static final Semaphore blocked = new Semaphore(0);
+ static CommitLogReplayer.MutationInitiator originalInitiator = null;
+ static final CommitLogReplayer.MutationInitiator mockInitiator = new CommitLogReplayer.MutationInitiator()
+ {
+ @Override
+ protected Future<Integer> initiateMutation(final Mutation mutation,
+ final long segmentId,
+ final int serializedSize,
+ final long entryLocation,
+ final CommitLogReplayer clr)
+ {
+ final Future<Integer> toWrap = super.initiateMutation(mutation,
+ segmentId,
+ serializedSize,
+ entryLocation,
+ clr);
+ return new Future<Integer>()
+ {
+
+ @Override
+ public boolean cancel(boolean mayInterruptIfRunning)
+ {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public boolean isCancelled()
+ {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public boolean isDone()
+ {
+ return blocker.availablePermits() > 0 && toWrap.isDone();
+ }
+
+ @Override
+ public Integer get() throws InterruptedException, ExecutionException
+ {
+ System.out.println("Got blocker once");
+ blocked.release();
+ blocker.acquire();
+ return toWrap.get();
+ }
+
+ @Override
+ public Integer get(long timeout, TimeUnit unit)
+ throws InterruptedException, ExecutionException, TimeoutException
+ {
+ blocked.release();
+ blocker.tryAcquire(1, timeout, unit);
+ return toWrap.get(timeout, unit);
+ }
+
+ };
+ }
+ };
private static final String KEYSPACE1 = "RecoveryManagerTest1";
private static final String CF_STANDARD1 = "Standard1";
@@ -86,6 +151,78 @@
}
@Test
+ public void testRecoverBlocksOnBytesOutstanding() throws Exception
+ {
+ long originalMaxOutstanding = CommitLogReplayer.MAX_OUTSTANDING_REPLAY_BYTES;
+ CommitLogReplayer.MAX_OUTSTANDING_REPLAY_BYTES = 1;
+ CommitLogReplayer.MutationInitiator originalInitiator = CommitLogReplayer.mutationInitiator;
+ CommitLogReplayer.mutationInitiator = mockInitiator;
+ try
+ {
+ CommitLog.instance.resetUnsafe(true);
+ Keyspace keyspace1 = Keyspace.open(KEYSPACE1);
+ Keyspace keyspace2 = Keyspace.open(KEYSPACE2);
+
+ UnfilteredRowIterator upd1 = Util.apply(new RowUpdateBuilder(keyspace1.getColumnFamilyStore(CF_STANDARD1).metadata, 1L, 0, "keymulti")
+ .clustering("col1").add("val", "1")
+ .build());
+
+ UnfilteredRowIterator upd2 = Util.apply(new RowUpdateBuilder(keyspace2.getColumnFamilyStore(CF_STANDARD3).metadata, 1L, 0, "keymulti")
+ .clustering("col2").add("val", "1")
+ .build());
+
+ keyspace1.getColumnFamilyStore("Standard1").clearUnsafe();
+ keyspace2.getColumnFamilyStore("Standard3").clearUnsafe();
+
+ DecoratedKey dk = Util.dk("keymulti");
+ Assert.assertTrue(Util.getAllUnfiltered(Util.cmd(keyspace1.getColumnFamilyStore(CF_STANDARD1), dk).build()).isEmpty());
+ Assert.assertTrue(Util.getAllUnfiltered(Util.cmd(keyspace2.getColumnFamilyStore(CF_STANDARD3), dk).build()).isEmpty());
+
+ final AtomicReference<Throwable> err = new AtomicReference<Throwable>();
+ Thread t = new Thread() {
+ @Override
+ public void run()
+ {
+ try
+ {
+ CommitLog.instance.resetUnsafe(false); // disassociate segments from live CL
+ }
+ catch (Throwable t)
+ {
+ err.set(t);
+ }
+ }
+ };
+ t.start();
+ Assert.assertTrue(blocked.tryAcquire(1, 20, TimeUnit.SECONDS));
+ Thread.sleep(100);
+ Assert.assertTrue(t.isAlive());
+ blocker.release(Integer.MAX_VALUE);
+ t.join(20 * 1000);
+
+ if (err.get() != null)
+ throw new RuntimeException(err.get());
+
+ if (t.isAlive())
+ {
+ Throwable toPrint = new Throwable();
+ toPrint.setStackTrace(Thread.getAllStackTraces().get(t));
+ toPrint.printStackTrace(System.out);
+ }
+ Assert.assertFalse(t.isAlive());
+
+ Assert.assertTrue(Util.equal(upd1, Util.getOnlyPartitionUnfiltered(Util.cmd(keyspace1.getColumnFamilyStore(CF_STANDARD1), dk).build()).unfilteredIterator()));
+ Assert.assertTrue(Util.equal(upd2, Util.getOnlyPartitionUnfiltered(Util.cmd(keyspace2.getColumnFamilyStore(CF_STANDARD3), dk).build()).unfilteredIterator()));
+ }
+ finally
+ {
+ CommitLogReplayer.mutationInitiator = originalInitiator;
+ CommitLogReplayer.MAX_OUTSTANDING_REPLAY_BYTES = originalMaxOutstanding;
+ }
+ }
+
+
+ @Test
public void testOne() throws IOException
{
CommitLog.instance.resetUnsafe(true);
diff --git a/test/unit/org/apache/cassandra/db/RepairedDataTombstonesTest.java b/test/unit/org/apache/cassandra/db/RepairedDataTombstonesTest.java
index 3a74029..ad009a4 100644
--- a/test/unit/org/apache/cassandra/db/RepairedDataTombstonesTest.java
+++ b/test/unit/org/apache/cassandra/db/RepairedDataTombstonesTest.java
@@ -177,7 +177,8 @@
Thread.sleep(1000);
ReadCommand cmd = Util.cmd(getCurrentColumnFamilyStore()).build();
int partitionsFound = 0;
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iterator = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iterator = cmd.executeLocally(executionController))
{
while (iterator.hasNext())
{
@@ -232,7 +233,8 @@
{
ReadCommand cmd = Util.cmd(getCurrentColumnFamilyStore()).build();
int foundRows = 0;
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iterator = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iterator = cmd.executeLocally(executionController))
{
while (iterator.hasNext())
{
@@ -263,7 +265,8 @@
{
ReadCommand cmd = Util.cmd(getCurrentColumnFamilyStore(), Util.dk(ByteBufferUtil.bytes(key))).build();
int foundRows = 0;
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iterator = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iterator = cmd.executeLocally(executionController))
{
while (iterator.hasNext())
{
diff --git a/test/unit/org/apache/cassandra/db/RowIndexEntryTest.java b/test/unit/org/apache/cassandra/db/RowIndexEntryTest.java
index 62c88a0..0c7ee59 100644
--- a/test/unit/org/apache/cassandra/db/RowIndexEntryTest.java
+++ b/test/unit/org/apache/cassandra/db/RowIndexEntryTest.java
@@ -135,7 +135,7 @@
File tempFile = File.createTempFile("row_index_entry_test", null);
tempFile.deleteOnExit();
SequentialWriter writer = SequentialWriter.open(tempFile);
- ColumnIndex columnIndex = ColumnIndex.writeAndBuildIndex(partition.unfilteredIterator(), writer, header, BigFormat.latestVersion);
+ ColumnIndex columnIndex = ColumnIndex.writeAndBuildIndex(partition.unfilteredIterator(), writer, header, Collections.emptySet(), BigFormat.latestVersion);
RowIndexEntry<IndexHelper.IndexInfo> withIndex = RowIndexEntry.create(0xdeadbeef, DeletionTime.LIVE, columnIndex);
IndexHelper.IndexInfo.Serializer indexSerializer = new IndexHelper.IndexInfo.Serializer(cfs.metadata, BigFormat.latestVersion, header);
diff --git a/test/unit/org/apache/cassandra/db/ScrubTest.java b/test/unit/org/apache/cassandra/db/ScrubTest.java
index d5baec8..824c533 100644
--- a/test/unit/org/apache/cassandra/db/ScrubTest.java
+++ b/test/unit/org/apache/cassandra/db/ScrubTest.java
@@ -633,14 +633,14 @@
{
SerializationHeader header = new SerializationHeader(true, metadata, metadata.partitionColumns(), EncodingStats.NO_STATS);
MetadataCollector collector = new MetadataCollector(metadata.comparator).sstableLevel(0);
- return new TestMultiWriter(new TestWriter(descriptor, keyCount, 0, metadata, collector, header, txn));
+ return new TestMultiWriter(new TestWriter(descriptor, keyCount, 0, metadata, collector, header, txn), txn);
}
private static class TestMultiWriter extends SimpleSSTableMultiWriter
{
- TestMultiWriter(SSTableWriter writer)
+ TestMultiWriter(SSTableWriter writer, LifecycleTransaction txn)
{
- super(writer);
+ super(writer, txn);
}
}
@@ -652,7 +652,7 @@
TestWriter(Descriptor descriptor, long keyCount, long repairedAt, CFMetaData metadata,
MetadataCollector collector, SerializationHeader header, LifecycleTransaction txn)
{
- super(descriptor, keyCount, repairedAt, metadata, collector, header, txn);
+ super(descriptor, keyCount, repairedAt, metadata, collector, header, Collections.emptySet(), txn);
}
@Override
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexTest.java b/test/unit/org/apache/cassandra/db/SecondaryIndexTest.java
index bbccc48..ee01a47 100644
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexTest.java
+++ b/test/unit/org/apache/cassandra/db/SecondaryIndexTest.java
@@ -119,7 +119,8 @@
.build();
Index.Searcher searcher = cfs.indexManager.getBestIndexFor(rc).searcherFor(rc);
- try (ReadOrderGroup orderGroup = rc.startOrderGroup(); UnfilteredPartitionIterator pi = searcher.search(orderGroup))
+ try (ReadExecutionController executionController = rc.executionController();
+ UnfilteredPartitionIterator pi = searcher.search(executionController))
{
assertTrue(pi.hasNext());
pi.next().close();
@@ -508,8 +509,8 @@
if (count != 0)
assertNotNull(searcher);
- try (ReadOrderGroup orderGroup = rc.startOrderGroup();
- PartitionIterator iter = UnfilteredPartitionIterators.filter(searcher.search(orderGroup),
+ try (ReadExecutionController executionController = rc.executionController();
+ PartitionIterator iter = UnfilteredPartitionIterators.filter(searcher.search(executionController),
FBUtilities.nowInSeconds()))
{
assertEquals(count, Util.size(iter));
@@ -519,8 +520,8 @@
private void assertIndexCfsIsEmpty(ColumnFamilyStore indexCfs)
{
PartitionRangeReadCommand command = (PartitionRangeReadCommand)Util.cmd(indexCfs).build();
- try (ReadOrderGroup orderGroup = command.startOrderGroup();
- PartitionIterator iter = UnfilteredPartitionIterators.filter(Util.executeLocally(command, indexCfs, orderGroup),
+ try (ReadExecutionController controller = command.executionController();
+ PartitionIterator iter = UnfilteredPartitionIterators.filter(Util.executeLocally(command, indexCfs, controller),
FBUtilities.nowInSeconds()))
{
assertFalse(iter.hasNext());
diff --git a/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java b/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
index 9af6028..1e4f696 100644
--- a/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
+++ b/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
@@ -110,7 +110,7 @@
cmd = ReadCommand.legacyReadCommandSerializer.deserialize(in, MessagingService.VERSION_21);
logger.debug("ReadCommand: {}", cmd);
- UnfilteredPartitionIterator partitionIterator = cmd.executeLocally(ReadOrderGroup.emptyGroup());
+ UnfilteredPartitionIterator partitionIterator = cmd.executeLocally(ReadExecutionController.empty());
ReadResponse response = ReadResponse.createDataResponse(partitionIterator, cmd);
logger.debug("creating response: {}", response);
@@ -155,7 +155,7 @@
sliceFilter);
// check raw iterator for static cell
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator pi = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController(); UnfilteredPartitionIterator pi = cmd.executeLocally(executionController))
{
checkForS(pi);
}
@@ -166,7 +166,7 @@
ReadResponse dst;
// check (de)serialized iterator for memtable static cell
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator pi = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController(); UnfilteredPartitionIterator pi = cmd.executeLocally(executionController))
{
response = ReadResponse.createDataResponse(pi, cmd);
}
@@ -182,7 +182,7 @@
// check (de)serialized iterator for sstable static cell
Schema.instance.getColumnFamilyStoreInstance(cfm.cfId).forceBlockingFlush();
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator pi = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController(); UnfilteredPartitionIterator pi = cmd.executeLocally(executionController))
{
response = ReadResponse.createDataResponse(pi, cmd);
}
diff --git a/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java b/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
index 3c0098b..7fee251 100644
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java
@@ -18,6 +18,7 @@
package org.apache.cassandra.db.compaction;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import org.junit.Test;
@@ -215,9 +216,9 @@
public boolean verifyStrategies(CompactionStrategyManager manager, Class<? extends AbstractCompactionStrategy> expected)
{
boolean found = false;
- for (AbstractCompactionStrategy actualStrategy : manager.getStrategies())
+ for (List<AbstractCompactionStrategy> strategies : manager.getStrategies())
{
- if (!actualStrategy.getClass().equals(expected))
+ if (!strategies.stream().allMatch((strategy) -> strategy.getClass().equals(expected)))
return false;
found = true;
}
diff --git a/test/unit/org/apache/cassandra/db/compaction/LeveledCompactionStrategyTest.java b/test/unit/org/apache/cassandra/db/compaction/LeveledCompactionStrategyTest.java
index f2ddb00..1676896 100644
--- a/test/unit/org/apache/cassandra/db/compaction/LeveledCompactionStrategyTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/LeveledCompactionStrategyTest.java
@@ -56,6 +56,7 @@
import org.apache.cassandra.service.ActiveRepairService;
import org.apache.cassandra.utils.FBUtilities;
+import static java.util.Collections.singleton;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
@@ -122,10 +123,11 @@
}
waitForLeveling(cfs);
- CompactionStrategyManager strategy = cfs.getCompactionStrategyManager();
+ CompactionStrategyManager strategyManager = cfs.getCompactionStrategyManager();
// Checking we're not completely bad at math
- int l1Count = strategy.getSSTableCountPerLevel()[1];
- int l2Count = strategy.getSSTableCountPerLevel()[2];
+
+ int l1Count = strategyManager.getSSTableCountPerLevel()[1];
+ int l2Count = strategyManager.getSSTableCountPerLevel()[2];
if (l1Count == 0 || l2Count == 0)
{
logger.error("L1 or L2 has 0 sstables. Expected > 0 on both.");
@@ -177,10 +179,10 @@
}
waitForLeveling(cfs);
- CompactionStrategyManager strategy = cfs.getCompactionStrategyManager();
+ CompactionStrategyManager strategyManager = cfs.getCompactionStrategyManager();
// Checking we're not completely bad at math
- assertTrue(strategy.getSSTableCountPerLevel()[1] > 0);
- assertTrue(strategy.getSSTableCountPerLevel()[2] > 0);
+ assertTrue(strategyManager.getSSTableCountPerLevel()[1] > 0);
+ assertTrue(strategyManager.getSSTableCountPerLevel()[2] > 0);
Range<Token> range = new Range<>(Util.token(""), Util.token(""));
int gcBefore = keyspace.getColumnFamilyStore(CF_STANDARDDLEVELED).gcBefore(FBUtilities.nowInSeconds());
@@ -196,7 +198,7 @@
*/
private void waitForLeveling(ColumnFamilyStore cfs) throws InterruptedException
{
- CompactionStrategyManager strategy = cfs.getCompactionStrategyManager();
+ CompactionStrategyManager strategy = cfs.getCompactionStrategyManager();
// L0 is the lowest priority, so when that's done, we know everything is done
while (strategy.getSSTableCountPerLevel()[0] > 1)
Thread.sleep(100);
@@ -224,7 +226,7 @@
}
waitForLeveling(cfs);
- LeveledCompactionStrategy strategy = (LeveledCompactionStrategy) (cfs.getCompactionStrategyManager()).getStrategies().get(1);
+ LeveledCompactionStrategy strategy = (LeveledCompactionStrategy) cfs.getCompactionStrategyManager().getStrategies().get(1).get(0);
assert strategy.getLevelSize(1) > 0;
// get LeveledScanner for level 1 sstables
@@ -260,7 +262,7 @@
cfs.forceBlockingFlush();
}
cfs.forceBlockingFlush();
- LeveledCompactionStrategy strategy = (LeveledCompactionStrategy) ( cfs.getCompactionStrategyManager()).getStrategies().get(1);
+ LeveledCompactionStrategy strategy = (LeveledCompactionStrategy) cfs.getCompactionStrategyManager().getStrategies().get(1).get(0);
cfs.forceMajorCompaction();
for (SSTableReader s : cfs.getLiveSSTables())
@@ -306,14 +308,14 @@
while(CompactionManager.instance.isCompacting(Arrays.asList(cfs)))
Thread.sleep(100);
- CompactionStrategyManager strategy = cfs.getCompactionStrategyManager();
- List<AbstractCompactionStrategy> strategies = strategy.getStrategies();
- LeveledCompactionStrategy repaired = (LeveledCompactionStrategy) strategies.get(0);
- LeveledCompactionStrategy unrepaired = (LeveledCompactionStrategy) strategies.get(1);
+ CompactionStrategyManager manager = cfs.getCompactionStrategyManager();
+ List<List<AbstractCompactionStrategy>> strategies = manager.getStrategies();
+ LeveledCompactionStrategy repaired = (LeveledCompactionStrategy) strategies.get(0).get(0);
+ LeveledCompactionStrategy unrepaired = (LeveledCompactionStrategy) strategies.get(1).get(0);
assertEquals(0, repaired.manifest.getLevelCount() );
assertEquals(2, unrepaired.manifest.getLevelCount());
- assertTrue(strategy.getSSTableCountPerLevel()[1] > 0);
- assertTrue(strategy.getSSTableCountPerLevel()[2] > 0);
+ assertTrue(manager.getSSTableCountPerLevel()[1] > 0);
+ assertTrue(manager.getSSTableCountPerLevel()[2] > 0);
for (SSTableReader sstable : cfs.getLiveSSTables())
assertFalse(sstable.isRepaired());
@@ -331,7 +333,7 @@
sstable1.reloadSSTableMetadata();
assertTrue(sstable1.isRepaired());
- strategy.handleNotification(new SSTableRepairStatusChanged(Arrays.asList(sstable1)), this);
+ manager.handleNotification(new SSTableRepairStatusChanged(Arrays.asList(sstable1)), this);
int repairedSSTableCount = 0;
for (List<SSTableReader> level : repaired.manifest.generations)
@@ -343,7 +345,7 @@
assertFalse(unrepaired.manifest.generations[2].contains(sstable1));
unrepaired.removeSSTable(sstable2);
- strategy.handleNotification(new SSTableAddedNotification(Collections.singleton(sstable2)), this);
+ manager.handleNotification(new SSTableAddedNotification(singleton(sstable2)), this);
assertTrue(unrepaired.manifest.getLevel(1).contains(sstable2));
assertFalse(repaired.manifest.getLevel(1).contains(sstable2));
}
diff --git a/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java b/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
index b264553..8a06bd4 100644
--- a/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
@@ -18,33 +18,32 @@
*/
package org.apache.cassandra.db.compaction;
-import java.io.IOException;
-import java.util.Collections;
-import java.util.Set;
-
+import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.db.lifecycle.SSTableSet;
+import org.apache.cassandra.db.rows.UnfilteredRowIterator;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.junit.BeforeClass;
import com.google.common.collect.Multimap;
import com.google.common.collect.Sets;
-import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.apache.cassandra.OrderedJUnit4ClassRunner;
import org.apache.cassandra.SchemaLoader;
import org.apache.cassandra.Util;
-import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.*;
-import org.apache.cassandra.db.rows.BTreeRow;
import org.apache.cassandra.db.filter.ColumnFilter;
-import org.apache.cassandra.db.lifecycle.SSTableSet;
-import org.apache.cassandra.db.rows.UnfilteredRowIterator;
-import org.apache.cassandra.db.marshal.*;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.io.sstable.ISSTableScanner;
import org.apache.cassandra.schema.KeyspaceParams;
import org.apache.cassandra.tools.SSTableExpiredBlockers;
import org.apache.cassandra.utils.ByteBufferUtil;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Set;
+
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
@@ -68,7 +67,6 @@
.addRegularColumn("col2", AsciiType.instance)
.addRegularColumn("col3", AsciiType.instance)
.addRegularColumn("col7", AsciiType.instance)
- .addRegularColumn("col8", MapType.getInstance(AsciiType.instance, AsciiType.instance, true))
.addRegularColumn("shadow", AsciiType.instance)
.build().gcGraceSeconds(0));
}
@@ -143,23 +141,9 @@
@Test
public void testSimpleExpire() throws InterruptedException
{
- testSimpleExpire(false);
- }
-
- @Test
- public void testBug10944() throws InterruptedException
- {
- // Reproduction for CASSANDRA-10944 (at the time of the bug)
- testSimpleExpire(true);
- }
-
- public void testSimpleExpire(boolean force10944Bug) throws InterruptedException
- {
ColumnFamilyStore cfs = Keyspace.open(KEYSPACE1).getColumnFamilyStore("Standard1");
- cfs.truncateBlocking();
cfs.disableAutoCompaction();
- // To reproduce #10944, we need our gcBefore to be equal to the locaDeletionTime. A gcGrace of 1 will (almost always) give us that.
- cfs.metadata.gcGraceSeconds(force10944Bug ? 1 : 0);
+ cfs.metadata.gcGraceSeconds(0);
long timestamp = System.currentTimeMillis();
String key = "ttl";
new RowUpdateBuilder(cfs.metadata, timestamp, 1, key)
@@ -172,16 +156,12 @@
new RowUpdateBuilder(cfs.metadata, timestamp, 1, key)
.add("col2", ByteBufferUtil.EMPTY_BYTE_BUFFER)
- .addMapEntry("col8", "bar", "foo")
- .delete("col1")
.build()
.applyUnsafe();
cfs.forceBlockingFlush();
- // To reproduce #10944, we need to avoid the optimization that get rid of full sstable because everything
- // is known to be gcAble, so keep some data non-expiring in that case.
- new RowUpdateBuilder(cfs.metadata, timestamp, force10944Bug ? 0 : 1, key)
+ new RowUpdateBuilder(cfs.metadata, timestamp, 1, key)
.add("col3", ByteBufferUtil.EMPTY_BYTE_BUFFER)
.build()
.applyUnsafe();
@@ -198,14 +178,13 @@
Thread.sleep(2000); // wait for ttl to expire
assertEquals(4, cfs.getLiveSSTables().size());
cfs.enableAutoCompaction(true);
- assertEquals(force10944Bug ? 1 : 0, cfs.getLiveSSTables().size());
+ assertEquals(0, cfs.getLiveSSTables().size());
}
@Test
public void testNoExpire() throws InterruptedException, IOException
{
ColumnFamilyStore cfs = Keyspace.open(KEYSPACE1).getColumnFamilyStore("Standard1");
- cfs.truncateBlocking();
cfs.disableAutoCompaction();
cfs.metadata.gcGraceSeconds(0);
long timestamp = System.currentTimeMillis();
diff --git a/test/unit/org/apache/cassandra/db/lifecycle/RealTransactionsTest.java b/test/unit/org/apache/cassandra/db/lifecycle/RealTransactionsTest.java
index 4fbbb36..bab9c90 100644
--- a/test/unit/org/apache/cassandra/db/lifecycle/RealTransactionsTest.java
+++ b/test/unit/org/apache/cassandra/db/lifecycle/RealTransactionsTest.java
@@ -168,6 +168,7 @@
0,
0,
SerializationHeader.make(cfs.metadata, txn.originals()),
+ cfs.indexManager.listIndexes(),
txn));
while (ci.hasNext())
{
diff --git a/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java b/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java
index 7b9b19c..4f49389 100644
--- a/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java
+++ b/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java
@@ -297,7 +297,7 @@
Assert.assertTrue(tracker.getView().flushingMemtables.contains(prev2));
SSTableReader reader = MockSchema.sstable(0, 10, false, cfs);
- tracker.replaceFlushed(prev2, Collections.singleton(reader));
+ tracker.replaceFlushed(prev2, singleton(reader));
Assert.assertEquals(1, tracker.getView().sstables.size());
Assert.assertEquals(1, listener.received.size());
Assert.assertEquals(singleton(reader), ((SSTableAddedNotification) listener.received.get(0)).added);
@@ -314,13 +314,13 @@
tracker.markFlushing(prev1);
reader = MockSchema.sstable(0, 10, true, cfs);
cfs.invalidate(false);
- tracker.replaceFlushed(prev1, Collections.singleton(reader));
+ tracker.replaceFlushed(prev1, singleton(reader));
Assert.assertEquals(0, tracker.getView().sstables.size());
Assert.assertEquals(0, tracker.getView().flushingMemtables.size());
Assert.assertEquals(0, cfs.metric.liveDiskSpaceUsed.getCount());
Assert.assertEquals(3, listener.received.size());
Assert.assertEquals(singleton(reader), ((SSTableAddedNotification) listener.received.get(0)).added);
- Assert.assertTrue(listener.received.get(1) instanceof SSTableDeletingNotification);
+ Assert.assertTrue(listener.received.get(1) instanceof SSTableDeletingNotification);
Assert.assertEquals(1, ((SSTableListChangedNotification) listener.received.get(2)).removed.size());
DatabaseDescriptor.setIncrementalBackupsEnabled(backups);
}
diff --git a/test/unit/org/apache/cassandra/db/lifecycle/ViewTest.java b/test/unit/org/apache/cassandra/db/lifecycle/ViewTest.java
index 523c203..e787cc4 100644
--- a/test/unit/org/apache/cassandra/db/lifecycle/ViewTest.java
+++ b/test/unit/org/apache/cassandra/db/lifecycle/ViewTest.java
@@ -42,6 +42,7 @@
import static com.google.common.collect.ImmutableSet.copyOf;
import static com.google.common.collect.ImmutableSet.of;
import static com.google.common.collect.Iterables.concat;
+import static java.util.Collections.singleton;
import static org.apache.cassandra.db.lifecycle.Helpers.emptySet;
public class ViewTest
@@ -195,7 +196,7 @@
Assert.assertEquals(memtable3, cur.getCurrentMemtable());
SSTableReader sstable = MockSchema.sstable(1, cfs);
- cur = View.replaceFlushed(memtable1, Collections.singleton(sstable)).apply(cur);
+ cur = View.replaceFlushed(memtable1, singleton(sstable)).apply(cur);
Assert.assertEquals(0, cur.flushingMemtables.size());
Assert.assertEquals(1, cur.liveMemtables.size());
Assert.assertEquals(memtable3, cur.getCurrentMemtable());
diff --git a/test/unit/org/apache/cassandra/db/marshal/AbstractCompositeTypeTest.java b/test/unit/org/apache/cassandra/db/marshal/AbstractCompositeTypeTest.java
new file mode 100644
index 0000000..0e91532
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/marshal/AbstractCompositeTypeTest.java
@@ -0,0 +1,35 @@
+package org.apache.cassandra.db.marshal;
+
+import org.junit.Test;
+import static org.junit.Assert.assertEquals;
+
+public class AbstractCompositeTypeTest
+{
+
+ @Test
+ public void testEscape()
+ {
+ assertEquals("", AbstractCompositeType.escape(""));
+ assertEquals("Ab!CdXy \\Z123-345", AbstractCompositeType.escape("Ab!CdXy \\Z123-345"));
+ assertEquals("Ab!CdXy \\Z123-345!!", AbstractCompositeType.escape("Ab!CdXy \\Z123-345!"));
+ assertEquals("Ab!CdXy \\Z123-345\\!", AbstractCompositeType.escape("Ab!CdXy \\Z123-345\\"));
+
+ assertEquals("A\\:b!CdXy \\\\:Z123-345", AbstractCompositeType.escape("A:b!CdXy \\:Z123-345"));
+ assertEquals("A\\:b!CdXy \\\\:Z123-345!!", AbstractCompositeType.escape("A:b!CdXy \\:Z123-345!"));
+ assertEquals("A\\:b!CdXy \\\\:Z123-345\\!", AbstractCompositeType.escape("A:b!CdXy \\:Z123-345\\"));
+
+ }
+
+ @Test
+ public void testUnescape()
+ {
+ assertEquals("", AbstractCompositeType.escape(""));
+ assertEquals("Ab!CdXy \\Z123-345", AbstractCompositeType.unescape("Ab!CdXy \\Z123-345"));
+ assertEquals("Ab!CdXy \\Z123-345!", AbstractCompositeType.unescape("Ab!CdXy \\Z123-345!!"));
+ assertEquals("Ab!CdXy \\Z123-345\\", AbstractCompositeType.unescape("Ab!CdXy \\Z123-345\\!"));
+
+ assertEquals("A:b!CdXy \\:Z123-345", AbstractCompositeType.unescape("A\\:b!CdXy \\\\:Z123-345"));
+ assertEquals("A:b!CdXy \\:Z123-345!", AbstractCompositeType.unescape("A\\:b!CdXy \\\\:Z123-345!!"));
+ assertEquals("A:b!CdXy \\:Z123-345\\", AbstractCompositeType.unescape("A\\:b!CdXy \\\\:Z123-345\\!"));
+ }
+}
diff --git a/test/unit/org/apache/cassandra/db/monitoring/MonitoringTaskTest.java b/test/unit/org/apache/cassandra/db/monitoring/MonitoringTaskTest.java
new file mode 100644
index 0000000..4490519
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/monitoring/MonitoringTaskTest.java
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.monitoring;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+public class MonitoringTaskTest
+{
+ private static final long timeout = 100;
+ private static final long MAX_SPIN_TIME_NANOS = TimeUnit.SECONDS.toNanos(5);
+
+ public static final int REPORT_INTERVAL_MS = 600000; // long enough so that it won't check unless told to do so
+ public static final int MAX_TIMEDOUT_OPERATIONS = -1; // unlimited
+
+ @BeforeClass
+ public static void setup()
+ {
+ MonitoringTask.instance = MonitoringTask.make(REPORT_INTERVAL_MS, MAX_TIMEDOUT_OPERATIONS);
+ }
+
+ private static final class TestMonitor extends MonitorableImpl
+ {
+ private final String name;
+
+ TestMonitor(String name, ConstructionTime constructionTime, long timeout)
+ {
+ this.name = name;
+ setMonitoringTime(constructionTime, timeout);
+ }
+
+ public String name()
+ {
+ return name;
+ }
+
+ @Override
+ public String toString()
+ {
+ return name();
+ }
+ }
+
+ private static void waitForOperationsToComplete(Monitorable... operations) throws InterruptedException
+ {
+ waitForOperationsToComplete(Arrays.asList(operations));
+ }
+
+ private static void waitForOperationsToComplete(List<Monitorable> operations) throws InterruptedException
+ {
+ long timeout = operations.stream().map(Monitorable::timeout).reduce(0L, Long::max);
+ Thread.sleep(timeout * 2 + ApproximateTime.precision());
+
+ long start = System.nanoTime();
+ while(System.nanoTime() - start <= MAX_SPIN_TIME_NANOS)
+ {
+ long numInProgress = operations.stream().filter(Monitorable::isInProgress).count();
+ if (numInProgress == 0)
+ return;
+
+ Thread.yield();
+ }
+ }
+
+ @Test
+ public void testAbort() throws InterruptedException
+ {
+ Monitorable operation = new TestMonitor("Test abort", new ConstructionTime(System.currentTimeMillis()), timeout);
+ waitForOperationsToComplete(operation);
+
+ assertTrue(operation.isAborted());
+ assertFalse(operation.isCompleted());
+ assertEquals(1, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testAbortIdemPotent() throws InterruptedException
+ {
+ Monitorable operation = new TestMonitor("Test abort", new ConstructionTime(System.currentTimeMillis()), timeout);
+ waitForOperationsToComplete(operation);
+
+ assertTrue(operation.abort());
+
+ assertTrue(operation.isAborted());
+ assertFalse(operation.isCompleted());
+ assertEquals(1, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testAbortCrossNode() throws InterruptedException
+ {
+ Monitorable operation = new TestMonitor("Test for cross node", new ConstructionTime(System.currentTimeMillis(), true), timeout);
+ waitForOperationsToComplete(operation);
+
+ assertTrue(operation.isAborted());
+ assertFalse(operation.isCompleted());
+ assertEquals(1, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testComplete() throws InterruptedException
+ {
+ Monitorable operation = new TestMonitor("Test complete", new ConstructionTime(System.currentTimeMillis()), timeout);
+ operation.complete();
+ waitForOperationsToComplete(operation);
+
+ assertFalse(operation.isAborted());
+ assertTrue(operation.isCompleted());
+ assertEquals(0, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testCompleteIdemPotent() throws InterruptedException
+ {
+ Monitorable operation = new TestMonitor("Test complete", new ConstructionTime(System.currentTimeMillis()), timeout);
+ operation.complete();
+ waitForOperationsToComplete(operation);
+
+ assertTrue(operation.complete());
+
+ assertFalse(operation.isAborted());
+ assertTrue(operation.isCompleted());
+ assertEquals(0, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testReport() throws InterruptedException
+ {
+ Monitorable operation = new TestMonitor("Test report", new ConstructionTime(System.currentTimeMillis()), timeout);
+ waitForOperationsToComplete(operation);
+
+ assertTrue(operation.isAborted());
+ assertFalse(operation.isCompleted());
+ MonitoringTask.instance.logFailedOperations(ApproximateTime.currentTimeMillis());
+ assertEquals(0, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testRealScheduling() throws InterruptedException
+ {
+ MonitoringTask.instance = MonitoringTask.make(10, -1);
+ try
+ {
+ Monitorable operation = new TestMonitor("Test report", new ConstructionTime(System.currentTimeMillis()), timeout);
+ waitForOperationsToComplete(operation);
+
+ assertTrue(operation.isAborted());
+ assertFalse(operation.isCompleted());
+
+ Thread.sleep(ApproximateTime.precision() + 500);
+ assertEquals(0, MonitoringTask.instance.getFailedOperations().size());
+ }
+ finally
+ {
+ MonitoringTask.instance = MonitoringTask.make(REPORT_INTERVAL_MS, MAX_TIMEDOUT_OPERATIONS);
+ }
+ }
+
+ @Test
+ public void testMultipleThreads() throws InterruptedException
+ {
+ final int opCount = 50;
+ final ExecutorService executorService = Executors.newFixedThreadPool(20);
+ final List<Monitorable> operations = Collections.synchronizedList(new ArrayList<>(opCount));
+
+ for (int i = 0; i < opCount; i++)
+ {
+ executorService.submit(() ->
+ operations.add(new TestMonitor(UUID.randomUUID().toString(), new ConstructionTime(), timeout))
+ );
+ }
+
+ executorService.shutdown();
+ assertTrue(executorService.awaitTermination(30, TimeUnit.SECONDS));
+ assertEquals(opCount, operations.size());
+
+ waitForOperationsToComplete(operations);
+ assertEquals(opCount, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testZeroMaxTimedoutOperations() throws InterruptedException
+ {
+ doTestMaxTimedoutOperations(0, 1, 0);
+ }
+
+ @Test
+ public void testMaxTimedoutOperationsExceeded() throws InterruptedException
+ {
+ doTestMaxTimedoutOperations(5, 10, 6);
+ }
+
+ private static void doTestMaxTimedoutOperations(int maxTimedoutOperations,
+ int numThreads,
+ int numExpectedOperations) throws InterruptedException
+ {
+ MonitoringTask.instance = MonitoringTask.make(REPORT_INTERVAL_MS, maxTimedoutOperations);
+ try
+ {
+ final int threadCount = numThreads;
+ ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
+ final CountDownLatch finished = new CountDownLatch(threadCount);
+
+ for (int i = 0; i < threadCount; i++)
+ {
+ final String operationName = "Operation " + Integer.toString(i+1);
+ final int numTimes = i + 1;
+ executorService.submit(() -> {
+ try
+ {
+ for (int j = 0; j < numTimes; j++)
+ {
+ Monitorable operation = new TestMonitor(operationName,
+ new ConstructionTime(System.currentTimeMillis()),
+ timeout);
+ waitForOperationsToComplete(operation);
+ }
+ }
+ catch (InterruptedException e)
+ {
+ e.printStackTrace();
+ fail("Unexpected exception");
+ }
+ finally
+ {
+ finished.countDown();
+ }
+ });
+ }
+
+ finished.await();
+ assertEquals(0, executorService.shutdownNow().size());
+
+ List<String> failedOperations = MonitoringTask.instance.getFailedOperations();
+ assertEquals(numExpectedOperations, failedOperations.size());
+ if (numExpectedOperations > 0)
+ assertTrue(failedOperations.get(numExpectedOperations - 1).startsWith("..."));
+ }
+ finally
+ {
+ MonitoringTask.instance = MonitoringTask.make(REPORT_INTERVAL_MS, MAX_TIMEDOUT_OPERATIONS);
+ }
+ }
+
+ @Test
+ public void testMultipleThreadsSameName() throws InterruptedException
+ {
+ final int threadCount = 50;
+ final List<Monitorable> operations = new ArrayList<>(threadCount);
+ ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
+ final CountDownLatch finished = new CountDownLatch(threadCount);
+
+ for (int i = 0; i < threadCount; i++)
+ {
+ executorService.submit(() -> {
+ try
+ {
+ Monitorable operation = new TestMonitor("Test testMultipleThreadsSameName",
+ new ConstructionTime(System.currentTimeMillis()),
+ timeout);
+ operations.add(operation);
+ }
+ finally
+ {
+ finished.countDown();
+ }
+ });
+ }
+
+ finished.await();
+ assertEquals(0, executorService.shutdownNow().size());
+
+ waitForOperationsToComplete(operations);
+ //MonitoringTask.instance.checkFailedOperations(ApproximateTime.currentTimeMillis());
+ assertEquals(1, MonitoringTask.instance.getFailedOperations().size());
+ }
+
+ @Test
+ public void testMultipleThreadsNoFailedOps() throws InterruptedException
+ {
+ final int threadCount = 50;
+ final List<Monitorable> operations = new ArrayList<>(threadCount);
+ ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
+ final CountDownLatch finished = new CountDownLatch(threadCount);
+
+ for (int i = 0; i < threadCount; i++)
+ {
+ executorService.submit(() -> {
+ try
+ {
+ Monitorable operation = new TestMonitor("Test thread " + Thread.currentThread().getName(),
+ new ConstructionTime(System.currentTimeMillis()),
+ timeout);
+ operations.add(operation);
+ operation.complete();
+ }
+ finally
+ {
+ finished.countDown();
+ }
+ });
+ }
+
+ finished.await();
+ assertEquals(0, executorService.shutdownNow().size());
+
+ waitForOperationsToComplete(operations);
+ assertEquals(0, MonitoringTask.instance.getFailedOperations().size());
+ }
+}
diff --git a/test/unit/org/apache/cassandra/dht/BootStrapperTest.java b/test/unit/org/apache/cassandra/dht/BootStrapperTest.java
index 9b1fa01..8454ec1 100644
--- a/test/unit/org/apache/cassandra/dht/BootStrapperTest.java
+++ b/test/unit/org/apache/cassandra/dht/BootStrapperTest.java
@@ -93,7 +93,7 @@
InetAddress myEndpoint = InetAddress.getByName("127.0.0.1");
assertEquals(numOldNodes, tmd.sortedTokens().size());
- RangeStreamer s = new RangeStreamer(tmd, null, myEndpoint, "Bootstrap", true, DatabaseDescriptor.getEndpointSnitch(), new StreamStateStore());
+ RangeStreamer s = new RangeStreamer(tmd, null, myEndpoint, "Bootstrap", true, DatabaseDescriptor.getEndpointSnitch(), new StreamStateStore(), false);
IFailureDetector mockFailureDetector = new IFailureDetector()
{
public boolean isAlive(InetAddress ep)
diff --git a/test/unit/org/apache/cassandra/dht/LengthPartitioner.java b/test/unit/org/apache/cassandra/dht/LengthPartitioner.java
index 9cefbf2..e2202fe 100644
--- a/test/unit/org/apache/cassandra/dht/LengthPartitioner.java
+++ b/test/unit/org/apache/cassandra/dht/LengthPartitioner.java
@@ -61,6 +61,12 @@
return MINIMUM;
}
+ @Override
+ public Token getMaximumToken()
+ {
+ return null;
+ }
+
public BigIntegerToken getRandomToken()
{
return new BigIntegerToken(BigInteger.valueOf(new Random().nextInt(15)));
diff --git a/test/unit/org/apache/cassandra/dht/SplitterTest.java b/test/unit/org/apache/cassandra/dht/SplitterTest.java
new file mode 100644
index 0000000..751a7d7
--- /dev/null
+++ b/test/unit/org/apache/cassandra/dht/SplitterTest.java
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.dht;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+import java.util.Set;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+public class SplitterTest
+{
+
+ @Test
+ public void randomSplitTestNoVNodesRandomPartitioner()
+ {
+ randomSplitTestNoVNodes(new RandomPartitioner());
+ }
+
+ @Test
+ public void randomSplitTestNoVNodesMurmur3Partitioner()
+ {
+ randomSplitTestNoVNodes(new Murmur3Partitioner());
+ }
+
+ @Test
+ public void randomSplitTestVNodesRandomPartitioner()
+ {
+ randomSplitTestVNodes(new RandomPartitioner());
+ }
+ @Test
+ public void randomSplitTestVNodesMurmur3Partitioner()
+ {
+ randomSplitTestVNodes(new Murmur3Partitioner());
+ }
+
+ public void randomSplitTestNoVNodes(IPartitioner partitioner)
+ {
+ Splitter splitter = partitioner.splitter().get();
+ Random r = new Random();
+ for (int i = 0; i < 10000; i++)
+ {
+ List<Range<Token>> localRanges = generateLocalRanges(1, r.nextInt(4)+1, splitter, r, partitioner instanceof RandomPartitioner);
+ List<Token> boundaries = splitter.splitOwnedRanges(r.nextInt(9) + 1, localRanges, false);
+ assertTrue("boundaries = "+boundaries+" ranges = "+localRanges, assertRangeSizeEqual(localRanges, boundaries, partitioner, splitter, true));
+ }
+ }
+
+ public void randomSplitTestVNodes(IPartitioner partitioner)
+ {
+ Splitter splitter = partitioner.splitter().get();
+ Random r = new Random();
+ for (int i = 0; i < 10000; i++)
+ {
+ // we need many tokens to be able to split evenly over the disks
+ int numTokens = 172 + r.nextInt(128);
+ int rf = r.nextInt(4) + 2;
+ int parts = r.nextInt(5)+1;
+ List<Range<Token>> localRanges = generateLocalRanges(numTokens, rf, splitter, r, partitioner instanceof RandomPartitioner);
+ List<Token> boundaries = splitter.splitOwnedRanges(parts, localRanges, true);
+ if (!assertRangeSizeEqual(localRanges, boundaries, partitioner, splitter, false))
+ fail(String.format("Could not split %d tokens with rf=%d into %d parts (localRanges=%s, boundaries=%s)", numTokens, rf, parts, localRanges, boundaries));
+ }
+ }
+
+ private boolean assertRangeSizeEqual(List<Range<Token>> localRanges, List<Token> tokens, IPartitioner partitioner, Splitter splitter, boolean splitIndividualRanges)
+ {
+ Token start = partitioner.getMinimumToken();
+ List<BigInteger> splits = new ArrayList<>();
+
+ for (int i = 0; i < tokens.size(); i++)
+ {
+ Token end = i == tokens.size() - 1 ? partitioner.getMaximumToken() : tokens.get(i);
+ splits.add(sumOwnedBetween(localRanges, start, end, splitter, splitIndividualRanges));
+ start = end;
+ }
+ // when we dont need to keep around full ranges, the difference is small between the partitions
+ BigDecimal delta = splitIndividualRanges ? BigDecimal.valueOf(0.001) : BigDecimal.valueOf(0.2);
+ boolean allBalanced = true;
+ for (BigInteger b : splits)
+ {
+ for (BigInteger i : splits)
+ {
+ BigDecimal bdb = new BigDecimal(b);
+ BigDecimal bdi = new BigDecimal(i);
+ BigDecimal q = bdb.divide(bdi, 2, BigDecimal.ROUND_HALF_DOWN);
+ if (q.compareTo(BigDecimal.ONE.add(delta)) > 0 || q.compareTo(BigDecimal.ONE.subtract(delta)) < 0)
+ allBalanced = false;
+ }
+ }
+ return allBalanced;
+ }
+
+ private BigInteger sumOwnedBetween(List<Range<Token>> localRanges, Token start, Token end, Splitter splitter, boolean splitIndividualRanges)
+ {
+ BigInteger sum = BigInteger.ZERO;
+ for (Range<Token> range : localRanges)
+ {
+ if (splitIndividualRanges)
+ {
+ Set<Range<Token>> intersections = new Range<>(start, end).intersectionWith(range);
+ for (Range<Token> intersection : intersections)
+ sum = sum.add(splitter.valueForToken(intersection.right).subtract(splitter.valueForToken(intersection.left)));
+ }
+ else
+ {
+ if (new Range<>(start, end).contains(range.left))
+ sum = sum.add(splitter.valueForToken(range.right).subtract(splitter.valueForToken(range.left)));
+ }
+ }
+ return sum;
+ }
+
+ private List<Range<Token>> generateLocalRanges(int numTokens, int rf, Splitter splitter, Random r, boolean randomPartitioner)
+ {
+ int localTokens = numTokens * rf;
+ List<Token> randomTokens = new ArrayList<>();
+
+ for (int i = 0; i < localTokens * 2; i++)
+ {
+ Token t = splitter.tokenForValue(randomPartitioner ? new BigInteger(127, r) : BigInteger.valueOf(r.nextLong()));
+ randomTokens.add(t);
+ }
+
+ Collections.sort(randomTokens);
+
+ List<Range<Token>> localRanges = new ArrayList<>(localTokens);
+ for (int i = 0; i < randomTokens.size() - 1; i++)
+ {
+ assert randomTokens.get(i).compareTo(randomTokens.get(i+1)) < 0;
+ localRanges.add(new Range<>(randomTokens.get(i), randomTokens.get(i+1)));
+ i++;
+ }
+ return localRanges;
+ }
+}
diff --git a/test/unit/org/apache/cassandra/hints/HintTest.java b/test/unit/org/apache/cassandra/hints/HintTest.java
index 4c7ec70..4d6ed15 100644
--- a/test/unit/org/apache/cassandra/hints/HintTest.java
+++ b/test/unit/org/apache/cassandra/hints/HintTest.java
@@ -222,8 +222,8 @@
{
ReadCommand cmd = cmd(key, table);
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup();
- PartitionIterator iterator = cmd.executeInternal(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ PartitionIterator iterator = cmd.executeInternal(executionController))
{
assertFalse(iterator.hasNext());
}
diff --git a/test/unit/org/apache/cassandra/index/CustomIndexTest.java b/test/unit/org/apache/cassandra/index/CustomIndexTest.java
index 3bfb6a5..c0349a1 100644
--- a/test/unit/org/apache/cassandra/index/CustomIndexTest.java
+++ b/test/unit/org/apache/cassandra/index/CustomIndexTest.java
@@ -19,7 +19,7 @@
import org.apache.cassandra.cql3.statements.SelectStatement;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.ReadCommand;
-import org.apache.cassandra.db.ReadOrderGroup;
+import org.apache.cassandra.db.ReadExecutionController;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.UTF8Type;
@@ -492,8 +492,8 @@
assertEquals(0, index.partitionDeletions.size());
ReadCommand cmd = Util.cmd(cfs, 0).build();
- try (ReadOrderGroup orderGroup = cmd.startOrderGroup();
- UnfilteredPartitionIterator iterator = cmd.executeLocally(orderGroup))
+ try (ReadExecutionController executionController = cmd.executionController();
+ UnfilteredPartitionIterator iterator = cmd.executeLocally(executionController))
{
assertTrue(iterator.hasNext());
cfs.indexManager.deletePartition(iterator.next(), FBUtilities.nowInSeconds());
diff --git a/test/unit/org/apache/cassandra/index/StubIndex.java b/test/unit/org/apache/cassandra/index/StubIndex.java
index 28ea097..5f9d1f3 100644
--- a/test/unit/org/apache/cassandra/index/StubIndex.java
+++ b/test/unit/org/apache/cassandra/index/StubIndex.java
@@ -194,7 +194,7 @@
public Searcher searcherFor(final ReadCommand command)
{
- return (orderGroup) -> Util.executeLocally((PartitionRangeReadCommand)command, baseCfs, orderGroup);
+ return (controller) -> Util.executeLocally((PartitionRangeReadCommand)command, baseCfs, controller);
}
public BiFunction<PartitionIterator, ReadCommand, PartitionIterator> postProcessorFor(ReadCommand readCommand)
diff --git a/test/unit/org/apache/cassandra/index/internal/CassandraIndexTest.java b/test/unit/org/apache/cassandra/index/internal/CassandraIndexTest.java
index c6783cc..64f11d1 100644
--- a/test/unit/org/apache/cassandra/index/internal/CassandraIndexTest.java
+++ b/test/unit/org/apache/cassandra/index/internal/CassandraIndexTest.java
@@ -504,8 +504,8 @@
indexKey,
ColumnFilter.all(indexCfs.metadata),
filter);
- try (ReadOrderGroup orderGroup = ReadOrderGroup.forCommand(command);
- UnfilteredRowIterator iter = command.queryMemtableAndDisk(indexCfs, orderGroup.indexReadOpOrderGroup()))
+ try (ReadExecutionController executionController = ReadExecutionController.forCommand(command);
+ UnfilteredRowIterator iter = command.queryMemtableAndDisk(indexCfs, executionController.indexReadOpOrderGroup()))
{
while( iter.hasNext())
{
diff --git a/test/unit/org/apache/cassandra/index/internal/CustomCassandraIndex.java b/test/unit/org/apache/cassandra/index/internal/CustomCassandraIndex.java
index a30cf4e..4045b6a 100644
--- a/test/unit/org/apache/cassandra/index/internal/CustomCassandraIndex.java
+++ b/test/unit/org/apache/cassandra/index/internal/CustomCassandraIndex.java
@@ -620,9 +620,9 @@
metadata.name,
getSSTableNames(sstables));
- SecondaryIndexBuilder builder = new SecondaryIndexBuilder(baseCfs,
- Collections.singleton(this),
- new ReducingKeyIterator(sstables));
+ SecondaryIndexBuilder builder = new CollatedViewIndexBuilder(baseCfs,
+ Collections.singleton(this),
+ new ReducingKeyIterator(sstables));
Future<?> future = CompactionManager.instance.submitIndexBuild(builder);
FBUtilities.waitOnFuture(future);
indexCfs.forceBlockingFlush();
diff --git a/test/unit/org/apache/cassandra/io/sstable/SSTableReaderTest.java b/test/unit/org/apache/cassandra/io/sstable/SSTableReaderTest.java
index c7f3c36..4546867 100644
--- a/test/unit/org/apache/cassandra/io/sstable/SSTableReaderTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/SSTableReaderTest.java
@@ -47,7 +47,6 @@
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.util.FileDataInput;
import org.apache.cassandra.io.util.MmappedRegions;
-import org.apache.cassandra.io.util.MmappedSegmentedFile;
import org.apache.cassandra.io.util.SegmentedFile;
import org.apache.cassandra.schema.CachingParams;
import org.apache.cassandra.schema.KeyspaceParams;
@@ -549,9 +548,9 @@
.build();
Index.Searcher searcher = indexedCFS.indexManager.getBestIndexFor(rc).searcherFor(rc);
assertNotNull(searcher);
- try (ReadOrderGroup orderGroup = ReadOrderGroup.forCommand(rc))
+ try (ReadExecutionController executionController = ReadExecutionController.forCommand(rc))
{
- assertEquals(1, Util.size(UnfilteredPartitionIterators.filter(searcher.search(orderGroup), rc.nowInSec())));
+ assertEquals(1, Util.size(UnfilteredPartitionIterators.filter(searcher.search(executionController), rc.nowInSec())));
}
}
diff --git a/test/unit/org/apache/cassandra/io/sstable/SSTableWriterTestBase.java b/test/unit/org/apache/cassandra/io/sstable/SSTableWriterTestBase.java
index 0af743d..70f154d 100644
--- a/test/unit/org/apache/cassandra/io/sstable/SSTableWriterTestBase.java
+++ b/test/unit/org/apache/cassandra/io/sstable/SSTableWriterTestBase.java
@@ -131,7 +131,7 @@
public static SSTableWriter getWriter(ColumnFamilyStore cfs, File directory, LifecycleTransaction txn)
{
String filename = cfs.getSSTablePath(directory);
- return SSTableWriter.create(filename, 0, 0, new SerializationHeader(true, cfs.metadata, cfs.metadata.partitionColumns(), EncodingStats.NO_STATS), txn);
+ return SSTableWriter.create(filename, 0, 0, new SerializationHeader(true, cfs.metadata, cfs.metadata.partitionColumns(), EncodingStats.NO_STATS), cfs.indexManager.listIndexes(), txn);
}
public static ByteBuffer random(int i, int size)
diff --git a/test/unit/org/apache/cassandra/io/sstable/format/SSTableFlushObserverTest.java b/test/unit/org/apache/cassandra/io/sstable/format/SSTableFlushObserverTest.java
new file mode 100644
index 0000000..29ad387
--- /dev/null
+++ b/test/unit/org/apache/cassandra/io/sstable/format/SSTableFlushObserverTest.java
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable.format;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+
+import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.Clustering;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.DeletionTime;
+import org.apache.cassandra.db.SerializationHeader;
+import org.apache.cassandra.db.compaction.OperationType;
+import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
+import org.apache.cassandra.db.marshal.Int32Type;
+import org.apache.cassandra.db.marshal.LongType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.db.rows.*;
+import org.apache.cassandra.io.FSReadError;
+import org.apache.cassandra.io.FSWriteError;
+import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.format.big.BigTableWriter;
+import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
+import org.apache.cassandra.io.util.FileDataInput;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.Pair;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+
+import junit.framework.Assert;
+import org.junit.Test;
+
+public class SSTableFlushObserverTest
+{
+ private static final String KS_NAME = "test";
+ private static final String CF_NAME = "flush_observer";
+
+ @Test
+ public void testFlushObserver()
+ {
+ CFMetaData cfm = CFMetaData.Builder.create(KS_NAME, CF_NAME)
+ .addPartitionKey("id", UTF8Type.instance)
+ .addRegularColumn("first_name", UTF8Type.instance)
+ .addRegularColumn("age", Int32Type.instance)
+ .addRegularColumn("height", LongType.instance)
+ .build();
+
+ LifecycleTransaction transaction = LifecycleTransaction.offline(OperationType.COMPACTION);
+ FlushObserver observer = new FlushObserver();
+
+ String sstableDirectory = DatabaseDescriptor.getAllDataFileLocations()[0];
+ File directory = new File(sstableDirectory + File.pathSeparator + KS_NAME + File.pathSeparator + CF_NAME);
+ directory.deleteOnExit();
+
+ if (!directory.exists() && !directory.mkdirs())
+ throw new FSWriteError(new IOException("failed to create tmp directory"), directory.getAbsolutePath());
+
+ SSTableFormat.Type sstableFormat = DatabaseDescriptor.getSSTableFormat();
+
+ BigTableWriter writer = new BigTableWriter(new Descriptor(sstableFormat.info.getLatestVersion().version,
+ directory,
+ KS_NAME, CF_NAME,
+ 0,
+ sstableFormat),
+ 10L, 0L, cfm,
+ new MetadataCollector(cfm.comparator).sstableLevel(0),
+ new SerializationHeader(true, cfm, cfm.partitionColumns(), EncodingStats.NO_STATS),
+ Collections.singletonList(observer),
+ transaction);
+
+ SSTableReader reader = null;
+ Multimap<ByteBuffer, Cell> expected = ArrayListMultimap.create();
+
+ try
+ {
+ final long now = System.currentTimeMillis();
+
+ ByteBuffer key = UTF8Type.instance.fromString("key1");
+ expected.putAll(key, Arrays.asList(BufferCell.live(cfm, getColumn(cfm, "first_name"), now,UTF8Type.instance.fromString("jack")),
+ BufferCell.live(cfm, getColumn(cfm, "age"), now, Int32Type.instance.decompose(27)),
+ BufferCell.live(cfm, getColumn(cfm, "height"), now, LongType.instance.decompose(183L))));
+
+ writer.append(new RowIterator(cfm, key.duplicate(), Collections.singletonList(buildRow(expected.get(key)))));
+
+ key = UTF8Type.instance.fromString("key2");
+ expected.putAll(key, Arrays.asList(BufferCell.live(cfm, getColumn(cfm, "first_name"), now,UTF8Type.instance.fromString("jim")),
+ BufferCell.live(cfm, getColumn(cfm, "age"), now, Int32Type.instance.decompose(30)),
+ BufferCell.live(cfm, getColumn(cfm, "height"), now, LongType.instance.decompose(180L))));
+
+ writer.append(new RowIterator(cfm, key, Collections.singletonList(buildRow(expected.get(key)))));
+
+ key = UTF8Type.instance.fromString("key3");
+ expected.putAll(key, Arrays.asList(BufferCell.live(cfm, getColumn(cfm, "first_name"), now,UTF8Type.instance.fromString("ken")),
+ BufferCell.live(cfm, getColumn(cfm, "age"), now, Int32Type.instance.decompose(30)),
+ BufferCell.live(cfm, getColumn(cfm, "height"), now, LongType.instance.decompose(178L))));
+
+ writer.append(new RowIterator(cfm, key, Collections.singletonList(buildRow(expected.get(key)))));
+
+ reader = writer.finish(true);
+ }
+ finally
+ {
+ FileUtils.closeQuietly(writer);
+ }
+
+ Assert.assertTrue(observer.isComplete);
+ Assert.assertEquals(expected.size(), observer.rows.size());
+
+ for (Pair<ByteBuffer, Long> e : observer.rows.keySet())
+ {
+ ByteBuffer key = e.left;
+ Long indexPosition = e.right;
+
+ try (FileDataInput index = reader.ifile.createReader(indexPosition))
+ {
+ ByteBuffer indexKey = ByteBufferUtil.readWithShortLength(index);
+ Assert.assertEquals(0, UTF8Type.instance.compare(key, indexKey));
+ }
+ catch (IOException ex)
+ {
+ throw new FSReadError(ex, reader.getIndexFilename());
+ }
+
+ Assert.assertEquals(expected.get(key), observer.rows.get(e));
+ }
+ }
+
+ private static class RowIterator extends AbstractUnfilteredRowIterator
+ {
+ private final Iterator<Unfiltered> rows;
+
+ public RowIterator(CFMetaData cfm, ByteBuffer key, Collection<Unfiltered> content)
+ {
+ super(cfm,
+ DatabaseDescriptor.getPartitioner().decorateKey(key),
+ DeletionTime.LIVE,
+ cfm.partitionColumns(),
+ BTreeRow.emptyRow(Clustering.STATIC_CLUSTERING),
+ false,
+ EncodingStats.NO_STATS);
+
+ rows = content.iterator();
+ }
+
+ @Override
+ protected Unfiltered computeNext()
+ {
+ return rows.hasNext() ? rows.next() : endOfData();
+ }
+ }
+
+ private static class FlushObserver implements SSTableFlushObserver
+ {
+ private final Multimap<Pair<ByteBuffer, Long>, Cell> rows = ArrayListMultimap.create();
+ private Pair<ByteBuffer, Long> currentKey;
+ private boolean isComplete;
+
+ @Override
+ public void begin()
+ {}
+
+ @Override
+ public void startPartition(DecoratedKey key, long indexPosition)
+ {
+ currentKey = Pair.create(key.getKey(), indexPosition);
+ }
+
+ @Override
+ public void nextCell(ColumnData cell)
+ {
+ rows.put(currentKey, (Cell) cell);
+ }
+
+ @Override
+ public void complete()
+ {
+ isComplete = true;
+ }
+ }
+
+ private static Row buildRow(Collection<Cell> cells)
+ {
+ Row.Builder rowBuilder = BTreeRow.sortedBuilder();
+ rowBuilder.newRow(Clustering.EMPTY);
+ cells.forEach(rowBuilder::addCell);
+ return rowBuilder.build();
+ }
+
+ private static ColumnDefinition getColumn(CFMetaData cfm, String name)
+ {
+ return cfm.getColumnDefinition(UTF8Type.instance.fromString(name));
+ }
+}
diff --git a/test/unit/org/apache/cassandra/io/util/FileUtilsTest.java b/test/unit/org/apache/cassandra/io/util/FileUtilsTest.java
index 7110504..ee33107 100644
--- a/test/unit/org/apache/cassandra/io/util/FileUtilsTest.java
+++ b/test/unit/org/apache/cassandra/io/util/FileUtilsTest.java
@@ -20,11 +20,20 @@
import java.io.File;
import java.io.IOException;
+import java.io.RandomAccessFile;
import java.nio.charset.Charset;
import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Arrays;
import org.junit.Test;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.schema.SchemaKeyspace;
+import org.apache.cassandra.utils.FBUtilities;
+
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
@@ -52,4 +61,47 @@
assertEquals(0, b.length);
}
+ @Test
+ public void testFolderSize() throws Exception
+ {
+ File folder = createFolder(Paths.get(DatabaseDescriptor.getAllDataFileLocations()[0], "testFolderSize"));
+ folder.deleteOnExit();
+
+ File childFolder = createFolder(Paths.get(folder.getPath(), "child"));
+
+ File[] files = {
+ createFile(new File(folder, "001"), 10000),
+ createFile(new File(folder, "002"), 1000),
+ createFile(new File(folder, "003"), 100),
+ createFile(new File(childFolder, "001"), 1000),
+ createFile(new File(childFolder, "002"), 2000),
+ };
+
+ assertEquals(0, FileUtils.folderSize(new File(folder, "i_dont_exist")));
+ assertEquals(files[0].length(), FileUtils.folderSize(files[0]));
+
+ long size = FileUtils.folderSize(folder);
+ assertEquals(Arrays.stream(files).mapToLong(f -> f.length()).sum(), size);
+ }
+
+ private File createFolder(Path path)
+ {
+ File folder = path.toFile();
+ FileUtils.createDirectory(folder);
+ return folder;
+ }
+
+ private File createFile(File file, long size)
+ {
+ try
+ {
+ RandomAccessFile f = new RandomAccessFile(file, "rw");
+ f.setLength(size);
+ }
+ catch (Exception e)
+ {
+ System.err.println(e);
+ }
+ return file;
+ }
}
diff --git a/test/unit/org/apache/cassandra/locator/NetworkTopologyStrategyTest.java b/test/unit/org/apache/cassandra/locator/NetworkTopologyStrategyTest.java
index bbfdd3b..3cba328 100644
--- a/test/unit/org/apache/cassandra/locator/NetworkTopologyStrategyTest.java
+++ b/test/unit/org/apache/cassandra/locator/NetworkTopologyStrategyTest.java
@@ -21,24 +21,26 @@
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
+import java.util.stream.Collectors;
import com.google.common.collect.HashMultimap;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Multimap;
+
import org.junit.Assert;
import org.junit.Test;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.Murmur3Partitioner;
import org.apache.cassandra.dht.OrderPreservingPartitioner.StringToken;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.locator.TokenMetadata.Topology;
+import org.apache.cassandra.service.StorageService;
public class NetworkTopologyStrategyTest
{
@@ -166,4 +168,203 @@
InetAddress add1 = InetAddress.getByAddress(bytes);
metadata.updateNormalToken(token1, add1);
}
+
+ @Test
+ public void testCalculateEndpoints() throws UnknownHostException
+ {
+ final int NODES = 100;
+ final int VNODES = 64;
+ final int RUNS = 10;
+ StorageService.instance.setPartitionerUnsafe(Murmur3Partitioner.instance);
+ Map<String, Integer> datacenters = ImmutableMap.of("rf1", 1, "rf3", 3, "rf5_1", 5, "rf5_2", 5, "rf5_3", 5);
+ List<InetAddress> nodes = new ArrayList<>(NODES);
+ for (byte i=0; i<NODES; ++i)
+ nodes.add(InetAddress.getByAddress(new byte[]{127, 0, 0, i}));
+ for (int run=0; run<RUNS; ++run)
+ {
+ Random rand = new Random();
+ IEndpointSnitch snitch = generateSnitch(datacenters, nodes, rand);
+ DatabaseDescriptor.setEndpointSnitch(snitch);
+
+ TokenMetadata meta = new TokenMetadata();
+ for (int i=0; i<NODES; ++i) // Nodes
+ for (int j=0; j<VNODES; ++j) // tokens/vnodes per node
+ meta.updateNormalToken(Murmur3Partitioner.instance.getRandomToken(rand), nodes.get(i));
+ testEquivalence(meta, snitch, datacenters, rand);
+ }
+ }
+
+ void testEquivalence(TokenMetadata tokenMetadata, IEndpointSnitch snitch, Map<String, Integer> datacenters, Random rand)
+ {
+ NetworkTopologyStrategy nts = new NetworkTopologyStrategy("ks", tokenMetadata, snitch,
+ datacenters.entrySet().stream().
+ collect(Collectors.toMap(x -> x.getKey(), x -> Integer.toString(x.getValue()))));
+ for (int i=0; i<1000; ++i)
+ {
+ Token token = Murmur3Partitioner.instance.getRandomToken(rand);
+ List<InetAddress> expected = calculateNaturalEndpoints(token, tokenMetadata, datacenters, snitch);
+ List<InetAddress> actual = nts.calculateNaturalEndpoints(token, tokenMetadata);
+ if (endpointsDiffer(expected, actual))
+ {
+ System.err.println("Endpoints mismatch for token " + token);
+ System.err.println(" expected: " + expected);
+ System.err.println(" actual : " + actual);
+ Assert.assertEquals("Endpoints for token " + token + " mismatch.", expected, actual);
+ }
+ }
+ }
+
+ private boolean endpointsDiffer(List<InetAddress> ep1, List<InetAddress> ep2)
+ {
+ // Because the old algorithm does not put the nodes in the correct order in the case where more replicas
+ // are required than there are racks in a dc, we accept different order as long as the primary
+ // replica is the same.
+ if (ep1.equals(ep2))
+ return false;
+ if (!ep1.get(0).equals(ep2.get(0)))
+ return true;
+ Set<InetAddress> s1 = new HashSet<>(ep1);
+ Set<InetAddress> s2 = new HashSet<>(ep2);
+ return !s1.equals(s2);
+ }
+
+ IEndpointSnitch generateSnitch(Map<String, Integer> datacenters, Collection<InetAddress> nodes, Random rand)
+ {
+ final Map<InetAddress, String> nodeToRack = new HashMap<>();
+ final Map<InetAddress, String> nodeToDC = new HashMap<>();
+ Map<String, List<String>> racksPerDC = new HashMap<>();
+ datacenters.forEach((dc, rf) -> racksPerDC.put(dc, randomRacks(rf, rand)));
+ int rf = datacenters.values().stream().mapToInt(x -> x).sum();
+ String[] dcs = new String[rf];
+ int pos = 0;
+ for (Map.Entry<String, Integer> dce : datacenters.entrySet())
+ {
+ for (int i = 0; i < dce.getValue(); ++i)
+ dcs[pos++] = dce.getKey();
+ }
+
+ for (InetAddress node : nodes)
+ {
+ String dc = dcs[rand.nextInt(rf)];
+ List<String> racks = racksPerDC.get(dc);
+ String rack = racks.get(rand.nextInt(racks.size()));
+ nodeToRack.put(node, rack);
+ nodeToDC.put(node, dc);
+ }
+
+ return new AbstractNetworkTopologySnitch()
+ {
+ public String getRack(InetAddress endpoint)
+ {
+ return nodeToRack.get(endpoint);
+ }
+
+ public String getDatacenter(InetAddress endpoint)
+ {
+ return nodeToDC.get(endpoint);
+ }
+ };
+ }
+
+ private List<String> randomRacks(int rf, Random rand)
+ {
+ int rc = rand.nextInt(rf * 3 - 1) + 1;
+ List<String> racks = new ArrayList<>(rc);
+ for (int i=0; i<rc; ++i)
+ racks.add(Integer.toString(i));
+ return racks;
+ }
+
+ // Copy of older endpoints calculation algorithm for comparison
+ public static List<InetAddress> calculateNaturalEndpoints(Token searchToken, TokenMetadata tokenMetadata, Map<String, Integer> datacenters, IEndpointSnitch snitch)
+ {
+ // we want to preserve insertion order so that the first added endpoint becomes primary
+ Set<InetAddress> replicas = new LinkedHashSet<>();
+ // replicas we have found in each DC
+ Map<String, Set<InetAddress>> dcReplicas = new HashMap<>(datacenters.size());
+ for (Map.Entry<String, Integer> dc : datacenters.entrySet())
+ dcReplicas.put(dc.getKey(), new HashSet<InetAddress>(dc.getValue()));
+
+ Topology topology = tokenMetadata.getTopology();
+ // all endpoints in each DC, so we can check when we have exhausted all the members of a DC
+ Multimap<String, InetAddress> allEndpoints = topology.getDatacenterEndpoints();
+ // all racks in a DC so we can check when we have exhausted all racks in a DC
+ Map<String, Multimap<String, InetAddress>> racks = topology.getDatacenterRacks();
+ assert !allEndpoints.isEmpty() && !racks.isEmpty() : "not aware of any cluster members";
+
+ // tracks the racks we have already placed replicas in
+ Map<String, Set<String>> seenRacks = new HashMap<>(datacenters.size());
+ for (Map.Entry<String, Integer> dc : datacenters.entrySet())
+ seenRacks.put(dc.getKey(), new HashSet<String>());
+
+ // tracks the endpoints that we skipped over while looking for unique racks
+ // when we relax the rack uniqueness we can append this to the current result so we don't have to wind back the iterator
+ Map<String, Set<InetAddress>> skippedDcEndpoints = new HashMap<>(datacenters.size());
+ for (Map.Entry<String, Integer> dc : datacenters.entrySet())
+ skippedDcEndpoints.put(dc.getKey(), new LinkedHashSet<InetAddress>());
+
+ Iterator<Token> tokenIter = TokenMetadata.ringIterator(tokenMetadata.sortedTokens(), searchToken, false);
+ while (tokenIter.hasNext() && !hasSufficientReplicas(dcReplicas, allEndpoints, datacenters))
+ {
+ Token next = tokenIter.next();
+ InetAddress ep = tokenMetadata.getEndpoint(next);
+ String dc = snitch.getDatacenter(ep);
+ // have we already found all replicas for this dc?
+ if (!datacenters.containsKey(dc) || hasSufficientReplicas(dc, dcReplicas, allEndpoints, datacenters))
+ continue;
+ // can we skip checking the rack?
+ if (seenRacks.get(dc).size() == racks.get(dc).keySet().size())
+ {
+ dcReplicas.get(dc).add(ep);
+ replicas.add(ep);
+ }
+ else
+ {
+ String rack = snitch.getRack(ep);
+ // is this a new rack?
+ if (seenRacks.get(dc).contains(rack))
+ {
+ skippedDcEndpoints.get(dc).add(ep);
+ }
+ else
+ {
+ dcReplicas.get(dc).add(ep);
+ replicas.add(ep);
+ seenRacks.get(dc).add(rack);
+ // if we've run out of distinct racks, add the hosts we skipped past already (up to RF)
+ if (seenRacks.get(dc).size() == racks.get(dc).keySet().size())
+ {
+ Iterator<InetAddress> skippedIt = skippedDcEndpoints.get(dc).iterator();
+ while (skippedIt.hasNext() && !hasSufficientReplicas(dc, dcReplicas, allEndpoints, datacenters))
+ {
+ InetAddress nextSkipped = skippedIt.next();
+ dcReplicas.get(dc).add(nextSkipped);
+ replicas.add(nextSkipped);
+ }
+ }
+ }
+ }
+ }
+
+ return new ArrayList<InetAddress>(replicas);
+ }
+
+ private static boolean hasSufficientReplicas(String dc, Map<String, Set<InetAddress>> dcReplicas, Multimap<String, InetAddress> allEndpoints, Map<String, Integer> datacenters)
+ {
+ return dcReplicas.get(dc).size() >= Math.min(allEndpoints.get(dc).size(), getReplicationFactor(dc, datacenters));
+ }
+
+ private static boolean hasSufficientReplicas(Map<String, Set<InetAddress>> dcReplicas, Multimap<String, InetAddress> allEndpoints, Map<String, Integer> datacenters)
+ {
+ for (String dc : datacenters.keySet())
+ if (!hasSufficientReplicas(dc, dcReplicas, allEndpoints, datacenters))
+ return false;
+ return true;
+ }
+
+ public static int getReplicationFactor(String dc, Map<String, Integer> datacenters)
+ {
+ Integer replicas = datacenters.get(dc);
+ return replicas == null ? 0 : replicas;
+ }
}
diff --git a/test/unit/org/apache/cassandra/metrics/CassandraMetricsRegistryTest.java b/test/unit/org/apache/cassandra/metrics/CassandraMetricsRegistryTest.java
new file mode 100644
index 0000000..0258b8c
--- /dev/null
+++ b/test/unit/org/apache/cassandra/metrics/CassandraMetricsRegistryTest.java
@@ -0,0 +1,34 @@
+package org.apache.cassandra.metrics;
+
+import org.junit.Test;
+import org.apache.cassandra.metrics.CassandraMetricsRegistry.MetricName;
+import static org.junit.Assert.*;
+
+
+public class CassandraMetricsRegistryTest
+{
+ // A class with a name ending in '$'
+ private static class StrangeName$
+ {
+ }
+
+ @Test
+ public void testChooseType()
+ {
+ assertEquals("StrangeName", MetricName.chooseType(null, StrangeName$.class));
+ assertEquals("StrangeName", MetricName.chooseType("", StrangeName$.class));
+ assertEquals("String", MetricName.chooseType(null, String.class));
+ assertEquals("String", MetricName.chooseType("", String.class));
+
+ assertEquals("a", MetricName.chooseType("a", StrangeName$.class));
+ assertEquals("b", MetricName.chooseType("b", String.class));
+ }
+
+ @Test
+ public void testMetricName()
+ {
+ MetricName name = new MetricName(StrangeName$.class, "NaMe", "ScOpE");
+ assertEquals("StrangeName", name.getType());
+ }
+
+}
diff --git a/test/unit/org/apache/cassandra/net/MessagingServiceTest.java b/test/unit/org/apache/cassandra/net/MessagingServiceTest.java
index 75c146e..3b9c957 100644
--- a/test/unit/org/apache/cassandra/net/MessagingServiceTest.java
+++ b/test/unit/org/apache/cassandra/net/MessagingServiceTest.java
@@ -15,22 +15,22 @@
{
MessagingService.Verb verb = MessagingService.Verb.READ;
- for (int i = 0; i < 5000; i++)
- messagingService.incrementDroppedMessages(verb, i % 2 == 0);
+ for (int i = 1; i <= 5000; i++)
+ messagingService.incrementDroppedMessages(verb, i, i % 2 == 0);
List<String> logs = messagingService.getDroppedMessagesLogs();
assertEquals(1, logs.size());
- assertEquals("READ messages were dropped in last 5000 ms: 2500 for internal timeout and 2500 for cross node timeout", logs.get(0));
+ assertEquals("READ messages were dropped in last 5000 ms: 2500 for internal timeout and 2500 for cross node timeout. Mean internal dropped latency: 2730 ms and Mean cross-node dropped latency: 2731 ms", logs.get(0));
assertEquals(5000, (int)messagingService.getDroppedMessages().get(verb.toString()));
logs = messagingService.getDroppedMessagesLogs();
assertEquals(0, logs.size());
for (int i = 0; i < 2500; i++)
- messagingService.incrementDroppedMessages(verb, i % 2 == 0);
+ messagingService.incrementDroppedMessages(verb, i, i % 2 == 0);
logs = messagingService.getDroppedMessagesLogs();
- assertEquals("READ messages were dropped in last 5000 ms: 1250 for internal timeout and 1250 for cross node timeout", logs.get(0));
+ assertEquals("READ messages were dropped in last 5000 ms: 1250 for internal timeout and 1250 for cross node timeout. Mean internal dropped latency: 2277 ms and Mean cross-node dropped latency: 2278 ms", logs.get(0));
assertEquals(7500, (int)messagingService.getDroppedMessages().get(verb.toString()));
}
diff --git a/test/unit/org/apache/cassandra/schema/IndexMetadataTest.java b/test/unit/org/apache/cassandra/schema/IndexMetadataTest.java
new file mode 100644
index 0000000..901a5aa
--- /dev/null
+++ b/test/unit/org/apache/cassandra/schema/IndexMetadataTest.java
@@ -0,0 +1,36 @@
+package org.apache.cassandra.schema;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class IndexMetadataTest {
+
+ @Test
+ public void testIsNameValidPositive()
+ {
+ assertTrue(IndexMetadata.isNameValid("abcdefghijklmnopqrstuvwxyz"));
+ assertTrue(IndexMetadata.isNameValid("ABCDEFGHIJKLMNOPQRSTUVWXYZ"));
+ assertTrue(IndexMetadata.isNameValid("_01234567890"));
+ }
+
+ @Test
+ public void testIsNameValidNegative()
+ {
+ assertFalse(IndexMetadata.isNameValid(null));
+ assertFalse(IndexMetadata.isNameValid(""));
+ assertFalse(IndexMetadata.isNameValid(" "));
+ assertFalse(IndexMetadata.isNameValid("@"));
+ assertFalse(IndexMetadata.isNameValid("!"));
+ }
+
+ @Test
+ public void testGetDefaultIndexName()
+ {
+ Assert.assertEquals("aB4__idx", IndexMetadata.getDefaultIndexName("a B-4@!_+", null));
+ Assert.assertEquals("34_Ddd_F6_idx", IndexMetadata.getDefaultIndexName("34_()Ddd", "#F%6*"));
+
+ }
+}
diff --git a/test/unit/org/apache/cassandra/security/CipherFactoryTest.java b/test/unit/org/apache/cassandra/security/CipherFactoryTest.java
new file mode 100644
index 0000000..53bccc7
--- /dev/null
+++ b/test/unit/org/apache/cassandra/security/CipherFactoryTest.java
@@ -0,0 +1,99 @@
+package org.apache.cassandra.security;
+
+import java.io.IOException;
+import java.security.SecureRandom;
+
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+
+import com.google.common.base.Charsets;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.cassandra.config.TransparentDataEncryptionOptions;
+
+public class CipherFactoryTest
+{
+ // http://www.gutenberg.org/files/4300/4300-h/4300-h.htm
+ static final String ULYSSEUS = "Stately, plump Buck Mulligan came from the stairhead, bearing a bowl of lather on which a mirror and a razor lay crossed. " +
+ "A yellow dressinggown, ungirdled, was sustained gently behind him on the mild morning air. He held the bowl aloft and intoned: " +
+ "-Introibo ad altare Dei.";
+ TransparentDataEncryptionOptions encryptionOptions;
+ CipherFactory cipherFactory;
+ SecureRandom secureRandom;
+
+ @Before
+ public void setup()
+ {
+ secureRandom = new SecureRandom(new byte[] {0,1,2,3,4,5,6,7,8,9} );
+ encryptionOptions = EncryptionContextGenerator.createEncryptionOptions();
+ cipherFactory = new CipherFactory(encryptionOptions);
+ }
+
+ @Test
+ public void roundTrip() throws IOException, BadPaddingException, IllegalBlockSizeException
+ {
+ Cipher encryptor = cipherFactory.getEncryptor(encryptionOptions.cipher, encryptionOptions.key_alias);
+ byte[] original = ULYSSEUS.getBytes(Charsets.UTF_8);
+ byte[] encrypted = encryptor.doFinal(original);
+
+ Cipher decryptor = cipherFactory.getDecryptor(encryptionOptions.cipher, encryptionOptions.key_alias, encryptor.getIV());
+ byte[] decrypted = decryptor.doFinal(encrypted);
+ Assert.assertEquals(ULYSSEUS, new String(decrypted, Charsets.UTF_8));
+ }
+
+ private byte[] nextIV()
+ {
+ byte[] b = new byte[16];
+ secureRandom.nextBytes(b);
+ return b;
+ }
+
+ @Test
+ public void buildCipher_SameParams() throws Exception
+ {
+ byte[] iv = nextIV();
+ Cipher c1 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, iv, Cipher.ENCRYPT_MODE);
+ Cipher c2 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, iv, Cipher.ENCRYPT_MODE);
+ Assert.assertTrue(c1 == c2);
+ }
+
+ @Test
+ public void buildCipher_DifferentModes() throws Exception
+ {
+ byte[] iv = nextIV();
+ Cipher c1 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, iv, Cipher.ENCRYPT_MODE);
+ Cipher c2 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, iv, Cipher.DECRYPT_MODE);
+ Assert.assertFalse(c1 == c2);
+ }
+
+ @Test
+ public void buildCipher_DifferentIVs() throws Exception
+ {
+ Cipher c1 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, nextIV(), Cipher.ENCRYPT_MODE);
+ Cipher c2 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, nextIV(), Cipher.DECRYPT_MODE);
+ Assert.assertFalse(c1 == c2);
+ }
+
+ @Test
+ public void buildCipher_DifferentAliases() throws Exception
+ {
+ Cipher c1 = cipherFactory.buildCipher(encryptionOptions.cipher, encryptionOptions.key_alias, nextIV(), Cipher.ENCRYPT_MODE);
+ Cipher c2 = cipherFactory.buildCipher(encryptionOptions.cipher, EncryptionContextGenerator.KEY_ALIAS_2, nextIV(), Cipher.DECRYPT_MODE);
+ Assert.assertFalse(c1 == c2);
+ }
+
+ @Test(expected = AssertionError.class)
+ public void getDecryptor_NullIv() throws IOException
+ {
+ cipherFactory.getDecryptor(encryptionOptions.cipher, encryptionOptions.key_alias, null);
+ }
+
+ @Test(expected = AssertionError.class)
+ public void getDecryptor_EmptyIv() throws IOException
+ {
+ cipherFactory.getDecryptor(encryptionOptions.cipher, encryptionOptions.key_alias, new byte[0]);
+ }
+}
diff --git a/test/unit/org/apache/cassandra/security/EncryptionContextGenerator.java b/test/unit/org/apache/cassandra/security/EncryptionContextGenerator.java
new file mode 100644
index 0000000..635889b
--- /dev/null
+++ b/test/unit/org/apache/cassandra/security/EncryptionContextGenerator.java
@@ -0,0 +1,54 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.cassandra.security;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.cassandra.config.ParameterizedClass;
+import org.apache.cassandra.config.TransparentDataEncryptionOptions;
+
+public class EncryptionContextGenerator
+{
+ public static final String KEY_ALIAS_1 = "testing:1";
+ public static final String KEY_ALIAS_2 = "testing:2";
+
+ public static EncryptionContext createContext(boolean init)
+ {
+ return new EncryptionContext(createEncryptionOptions(), init);
+ }
+
+ public static TransparentDataEncryptionOptions createEncryptionOptions()
+ {
+ Map<String,String> params = new HashMap<>();
+ params.put("keystore", "test/conf/cassandra.keystore");
+ params.put("keystore_password", "cassandra");
+ params.put("store_type", "JCEKS");
+ ParameterizedClass keyProvider = new ParameterizedClass(JKSKeyProvider.class.getName(), params);
+
+ return new TransparentDataEncryptionOptions("AES/CBC/PKCS5Padding", KEY_ALIAS_1, keyProvider);
+ }
+
+ public static EncryptionContext createDisabledContext()
+ {
+ return new EncryptionContext();
+ }
+}
diff --git a/test/unit/org/apache/cassandra/security/JKSKeyProviderTest.java b/test/unit/org/apache/cassandra/security/JKSKeyProviderTest.java
new file mode 100644
index 0000000..081f688
--- /dev/null
+++ b/test/unit/org/apache/cassandra/security/JKSKeyProviderTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.security;
+
+import java.io.IOException;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.cassandra.config.TransparentDataEncryptionOptions;
+
+public class JKSKeyProviderTest
+{
+ JKSKeyProvider jksKeyProvider;
+ TransparentDataEncryptionOptions tdeOptions;
+
+ @Before
+ public void setup()
+ {
+ tdeOptions = EncryptionContextGenerator.createEncryptionOptions();
+ jksKeyProvider = new JKSKeyProvider(tdeOptions);
+ }
+
+ @Test
+ public void getSecretKey_WithKeyPassword() throws IOException
+ {
+ Assert.assertNotNull(jksKeyProvider.getSecretKey(tdeOptions.key_alias));
+ }
+
+ @Test
+ public void getSecretKey_WithoutKeyPassword() throws IOException
+ {
+ tdeOptions.remove("key_password");
+ Assert.assertNotNull(jksKeyProvider.getSecretKey(tdeOptions.key_alias));
+ }
+}
diff --git a/test/unit/org/apache/cassandra/service/DataResolverTest.java b/test/unit/org/apache/cassandra/service/DataResolverTest.java
index 997f4e4..b753362 100644
--- a/test/unit/org/apache/cassandra/service/DataResolverTest.java
+++ b/test/unit/org/apache/cassandra/service/DataResolverTest.java
@@ -715,7 +715,8 @@
ReadResponse.createRemoteDataResponse(partitionIterator, cmd),
Collections.EMPTY_MAP,
MessagingService.Verb.REQUEST_RESPONSE,
- MessagingService.current_version);
+ MessagingService.current_version,
+ MessageIn.createTimestamp());
}
private RangeTombstone tombstone(Object start, Object end, long markedForDeleteAt, int localDeletionTime)
diff --git a/test/unit/org/apache/cassandra/service/QueryPagerTest.java b/test/unit/org/apache/cassandra/service/QueryPagerTest.java
index bfc66e0..2f2a236 100644
--- a/test/unit/org/apache/cassandra/service/QueryPagerTest.java
+++ b/test/unit/org/apache/cassandra/service/QueryPagerTest.java
@@ -122,7 +122,8 @@
StringBuilder sb = new StringBuilder();
List<FilteredPartition> partitionList = new ArrayList<>();
int rows = 0;
- try (ReadOrderGroup orderGroup = pager.startOrderGroup(); PartitionIterator iterator = pager.fetchPageInternal(toQuery, orderGroup))
+ try (ReadExecutionController executionController = pager.executionController();
+ PartitionIterator iterator = pager.fetchPageInternal(toQuery, executionController))
{
while (iterator.hasNext())
{
diff --git a/test/unit/org/apache/cassandra/utils/CassandraVersionTest.java b/test/unit/org/apache/cassandra/utils/CassandraVersionTest.java
index 145b735..73562b7 100644
--- a/test/unit/org/apache/cassandra/utils/CassandraVersionTest.java
+++ b/test/unit/org/apache/cassandra/utils/CassandraVersionTest.java
@@ -17,9 +17,16 @@
*/
package org.apache.cassandra.utils;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Arrays;
import org.junit.Test;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.junit.matchers.JUnitMatchers.containsString;
public class CassandraVersionTest
{
@@ -29,14 +36,18 @@
CassandraVersion version;
version = new CassandraVersion("1.2.3");
- assert version.major == 1 && version.minor == 2 && version.patch == 3;
+ assertTrue(version.major == 1 && version.minor == 2 && version.patch == 3);
version = new CassandraVersion("1.2.3-foo.2+Bar");
- assert version.major == 1 && version.minor == 2 && version.patch == 3;
+ assertTrue(version.major == 1 && version.minor == 2 && version.patch == 3);
// CassandraVersion can parse 4th '.' as build number
version = new CassandraVersion("1.2.3.456");
- assert version.major == 1 && version.minor == 2 && version.patch == 3;
+ assertTrue(version.major == 1 && version.minor == 2 && version.patch == 3);
+
+ // support for tick-tock release
+ version = new CassandraVersion("3.2");
+ assertTrue(version.major == 3 && version.minor == 2 && version.patch == 0);
}
@Test
@@ -46,32 +57,32 @@
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.2.4");
- assert v1.compareTo(v2) == -1;
+ assertTrue(v1.compareTo(v2) == -1);
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.2.3");
- assert v1.compareTo(v2) == 0;
+ assertTrue(v1.compareTo(v2) == 0);
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("2.0.0");
- assert v1.compareTo(v2) == -1;
- assert v2.compareTo(v1) == 1;
+ assertTrue(v1.compareTo(v2) == -1);
+ assertTrue(v2.compareTo(v1) == 1);
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.2.3-alpha");
- assert v1.compareTo(v2) == 1;
+ assertTrue(v1.compareTo(v2) == 1);
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.2.3+foo");
- assert v1.compareTo(v2) == -1;
+ assertTrue(v1.compareTo(v2) == -1);
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.2.3-alpha+foo");
- assert v1.compareTo(v2) == 1;
+ assertTrue(v1.compareTo(v2) == 1);
v1 = new CassandraVersion("1.2.3-alpha+1");
v2 = new CassandraVersion("1.2.3-alpha+2");
- assert v1.compareTo(v2) == -1;
+ assertTrue(v1.compareTo(v2) == -1);
}
@Test
@@ -80,33 +91,32 @@
CassandraVersion v1, v2;
v1 = new CassandraVersion("3.0.2");
- assert v1.isSupportedBy(v1);
+ assertTrue(v1.isSupportedBy(v1));
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.2.4");
- assert v1.isSupportedBy(v2);
- assert !v2.isSupportedBy(v1);
+ assertTrue(v1.isSupportedBy(v2));
+ assertTrue(!v2.isSupportedBy(v1));
v1 = new CassandraVersion("1.2.3");
v2 = new CassandraVersion("1.3.3");
- assert v1.isSupportedBy(v2);
- assert !v2.isSupportedBy(v1);
+ assertTrue(v1.isSupportedBy(v2));
+ assertTrue(!v2.isSupportedBy(v1));
v1 = new CassandraVersion("2.2.3");
v2 = new CassandraVersion("1.3.3");
- assert !v1.isSupportedBy(v2);
- assert !v2.isSupportedBy(v1);
+ assertTrue(!v1.isSupportedBy(v2));
+ assertTrue(!v2.isSupportedBy(v1));
v1 = new CassandraVersion("3.1.0");
v2 = new CassandraVersion("3.0.1");
- assert !v1.isSupportedBy(v2);
- assert v2.isSupportedBy(v1);
+ assertTrue(!v1.isSupportedBy(v2));
+ assertTrue(v2.isSupportedBy(v1));
}
@Test
public void testInvalid()
{
- assertThrows("1.0");
assertThrows("1.0.0a");
assertThrows("1.a.4");
assertThrows("1.0.0-foo&");
@@ -132,15 +142,74 @@
prev = next;
next = new CassandraVersion("2.2.0");
assertTrue(prev.compareTo(next) < 0);
- }
+ prev = next;
+ next = new CassandraVersion("3.1");
+ assertTrue(prev.compareTo(next) < 0);
+
+ prev = next;
+ next = new CassandraVersion("3.1.1");
+ assertTrue(prev.compareTo(next) < 0);
+
+ prev = next;
+ next = new CassandraVersion("3.2-rc1-SNAPSHOT");
+ assertTrue(prev.compareTo(next) < 0);
+
+ prev = next;
+ next = new CassandraVersion("3.2");
+ assertTrue(prev.compareTo(next) < 0);
+ }
+
private static void assertThrows(String str)
{
try
{
new CassandraVersion(str);
- assert false;
+ fail();
}
catch (IllegalArgumentException e) {}
}
+
+ @Test
+ public void testParseIdentifiersPositive() throws Throwable
+ {
+ String[] result = parseIdentifiers("DUMMY", "+a.b.cde.f_g.");
+ String[] expected = {"a", "b", "cde", "f_g"};
+ assertArrayEquals(expected, result);
+ }
+
+ @Test
+ public void testParseIdentifiersNegative() throws Throwable
+ {
+ String version = "DUMMY";
+ try
+ {
+ parseIdentifiers(version, "+a. .b");
+
+ }
+ catch (IllegalArgumentException e)
+ {
+ assertThat(e.getMessage(), containsString(version));
+ }
+ }
+ private static String[] parseIdentifiers(String version, String str) throws Throwable
+ {
+ String name = "parseIdentifiers";
+ Class[] args = {String.class, String.class};
+ for (Method m: CassandraVersion.class.getDeclaredMethods())
+ {
+ if (name.equals(m.getName()) &&
+ Arrays.equals(args, m.getParameterTypes()))
+ {
+ m.setAccessible(true);
+ try
+ {
+ return (String[]) m.invoke(null, version, str);
+ } catch (InvocationTargetException e){
+ throw e.getTargetException();
+ }
+ }
+ }
+ throw new NoSuchMethodException(CassandraVersion.class + "." + name + Arrays.toString(args));
+ }
}
diff --git a/test/unit/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillisTest.java b/test/unit/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillisTest.java
index 1662e77..b3dfad3 100644
--- a/test/unit/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillisTest.java
+++ b/test/unit/org/apache/cassandra/utils/NanoTimeToCurrentTimeMillisTest.java
@@ -34,10 +34,7 @@
now = Math.max(now, System.currentTimeMillis());
if (ii % 10000 == 0)
{
- synchronized (NanoTimeToCurrentTimeMillis.TIMESTAMP_UPDATE)
- {
- NanoTimeToCurrentTimeMillis.TIMESTAMP_UPDATE.notify();
- }
+ NanoTimeToCurrentTimeMillis.updateNow();
Thread.sleep(1);
}
diff --git a/test/unit/org/apache/cassandra/utils/UUIDTests.java b/test/unit/org/apache/cassandra/utils/UUIDTests.java
index 83e421a..ebe5fd1 100644
--- a/test/unit/org/apache/cassandra/utils/UUIDTests.java
+++ b/test/unit/org/apache/cassandra/utils/UUIDTests.java
@@ -48,7 +48,6 @@
assert one.timestamp() < two.timestamp();
}
-
@Test
public void testDecomposeAndRaw()
{
@@ -59,6 +58,15 @@
}
@Test
+ public void testToFromByteBuffer()
+ {
+ UUID a = UUIDGen.getTimeUUID();
+ ByteBuffer bb = UUIDGen.toByteBuffer(a);
+ UUID b = UUIDGen.getUUID(bb);
+ assert a.equals(b);
+ }
+
+ @Test
public void testTimeUUIDType()
{
TimeUUIDType comp = TimeUUIDType.instance;
diff --git a/test/unit/org/apache/cassandra/utils/btree/BTreeRemovalTest.java b/test/unit/org/apache/cassandra/utils/btree/BTreeRemovalTest.java
new file mode 100644
index 0000000..a9cf383
--- /dev/null
+++ b/test/unit/org/apache/cassandra/utils/btree/BTreeRemovalTest.java
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.utils.btree;
+
+import static org.apache.cassandra.utils.btree.BTreeRemoval.remove;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+import java.util.Comparator;
+import java.util.Random;
+import java.util.SortedSet;
+import java.util.TreeSet;
+
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
+public class BTreeRemovalTest
+{
+ static
+ {
+ System.setProperty("cassandra.btree.fanfactor", "8");
+ }
+
+ private static final Comparator<Integer> CMP = new Comparator<Integer>()
+ {
+ public int compare(Integer o1, Integer o2)
+ {
+ return Integer.compare(o1, o2);
+ }
+ };
+
+ private static Object[] copy(final Object[] btree)
+ {
+ final Object[] result = new Object[btree.length];
+ System.arraycopy(btree, 0, result, 0, btree.length);
+ if (!BTree.isLeaf(btree))
+ {
+ for (int i = BTree.getChildStart(btree); i < BTree.getChildEnd(btree); ++i)
+ result[i] = copy((Object[]) btree[i]);
+ final int[] sizeMap = BTree.getSizeMap(btree);
+ final int[] resultSizeMap = new int[sizeMap.length];
+ System.arraycopy(sizeMap, 0, resultSizeMap, 0, sizeMap.length);
+ result[result.length - 1] = resultSizeMap;
+ }
+ return result;
+ }
+
+ private static Object[] assertRemove(final Object[] btree, final int key)
+ {
+ final Object[] btreeBeforeRemoval = copy(btree);
+ final Object[] result = remove(btree, CMP, key);
+ assertBTree(btreeBeforeRemoval, btree);
+ assertTrue(BTree.isWellFormed(result, CMP));
+ assertEquals(BTree.size(btree) - 1, BTree.size(result));
+ assertNull(BTree.find(result, CMP, key));
+
+ for (Integer k : BTree.<Integer>iterable(btree))
+ if (k != key)
+ assertNotNull(BTree.find(result, CMP, k));
+
+ return result;
+ }
+
+ private static void assertBTree(final Object[] expected, final Object[] result)
+ {
+ assertEquals(BTree.isEmpty(expected), BTree.isEmpty(result));
+ assertEquals(BTree.isLeaf(expected), BTree.isLeaf(result));
+ assertEquals(expected.length, result.length);
+ if (BTree.isLeaf(expected))
+ {
+ assertArrayEquals(expected, result);
+ }
+ else
+ {
+ for (int i = 0; i < BTree.getBranchKeyEnd(expected); ++i)
+ assertEquals(expected[i], result[i]);
+ for (int i = BTree.getChildStart(expected); i < BTree.getChildEnd(expected); ++i)
+ assertBTree((Object[]) expected[i], (Object[]) result[i]);
+ assertArrayEquals(BTree.getSizeMap(expected), BTree.getSizeMap(result));
+ }
+ }
+
+ private static Object[] generateLeaf(int from, int size)
+ {
+ final Object[] result = new Object[(size & 1) == 1 ? size : size + 1];
+ for (int i = 0; i < size; ++i)
+ result[i] = from + i;
+ return result;
+ }
+
+ private static Object[] generateBranch(int[] keys, Object[][] children)
+ {
+ assert keys.length > 0;
+ assert children.length > 1;
+ assert children.length == keys.length + 1;
+ final Object[] result = new Object[keys.length + children.length + 1];
+ for (int i = 0; i < keys.length; ++i)
+ result[i] = keys[i];
+ for (int i = 0; i < children.length; ++i)
+ result[keys.length + i] = children[i];
+ final int[] sizeMap = new int[children.length];
+ sizeMap[0] = BTree.size(children[0]);
+ for (int i = 1; i < children.length; ++i)
+ sizeMap[i] = sizeMap[i - 1] + BTree.size(children[i]) + 1;
+ result[result.length - 1] = sizeMap;
+ return result;
+ }
+
+ private static Object[] generateSampleTwoLevelsTree(final int[] leafSizes)
+ {
+ assert leafSizes.length > 1;
+ final Object[][] leaves = new Object[leafSizes.length][];
+ for (int i = 0; i < leaves.length; ++i)
+ leaves[i] = generateLeaf(10 * i + 1, leafSizes[i]);
+ final int[] keys = new int[leafSizes.length - 1];
+ for (int i = 0; i < keys.length; ++i)
+ keys[i] = 10 * (i + 1);
+ final Object[] btree = generateBranch(keys, leaves);
+ assertTrue(BTree.isWellFormed(btree, CMP));
+ return btree;
+ }
+
+ private static Object[] generateSampleThreeLevelsTree(final int[] middleNodeSizes)
+ {
+ assert middleNodeSizes.length > 1;
+ final Object[][] middleNodes = new Object[middleNodeSizes.length][];
+ for (int i = 0; i < middleNodes.length; ++i)
+ {
+ final Object[][] leaves = new Object[middleNodeSizes[i]][];
+ for (int j = 0; j < middleNodeSizes[i]; ++j)
+ leaves[j] = generateLeaf(100 * i + 10 * j + 1, 4);
+ final int[] keys = new int[middleNodeSizes[i] - 1];
+ for (int j = 0; j < keys.length; ++j)
+ keys[j] = 100 * i + 10 * (j + 1);
+ middleNodes[i] = generateBranch(keys, leaves);
+ }
+ final int[] keys = new int[middleNodeSizes.length - 1];
+ for (int i = 0; i < keys.length; ++i)
+ keys[i] = 100 * (i + 1);
+ final Object[] btree = generateBranch(keys, middleNodes);
+ assertTrue(BTree.isWellFormed(btree, CMP));
+ return btree;
+ }
+
+ @Test
+ public void testRemoveFromEmpty()
+ {
+ assertBTree(BTree.empty(), remove(BTree.empty(), CMP, 1));
+ }
+
+ @Test
+ public void testRemoveNonexistingElement()
+ {
+ final Object[] btree = new Object[] {1, 2, 3, 4, null};
+ assertBTree(btree, remove(btree, CMP, 5));
+ }
+
+ @Test
+ public void testRemoveLastElement()
+ {
+ final Object[] btree = new Object[] {1};
+ assertBTree(BTree.empty(), remove(btree, CMP, 1));
+ }
+
+ @Test
+ public void testRemoveFromRootWhichIsALeaf()
+ {
+ for (int size = 1; size < 9; ++size)
+ {
+ final Object[] btree = new Object[(size & 1) == 1 ? size : size + 1];
+ for (int i = 0; i < size; ++i)
+ btree[i] = i + 1;
+ for (int i = 0; i < size; ++i)
+ {
+ final Object[] result = remove(btree, CMP, i + 1);
+ assertTrue("size " + size, BTree.isWellFormed(result, CMP));
+ for (int j = 0; j < i; ++j)
+ assertEquals("size " + size + "elem " + j, btree[j], result[j]);
+ for (int j = i; j < size - 1; ++j)
+ assertEquals("size " + size + "elem " + j, btree[j + 1], result[j]);
+ for (int j = size - 1; j < result.length; ++j)
+ assertNull("size " + size + "elem " + j, result[j]);
+ }
+
+ {
+ final Object[] result = remove(btree, CMP, 0);
+ assertTrue("size " + size, BTree.isWellFormed(result, CMP));
+ assertBTree(btree, result);
+ }
+
+ {
+ final Object[] result = remove(btree, CMP, size + 1);
+ assertTrue("size " + size, BTree.isWellFormed(result, CMP));
+ assertBTree(btree, result);
+ }
+ }
+ }
+
+ @Test
+ public void testRemoveFromNonMinimalLeaf()
+ {
+ for (int size = 5; size < 9; ++size)
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {size, 4, 4, 4, 4});
+
+ for (int i = 1; i < size + 1; ++i)
+ assertRemove(btree, i);
+ }
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafRotateLeft()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 5, 5, 5, 5});
+
+ for (int i = 11; i < 15; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafRotateRight1()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 5, 5, 5, 5});
+
+ for (int i = 1; i < 5; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafRotateRight2()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 4, 5, 5, 5});
+
+ for (int i = 11; i < 15; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafMergeWithLeft1()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 4, 4, 4, 4});
+
+ for (int i = 11; i < 15; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafMergeWithLeft2()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 4, 4, 4, 4});
+
+ for (int i = 41; i < 45; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafMergeWithRight()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 4, 4, 4, 4});
+
+ for (int i = 1; i < 5; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWhenSingleKeyRootMergeWithLeft()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 4});
+
+ for (int i = 1; i < 5; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWhenSingleKeyRootMergeWithRight()
+ {
+ final Object[] btree = generateSampleTwoLevelsTree(new int[] {4, 4});
+
+ for (int i = 11; i < 15; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWithBranchLeftRotation()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {6, 5, 5, 5, 5});
+ for (int i = 101; i < 105; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWithBranchRightRotation1()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 6, 5, 5, 5});
+ for (int i = 1; i < 5; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWithBranchRightRotation2()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 5, 6, 5, 5});
+ for (int i = 101; i < 105; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWithBranchMergeWithLeft1()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 5, 5, 5, 5});
+ for (int i = 101; i < 105; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWithBranchMergeWithLeft2()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 5, 5, 5, 5});
+ for (int i = 401; i < 405; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMinimalLeafWithBranchMergeWithRight()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 5, 5, 5, 5});
+ for (int i = 1; i < 5; ++i)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromMiddleBranch()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 5, 5, 5, 5});
+ for (int i = 10; i < 50; i += 10)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void testRemoveFromRootBranch()
+ {
+ final Object[] btree = generateSampleThreeLevelsTree(new int[] {5, 5, 5, 5, 5});
+ for (int i = 100; i < 500; i += 100)
+ assertRemove(btree, i);
+ }
+
+ @Test
+ public void randomizedTest()
+ {
+ Random rand = new Random(2);
+ SortedSet<Integer> data = new TreeSet<>();
+ for (int i = 0; i < 1000; ++i)
+ data.add(rand.nextInt());
+ Object[] btree = BTree.build(data, UpdateFunction.<Integer>noOp());
+
+ assertTrue(BTree.isWellFormed(btree, CMP));
+ assertTrue(Iterables.elementsEqual(data, BTree.iterable(btree)));
+ while (btree != BTree.empty())
+ {
+ int idx = rand.nextInt(BTree.size(btree));
+ Integer val = BTree.findByIndex(btree, idx);
+ assertTrue(data.remove(val));
+ btree = assertRemove(btree, val);
+ }
+ }
+}
diff --git a/tools/bin/cassandra-stress b/tools/bin/cassandra-stress
index 14fbd09..1ffb598 100755
--- a/tools/bin/cassandra-stress
+++ b/tools/bin/cassandra-stress
@@ -17,24 +17,25 @@
# limitations under the License.
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
- for include in "`dirname $0`/cassandra.in.sh" \
- "$HOME/.cassandra.in.sh" \
- /usr/share/cassandra/cassandra.in.sh \
+ for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
- /opt/cassandra/cassandra.in.sh; do
- if [ -r $include ]; then
- . $include
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
+ if [ -r "$include" ]; then
+ . "$include"
break
fi
done
-elif [ -r $CASSANDRA_INCLUDE ]; then
- . $CASSANDRA_INCLUDE
+elif [ -r "$CASSANDRA_INCLUDE" ]; then
+ . "$CASSANDRA_INCLUDE"
fi
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
+# Use JAVA_HOME if set, otherwise look for java in PATH
+if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
else
- JAVA=`which java`
+ JAVA="`which java`"
fi
if [ "x$JAVA" = "x" ]; then
@@ -42,6 +43,14 @@
exit 1
fi
-"$JAVA" -server -ea -cp "$CLASSPATH" \
- -Dcassandra.storagedir="$cassandra_storagedir" \
- org.apache.cassandra.stress.Stress $@
+if [ -z "$CLASSPATH" ]; then
+ echo "You must set the CLASSPATH var" >&2
+ exit 1
+fi
+
+"$JAVA" -server -ea -cp "$CLASSPATH" $JVM_OPTS \
+ -Dcassandra.storagedir="$cassandra_storagedir" \
+ -Dlogback.configurationFile=logback-tools.xml \
+ org.apache.cassandra.stress.Stress $@
+
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/cassandra-stressd b/tools/bin/cassandra-stressd
index 50dedee..229ba0b 100755
--- a/tools/bin/cassandra-stressd
+++ b/tools/bin/cassandra-stressd
@@ -17,31 +17,26 @@
# limitations under the License.
DESC="Cassandra Stress Test Daemon"
-if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
- for include in "`dirname $0`/cassandra.in.sh" \
- "$HOME/.cassandra.in.sh" \
- /usr/share/cassandra/cassandra.in.sh \
+if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
+ for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
- /opt/cassandra/cassandra.in.sh; do
- if [ -r $include ]; then
- . $include
- break
- fi
- done
-elif [ -r $CASSANDRA_INCLUDE ]; then
- . $CASSANDRA_INCLUDE
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
+ if [ -r "$include" ]; then
+ . "$include"
+ break
+ fi
+ done
+elif [ -r "$CASSANDRA_INCLUDE" ]; then
+ . "$CASSANDRA_INCLUDE"
fi
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
+# Use JAVA_HOME if set, otherwise look for java in PATH
+if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
else
- JAVA=`which java`
-fi
-
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
-else
- JAVA=`which java`
+ JAVA="`which java`"
fi
if [ "x$JAVA" = "x" ]; then
@@ -49,12 +44,18 @@
exit 1
fi
+if [ -z "$CLASSPATH" ]; then
+ echo "You must set the CLASSPATH var" >&2
+ exit 1
+fi
+
case "$1" in
start)
echo "Starting $DESC: "
- "$JAVA" -server -cp "$CLASSPATH" \
- -Dcassandra.storagedir="$cassandra_storagedir" \
- org.apache.cassandra.stress.StressServer $@ 1> ./stressd.out.log 2> ./stressd.err.log &
+ "$JAVA" -server -cp "$CLASSPATH" $JVM_OPTS \
+ -Dcassandra.storagedir="$cassandra_storagedir" \
+ -Dlogback.configurationFile=logback-tools.xml \
+ org.apache.cassandra.stress.StressServer $@ 1> ./stressd.out.log 2> ./stressd.err.log &
echo $! > ./stressd.pid
echo "done."
;;
@@ -86,3 +87,4 @@
;;
esac
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/sstableexpiredblockers b/tools/bin/sstableexpiredblockers
index 0027208..cefdfe3 100755
--- a/tools/bin/sstableexpiredblockers
+++ b/tools/bin/sstableexpiredblockers
@@ -20,7 +20,7 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
"`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
@@ -38,6 +38,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
@@ -47,8 +52,9 @@
MAX_HEAP_SIZE="256M"
fi
-"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" -Xmx$MAX_HEAP_SIZE \
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.SSTableExpiredBlockers "$@"
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/sstablelevelreset b/tools/bin/sstablelevelreset
index 1756a21..8191f3d 100755
--- a/tools/bin/sstablelevelreset
+++ b/tools/bin/sstablelevelreset
@@ -17,26 +17,30 @@
# limitations under the License.
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
- for include in "`dirname $0`/cassandra.in.sh" \
- "$HOME/.cassandra.in.sh" \
- /usr/share/cassandra/cassandra.in.sh \
+ for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
- /opt/cassandra/cassandra.in.sh; do
- if [ -r $include ]; then
- . $include
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
+ if [ -r "$include" ]; then
+ . "$include"
break
fi
done
-elif [ -r $CASSANDRA_INCLUDE ]; then
- . $CASSANDRA_INCLUDE
+elif [ -r "$CASSANDRA_INCLUDE" ]; then
+ . "$CASSANDRA_INCLUDE"
fi
-
# Use JAVA_HOME if set, otherwise look for java in PATH
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
+if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
else
- JAVA=`which java`
+ JAVA="`which java`"
+fi
+
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
fi
if [ -z "$CLASSPATH" ]; then
@@ -44,7 +48,13 @@
exit 1
fi
-"$JAVA" -cp "$CLASSPATH" -Dstorage-config=$CASSANDRA_CONF \
+if [ "x$MAX_HEAP_SIZE" = "x" ]; then
+ MAX_HEAP_SIZE="256M"
+fi
+
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.SSTableLevelResetter "$@"
+
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/sstablemetadata b/tools/bin/sstablemetadata
index c5ff832..5316869 100755
--- a/tools/bin/sstablemetadata
+++ b/tools/bin/sstablemetadata
@@ -17,11 +17,11 @@
# limitations under the License.
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
- for include in "`dirname "$0"`/cassandra.in.sh" \
- "$HOME/.cassandra.in.sh" \
- /usr/share/cassandra/cassandra.in.sh \
+ for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
- /opt/cassandra/cassandra.in.sh; do
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
break
@@ -31,7 +31,6 @@
. "$CASSANDRA_INCLUDE"
fi
-
# Use JAVA_HOME if set, otherwise look for java in PATH
if [ -x "$JAVA_HOME/bin/java" ]; then
JAVA="$JAVA_HOME/bin/java"
@@ -39,12 +38,23 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
fi
-"$JAVA" -cp "$CLASSPATH" \
+if [ "x$MAX_HEAP_SIZE" = "x" ]; then
+ MAX_HEAP_SIZE="256M"
+fi
+
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.SSTableMetadataViewer "$@"
+
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/sstableofflinerelevel b/tools/bin/sstableofflinerelevel
index bfe28a0..a4ba6cb 100755
--- a/tools/bin/sstableofflinerelevel
+++ b/tools/bin/sstableofflinerelevel
@@ -20,7 +20,7 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
"`dirname "$0"`/cassandra.in.sh"; do
if [ -r "$include" ]; then
. "$include"
@@ -38,6 +38,11 @@
JAVA="`which java`"
fi
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
@@ -47,8 +52,9 @@
MAX_HEAP_SIZE="256M"
fi
-"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" -Xmx$MAX_HEAP_SIZE \
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.SSTableOfflineRelevel "$@"
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/sstablerepairedset b/tools/bin/sstablerepairedset
index 1aa30a3..efb8a70 100755
--- a/tools/bin/sstablerepairedset
+++ b/tools/bin/sstablerepairedset
@@ -16,32 +16,45 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-if [ "x$CLASSPATH" = "x" ]; then
-
- # execute from the build dir.
- if [ -d `dirname $0`/../../build/classes ]; then
- for directory in `dirname $0`/../../build/classes/*; do
- CLASSPATH=$CLASSPATH:$directory
- done
- else
- if [ -f `dirname $0`/../lib/stress.jar ]; then
- CLASSPATH=`dirname $0`/../lib/stress.jar
+if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
+ for include in /usr/share/cassandra/cassandra.in.sh \
+ /usr/local/share/cassandra/cassandra.in.sh \
+ /opt/cassandra/cassandra.in.sh \
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
+ if [ -r "$include" ]; then
+ . "$include"
+ break
fi
- fi
-
- for jar in `dirname $0`/../../lib/*.jar; do
- CLASSPATH=$CLASSPATH:$jar
done
+elif [ -r "$CASSANDRA_INCLUDE" ]; then
+ . "$CASSANDRA_INCLUDE"
fi
# Use JAVA_HOME if set, otherwise look for java in PATH
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
+if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
else
- JAVA=`which java`
+ JAVA="`which java`"
fi
-$JAVA -cp $CLASSPATH \
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
+fi
+
+if [ -z "$CLASSPATH" ]; then
+ echo "You must set the CLASSPATH var" >&2
+ exit 1
+fi
+
+if [ "x$MAX_HEAP_SIZE" = "x" ]; then
+ MAX_HEAP_SIZE="256M"
+fi
+
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.SSTableRepairedAtSetter "$@"
+
+# vi:ai sw=4 ts=4 tw=0 et
diff --git a/tools/bin/sstablesplit b/tools/bin/sstablesplit
index 51da481..90d91ce 100755
--- a/tools/bin/sstablesplit
+++ b/tools/bin/sstablesplit
@@ -20,22 +20,27 @@
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
- ~/.cassandra.in.sh \
- `dirname $0`/cassandra.in.sh; do
- if [ -r $include ]; then
- . $include
+ "$HOME/.cassandra.in.sh" \
+ "`dirname "$0"`/cassandra.in.sh"; do
+ if [ -r "$include" ]; then
+ . "$include"
break
fi
done
-elif [ -r $CASSANDRA_INCLUDE ]; then
- . $CASSANDRA_INCLUDE
+elif [ -r "$CASSANDRA_INCLUDE" ]; then
+ . "$CASSANDRA_INCLUDE"
fi
# Use JAVA_HOME if set, otherwise look for java in PATH
-if [ -x $JAVA_HOME/bin/java ]; then
- JAVA=$JAVA_HOME/bin/java
+if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
else
- JAVA=`which java`
+ JAVA="`which java`"
+fi
+
+if [ "x$JAVA" = "x" ]; then
+ echo "Java executable not found (hint: set JAVA_HOME)" >&2
+ exit 1
fi
if [ -z "$CLASSPATH" ]; then
@@ -43,7 +48,11 @@
exit 1
fi
-"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx256M \
+if [ "x$MAX_HEAP_SIZE" = "x" ]; then
+ MAX_HEAP_SIZE="256M"
+fi
+
+"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.StandaloneSplitter "$@"
diff --git a/tools/stress/src/org/apache/cassandra/stress/Operation.java b/tools/stress/src/org/apache/cassandra/stress/Operation.java
index 4123911..139dd53 100644
--- a/tools/stress/src/org/apache/cassandra/stress/Operation.java
+++ b/tools/stress/src/org/apache/cassandra/stress/Operation.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -15,6 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
package org.apache.cassandra.stress;
import java.io.IOException;
@@ -24,7 +25,6 @@
import com.google.common.util.concurrent.RateLimiter;
import org.apache.cassandra.stress.generate.*;
-import org.apache.cassandra.stress.settings.OptionDistribution;
import org.apache.cassandra.stress.settings.OptionRatioDistribution;
import org.apache.cassandra.stress.settings.SettingsLog;
import org.apache.cassandra.stress.settings.StressSettings;
@@ -141,11 +141,13 @@
*/
public abstract void run(ThriftClient client) throws IOException;
- public void run(SimpleClient client) throws IOException {
+ public void run(SimpleClient client) throws IOException
+ {
throw new UnsupportedOperationException();
}
- public void run(JavaDriverClient client) throws IOException {
+ public void run(JavaDriverClient client) throws IOException
+ {
throw new UnsupportedOperationException();
}
diff --git a/tools/stress/src/org/apache/cassandra/stress/Stress.java b/tools/stress/src/org/apache/cassandra/stress/Stress.java
index bc6d027..aeddb4a 100644
--- a/tools/stress/src/org/apache/cassandra/stress/Stress.java
+++ b/tools/stress/src/org/apache/cassandra/stress/Stress.java
@@ -24,6 +24,7 @@
import org.apache.cassandra.stress.settings.StressSettings;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.WindowsTimer;
+import org.apache.cassandra.stress.util.MultiPrintStream;
public final class Stress
{
@@ -72,7 +73,10 @@
return;
}
- PrintStream logout = settings.log.getOutput();
+ MultiPrintStream logout = settings.log.getOutput();
+ if (settings.graph.inGraphMode()) {
+ logout.addStream(new PrintStream(settings.graph.temporaryLogFile));
+ }
if (settings.sendToDaemon != null)
{
@@ -115,6 +119,9 @@
{
StressAction stressAction = new StressAction(settings, logout);
stressAction.run();
+ logout.flush();
+ if (settings.graph.inGraphMode())
+ new StressGraph(settings, arguments).generateGraph();
}
}
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressAction.java b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
index 158a278..3150e14 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressAction.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressAction.java
@@ -34,7 +34,6 @@
import org.apache.cassandra.stress.settings.StressSettings;
import org.apache.cassandra.stress.util.JavaDriverClient;
import org.apache.cassandra.stress.util.ThriftClient;
-import org.apache.cassandra.stress.util.TimingInterval;
import org.apache.cassandra.transport.SimpleClient;
public class StressAction implements Runnable
@@ -118,6 +117,7 @@
List<String> runIds = new ArrayList<>();
do
{
+ output.println("");
output.println(String.format("Running with %d threadCount", threadCount));
if (settings.command.truncate == SettingsCommand.TruncateWhen.ALWAYS)
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressGraph.java b/tools/stress/src/org/apache/cassandra/stress/StressGraph.java
new file mode 100644
index 0000000..ebaa0ae
--- /dev/null
+++ b/tools/stress/src/org/apache/cassandra/stress/StressGraph.java
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.stress;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.PrintWriter;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.io.ByteStreams;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.cassandra.stress.settings.StressSettings;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
+import org.json.simple.JSONValue;
+
+
+public class StressGraph
+{
+ private StressSettings stressSettings;
+ private enum ReadingMode
+ {
+ START,
+ METRICS,
+ AGGREGATES,
+ NEXTITERATION
+ }
+ private String[] stressArguments;
+
+ public StressGraph(StressSettings stressSetttings, String[] stressArguments)
+ {
+ this.stressSettings = stressSetttings;
+ this.stressArguments = stressArguments;
+ }
+
+ public void generateGraph()
+ {
+ File htmlFile = new File(stressSettings.graph.file);
+ JSONObject stats;
+ if (htmlFile.isFile())
+ {
+ try
+ {
+ String html = new String(Files.readAllBytes(Paths.get(htmlFile.toURI())), StandardCharsets.UTF_8);
+ stats = parseExistingStats(html);
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException("Couldn't load existing stats html.");
+ }
+ stats = this.createJSONStats(stats);
+ }
+ else
+ {
+ stats = this.createJSONStats(null);
+ }
+
+ try
+ {
+ PrintWriter out = new PrintWriter(htmlFile);
+ String statsBlock = "/* stats start */\nstats = " + stats.toJSONString() + ";\n/* stats end */\n";
+ String html = getGraphHTML().replaceFirst("/\\* stats start \\*/\n\n/\\* stats end \\*/\n", statsBlock);
+ out.write(html);
+ out.close();
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException("Couldn't write stats html.");
+ }
+ }
+
+ private JSONObject parseExistingStats(String html)
+ {
+ JSONObject stats;
+
+ Pattern pattern = Pattern.compile("(?s).*/\\* stats start \\*/\\nstats = (.*);\\n/\\* stats end \\*/.*");
+ Matcher matcher = pattern.matcher(html);
+ matcher.matches();
+ stats = (JSONObject) JSONValue.parse(matcher.group(1));
+
+ return stats;
+ }
+
+ private String getGraphHTML()
+ {
+ InputStream graphHTMLRes = StressGraph.class.getClassLoader().getResourceAsStream("org/apache/cassandra/stress/graph/graph.html");
+ String graphHTML;
+ try
+ {
+ graphHTML = new String(ByteStreams.toByteArray(graphHTMLRes));
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException(e);
+ }
+ return graphHTML;
+ }
+
+ /** Parse log and append to stats array */
+ private JSONArray parseLogStats(InputStream log, JSONArray stats) {
+ BufferedReader reader = new BufferedReader(new InputStreamReader(log));
+ JSONObject json = new JSONObject();
+ JSONArray intervals = new JSONArray();
+ boolean runningMultipleThreadCounts = false;
+ String currentThreadCount = null;
+ Pattern threadCountMessage = Pattern.compile("Running ([A-Z]+) with ([0-9]+) threads .*");
+ ReadingMode mode = ReadingMode.START;
+
+ try
+ {
+ String line;
+ while ((line = reader.readLine()) != null)
+ {
+ // Detect if we are running multiple thread counts:
+ if (line.startsWith("Thread count was not specified"))
+ runningMultipleThreadCounts = true;
+
+ if (runningMultipleThreadCounts)
+ {
+ // Detect thread count:
+ Matcher tc = threadCountMessage.matcher(line);
+ if (tc.matches())
+ {
+ currentThreadCount = tc.group(2);
+ }
+ }
+
+ // Detect mode changes
+ if (line.equals(StressMetrics.HEAD))
+ {
+ mode = ReadingMode.METRICS;
+ continue;
+ }
+ else if (line.equals("Results:"))
+ {
+ mode = ReadingMode.AGGREGATES;
+ continue;
+ }
+ else if (mode == ReadingMode.AGGREGATES && line.equals(""))
+ {
+ mode = ReadingMode.NEXTITERATION;
+ }
+ else if (line.equals("END") || line.equals("FAILURE"))
+ {
+ break;
+ }
+
+ // Process lines
+ if (mode == ReadingMode.METRICS)
+ {
+ JSONArray metrics = new JSONArray();
+ String[] parts = line.split(",");
+ if (parts.length != StressMetrics.HEADMETRICS.length)
+ {
+ continue;
+ }
+ for (String m : parts)
+ {
+ try
+ {
+ metrics.add(new BigDecimal(m.trim()));
+ }
+ catch (NumberFormatException e)
+ {
+ metrics.add(null);
+ }
+ }
+ intervals.add(metrics);
+ }
+ else if (mode == ReadingMode.AGGREGATES)
+ {
+ String[] parts = line.split(":",2);
+ if (parts.length != 2)
+ {
+ continue;
+ }
+ json.put(parts[0].trim(), parts[1].trim());
+ }
+ else if (mode == ReadingMode.NEXTITERATION)
+ {
+ //Wrap up the results of this test and append to the array.
+ json.put("metrics", Arrays.asList(StressMetrics.HEADMETRICS));
+ json.put("test", stressSettings.graph.operation);
+ if (currentThreadCount == null)
+ json.put("revision", stressSettings.graph.revision);
+ else
+ json.put("revision", String.format("%s - %s threads", stressSettings.graph.revision, currentThreadCount));
+ json.put("command", StringUtils.join(stressArguments, " "));
+ json.put("intervals", intervals);
+ stats.add(json);
+
+ //Start fresh for next iteration:
+ json = new JSONObject();
+ intervals = new JSONArray();
+ mode = ReadingMode.START;
+ }
+ }
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException("Couldn't read from temporary stress log file");
+ }
+ stats.add(json);
+ return stats;
+ }
+
+ private JSONObject createJSONStats(JSONObject json)
+ {
+ try (InputStream logStream = new FileInputStream(stressSettings.graph.temporaryLogFile))
+ {
+ JSONArray stats;
+ if (json == null)
+ {
+ json = new JSONObject();
+ stats = new JSONArray();
+ }
+ else
+ {
+ stats = (JSONArray) json.get("stats");
+ }
+
+ stats = parseLogStats(logStream, stats);
+
+ json.put("title", stressSettings.graph.title);
+ json.put("stats", stats);
+ return json;
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+}
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java b/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
index a640058..0190ed8 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
@@ -177,10 +177,12 @@
public static final String HEADFORMAT = "%-10s%10s,%8s,%8s,%8s,%8s,%8s,%8s,%8s,%8s,%8s,%7s,%9s,%7s,%7s,%8s,%8s,%8s,%8s";
public static final String ROWFORMAT = "%-10s%10d,%8.0f,%8.0f,%8.0f,%8.1f,%8.1f,%8.1f,%8.1f,%8.1f,%8.1f,%7.1f,%9.5f,%7d,%7.0f,%8.0f,%8.0f,%8.0f,%8.0f";
+ public static final String[] HEADMETRICS = new String[]{"type", "total ops","op/s","pk/s","row/s","mean","med",".95",".99",".999","max","time","stderr", "errors", "gc: #", "max ms", "sum ms", "sdv ms", "mb"};
+ public static final String HEAD = String.format(HEADFORMAT, (Object[]) HEADMETRICS);
private static void printHeader(String prefix, PrintStream output)
{
- output.println(prefix + String.format(HEADFORMAT, "type,", "total ops","op/s","pk/s","row/s","mean","med",".95",".99",".999","max","time","stderr", "errors", "gc: #", "max ms", "sum ms", "sdv ms", "mb"));
+ output.println(prefix + HEAD);
}
private static void printRow(String prefix, String type, TimingInterval interval, TimingInterval total, JmxCollector.GcStats gcStats, Uncertainty opRateUncertainty, PrintStream output)
@@ -233,6 +235,7 @@
output.println(String.format("stdev gc time(ms) : %.0f", totalGcStats.sdvms));
output.println("Total operation time : " + DurationFormatUtils.formatDuration(
history.runTime(), "HH:mm:ss", true));
+ output.println(""); // Newline is important here to separate the aggregates section from the END or the next stress iteration
}
public static void summarise(List<String> ids, List<StressMetrics> summarise, PrintStream out, int historySampleCount)
diff --git a/tools/stress/src/org/apache/cassandra/stress/operations/predefined/CqlOperation.java b/tools/stress/src/org/apache/cassandra/stress/operations/predefined/CqlOperation.java
index afdc0b1..097c1a0 100644
--- a/tools/stress/src/org/apache/cassandra/stress/operations/predefined/CqlOperation.java
+++ b/tools/stress/src/org/apache/cassandra/stress/operations/predefined/CqlOperation.java
@@ -49,6 +49,9 @@
public abstract class CqlOperation<V> extends PredefinedOperation
{
+ public static final ByteBuffer[][] EMPTY_BYTE_BUFFERS = new ByteBuffer[0][];
+ public static final byte[][] EMPTY_BYTE_ARRAYS = new byte[0][];
+
protected abstract List<Object> getQueryParameters(byte[] key);
protected abstract String buildQuery();
protected abstract CqlRunOp<V> buildRunOp(ClientWrapper client, String query, Object queryId, List<Object> params, ByteBuffer key);
@@ -455,7 +458,7 @@
public ByteBuffer[][] apply(ResultSet result)
{
if (result == null)
- return new ByteBuffer[0][];
+ return EMPTY_BYTE_BUFFERS;
List<Row> rows = result.all();
ByteBuffer[][] r = new ByteBuffer[rows.size()][];
@@ -481,7 +484,7 @@
public ByteBuffer[][] apply(ResultMessage result)
{
if (!(result instanceof ResultMessage.Rows))
- return new ByteBuffer[0][];
+ return EMPTY_BYTE_BUFFERS;
ResultMessage.Rows rows = ((ResultMessage.Rows) result);
ByteBuffer[][] r = new ByteBuffer[rows.result.size()][];
@@ -536,7 +539,7 @@
{
if (result == null)
- return new byte[0][];
+ return EMPTY_BYTE_ARRAYS;
List<Row> rows = result.all();
byte[][] r = new byte[rows.size()][];
for (int i = 0 ; i < r.length ; i++)
diff --git a/tools/stress/src/org/apache/cassandra/stress/operations/predefined/PredefinedOperation.java b/tools/stress/src/org/apache/cassandra/stress/operations/predefined/PredefinedOperation.java
index 66f232a..b435abb 100644
--- a/tools/stress/src/org/apache/cassandra/stress/operations/predefined/PredefinedOperation.java
+++ b/tools/stress/src/org/apache/cassandra/stress/operations/predefined/PredefinedOperation.java
@@ -34,6 +34,7 @@
public abstract class PredefinedOperation extends Operation
{
+ public static final byte[] EMPTY_BYTE_ARRAY = {};
public final Command type;
private final Distribution columnCount;
private Object cqlCache;
@@ -106,7 +107,7 @@
{
predicate.setSlice_range(new SliceRange()
.setStart(settings.columns.names.get(lb))
- .setFinish(new byte[] {})
+ .setFinish(EMPTY_BYTE_ARRAY)
.setReversed(false)
.setCount(count())
);
diff --git a/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/SchemaStatement.java b/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/SchemaStatement.java
index 49891ec..1c88490 100644
--- a/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/SchemaStatement.java
+++ b/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/SchemaStatement.java
@@ -20,8 +20,6 @@
*
*/
-
-import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
@@ -37,7 +35,6 @@
import org.apache.cassandra.stress.settings.StressSettings;
import org.apache.cassandra.stress.util.JavaDriverClient;
import org.apache.cassandra.stress.util.Timer;
-import org.apache.cassandra.transport.SimpleClient;
public abstract class SchemaStatement extends Operation
{
@@ -91,12 +88,6 @@
return args;
}
- @Override
- public void run(SimpleClient client) throws IOException
- {
- throw new UnsupportedOperationException();
- }
-
abstract class Runner implements RunOp
{
int partitionCount;
diff --git a/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/ValidatingSchemaQuery.java b/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/ValidatingSchemaQuery.java
index 8bdde51..c07328a 100644
--- a/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/ValidatingSchemaQuery.java
+++ b/tools/stress/src/org/apache/cassandra/stress/operations/userdefined/ValidatingSchemaQuery.java
@@ -26,7 +26,6 @@
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
-import java.util.Random;
import java.util.concurrent.ThreadLocalRandom;
import com.datastax.driver.core.*;
@@ -42,13 +41,11 @@
import org.apache.cassandra.thrift.CqlResult;
import org.apache.cassandra.thrift.CqlRow;
import org.apache.cassandra.thrift.ThriftConversion;
-import org.apache.cassandra.transport.SimpleClient;
import org.apache.cassandra.utils.Pair;
import org.apache.thrift.TException;
public class ValidatingSchemaQuery extends Operation
{
- final Random random = new Random();
private Pair<Row, Row> bounds;
final int clusteringComponents;
@@ -57,12 +54,6 @@
final int[] argumentIndex;
final Object[] bindBuffer;
- @Override
- public void run(SimpleClient client) throws IOException
- {
- throw new UnsupportedOperationException();
- }
-
private ValidatingSchemaQuery(Timer timer, StressSettings settings, PartitionGenerator generator, SeedManager seedManager, ValidatingStatement[] statements, ConsistencyLevel cl, int clusteringComponents)
{
super(timer, settings, new DataSpec(generator, seedManager, new DistributionFixed(1), settings.insert.rowPopulationRatio.get(), 1));
@@ -280,14 +271,14 @@
{
StringBuilder cc = new StringBuilder();
StringBuilder arg = new StringBuilder();
- cc.append("("); arg.append("(");
+ cc.append('('); arg.append('(');
for (int d = 0 ; d <= depth ; d++)
{
- if (d > 0) { cc.append(","); arg.append(","); }
+ if (d > 0) { cc.append(','); arg.append(','); }
cc.append(metadata.getClusteringColumns().get(d).getName());
- arg.append("?");
+ arg.append('?');
}
- cc.append(")"); arg.append(")");
+ cc.append(')'); arg.append(')');
ValidatingStatement[] statements = new ValidatingStatement[depth < maxDepth ? 1 : 4];
int i = 0;
diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/CliOption.java b/tools/stress/src/org/apache/cassandra/stress/settings/CliOption.java
index 5ec56dc..30933d9 100644
--- a/tools/stress/src/org/apache/cassandra/stress/settings/CliOption.java
+++ b/tools/stress/src/org/apache/cassandra/stress/settings/CliOption.java
@@ -38,7 +38,8 @@
LOG("Where to log progress to, and the interval at which to do it", SettingsLog.helpPrinter()),
TRANSPORT("Custom transport factories", SettingsTransport.helpPrinter()),
PORT("The port to connect to cassandra nodes on", SettingsPort.helpPrinter()),
- SENDTO("-send-to", "Specify a stress server to send this command to", SettingsMisc.sendToDaemonHelpPrinter())
+ SENDTO("-send-to", "Specify a stress server to send this command to", SettingsMisc.sendToDaemonHelpPrinter()),
+ GRAPH("-graph", "Graph recorded metrics", SettingsGraph.helpPrinter())
;
private static final Map<String, CliOption> LOOKUP;
diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsGraph.java b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsGraph.java
new file mode 100644
index 0000000..22261c1
--- /dev/null
+++ b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsGraph.java
@@ -0,0 +1,125 @@
+package org.apache.cassandra.stress.settings;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+public class SettingsGraph implements Serializable
+{
+ public final String file;
+ public final String revision;
+ public final String title;
+ public final String operation;
+ public final File temporaryLogFile;
+
+ public SettingsGraph(GraphOptions options, SettingsCommand stressCommand)
+ {
+ file = options.file.value();
+ revision = options.revision.value();
+ title = options.revision.value() == null
+ ? "cassandra-stress - " + new SimpleDateFormat("yyyy-mm-dd hh:mm:ss").format(new Date())
+ : options.title.value();
+
+ operation = options.operation.value() == null
+ ? stressCommand.type.name()
+ : options.operation.value();
+
+ if (inGraphMode())
+ {
+ try
+ {
+ temporaryLogFile = File.createTempFile("cassandra-stress", ".log");
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException("Cannot open temporary file");
+ }
+ }
+ else
+ {
+ temporaryLogFile = null;
+ }
+ }
+
+ public boolean inGraphMode()
+ {
+ return this.file == null ? false : true;
+ }
+
+ // Option Declarations
+ private static final class GraphOptions extends GroupedOptions
+ {
+ final OptionSimple file = new OptionSimple("file=", ".*", null, "HTML file to create or append to", true);
+ final OptionSimple revision = new OptionSimple("revision=", ".*", "unknown", "Unique name to assign to the current configuration being stressed", false);
+ final OptionSimple title = new OptionSimple("title=", ".*", null, "Title for chart (current date by default)", false);
+ final OptionSimple operation = new OptionSimple("op=", ".*", null, "Alternative name for current operation (stress op name used by default)", false);
+
+ @Override
+ public List<? extends Option> options()
+ {
+ return Arrays.asList(file, revision, title, operation);
+ }
+ }
+
+ // CLI Utility Methods
+ public static SettingsGraph get(Map<String, String[]> clArgs, SettingsCommand stressCommand)
+ {
+ String[] params = clArgs.remove("-graph");
+ if (params == null)
+ {
+ return new SettingsGraph(new GraphOptions(), stressCommand);
+ }
+ GraphOptions options = GroupedOptions.select(params, new GraphOptions());
+ if (options == null)
+ {
+ printHelp();
+ System.out.println("Invalid -graph options provided, see output for valid options");
+ System.exit(1);
+ }
+ return new SettingsGraph(options, stressCommand);
+ }
+
+ public static void printHelp()
+ {
+ GroupedOptions.printOptions(System.out, "-graph", new GraphOptions());
+ }
+
+ public static Runnable helpPrinter()
+ {
+ return new Runnable()
+ {
+ @Override
+ public void run()
+ {
+ printHelp();
+ }
+ };
+ }
+}
+
diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsLog.java b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsLog.java
index 5657fb2..e5f155b 100644
--- a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsLog.java
+++ b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsLog.java
@@ -21,10 +21,9 @@
*/
-import java.io.File;
-import java.io.FileNotFoundException;
-import java.io.PrintStream;
-import java.io.Serializable;
+import org.apache.cassandra.stress.util.MultiPrintStream;
+
+import java.io.*;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
@@ -62,9 +61,15 @@
level = Level.valueOf(options.level.value().toUpperCase());
}
- public PrintStream getOutput() throws FileNotFoundException
+ public MultiPrintStream getOutput() throws FileNotFoundException
{
- return file == null ? new PrintStream(System.out) : new PrintStream(file);
+ // Always print to stdout regardless of whether we're graphing or not
+ MultiPrintStream stream = new MultiPrintStream(new PrintStream(System.out));
+
+ if (file != null)
+ stream.addStream(new PrintStream(file));
+
+ return stream;
}
// Option Declarations
diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/StressSettings.java b/tools/stress/src/org/apache/cassandra/stress/settings/StressSettings.java
index 335ca92..26f65b5 100644
--- a/tools/stress/src/org/apache/cassandra/stress/settings/StressSettings.java
+++ b/tools/stress/src/org/apache/cassandra/stress/settings/StressSettings.java
@@ -54,8 +54,23 @@
public final SettingsTransport transport;
public final SettingsPort port;
public final String sendToDaemon;
+ public final SettingsGraph graph;
- public StressSettings(SettingsCommand command, SettingsRate rate, SettingsPopulation generate, SettingsInsert insert, SettingsColumn columns, SettingsSamples samples, SettingsErrors errors, SettingsLog log, SettingsMode mode, SettingsNode node, SettingsSchema schema, SettingsTransport transport, SettingsPort port, String sendToDaemon)
+ public StressSettings(SettingsCommand command,
+ SettingsRate rate,
+ SettingsPopulation generate,
+ SettingsInsert insert,
+ SettingsColumn columns,
+ SettingsSamples samples,
+ SettingsErrors errors,
+ SettingsLog log,
+ SettingsMode mode,
+ SettingsNode node,
+ SettingsSchema schema,
+ SettingsTransport transport,
+ SettingsPort port,
+ String sendToDaemon,
+ SettingsGraph graph)
{
this.command = command;
this.rate = rate;
@@ -71,6 +86,7 @@
this.transport = transport;
this.port = port;
this.sendToDaemon = sendToDaemon;
+ this.graph = graph;
}
private SmartThriftClient tclient;
@@ -262,6 +278,7 @@
SettingsNode node = SettingsNode.get(clArgs);
SettingsSchema schema = SettingsSchema.get(clArgs, command);
SettingsTransport transport = SettingsTransport.get(clArgs);
+ SettingsGraph graph = SettingsGraph.get(clArgs, command);
if (!clArgs.isEmpty())
{
printHelp();
@@ -278,7 +295,8 @@
}
System.exit(1);
}
- return new StressSettings(command, rate, generate, insert, columns, samples, errors, log, mode, node, schema, transport, port, sendToDaemon);
+
+ return new StressSettings(command, rate, generate, insert, columns, samples, errors, log, mode, node, schema, transport, port, sendToDaemon, graph);
}
private static Map<String, String[]> parseMap(String[] args)
diff --git a/tools/stress/src/org/apache/cassandra/stress/util/MultiPrintStream.java b/tools/stress/src/org/apache/cassandra/stress/util/MultiPrintStream.java
new file mode 100644
index 0000000..d299013
--- /dev/null
+++ b/tools/stress/src/org/apache/cassandra/stress/util/MultiPrintStream.java
@@ -0,0 +1,288 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.cassandra.stress.util;
+
+import java.io.*;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+
+/** PrintStream that multiplexes to multiple streams */
+public class MultiPrintStream extends PrintStream
+{
+ private List<PrintStream> newStreams;
+
+ public MultiPrintStream(PrintStream baseStream)
+ {
+ super(baseStream);
+ this.newStreams = new ArrayList();
+ }
+
+ public void addStream(PrintStream printStream)
+ {
+ newStreams.add(printStream);
+ }
+
+ @Override
+ public void flush()
+ {
+ super.flush();
+ for (PrintStream s : newStreams)
+ s.flush();
+ }
+
+ @Override
+ public void close()
+ {
+ super.close();
+ for (PrintStream s : newStreams)
+ s.close();
+ }
+
+ @Override
+ public boolean checkError()
+ {
+ boolean error = super.checkError();
+ for (PrintStream s : newStreams)
+ {
+ if (s.checkError())
+ error = true;
+ }
+ return error;
+ }
+
+ @Override
+ public void write(int b)
+ {
+ super.write(b);
+ for (PrintStream s: newStreams)
+ s.write(b);
+ }
+
+ @Override
+ public void write(byte[] buf, int off, int len)
+ {
+ super.write(buf, off, len);
+ for (PrintStream s: newStreams)
+ s.write(buf, off, len);
+ }
+
+ @Override
+ public void print(boolean b)
+ {
+ super.print(b);
+ for (PrintStream s: newStreams)
+ s.print(b);
+ }
+
+ @Override
+ public void print(char c)
+ {
+ super.print(c);
+ for (PrintStream s: newStreams)
+ s.print(c);
+ }
+
+ @Override
+ public void print(int i)
+ {
+ super.print(i);
+ for (PrintStream s: newStreams)
+ s.print(i);
+ }
+
+ @Override
+ public void print(long l)
+ {
+ super.print(l);
+ for (PrintStream s: newStreams)
+ s.print(l);
+ }
+
+ @Override
+ public void print(float f)
+ {
+ super.print(f);
+ for (PrintStream s: newStreams)
+ s.print(f);
+ }
+
+ @Override
+ public void print(double d)
+ {
+ super.print(d);
+ for (PrintStream s: newStreams)
+ s.print(d);
+ }
+
+ @Override
+ public void print(char[] s)
+ {
+ super.print(s);
+ for (PrintStream stream: newStreams)
+ stream.print(s);
+ }
+
+ @Override
+ public void print(String s)
+ {
+ super.print(s);
+ for (PrintStream stream: newStreams)
+ stream.print(s);
+ }
+
+ @Override
+ public void print(Object obj)
+ {
+ super.print(obj);
+ for (PrintStream s: newStreams)
+ s.print(obj);
+ }
+
+ @Override
+ public void println()
+ {
+ super.println();
+ for (PrintStream s: newStreams)
+ s.println();
+ }
+
+ @Override
+ public void println(boolean x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(char x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(int x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(long x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(float x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(double x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(char[] x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(String x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public void println(Object x)
+ {
+ super.println(x);
+ for (PrintStream s: newStreams)
+ s.println(x);
+ }
+
+ @Override
+ public PrintStream printf(String format, Object... args)
+ {
+ for (PrintStream s: newStreams)
+ s.printf(format, args);
+ return super.printf(format, args);
+ }
+
+ @Override
+ public PrintStream printf(Locale l, String format, Object... args)
+ {
+ for (PrintStream s: newStreams)
+ s.printf(l, format, args);
+ return super.printf(l, format, args);
+ }
+
+ @Override
+ public PrintStream append(CharSequence csq)
+ {
+ for (PrintStream s: newStreams)
+ s.append(csq);
+ return super.append(csq);
+ }
+
+ @Override
+ public PrintStream append(CharSequence csq, int start, int end)
+ {
+ for (PrintStream s: newStreams)
+ s.append(csq, start, end);
+ return super.append(csq, start, end);
+ }
+
+ @Override
+ public PrintStream append(char c)
+ {
+ for (PrintStream s: newStreams)
+ s.append(c);
+ return super.append(c);
+ }
+
+ @Override
+ public void write(byte[] b) throws IOException
+ {
+ super.write(b);
+ for (PrintStream s: newStreams)
+ s.write(b);
+ }
+
+}
diff --git a/tools/stress/src/org/apache/cassandra/stress/util/TimingInterval.java b/tools/stress/src/org/apache/cassandra/stress/util/TimingInterval.java
index 6be71c8..89995dd 100644
--- a/tools/stress/src/org/apache/cassandra/stress/util/TimingInterval.java
+++ b/tools/stress/src/org/apache/cassandra/stress/util/TimingInterval.java
@@ -27,6 +27,7 @@
// used for both single timer results and merged timer results
public final class TimingInterval
{
+ public static final long[] EMPTY_SAMPLE = new long[0];
// nanos
private final long start;
private final long end;
@@ -56,7 +57,7 @@
maxLatency = totalLatency = 0;
partitionCount = rowCount = operationCount = errorCount = 0;
pauseStart = pauseLength = 0;
- sample = new SampleOfLongs(new long[0], 1d);
+ sample = new SampleOfLongs(EMPTY_SAMPLE, 1d);
}
TimingInterval(long start, long end, long maxLatency, long pauseStart, long pauseLength, long partitionCount,
diff --git a/tools/stress/src/resources/org/apache/cassandra/stress/graph/graph.html b/tools/stress/src/resources/org/apache/cassandra/stress/graph/graph.html
new file mode 100644
index 0000000..54cd469
--- /dev/null
+++ b/tools/stress/src/resources/org/apache/cassandra/stress/graph/graph.html
@@ -0,0 +1,659 @@
+<!DOCTYPE html>
+
+<!-- cstar_perf (https://github.com/datastax/cstar_perf) graphing
+utility adapted for use directly with command line cassandra-stress -->
+
+<head>
+ <meta charset="utf-8">
+ <script language="javascript" type="text/javascript">
+ <!--
+
+/* stats start */
+
+/* stats end */
+
+/*! jQuery v1.11.1 | (c) 2005, 2014 jQuery Foundation, Inc. | jquery.org/license */
+!function(a,b){"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){var c=[],d=c.slice,e=c.concat,f=c.push,g=c.indexOf,h={},i=h.toString,j=h.hasOwnProperty,k={},l="1.11.1",m=function(a,b){return new m.fn.init(a,b)},n=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,o=/^-ms-/,p=/-([\da-z])/gi,q=function(a,b){return b.toUpperCase()};m.fn=m.prototype={jquery:l,constructor:m,selector:"",length:0,toArray:function(){return d.call(this)},get:function(a){return null!=a?0>a?this[a+this.length]:this[a]:d.call(this)},pushStack:function(a){var b=m.merge(this.constructor(),a);return b.prevObject=this,b.context=this.context,b},each:function(a,b){return m.each(this,a,b)},map:function(a){return this.pushStack(m.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(d.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(0>a?b:0);return this.pushStack(c>=0&&b>c?[this[c]]:[])},end:function(){return this.prevObject||this.constructor(null)},push:f,sort:c.sort,splice:c.splice},m.extend=m.fn.extend=function(){var a,b,c,d,e,f,g=arguments[0]||{},h=1,i=arguments.length,j=!1;for("boolean"==typeof g&&(j=g,g=arguments[h]||{},h++),"object"==typeof g||m.isFunction(g)||(g={}),h===i&&(g=this,h--);i>h;h++)if(null!=(e=arguments[h]))for(d in e)a=g[d],c=e[d],g!==c&&(j&&c&&(m.isPlainObject(c)||(b=m.isArray(c)))?(b?(b=!1,f=a&&m.isArray(a)?a:[]):f=a&&m.isPlainObject(a)?a:{},g[d]=m.extend(j,f,c)):void 0!==c&&(g[d]=c));return g},m.extend({expando:"jQuery"+(l+Math.random()).replace(/\D/g,""),isReady:!0,error:function(a){throw new Error(a)},noop:function(){},isFunction:function(a){return"function"===m.type(a)},isArray:Array.isArray||function(a){return"array"===m.type(a)},isWindow:function(a){return null!=a&&a==a.window},isNumeric:function(a){return!m.isArray(a)&&a-parseFloat(a)>=0},isEmptyObject:function(a){var b;for(b in a)return!1;return!0},isPlainObject:function(a){var b;if(!a||"object"!==m.type(a)||a.nodeType||m.isWindow(a))return!1;try{if(a.constructor&&!j.call(a,"constructor")&&!j.call(a.constructor.prototype,"isPrototypeOf"))return!1}catch(c){return!1}if(k.ownLast)for(b in a)return j.call(a,b);for(b in a);return void 0===b||j.call(a,b)},type:function(a){return null==a?a+"":"object"==typeof a||"function"==typeof a?h[i.call(a)]||"object":typeof a},globalEval:function(b){b&&m.trim(b)&&(a.execScript||function(b){a.eval.call(a,b)})(b)},camelCase:function(a){return a.replace(o,"ms-").replace(p,q)},nodeName:function(a,b){return a.nodeName&&a.nodeName.toLowerCase()===b.toLowerCase()},each:function(a,b,c){var d,e=0,f=a.length,g=r(a);if(c){if(g){for(;f>e;e++)if(d=b.apply(a[e],c),d===!1)break}else for(e in a)if(d=b.apply(a[e],c),d===!1)break}else if(g){for(;f>e;e++)if(d=b.call(a[e],e,a[e]),d===!1)break}else for(e in a)if(d=b.call(a[e],e,a[e]),d===!1)break;return a},trim:function(a){return null==a?"":(a+"").replace(n,"")},makeArray:function(a,b){var c=b||[];return null!=a&&(r(Object(a))?m.merge(c,"string"==typeof a?[a]:a):f.call(c,a)),c},inArray:function(a,b,c){var d;if(b){if(g)return g.call(b,a,c);for(d=b.length,c=c?0>c?Math.max(0,d+c):c:0;d>c;c++)if(c in b&&b[c]===a)return c}return-1},merge:function(a,b){var c=+b.length,d=0,e=a.length;while(c>d)a[e++]=b[d++];if(c!==c)while(void 0!==b[d])a[e++]=b[d++];return a.length=e,a},grep:function(a,b,c){for(var d,e=[],f=0,g=a.length,h=!c;g>f;f++)d=!b(a[f],f),d!==h&&e.push(a[f]);return e},map:function(a,b,c){var d,f=0,g=a.length,h=r(a),i=[];if(h)for(;g>f;f++)d=b(a[f],f,c),null!=d&&i.push(d);else for(f in a)d=b(a[f],f,c),null!=d&&i.push(d);return e.apply([],i)},guid:1,proxy:function(a,b){var c,e,f;return"string"==typeof b&&(f=a[b],b=a,a=f),m.isFunction(a)?(c=d.call(arguments,2),e=function(){return a.apply(b||this,c.concat(d.call(arguments)))},e.guid=a.guid=a.guid||m.guid++,e):void 0},now:function(){return+new Date},support:k}),m.each("Boolean Number String Function Array Date RegExp Object Error".split(" "),function(a,b){h["[object "+b+"]"]=b.toLowerCase()});function r(a){var b=a.length,c=m.type(a);return"function"===c||m.isWindow(a)?!1:1===a.nodeType&&b?!0:"array"===c||0===b||"number"==typeof b&&b>0&&b-1 in a}var s=function(a){var b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u="sizzle"+-new Date,v=a.document,w=0,x=0,y=gb(),z=gb(),A=gb(),B=function(a,b){return a===b&&(l=!0),0},C="undefined",D=1<<31,E={}.hasOwnProperty,F=[],G=F.pop,H=F.push,I=F.push,J=F.slice,K=F.indexOf||function(a){for(var b=0,c=this.length;c>b;b++)if(this[b]===a)return b;return-1},L="checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|ismap|loop|multiple|open|readonly|required|scoped",M="[\\x20\\t\\r\\n\\f]",N="(?:\\\\.|[\\w-]|[^\\x00-\\xa0])+",O=N.replace("w","w#"),P="\\["+M+"*("+N+")(?:"+M+"*([*^$|!~]?=)"+M+"*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|("+O+"))|)"+M+"*\\]",Q=":("+N+")(?:\\((('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|((?:\\\\.|[^\\\\()[\\]]|"+P+")*)|.*)\\)|)",R=new RegExp("^"+M+"+|((?:^|[^\\\\])(?:\\\\.)*)"+M+"+$","g"),S=new RegExp("^"+M+"*,"+M+"*"),T=new RegExp("^"+M+"*([>+~]|"+M+")"+M+"*"),U=new RegExp("="+M+"*([^\\]'\"]*?)"+M+"*\\]","g"),V=new RegExp(Q),W=new RegExp("^"+O+"$"),X={ID:new RegExp("^#("+N+")"),CLASS:new RegExp("^\\.("+N+")"),TAG:new RegExp("^("+N.replace("w","w*")+")"),ATTR:new RegExp("^"+P),PSEUDO:new RegExp("^"+Q),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+L+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+M+"*((?:-\\d)?\\d*)"+M+"*\\)|)(?=[^-]|$)","i")},Y=/^(?:input|select|textarea|button)$/i,Z=/^h\d$/i,$=/^[^{]+\{\s*\[native \w/,_=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,ab=/[+~]/,bb=/'|\\/g,cb=new RegExp("\\\\([\\da-f]{1,6}"+M+"?|("+M+")|.)","ig"),db=function(a,b,c){var d="0x"+b-65536;return d!==d||c?b:0>d?String.fromCharCode(d+65536):String.fromCharCode(d>>10|55296,1023&d|56320)};try{I.apply(F=J.call(v.childNodes),v.childNodes),F[v.childNodes.length].nodeType}catch(eb){I={apply:F.length?function(a,b){H.apply(a,J.call(b))}:function(a,b){var c=a.length,d=0;while(a[c++]=b[d++]);a.length=c-1}}}function fb(a,b,d,e){var f,h,j,k,l,o,r,s,w,x;if((b?b.ownerDocument||b:v)!==n&&m(b),b=b||n,d=d||[],!a||"string"!=typeof a)return d;if(1!==(k=b.nodeType)&&9!==k)return[];if(p&&!e){if(f=_.exec(a))if(j=f[1]){if(9===k){if(h=b.getElementById(j),!h||!h.parentNode)return d;if(h.id===j)return d.push(h),d}else if(b.ownerDocument&&(h=b.ownerDocument.getElementById(j))&&t(b,h)&&h.id===j)return d.push(h),d}else{if(f[2])return I.apply(d,b.getElementsByTagName(a)),d;if((j=f[3])&&c.getElementsByClassName&&b.getElementsByClassName)return I.apply(d,b.getElementsByClassName(j)),d}if(c.qsa&&(!q||!q.test(a))){if(s=r=u,w=b,x=9===k&&a,1===k&&"object"!==b.nodeName.toLowerCase()){o=g(a),(r=b.getAttribute("id"))?s=r.replace(bb,"\\$&"):b.setAttribute("id",s),s="[id='"+s+"'] ",l=o.length;while(l--)o[l]=s+qb(o[l]);w=ab.test(a)&&ob(b.parentNode)||b,x=o.join(",")}if(x)try{return I.apply(d,w.querySelectorAll(x)),d}catch(y){}finally{r||b.removeAttribute("id")}}}return i(a.replace(R,"$1"),b,d,e)}function gb(){var a=[];function b(c,e){return a.push(c+" ")>d.cacheLength&&delete b[a.shift()],b[c+" "]=e}return b}function hb(a){return a[u]=!0,a}function ib(a){var b=n.createElement("div");try{return!!a(b)}catch(c){return!1}finally{b.parentNode&&b.parentNode.removeChild(b),b=null}}function jb(a,b){var c=a.split("|"),e=a.length;while(e--)d.attrHandle[c[e]]=b}function kb(a,b){var c=b&&a,d=c&&1===a.nodeType&&1===b.nodeType&&(~b.sourceIndex||D)-(~a.sourceIndex||D);if(d)return d;if(c)while(c=c.nextSibling)if(c===b)return-1;return a?1:-1}function lb(a){return function(b){var c=b.nodeName.toLowerCase();return"input"===c&&b.type===a}}function mb(a){return function(b){var c=b.nodeName.toLowerCase();return("input"===c||"button"===c)&&b.type===a}}function nb(a){return hb(function(b){return b=+b,hb(function(c,d){var e,f=a([],c.length,b),g=f.length;while(g--)c[e=f[g]]&&(c[e]=!(d[e]=c[e]))})})}function ob(a){return a&&typeof a.getElementsByTagName!==C&&a}c=fb.support={},f=fb.isXML=function(a){var b=a&&(a.ownerDocument||a).documentElement;return b?"HTML"!==b.nodeName:!1},m=fb.setDocument=function(a){var b,e=a?a.ownerDocument||a:v,g=e.defaultView;return e!==n&&9===e.nodeType&&e.documentElement?(n=e,o=e.documentElement,p=!f(e),g&&g!==g.top&&(g.addEventListener?g.addEventListener("unload",function(){m()},!1):g.attachEvent&&g.attachEvent("onunload",function(){m()})),c.attributes=ib(function(a){return a.className="i",!a.getAttribute("className")}),c.getElementsByTagName=ib(function(a){return a.appendChild(e.createComment("")),!a.getElementsByTagName("*").length}),c.getElementsByClassName=$.test(e.getElementsByClassName)&&ib(function(a){return a.innerHTML="<div class='a'></div><div class='a i'></div>",a.firstChild.className="i",2===a.getElementsByClassName("i").length}),c.getById=ib(function(a){return o.appendChild(a).id=u,!e.getElementsByName||!e.getElementsByName(u).length}),c.getById?(d.find.ID=function(a,b){if(typeof b.getElementById!==C&&p){var c=b.getElementById(a);return c&&c.parentNode?[c]:[]}},d.filter.ID=function(a){var b=a.replace(cb,db);return function(a){return a.getAttribute("id")===b}}):(delete d.find.ID,d.filter.ID=function(a){var b=a.replace(cb,db);return function(a){var c=typeof a.getAttributeNode!==C&&a.getAttributeNode("id");return c&&c.value===b}}),d.find.TAG=c.getElementsByTagName?function(a,b){return typeof b.getElementsByTagName!==C?b.getElementsByTagName(a):void 0}:function(a,b){var c,d=[],e=0,f=b.getElementsByTagName(a);if("*"===a){while(c=f[e++])1===c.nodeType&&d.push(c);return d}return f},d.find.CLASS=c.getElementsByClassName&&function(a,b){return typeof b.getElementsByClassName!==C&&p?b.getElementsByClassName(a):void 0},r=[],q=[],(c.qsa=$.test(e.querySelectorAll))&&(ib(function(a){a.innerHTML="<select msallowclip=''><option selected=''></option></select>",a.querySelectorAll("[msallowclip^='']").length&&q.push("[*^$]="+M+"*(?:''|\"\")"),a.querySelectorAll("[selected]").length||q.push("\\["+M+"*(?:value|"+L+")"),a.querySelectorAll(":checked").length||q.push(":checked")}),ib(function(a){var b=e.createElement("input");b.setAttribute("type","hidden"),a.appendChild(b).setAttribute("name","D"),a.querySelectorAll("[name=d]").length&&q.push("name"+M+"*[*^$|!~]?="),a.querySelectorAll(":enabled").length||q.push(":enabled",":disabled"),a.querySelectorAll("*,:x"),q.push(",.*:")})),(c.matchesSelector=$.test(s=o.matches||o.webkitMatchesSelector||o.mozMatchesSelector||o.oMatchesSelector||o.msMatchesSelector))&&ib(function(a){c.disconnectedMatch=s.call(a,"div"),s.call(a,"[s!='']:x"),r.push("!=",Q)}),q=q.length&&new RegExp(q.join("|")),r=r.length&&new RegExp(r.join("|")),b=$.test(o.compareDocumentPosition),t=b||$.test(o.contains)?function(a,b){var c=9===a.nodeType?a.documentElement:a,d=b&&b.parentNode;return a===d||!(!d||1!==d.nodeType||!(c.contains?c.contains(d):a.compareDocumentPosition&&16&a.compareDocumentPosition(d)))}:function(a,b){if(b)while(b=b.parentNode)if(b===a)return!0;return!1},B=b?function(a,b){if(a===b)return l=!0,0;var d=!a.compareDocumentPosition-!b.compareDocumentPosition;return d?d:(d=(a.ownerDocument||a)===(b.ownerDocument||b)?a.compareDocumentPosition(b):1,1&d||!c.sortDetached&&b.compareDocumentPosition(a)===d?a===e||a.ownerDocument===v&&t(v,a)?-1:b===e||b.ownerDocument===v&&t(v,b)?1:k?K.call(k,a)-K.call(k,b):0:4&d?-1:1)}:function(a,b){if(a===b)return l=!0,0;var c,d=0,f=a.parentNode,g=b.parentNode,h=[a],i=[b];if(!f||!g)return a===e?-1:b===e?1:f?-1:g?1:k?K.call(k,a)-K.call(k,b):0;if(f===g)return kb(a,b);c=a;while(c=c.parentNode)h.unshift(c);c=b;while(c=c.parentNode)i.unshift(c);while(h[d]===i[d])d++;return d?kb(h[d],i[d]):h[d]===v?-1:i[d]===v?1:0},e):n},fb.matches=function(a,b){return fb(a,null,null,b)},fb.matchesSelector=function(a,b){if((a.ownerDocument||a)!==n&&m(a),b=b.replace(U,"='$1']"),!(!c.matchesSelector||!p||r&&r.test(b)||q&&q.test(b)))try{var d=s.call(a,b);if(d||c.disconnectedMatch||a.document&&11!==a.document.nodeType)return d}catch(e){}return fb(b,n,null,[a]).length>0},fb.contains=function(a,b){return(a.ownerDocument||a)!==n&&m(a),t(a,b)},fb.attr=function(a,b){(a.ownerDocument||a)!==n&&m(a);var e=d.attrHandle[b.toLowerCase()],f=e&&E.call(d.attrHandle,b.toLowerCase())?e(a,b,!p):void 0;return void 0!==f?f:c.attributes||!p?a.getAttribute(b):(f=a.getAttributeNode(b))&&f.specified?f.value:null},fb.error=function(a){throw new Error("Syntax error, unrecognized expression: "+a)},fb.uniqueSort=function(a){var b,d=[],e=0,f=0;if(l=!c.detectDuplicates,k=!c.sortStable&&a.slice(0),a.sort(B),l){while(b=a[f++])b===a[f]&&(e=d.push(f));while(e--)a.splice(d[e],1)}return k=null,a},e=fb.getText=function(a){var b,c="",d=0,f=a.nodeType;if(f){if(1===f||9===f||11===f){if("string"==typeof a.textContent)return a.textContent;for(a=a.firstChild;a;a=a.nextSibling)c+=e(a)}else if(3===f||4===f)return a.nodeValue}else while(b=a[d++])c+=e(b);return c},d=fb.selectors={cacheLength:50,createPseudo:hb,match:X,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(a){return a[1]=a[1].replace(cb,db),a[3]=(a[3]||a[4]||a[5]||"").replace(cb,db),"~="===a[2]&&(a[3]=" "+a[3]+" "),a.slice(0,4)},CHILD:function(a){return a[1]=a[1].toLowerCase(),"nth"===a[1].slice(0,3)?(a[3]||fb.error(a[0]),a[4]=+(a[4]?a[5]+(a[6]||1):2*("even"===a[3]||"odd"===a[3])),a[5]=+(a[7]+a[8]||"odd"===a[3])):a[3]&&fb.error(a[0]),a},PSEUDO:function(a){var b,c=!a[6]&&a[2];return X.CHILD.test(a[0])?null:(a[3]?a[2]=a[4]||a[5]||"":c&&V.test(c)&&(b=g(c,!0))&&(b=c.indexOf(")",c.length-b)-c.length)&&(a[0]=a[0].slice(0,b),a[2]=c.slice(0,b)),a.slice(0,3))}},filter:{TAG:function(a){var b=a.replace(cb,db).toLowerCase();return"*"===a?function(){return!0}:function(a){return a.nodeName&&a.nodeName.toLowerCase()===b}},CLASS:function(a){var b=y[a+" "];return b||(b=new RegExp("(^|"+M+")"+a+"("+M+"|$)"))&&y(a,function(a){return b.test("string"==typeof a.className&&a.className||typeof a.getAttribute!==C&&a.getAttribute("class")||"")})},ATTR:function(a,b,c){return function(d){var e=fb.attr(d,a);return null==e?"!="===b:b?(e+="","="===b?e===c:"!="===b?e!==c:"^="===b?c&&0===e.indexOf(c):"*="===b?c&&e.indexOf(c)>-1:"$="===b?c&&e.slice(-c.length)===c:"~="===b?(" "+e+" ").indexOf(c)>-1:"|="===b?e===c||e.slice(0,c.length+1)===c+"-":!1):!0}},CHILD:function(a,b,c,d,e){var f="nth"!==a.slice(0,3),g="last"!==a.slice(-4),h="of-type"===b;return 1===d&&0===e?function(a){return!!a.parentNode}:function(b,c,i){var j,k,l,m,n,o,p=f!==g?"nextSibling":"previousSibling",q=b.parentNode,r=h&&b.nodeName.toLowerCase(),s=!i&&!h;if(q){if(f){while(p){l=b;while(l=l[p])if(h?l.nodeName.toLowerCase()===r:1===l.nodeType)return!1;o=p="only"===a&&!o&&"nextSibling"}return!0}if(o=[g?q.firstChild:q.lastChild],g&&s){k=q[u]||(q[u]={}),j=k[a]||[],n=j[0]===w&&j[1],m=j[0]===w&&j[2],l=n&&q.childNodes[n];while(l=++n&&l&&l[p]||(m=n=0)||o.pop())if(1===l.nodeType&&++m&&l===b){k[a]=[w,n,m];break}}else if(s&&(j=(b[u]||(b[u]={}))[a])&&j[0]===w)m=j[1];else while(l=++n&&l&&l[p]||(m=n=0)||o.pop())if((h?l.nodeName.toLowerCase()===r:1===l.nodeType)&&++m&&(s&&((l[u]||(l[u]={}))[a]=[w,m]),l===b))break;return m-=e,m===d||m%d===0&&m/d>=0}}},PSEUDO:function(a,b){var c,e=d.pseudos[a]||d.setFilters[a.toLowerCase()]||fb.error("unsupported pseudo: "+a);return e[u]?e(b):e.length>1?(c=[a,a,"",b],d.setFilters.hasOwnProperty(a.toLowerCase())?hb(function(a,c){var d,f=e(a,b),g=f.length;while(g--)d=K.call(a,f[g]),a[d]=!(c[d]=f[g])}):function(a){return e(a,0,c)}):e}},pseudos:{not:hb(function(a){var b=[],c=[],d=h(a.replace(R,"$1"));return d[u]?hb(function(a,b,c,e){var f,g=d(a,null,e,[]),h=a.length;while(h--)(f=g[h])&&(a[h]=!(b[h]=f))}):function(a,e,f){return b[0]=a,d(b,null,f,c),!c.pop()}}),has:hb(function(a){return function(b){return fb(a,b).length>0}}),contains:hb(function(a){return function(b){return(b.textContent||b.innerText||e(b)).indexOf(a)>-1}}),lang:hb(function(a){return W.test(a||"")||fb.error("unsupported lang: "+a),a=a.replace(cb,db).toLowerCase(),function(b){var c;do if(c=p?b.lang:b.getAttribute("xml:lang")||b.getAttribute("lang"))return c=c.toLowerCase(),c===a||0===c.indexOf(a+"-");while((b=b.parentNode)&&1===b.nodeType);return!1}}),target:function(b){var c=a.location&&a.location.hash;return c&&c.slice(1)===b.id},root:function(a){return a===o},focus:function(a){return a===n.activeElement&&(!n.hasFocus||n.hasFocus())&&!!(a.type||a.href||~a.tabIndex)},enabled:function(a){return a.disabled===!1},disabled:function(a){return a.disabled===!0},checked:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&!!a.checked||"option"===b&&!!a.selected},selected:function(a){return a.parentNode&&a.parentNode.selectedIndex,a.selected===!0},empty:function(a){for(a=a.firstChild;a;a=a.nextSibling)if(a.nodeType<6)return!1;return!0},parent:function(a){return!d.pseudos.empty(a)},header:function(a){return Z.test(a.nodeName)},input:function(a){return Y.test(a.nodeName)},button:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&"button"===a.type||"button"===b},text:function(a){var b;return"input"===a.nodeName.toLowerCase()&&"text"===a.type&&(null==(b=a.getAttribute("type"))||"text"===b.toLowerCase())},first:nb(function(){return[0]}),last:nb(function(a,b){return[b-1]}),eq:nb(function(a,b,c){return[0>c?c+b:c]}),even:nb(function(a,b){for(var c=0;b>c;c+=2)a.push(c);return a}),odd:nb(function(a,b){for(var c=1;b>c;c+=2)a.push(c);return a}),lt:nb(function(a,b,c){for(var d=0>c?c+b:c;--d>=0;)a.push(d);return a}),gt:nb(function(a,b,c){for(var d=0>c?c+b:c;++d<b;)a.push(d);return a})}},d.pseudos.nth=d.pseudos.eq;for(b in{radio:!0,checkbox:!0,file:!0,password:!0,image:!0})d.pseudos[b]=lb(b);for(b in{submit:!0,reset:!0})d.pseudos[b]=mb(b);function pb(){}pb.prototype=d.filters=d.pseudos,d.setFilters=new pb,g=fb.tokenize=function(a,b){var c,e,f,g,h,i,j,k=z[a+" "];if(k)return b?0:k.slice(0);h=a,i=[],j=d.preFilter;while(h){(!c||(e=S.exec(h)))&&(e&&(h=h.slice(e[0].length)||h),i.push(f=[])),c=!1,(e=T.exec(h))&&(c=e.shift(),f.push({value:c,type:e[0].replace(R," ")}),h=h.slice(c.length));for(g in d.filter)!(e=X[g].exec(h))||j[g]&&!(e=j[g](e))||(c=e.shift(),f.push({value:c,type:g,matches:e}),h=h.slice(c.length));if(!c)break}return b?h.length:h?fb.error(a):z(a,i).slice(0)};function qb(a){for(var b=0,c=a.length,d="";c>b;b++)d+=a[b].value;return d}function rb(a,b,c){var d=b.dir,e=c&&"parentNode"===d,f=x++;return b.first?function(b,c,f){while(b=b[d])if(1===b.nodeType||e)return a(b,c,f)}:function(b,c,g){var h,i,j=[w,f];if(g){while(b=b[d])if((1===b.nodeType||e)&&a(b,c,g))return!0}else while(b=b[d])if(1===b.nodeType||e){if(i=b[u]||(b[u]={}),(h=i[d])&&h[0]===w&&h[1]===f)return j[2]=h[2];if(i[d]=j,j[2]=a(b,c,g))return!0}}}function sb(a){return a.length>1?function(b,c,d){var e=a.length;while(e--)if(!a[e](b,c,d))return!1;return!0}:a[0]}function tb(a,b,c){for(var d=0,e=b.length;e>d;d++)fb(a,b[d],c);return c}function ub(a,b,c,d,e){for(var f,g=[],h=0,i=a.length,j=null!=b;i>h;h++)(f=a[h])&&(!c||c(f,d,e))&&(g.push(f),j&&b.push(h));return g}function vb(a,b,c,d,e,f){return d&&!d[u]&&(d=vb(d)),e&&!e[u]&&(e=vb(e,f)),hb(function(f,g,h,i){var j,k,l,m=[],n=[],o=g.length,p=f||tb(b||"*",h.nodeType?[h]:h,[]),q=!a||!f&&b?p:ub(p,m,a,h,i),r=c?e||(f?a:o||d)?[]:g:q;if(c&&c(q,r,h,i),d){j=ub(r,n),d(j,[],h,i),k=j.length;while(k--)(l=j[k])&&(r[n[k]]=!(q[n[k]]=l))}if(f){if(e||a){if(e){j=[],k=r.length;while(k--)(l=r[k])&&j.push(q[k]=l);e(null,r=[],j,i)}k=r.length;while(k--)(l=r[k])&&(j=e?K.call(f,l):m[k])>-1&&(f[j]=!(g[j]=l))}}else r=ub(r===g?r.splice(o,r.length):r),e?e(null,g,r,i):I.apply(g,r)})}function wb(a){for(var b,c,e,f=a.length,g=d.relative[a[0].type],h=g||d.relative[" "],i=g?1:0,k=rb(function(a){return a===b},h,!0),l=rb(function(a){return K.call(b,a)>-1},h,!0),m=[function(a,c,d){return!g&&(d||c!==j)||((b=c).nodeType?k(a,c,d):l(a,c,d))}];f>i;i++)if(c=d.relative[a[i].type])m=[rb(sb(m),c)];else{if(c=d.filter[a[i].type].apply(null,a[i].matches),c[u]){for(e=++i;f>e;e++)if(d.relative[a[e].type])break;return vb(i>1&&sb(m),i>1&&qb(a.slice(0,i-1).concat({value:" "===a[i-2].type?"*":""})).replace(R,"$1"),c,e>i&&wb(a.slice(i,e)),f>e&&wb(a=a.slice(e)),f>e&&qb(a))}m.push(c)}return sb(m)}function xb(a,b){var c=b.length>0,e=a.length>0,f=function(f,g,h,i,k){var l,m,o,p=0,q="0",r=f&&[],s=[],t=j,u=f||e&&d.find.TAG("*",k),v=w+=null==t?1:Math.random()||.1,x=u.length;for(k&&(j=g!==n&&g);q!==x&&null!=(l=u[q]);q++){if(e&&l){m=0;while(o=a[m++])if(o(l,g,h)){i.push(l);break}k&&(w=v)}c&&((l=!o&&l)&&p--,f&&r.push(l))}if(p+=q,c&&q!==p){m=0;while(o=b[m++])o(r,s,g,h);if(f){if(p>0)while(q--)r[q]||s[q]||(s[q]=G.call(i));s=ub(s)}I.apply(i,s),k&&!f&&s.length>0&&p+b.length>1&&fb.uniqueSort(i)}return k&&(w=v,j=t),r};return c?hb(f):f}return h=fb.compile=function(a,b){var c,d=[],e=[],f=A[a+" "];if(!f){b||(b=g(a)),c=b.length;while(c--)f=wb(b[c]),f[u]?d.push(f):e.push(f);f=A(a,xb(e,d)),f.selector=a}return f},i=fb.select=function(a,b,e,f){var i,j,k,l,m,n="function"==typeof a&&a,o=!f&&g(a=n.selector||a);if(e=e||[],1===o.length){if(j=o[0]=o[0].slice(0),j.length>2&&"ID"===(k=j[0]).type&&c.getById&&9===b.nodeType&&p&&d.relative[j[1].type]){if(b=(d.find.ID(k.matches[0].replace(cb,db),b)||[])[0],!b)return e;n&&(b=b.parentNode),a=a.slice(j.shift().value.length)}i=X.needsContext.test(a)?0:j.length;while(i--){if(k=j[i],d.relative[l=k.type])break;if((m=d.find[l])&&(f=m(k.matches[0].replace(cb,db),ab.test(j[0].type)&&ob(b.parentNode)||b))){if(j.splice(i,1),a=f.length&&qb(j),!a)return I.apply(e,f),e;break}}}return(n||h(a,o))(f,b,!p,e,ab.test(a)&&ob(b.parentNode)||b),e},c.sortStable=u.split("").sort(B).join("")===u,c.detectDuplicates=!!l,m(),c.sortDetached=ib(function(a){return 1&a.compareDocumentPosition(n.createElement("div"))}),ib(function(a){return a.innerHTML="<a href='#'></a>","#"===a.firstChild.getAttribute("href")})||jb("type|href|height|width",function(a,b,c){return c?void 0:a.getAttribute(b,"type"===b.toLowerCase()?1:2)}),c.attributes&&ib(function(a){return a.innerHTML="<input/>",a.firstChild.setAttribute("value",""),""===a.firstChild.getAttribute("value")})||jb("value",function(a,b,c){return c||"input"!==a.nodeName.toLowerCase()?void 0:a.defaultValue}),ib(function(a){return null==a.getAttribute("disabled")})||jb(L,function(a,b,c){var d;return c?void 0:a[b]===!0?b.toLowerCase():(d=a.getAttributeNode(b))&&d.specified?d.value:null}),fb}(a);m.find=s,m.expr=s.selectors,m.expr[":"]=m.expr.pseudos,m.unique=s.uniqueSort,m.text=s.getText,m.isXMLDoc=s.isXML,m.contains=s.contains;var t=m.expr.match.needsContext,u=/^<(\w+)\s*\/?>(?:<\/\1>|)$/,v=/^.[^:#\[\.,]*$/;function w(a,b,c){if(m.isFunction(b))return m.grep(a,function(a,d){return!!b.call(a,d,a)!==c});if(b.nodeType)return m.grep(a,function(a){return a===b!==c});if("string"==typeof b){if(v.test(b))return m.filter(b,a,c);b=m.filter(b,a)}return m.grep(a,function(a){return m.inArray(a,b)>=0!==c})}m.filter=function(a,b,c){var d=b[0];return c&&(a=":not("+a+")"),1===b.length&&1===d.nodeType?m.find.matchesSelector(d,a)?[d]:[]:m.find.matches(a,m.grep(b,function(a){return 1===a.nodeType}))},m.fn.extend({find:function(a){var b,c=[],d=this,e=d.length;if("string"!=typeof a)return this.pushStack(m(a).filter(function(){for(b=0;e>b;b++)if(m.contains(d[b],this))return!0}));for(b=0;e>b;b++)m.find(a,d[b],c);return c=this.pushStack(e>1?m.unique(c):c),c.selector=this.selector?this.selector+" "+a:a,c},filter:function(a){return this.pushStack(w(this,a||[],!1))},not:function(a){return this.pushStack(w(this,a||[],!0))},is:function(a){return!!w(this,"string"==typeof a&&t.test(a)?m(a):a||[],!1).length}});var x,y=a.document,z=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]*))$/,A=m.fn.init=function(a,b){var c,d;if(!a)return this;if("string"==typeof a){if(c="<"===a.charAt(0)&&">"===a.charAt(a.length-1)&&a.length>=3?[null,a,null]:z.exec(a),!c||!c[1]&&b)return!b||b.jquery?(b||x).find(a):this.constructor(b).find(a);if(c[1]){if(b=b instanceof m?b[0]:b,m.merge(this,m.parseHTML(c[1],b&&b.nodeType?b.ownerDocument||b:y,!0)),u.test(c[1])&&m.isPlainObject(b))for(c in b)m.isFunction(this[c])?this[c](b[c]):this.attr(c,b[c]);return this}if(d=y.getElementById(c[2]),d&&d.parentNode){if(d.id!==c[2])return x.find(a);this.length=1,this[0]=d}return this.context=y,this.selector=a,this}return a.nodeType?(this.context=this[0]=a,this.length=1,this):m.isFunction(a)?"undefined"!=typeof x.ready?x.ready(a):a(m):(void 0!==a.selector&&(this.selector=a.selector,this.context=a.context),m.makeArray(a,this))};A.prototype=m.fn,x=m(y);var B=/^(?:parents|prev(?:Until|All))/,C={children:!0,contents:!0,next:!0,prev:!0};m.extend({dir:function(a,b,c){var d=[],e=a[b];while(e&&9!==e.nodeType&&(void 0===c||1!==e.nodeType||!m(e).is(c)))1===e.nodeType&&d.push(e),e=e[b];return d},sibling:function(a,b){for(var c=[];a;a=a.nextSibling)1===a.nodeType&&a!==b&&c.push(a);return c}}),m.fn.extend({has:function(a){var b,c=m(a,this),d=c.length;return this.filter(function(){for(b=0;d>b;b++)if(m.contains(this,c[b]))return!0})},closest:function(a,b){for(var c,d=0,e=this.length,f=[],g=t.test(a)||"string"!=typeof a?m(a,b||this.context):0;e>d;d++)for(c=this[d];c&&c!==b;c=c.parentNode)if(c.nodeType<11&&(g?g.index(c)>-1:1===c.nodeType&&m.find.matchesSelector(c,a))){f.push(c);break}return this.pushStack(f.length>1?m.unique(f):f)},index:function(a){return a?"string"==typeof a?m.inArray(this[0],m(a)):m.inArray(a.jquery?a[0]:a,this):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(a,b){return this.pushStack(m.unique(m.merge(this.get(),m(a,b))))},addBack:function(a){return this.add(null==a?this.prevObject:this.prevObject.filter(a))}});function D(a,b){do a=a[b];while(a&&1!==a.nodeType);return a}m.each({parent:function(a){var b=a.parentNode;return b&&11!==b.nodeType?b:null},parents:function(a){return m.dir(a,"parentNode")},parentsUntil:function(a,b,c){return m.dir(a,"parentNode",c)},next:function(a){return D(a,"nextSibling")},prev:function(a){return D(a,"previousSibling")},nextAll:function(a){return m.dir(a,"nextSibling")},prevAll:function(a){return m.dir(a,"previousSibling")},nextUntil:function(a,b,c){return m.dir(a,"nextSibling",c)},prevUntil:function(a,b,c){return m.dir(a,"previousSibling",c)},siblings:function(a){return m.sibling((a.parentNode||{}).firstChild,a)},children:function(a){return m.sibling(a.firstChild)},contents:function(a){return m.nodeName(a,"iframe")?a.contentDocument||a.contentWindow.document:m.merge([],a.childNodes)}},function(a,b){m.fn[a]=function(c,d){var e=m.map(this,b,c);return"Until"!==a.slice(-5)&&(d=c),d&&"string"==typeof d&&(e=m.filter(d,e)),this.length>1&&(C[a]||(e=m.unique(e)),B.test(a)&&(e=e.reverse())),this.pushStack(e)}});var E=/\S+/g,F={};function G(a){var b=F[a]={};return m.each(a.match(E)||[],function(a,c){b[c]=!0}),b}m.Callbacks=function(a){a="string"==typeof a?F[a]||G(a):m.extend({},a);var b,c,d,e,f,g,h=[],i=!a.once&&[],j=function(l){for(c=a.memory&&l,d=!0,f=g||0,g=0,e=h.length,b=!0;h&&e>f;f++)if(h[f].apply(l[0],l[1])===!1&&a.stopOnFalse){c=!1;break}b=!1,h&&(i?i.length&&j(i.shift()):c?h=[]:k.disable())},k={add:function(){if(h){var d=h.length;!function f(b){m.each(b,function(b,c){var d=m.type(c);"function"===d?a.unique&&k.has(c)||h.push(c):c&&c.length&&"string"!==d&&f(c)})}(arguments),b?e=h.length:c&&(g=d,j(c))}return this},remove:function(){return h&&m.each(arguments,function(a,c){var d;while((d=m.inArray(c,h,d))>-1)h.splice(d,1),b&&(e>=d&&e--,f>=d&&f--)}),this},has:function(a){return a?m.inArray(a,h)>-1:!(!h||!h.length)},empty:function(){return h=[],e=0,this},disable:function(){return h=i=c=void 0,this},disabled:function(){return!h},lock:function(){return i=void 0,c||k.disable(),this},locked:function(){return!i},fireWith:function(a,c){return!h||d&&!i||(c=c||[],c=[a,c.slice?c.slice():c],b?i.push(c):j(c)),this},fire:function(){return k.fireWith(this,arguments),this},fired:function(){return!!d}};return k},m.extend({Deferred:function(a){var b=[["resolve","done",m.Callbacks("once memory"),"resolved"],["reject","fail",m.Callbacks("once memory"),"rejected"],["notify","progress",m.Callbacks("memory")]],c="pending",d={state:function(){return c},always:function(){return e.done(arguments).fail(arguments),this},then:function(){var a=arguments;return m.Deferred(function(c){m.each(b,function(b,f){var g=m.isFunction(a[b])&&a[b];e[f[1]](function(){var a=g&&g.apply(this,arguments);a&&m.isFunction(a.promise)?a.promise().done(c.resolve).fail(c.reject).progress(c.notify):c[f[0]+"With"](this===d?c.promise():this,g?[a]:arguments)})}),a=null}).promise()},promise:function(a){return null!=a?m.extend(a,d):d}},e={};return d.pipe=d.then,m.each(b,function(a,f){var g=f[2],h=f[3];d[f[1]]=g.add,h&&g.add(function(){c=h},b[1^a][2].disable,b[2][2].lock),e[f[0]]=function(){return e[f[0]+"With"](this===e?d:this,arguments),this},e[f[0]+"With"]=g.fireWith}),d.promise(e),a&&a.call(e,e),e},when:function(a){var b=0,c=d.call(arguments),e=c.length,f=1!==e||a&&m.isFunction(a.promise)?e:0,g=1===f?a:m.Deferred(),h=function(a,b,c){return function(e){b[a]=this,c[a]=arguments.length>1?d.call(arguments):e,c===i?g.notifyWith(b,c):--f||g.resolveWith(b,c)}},i,j,k;if(e>1)for(i=new Array(e),j=new Array(e),k=new Array(e);e>b;b++)c[b]&&m.isFunction(c[b].promise)?c[b].promise().done(h(b,k,c)).fail(g.reject).progress(h(b,j,i)):--f;return f||g.resolveWith(k,c),g.promise()}});var H;m.fn.ready=function(a){return m.ready.promise().done(a),this},m.extend({isReady:!1,readyWait:1,holdReady:function(a){a?m.readyWait++:m.ready(!0)},ready:function(a){if(a===!0?!--m.readyWait:!m.isReady){if(!y.body)return setTimeout(m.ready);m.isReady=!0,a!==!0&&--m.readyWait>0||(H.resolveWith(y,[m]),m.fn.triggerHandler&&(m(y).triggerHandler("ready"),m(y).off("ready")))}}});function I(){y.addEventListener?(y.removeEventListener("DOMContentLoaded",J,!1),a.removeEventListener("load",J,!1)):(y.detachEvent("onreadystatechange",J),a.detachEvent("onload",J))}function J(){(y.addEventListener||"load"===event.type||"complete"===y.readyState)&&(I(),m.ready())}m.ready.promise=function(b){if(!H)if(H=m.Deferred(),"complete"===y.readyState)setTimeout(m.ready);else if(y.addEventListener)y.addEventListener("DOMContentLoaded",J,!1),a.addEventListener("load",J,!1);else{y.attachEvent("onreadystatechange",J),a.attachEvent("onload",J);var c=!1;try{c=null==a.frameElement&&y.documentElement}catch(d){}c&&c.doScroll&&!function e(){if(!m.isReady){try{c.doScroll("left")}catch(a){return setTimeout(e,50)}I(),m.ready()}}()}return H.promise(b)};var K="undefined",L;for(L in m(k))break;k.ownLast="0"!==L,k.inlineBlockNeedsLayout=!1,m(function(){var a,b,c,d;c=y.getElementsByTagName("body")[0],c&&c.style&&(b=y.createElement("div"),d=y.createElement("div"),d.style.cssText="position:absolute;border:0;width:0;height:0;top:0;left:-9999px",c.appendChild(d).appendChild(b),typeof b.style.zoom!==K&&(b.style.cssText="display:inline;margin:0;border:0;padding:1px;width:1px;zoom:1",k.inlineBlockNeedsLayout=a=3===b.offsetWidth,a&&(c.style.zoom=1)),c.removeChild(d))}),function(){var a=y.createElement("div");if(null==k.deleteExpando){k.deleteExpando=!0;try{delete a.test}catch(b){k.deleteExpando=!1}}a=null}(),m.acceptData=function(a){var b=m.noData[(a.nodeName+" ").toLowerCase()],c=+a.nodeType||1;return 1!==c&&9!==c?!1:!b||b!==!0&&a.getAttribute("classid")===b};var M=/^(?:\{[\w\W]*\}|\[[\w\W]*\])$/,N=/([A-Z])/g;function O(a,b,c){if(void 0===c&&1===a.nodeType){var d="data-"+b.replace(N,"-$1").toLowerCase();if(c=a.getAttribute(d),"string"==typeof c){try{c="true"===c?!0:"false"===c?!1:"null"===c?null:+c+""===c?+c:M.test(c)?m.parseJSON(c):c}catch(e){}m.data(a,b,c)}else c=void 0}return c}function P(a){var b;for(b in a)if(("data"!==b||!m.isEmptyObject(a[b]))&&"toJSON"!==b)return!1;return!0}function Q(a,b,d,e){if(m.acceptData(a)){var f,g,h=m.expando,i=a.nodeType,j=i?m.cache:a,k=i?a[h]:a[h]&&h;
+if(k&&j[k]&&(e||j[k].data)||void 0!==d||"string"!=typeof b)return k||(k=i?a[h]=c.pop()||m.guid++:h),j[k]||(j[k]=i?{}:{toJSON:m.noop}),("object"==typeof b||"function"==typeof b)&&(e?j[k]=m.extend(j[k],b):j[k].data=m.extend(j[k].data,b)),g=j[k],e||(g.data||(g.data={}),g=g.data),void 0!==d&&(g[m.camelCase(b)]=d),"string"==typeof b?(f=g[b],null==f&&(f=g[m.camelCase(b)])):f=g,f}}function R(a,b,c){if(m.acceptData(a)){var d,e,f=a.nodeType,g=f?m.cache:a,h=f?a[m.expando]:m.expando;if(g[h]){if(b&&(d=c?g[h]:g[h].data)){m.isArray(b)?b=b.concat(m.map(b,m.camelCase)):b in d?b=[b]:(b=m.camelCase(b),b=b in d?[b]:b.split(" ")),e=b.length;while(e--)delete d[b[e]];if(c?!P(d):!m.isEmptyObject(d))return}(c||(delete g[h].data,P(g[h])))&&(f?m.cleanData([a],!0):k.deleteExpando||g!=g.window?delete g[h]:g[h]=null)}}}m.extend({cache:{},noData:{"applet ":!0,"embed ":!0,"object ":"clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"},hasData:function(a){return a=a.nodeType?m.cache[a[m.expando]]:a[m.expando],!!a&&!P(a)},data:function(a,b,c){return Q(a,b,c)},removeData:function(a,b){return R(a,b)},_data:function(a,b,c){return Q(a,b,c,!0)},_removeData:function(a,b){return R(a,b,!0)}}),m.fn.extend({data:function(a,b){var c,d,e,f=this[0],g=f&&f.attributes;if(void 0===a){if(this.length&&(e=m.data(f),1===f.nodeType&&!m._data(f,"parsedAttrs"))){c=g.length;while(c--)g[c]&&(d=g[c].name,0===d.indexOf("data-")&&(d=m.camelCase(d.slice(5)),O(f,d,e[d])));m._data(f,"parsedAttrs",!0)}return e}return"object"==typeof a?this.each(function(){m.data(this,a)}):arguments.length>1?this.each(function(){m.data(this,a,b)}):f?O(f,a,m.data(f,a)):void 0},removeData:function(a){return this.each(function(){m.removeData(this,a)})}}),m.extend({queue:function(a,b,c){var d;return a?(b=(b||"fx")+"queue",d=m._data(a,b),c&&(!d||m.isArray(c)?d=m._data(a,b,m.makeArray(c)):d.push(c)),d||[]):void 0},dequeue:function(a,b){b=b||"fx";var c=m.queue(a,b),d=c.length,e=c.shift(),f=m._queueHooks(a,b),g=function(){m.dequeue(a,b)};"inprogress"===e&&(e=c.shift(),d--),e&&("fx"===b&&c.unshift("inprogress"),delete f.stop,e.call(a,g,f)),!d&&f&&f.empty.fire()},_queueHooks:function(a,b){var c=b+"queueHooks";return m._data(a,c)||m._data(a,c,{empty:m.Callbacks("once memory").add(function(){m._removeData(a,b+"queue"),m._removeData(a,c)})})}}),m.fn.extend({queue:function(a,b){var c=2;return"string"!=typeof a&&(b=a,a="fx",c--),arguments.length<c?m.queue(this[0],a):void 0===b?this:this.each(function(){var c=m.queue(this,a,b);m._queueHooks(this,a),"fx"===a&&"inprogress"!==c[0]&&m.dequeue(this,a)})},dequeue:function(a){return this.each(function(){m.dequeue(this,a)})},clearQueue:function(a){return this.queue(a||"fx",[])},promise:function(a,b){var c,d=1,e=m.Deferred(),f=this,g=this.length,h=function(){--d||e.resolveWith(f,[f])};"string"!=typeof a&&(b=a,a=void 0),a=a||"fx";while(g--)c=m._data(f[g],a+"queueHooks"),c&&c.empty&&(d++,c.empty.add(h));return h(),e.promise(b)}});var S=/[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/.source,T=["Top","Right","Bottom","Left"],U=function(a,b){return a=b||a,"none"===m.css(a,"display")||!m.contains(a.ownerDocument,a)},V=m.access=function(a,b,c,d,e,f,g){var h=0,i=a.length,j=null==c;if("object"===m.type(c)){e=!0;for(h in c)m.access(a,b,h,c[h],!0,f,g)}else if(void 0!==d&&(e=!0,m.isFunction(d)||(g=!0),j&&(g?(b.call(a,d),b=null):(j=b,b=function(a,b,c){return j.call(m(a),c)})),b))for(;i>h;h++)b(a[h],c,g?d:d.call(a[h],h,b(a[h],c)));return e?a:j?b.call(a):i?b(a[0],c):f},W=/^(?:checkbox|radio)$/i;!function(){var a=y.createElement("input"),b=y.createElement("div"),c=y.createDocumentFragment();if(b.innerHTML=" <link/><table></table><a href='/a'>a</a><input type='checkbox'/>",k.leadingWhitespace=3===b.firstChild.nodeType,k.tbody=!b.getElementsByTagName("tbody").length,k.htmlSerialize=!!b.getElementsByTagName("link").length,k.html5Clone="<:nav></:nav>"!==y.createElement("nav").cloneNode(!0).outerHTML,a.type="checkbox",a.checked=!0,c.appendChild(a),k.appendChecked=a.checked,b.innerHTML="<textarea>x</textarea>",k.noCloneChecked=!!b.cloneNode(!0).lastChild.defaultValue,c.appendChild(b),b.innerHTML="<input type='radio' checked='checked' name='t'/>",k.checkClone=b.cloneNode(!0).cloneNode(!0).lastChild.checked,k.noCloneEvent=!0,b.attachEvent&&(b.attachEvent("onclick",function(){k.noCloneEvent=!1}),b.cloneNode(!0).click()),null==k.deleteExpando){k.deleteExpando=!0;try{delete b.test}catch(d){k.deleteExpando=!1}}}(),function(){var b,c,d=y.createElement("div");for(b in{submit:!0,change:!0,focusin:!0})c="on"+b,(k[b+"Bubbles"]=c in a)||(d.setAttribute(c,"t"),k[b+"Bubbles"]=d.attributes[c].expando===!1);d=null}();var X=/^(?:input|select|textarea)$/i,Y=/^key/,Z=/^(?:mouse|pointer|contextmenu)|click/,$=/^(?:focusinfocus|focusoutblur)$/,_=/^([^.]*)(?:\.(.+)|)$/;function ab(){return!0}function bb(){return!1}function cb(){try{return y.activeElement}catch(a){}}m.event={global:{},add:function(a,b,c,d,e){var f,g,h,i,j,k,l,n,o,p,q,r=m._data(a);if(r){c.handler&&(i=c,c=i.handler,e=i.selector),c.guid||(c.guid=m.guid++),(g=r.events)||(g=r.events={}),(k=r.handle)||(k=r.handle=function(a){return typeof m===K||a&&m.event.triggered===a.type?void 0:m.event.dispatch.apply(k.elem,arguments)},k.elem=a),b=(b||"").match(E)||[""],h=b.length;while(h--)f=_.exec(b[h])||[],o=q=f[1],p=(f[2]||"").split(".").sort(),o&&(j=m.event.special[o]||{},o=(e?j.delegateType:j.bindType)||o,j=m.event.special[o]||{},l=m.extend({type:o,origType:q,data:d,handler:c,guid:c.guid,selector:e,needsContext:e&&m.expr.match.needsContext.test(e),namespace:p.join(".")},i),(n=g[o])||(n=g[o]=[],n.delegateCount=0,j.setup&&j.setup.call(a,d,p,k)!==!1||(a.addEventListener?a.addEventListener(o,k,!1):a.attachEvent&&a.attachEvent("on"+o,k))),j.add&&(j.add.call(a,l),l.handler.guid||(l.handler.guid=c.guid)),e?n.splice(n.delegateCount++,0,l):n.push(l),m.event.global[o]=!0);a=null}},remove:function(a,b,c,d,e){var f,g,h,i,j,k,l,n,o,p,q,r=m.hasData(a)&&m._data(a);if(r&&(k=r.events)){b=(b||"").match(E)||[""],j=b.length;while(j--)if(h=_.exec(b[j])||[],o=q=h[1],p=(h[2]||"").split(".").sort(),o){l=m.event.special[o]||{},o=(d?l.delegateType:l.bindType)||o,n=k[o]||[],h=h[2]&&new RegExp("(^|\\.)"+p.join("\\.(?:.*\\.|)")+"(\\.|$)"),i=f=n.length;while(f--)g=n[f],!e&&q!==g.origType||c&&c.guid!==g.guid||h&&!h.test(g.namespace)||d&&d!==g.selector&&("**"!==d||!g.selector)||(n.splice(f,1),g.selector&&n.delegateCount--,l.remove&&l.remove.call(a,g));i&&!n.length&&(l.teardown&&l.teardown.call(a,p,r.handle)!==!1||m.removeEvent(a,o,r.handle),delete k[o])}else for(o in k)m.event.remove(a,o+b[j],c,d,!0);m.isEmptyObject(k)&&(delete r.handle,m._removeData(a,"events"))}},trigger:function(b,c,d,e){var f,g,h,i,k,l,n,o=[d||y],p=j.call(b,"type")?b.type:b,q=j.call(b,"namespace")?b.namespace.split("."):[];if(h=l=d=d||y,3!==d.nodeType&&8!==d.nodeType&&!$.test(p+m.event.triggered)&&(p.indexOf(".")>=0&&(q=p.split("."),p=q.shift(),q.sort()),g=p.indexOf(":")<0&&"on"+p,b=b[m.expando]?b:new m.Event(p,"object"==typeof b&&b),b.isTrigger=e?2:3,b.namespace=q.join("."),b.namespace_re=b.namespace?new RegExp("(^|\\.)"+q.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,b.result=void 0,b.target||(b.target=d),c=null==c?[b]:m.makeArray(c,[b]),k=m.event.special[p]||{},e||!k.trigger||k.trigger.apply(d,c)!==!1)){if(!e&&!k.noBubble&&!m.isWindow(d)){for(i=k.delegateType||p,$.test(i+p)||(h=h.parentNode);h;h=h.parentNode)o.push(h),l=h;l===(d.ownerDocument||y)&&o.push(l.defaultView||l.parentWindow||a)}n=0;while((h=o[n++])&&!b.isPropagationStopped())b.type=n>1?i:k.bindType||p,f=(m._data(h,"events")||{})[b.type]&&m._data(h,"handle"),f&&f.apply(h,c),f=g&&h[g],f&&f.apply&&m.acceptData(h)&&(b.result=f.apply(h,c),b.result===!1&&b.preventDefault());if(b.type=p,!e&&!b.isDefaultPrevented()&&(!k._default||k._default.apply(o.pop(),c)===!1)&&m.acceptData(d)&&g&&d[p]&&!m.isWindow(d)){l=d[g],l&&(d[g]=null),m.event.triggered=p;try{d[p]()}catch(r){}m.event.triggered=void 0,l&&(d[g]=l)}return b.result}},dispatch:function(a){a=m.event.fix(a);var b,c,e,f,g,h=[],i=d.call(arguments),j=(m._data(this,"events")||{})[a.type]||[],k=m.event.special[a.type]||{};if(i[0]=a,a.delegateTarget=this,!k.preDispatch||k.preDispatch.call(this,a)!==!1){h=m.event.handlers.call(this,a,j),b=0;while((f=h[b++])&&!a.isPropagationStopped()){a.currentTarget=f.elem,g=0;while((e=f.handlers[g++])&&!a.isImmediatePropagationStopped())(!a.namespace_re||a.namespace_re.test(e.namespace))&&(a.handleObj=e,a.data=e.data,c=((m.event.special[e.origType]||{}).handle||e.handler).apply(f.elem,i),void 0!==c&&(a.result=c)===!1&&(a.preventDefault(),a.stopPropagation()))}return k.postDispatch&&k.postDispatch.call(this,a),a.result}},handlers:function(a,b){var c,d,e,f,g=[],h=b.delegateCount,i=a.target;if(h&&i.nodeType&&(!a.button||"click"!==a.type))for(;i!=this;i=i.parentNode||this)if(1===i.nodeType&&(i.disabled!==!0||"click"!==a.type)){for(e=[],f=0;h>f;f++)d=b[f],c=d.selector+" ",void 0===e[c]&&(e[c]=d.needsContext?m(c,this).index(i)>=0:m.find(c,this,null,[i]).length),e[c]&&e.push(d);e.length&&g.push({elem:i,handlers:e})}return h<b.length&&g.push({elem:this,handlers:b.slice(h)}),g},fix:function(a){if(a[m.expando])return a;var b,c,d,e=a.type,f=a,g=this.fixHooks[e];g||(this.fixHooks[e]=g=Z.test(e)?this.mouseHooks:Y.test(e)?this.keyHooks:{}),d=g.props?this.props.concat(g.props):this.props,a=new m.Event(f),b=d.length;while(b--)c=d[b],a[c]=f[c];return a.target||(a.target=f.srcElement||y),3===a.target.nodeType&&(a.target=a.target.parentNode),a.metaKey=!!a.metaKey,g.filter?g.filter(a,f):a},props:"altKey bubbles cancelable ctrlKey currentTarget eventPhase metaKey relatedTarget shiftKey target timeStamp view which".split(" "),fixHooks:{},keyHooks:{props:"char charCode key keyCode".split(" "),filter:function(a,b){return null==a.which&&(a.which=null!=b.charCode?b.charCode:b.keyCode),a}},mouseHooks:{props:"button buttons clientX clientY fromElement offsetX offsetY pageX pageY screenX screenY toElement".split(" "),filter:function(a,b){var c,d,e,f=b.button,g=b.fromElement;return null==a.pageX&&null!=b.clientX&&(d=a.target.ownerDocument||y,e=d.documentElement,c=d.body,a.pageX=b.clientX+(e&&e.scrollLeft||c&&c.scrollLeft||0)-(e&&e.clientLeft||c&&c.clientLeft||0),a.pageY=b.clientY+(e&&e.scrollTop||c&&c.scrollTop||0)-(e&&e.clientTop||c&&c.clientTop||0)),!a.relatedTarget&&g&&(a.relatedTarget=g===a.target?b.toElement:g),a.which||void 0===f||(a.which=1&f?1:2&f?3:4&f?2:0),a}},special:{load:{noBubble:!0},focus:{trigger:function(){if(this!==cb()&&this.focus)try{return this.focus(),!1}catch(a){}},delegateType:"focusin"},blur:{trigger:function(){return this===cb()&&this.blur?(this.blur(),!1):void 0},delegateType:"focusout"},click:{trigger:function(){return m.nodeName(this,"input")&&"checkbox"===this.type&&this.click?(this.click(),!1):void 0},_default:function(a){return m.nodeName(a.target,"a")}},beforeunload:{postDispatch:function(a){void 0!==a.result&&a.originalEvent&&(a.originalEvent.returnValue=a.result)}}},simulate:function(a,b,c,d){var e=m.extend(new m.Event,c,{type:a,isSimulated:!0,originalEvent:{}});d?m.event.trigger(e,null,b):m.event.dispatch.call(b,e),e.isDefaultPrevented()&&c.preventDefault()}},m.removeEvent=y.removeEventListener?function(a,b,c){a.removeEventListener&&a.removeEventListener(b,c,!1)}:function(a,b,c){var d="on"+b;a.detachEvent&&(typeof a[d]===K&&(a[d]=null),a.detachEvent(d,c))},m.Event=function(a,b){return this instanceof m.Event?(a&&a.type?(this.originalEvent=a,this.type=a.type,this.isDefaultPrevented=a.defaultPrevented||void 0===a.defaultPrevented&&a.returnValue===!1?ab:bb):this.type=a,b&&m.extend(this,b),this.timeStamp=a&&a.timeStamp||m.now(),void(this[m.expando]=!0)):new m.Event(a,b)},m.Event.prototype={isDefaultPrevented:bb,isPropagationStopped:bb,isImmediatePropagationStopped:bb,preventDefault:function(){var a=this.originalEvent;this.isDefaultPrevented=ab,a&&(a.preventDefault?a.preventDefault():a.returnValue=!1)},stopPropagation:function(){var a=this.originalEvent;this.isPropagationStopped=ab,a&&(a.stopPropagation&&a.stopPropagation(),a.cancelBubble=!0)},stopImmediatePropagation:function(){var a=this.originalEvent;this.isImmediatePropagationStopped=ab,a&&a.stopImmediatePropagation&&a.stopImmediatePropagation(),this.stopPropagation()}},m.each({mouseenter:"mouseover",mouseleave:"mouseout",pointerenter:"pointerover",pointerleave:"pointerout"},function(a,b){m.event.special[a]={delegateType:b,bindType:b,handle:function(a){var c,d=this,e=a.relatedTarget,f=a.handleObj;return(!e||e!==d&&!m.contains(d,e))&&(a.type=f.origType,c=f.handler.apply(this,arguments),a.type=b),c}}}),k.submitBubbles||(m.event.special.submit={setup:function(){return m.nodeName(this,"form")?!1:void m.event.add(this,"click._submit keypress._submit",function(a){var b=a.target,c=m.nodeName(b,"input")||m.nodeName(b,"button")?b.form:void 0;c&&!m._data(c,"submitBubbles")&&(m.event.add(c,"submit._submit",function(a){a._submit_bubble=!0}),m._data(c,"submitBubbles",!0))})},postDispatch:function(a){a._submit_bubble&&(delete a._submit_bubble,this.parentNode&&!a.isTrigger&&m.event.simulate("submit",this.parentNode,a,!0))},teardown:function(){return m.nodeName(this,"form")?!1:void m.event.remove(this,"._submit")}}),k.changeBubbles||(m.event.special.change={setup:function(){return X.test(this.nodeName)?(("checkbox"===this.type||"radio"===this.type)&&(m.event.add(this,"propertychange._change",function(a){"checked"===a.originalEvent.propertyName&&(this._just_changed=!0)}),m.event.add(this,"click._change",function(a){this._just_changed&&!a.isTrigger&&(this._just_changed=!1),m.event.simulate("change",this,a,!0)})),!1):void m.event.add(this,"beforeactivate._change",function(a){var b=a.target;X.test(b.nodeName)&&!m._data(b,"changeBubbles")&&(m.event.add(b,"change._change",function(a){!this.parentNode||a.isSimulated||a.isTrigger||m.event.simulate("change",this.parentNode,a,!0)}),m._data(b,"changeBubbles",!0))})},handle:function(a){var b=a.target;return this!==b||a.isSimulated||a.isTrigger||"radio"!==b.type&&"checkbox"!==b.type?a.handleObj.handler.apply(this,arguments):void 0},teardown:function(){return m.event.remove(this,"._change"),!X.test(this.nodeName)}}),k.focusinBubbles||m.each({focus:"focusin",blur:"focusout"},function(a,b){var c=function(a){m.event.simulate(b,a.target,m.event.fix(a),!0)};m.event.special[b]={setup:function(){var d=this.ownerDocument||this,e=m._data(d,b);e||d.addEventListener(a,c,!0),m._data(d,b,(e||0)+1)},teardown:function(){var d=this.ownerDocument||this,e=m._data(d,b)-1;e?m._data(d,b,e):(d.removeEventListener(a,c,!0),m._removeData(d,b))}}}),m.fn.extend({on:function(a,b,c,d,e){var f,g;if("object"==typeof a){"string"!=typeof b&&(c=c||b,b=void 0);for(f in a)this.on(f,b,c,a[f],e);return this}if(null==c&&null==d?(d=b,c=b=void 0):null==d&&("string"==typeof b?(d=c,c=void 0):(d=c,c=b,b=void 0)),d===!1)d=bb;else if(!d)return this;return 1===e&&(g=d,d=function(a){return m().off(a),g.apply(this,arguments)},d.guid=g.guid||(g.guid=m.guid++)),this.each(function(){m.event.add(this,a,d,c,b)})},one:function(a,b,c,d){return this.on(a,b,c,d,1)},off:function(a,b,c){var d,e;if(a&&a.preventDefault&&a.handleObj)return d=a.handleObj,m(a.delegateTarget).off(d.namespace?d.origType+"."+d.namespace:d.origType,d.selector,d.handler),this;if("object"==typeof a){for(e in a)this.off(e,b,a[e]);return this}return(b===!1||"function"==typeof b)&&(c=b,b=void 0),c===!1&&(c=bb),this.each(function(){m.event.remove(this,a,c,b)})},trigger:function(a,b){return this.each(function(){m.event.trigger(a,b,this)})},triggerHandler:function(a,b){var c=this[0];return c?m.event.trigger(a,b,c,!0):void 0}});function db(a){var b=eb.split("|"),c=a.createDocumentFragment();if(c.createElement)while(b.length)c.createElement(b.pop());return c}var eb="abbr|article|aside|audio|bdi|canvas|data|datalist|details|figcaption|figure|footer|header|hgroup|mark|meter|nav|output|progress|section|summary|time|video",fb=/ jQuery\d+="(?:null|\d+)"/g,gb=new RegExp("<(?:"+eb+")[\\s/>]","i"),hb=/^\s+/,ib=/<(?!area|br|col|embed|hr|img|input|link|meta|param)(([\w:]+)[^>]*)\/>/gi,jb=/<([\w:]+)/,kb=/<tbody/i,lb=/<|&#?\w+;/,mb=/<(?:script|style|link)/i,nb=/checked\s*(?:[^=]|=\s*.checked.)/i,ob=/^$|\/(?:java|ecma)script/i,pb=/^true\/(.*)/,qb=/^\s*<!(?:\[CDATA\[|--)|(?:\]\]|--)>\s*$/g,rb={option:[1,"<select multiple='multiple'>","</select>"],legend:[1,"<fieldset>","</fieldset>"],area:[1,"<map>","</map>"],param:[1,"<object>","</object>"],thead:[1,"<table>","</table>"],tr:[2,"<table><tbody>","</tbody></table>"],col:[2,"<table><tbody></tbody><colgroup>","</colgroup></table>"],td:[3,"<table><tbody><tr>","</tr></tbody></table>"],_default:k.htmlSerialize?[0,"",""]:[1,"X<div>","</div>"]},sb=db(y),tb=sb.appendChild(y.createElement("div"));rb.optgroup=rb.option,rb.tbody=rb.tfoot=rb.colgroup=rb.caption=rb.thead,rb.th=rb.td;function ub(a,b){var c,d,e=0,f=typeof a.getElementsByTagName!==K?a.getElementsByTagName(b||"*"):typeof a.querySelectorAll!==K?a.querySelectorAll(b||"*"):void 0;if(!f)for(f=[],c=a.childNodes||a;null!=(d=c[e]);e++)!b||m.nodeName(d,b)?f.push(d):m.merge(f,ub(d,b));return void 0===b||b&&m.nodeName(a,b)?m.merge([a],f):f}function vb(a){W.test(a.type)&&(a.defaultChecked=a.checked)}function wb(a,b){return m.nodeName(a,"table")&&m.nodeName(11!==b.nodeType?b:b.firstChild,"tr")?a.getElementsByTagName("tbody")[0]||a.appendChild(a.ownerDocument.createElement("tbody")):a}function xb(a){return a.type=(null!==m.find.attr(a,"type"))+"/"+a.type,a}function yb(a){var b=pb.exec(a.type);return b?a.type=b[1]:a.removeAttribute("type"),a}function zb(a,b){for(var c,d=0;null!=(c=a[d]);d++)m._data(c,"globalEval",!b||m._data(b[d],"globalEval"))}function Ab(a,b){if(1===b.nodeType&&m.hasData(a)){var c,d,e,f=m._data(a),g=m._data(b,f),h=f.events;if(h){delete g.handle,g.events={};for(c in h)for(d=0,e=h[c].length;e>d;d++)m.event.add(b,c,h[c][d])}g.data&&(g.data=m.extend({},g.data))}}function Bb(a,b){var c,d,e;if(1===b.nodeType){if(c=b.nodeName.toLowerCase(),!k.noCloneEvent&&b[m.expando]){e=m._data(b);for(d in e.events)m.removeEvent(b,d,e.handle);b.removeAttribute(m.expando)}"script"===c&&b.text!==a.text?(xb(b).text=a.text,yb(b)):"object"===c?(b.parentNode&&(b.outerHTML=a.outerHTML),k.html5Clone&&a.innerHTML&&!m.trim(b.innerHTML)&&(b.innerHTML=a.innerHTML)):"input"===c&&W.test(a.type)?(b.defaultChecked=b.checked=a.checked,b.value!==a.value&&(b.value=a.value)):"option"===c?b.defaultSelected=b.selected=a.defaultSelected:("input"===c||"textarea"===c)&&(b.defaultValue=a.defaultValue)}}m.extend({clone:function(a,b,c){var d,e,f,g,h,i=m.contains(a.ownerDocument,a);if(k.html5Clone||m.isXMLDoc(a)||!gb.test("<"+a.nodeName+">")?f=a.cloneNode(!0):(tb.innerHTML=a.outerHTML,tb.removeChild(f=tb.firstChild)),!(k.noCloneEvent&&k.noCloneChecked||1!==a.nodeType&&11!==a.nodeType||m.isXMLDoc(a)))for(d=ub(f),h=ub(a),g=0;null!=(e=h[g]);++g)d[g]&&Bb(e,d[g]);if(b)if(c)for(h=h||ub(a),d=d||ub(f),g=0;null!=(e=h[g]);g++)Ab(e,d[g]);else Ab(a,f);return d=ub(f,"script"),d.length>0&&zb(d,!i&&ub(a,"script")),d=h=e=null,f},buildFragment:function(a,b,c,d){for(var e,f,g,h,i,j,l,n=a.length,o=db(b),p=[],q=0;n>q;q++)if(f=a[q],f||0===f)if("object"===m.type(f))m.merge(p,f.nodeType?[f]:f);else if(lb.test(f)){h=h||o.appendChild(b.createElement("div")),i=(jb.exec(f)||["",""])[1].toLowerCase(),l=rb[i]||rb._default,h.innerHTML=l[1]+f.replace(ib,"<$1></$2>")+l[2],e=l[0];while(e--)h=h.lastChild;if(!k.leadingWhitespace&&hb.test(f)&&p.push(b.createTextNode(hb.exec(f)[0])),!k.tbody){f="table"!==i||kb.test(f)?"<table>"!==l[1]||kb.test(f)?0:h:h.firstChild,e=f&&f.childNodes.length;while(e--)m.nodeName(j=f.childNodes[e],"tbody")&&!j.childNodes.length&&f.removeChild(j)}m.merge(p,h.childNodes),h.textContent="";while(h.firstChild)h.removeChild(h.firstChild);h=o.lastChild}else p.push(b.createTextNode(f));h&&o.removeChild(h),k.appendChecked||m.grep(ub(p,"input"),vb),q=0;while(f=p[q++])if((!d||-1===m.inArray(f,d))&&(g=m.contains(f.ownerDocument,f),h=ub(o.appendChild(f),"script"),g&&zb(h),c)){e=0;while(f=h[e++])ob.test(f.type||"")&&c.push(f)}return h=null,o},cleanData:function(a,b){for(var d,e,f,g,h=0,i=m.expando,j=m.cache,l=k.deleteExpando,n=m.event.special;null!=(d=a[h]);h++)if((b||m.acceptData(d))&&(f=d[i],g=f&&j[f])){if(g.events)for(e in g.events)n[e]?m.event.remove(d,e):m.removeEvent(d,e,g.handle);j[f]&&(delete j[f],l?delete d[i]:typeof d.removeAttribute!==K?d.removeAttribute(i):d[i]=null,c.push(f))}}}),m.fn.extend({text:function(a){return V(this,function(a){return void 0===a?m.text(this):this.empty().append((this[0]&&this[0].ownerDocument||y).createTextNode(a))},null,a,arguments.length)},append:function(){return this.domManip(arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=wb(this,a);b.appendChild(a)}})},prepend:function(){return this.domManip(arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=wb(this,a);b.insertBefore(a,b.firstChild)}})},before:function(){return this.domManip(arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this)})},after:function(){return this.domManip(arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this.nextSibling)})},remove:function(a,b){for(var c,d=a?m.filter(a,this):this,e=0;null!=(c=d[e]);e++)b||1!==c.nodeType||m.cleanData(ub(c)),c.parentNode&&(b&&m.contains(c.ownerDocument,c)&&zb(ub(c,"script")),c.parentNode.removeChild(c));return this},empty:function(){for(var a,b=0;null!=(a=this[b]);b++){1===a.nodeType&&m.cleanData(ub(a,!1));while(a.firstChild)a.removeChild(a.firstChild);a.options&&m.nodeName(a,"select")&&(a.options.length=0)}return this},clone:function(a,b){return a=null==a?!1:a,b=null==b?a:b,this.map(function(){return m.clone(this,a,b)})},html:function(a){return V(this,function(a){var b=this[0]||{},c=0,d=this.length;if(void 0===a)return 1===b.nodeType?b.innerHTML.replace(fb,""):void 0;if(!("string"!=typeof a||mb.test(a)||!k.htmlSerialize&&gb.test(a)||!k.leadingWhitespace&&hb.test(a)||rb[(jb.exec(a)||["",""])[1].toLowerCase()])){a=a.replace(ib,"<$1></$2>");try{for(;d>c;c++)b=this[c]||{},1===b.nodeType&&(m.cleanData(ub(b,!1)),b.innerHTML=a);b=0}catch(e){}}b&&this.empty().append(a)},null,a,arguments.length)},replaceWith:function(){var a=arguments[0];return this.domManip(arguments,function(b){a=this.parentNode,m.cleanData(ub(this)),a&&a.replaceChild(b,this)}),a&&(a.length||a.nodeType)?this:this.remove()},detach:function(a){return this.remove(a,!0)},domManip:function(a,b){a=e.apply([],a);var c,d,f,g,h,i,j=0,l=this.length,n=this,o=l-1,p=a[0],q=m.isFunction(p);if(q||l>1&&"string"==typeof p&&!k.checkClone&&nb.test(p))return this.each(function(c){var d=n.eq(c);q&&(a[0]=p.call(this,c,d.html())),d.domManip(a,b)});if(l&&(i=m.buildFragment(a,this[0].ownerDocument,!1,this),c=i.firstChild,1===i.childNodes.length&&(i=c),c)){for(g=m.map(ub(i,"script"),xb),f=g.length;l>j;j++)d=i,j!==o&&(d=m.clone(d,!0,!0),f&&m.merge(g,ub(d,"script"))),b.call(this[j],d,j);if(f)for(h=g[g.length-1].ownerDocument,m.map(g,yb),j=0;f>j;j++)d=g[j],ob.test(d.type||"")&&!m._data(d,"globalEval")&&m.contains(h,d)&&(d.src?m._evalUrl&&m._evalUrl(d.src):m.globalEval((d.text||d.textContent||d.innerHTML||"").replace(qb,"")));i=c=null}return this}}),m.each({appendTo:"append",prependTo:"prepend",insertBefore:"before",insertAfter:"after",replaceAll:"replaceWith"},function(a,b){m.fn[a]=function(a){for(var c,d=0,e=[],g=m(a),h=g.length-1;h>=d;d++)c=d===h?this:this.clone(!0),m(g[d])[b](c),f.apply(e,c.get());return this.pushStack(e)}});var Cb,Db={};function Eb(b,c){var d,e=m(c.createElement(b)).appendTo(c.body),f=a.getDefaultComputedStyle&&(d=a.getDefaultComputedStyle(e[0]))?d.display:m.css(e[0],"display");return e.detach(),f}function Fb(a){var b=y,c=Db[a];return c||(c=Eb(a,b),"none"!==c&&c||(Cb=(Cb||m("<iframe frameborder='0' width='0' height='0'/>")).appendTo(b.documentElement),b=(Cb[0].contentWindow||Cb[0].contentDocument).document,b.write(),b.close(),c=Eb(a,b),Cb.detach()),Db[a]=c),c}!function(){var a;k.shrinkWrapBlocks=function(){if(null!=a)return a;a=!1;var b,c,d;return c=y.getElementsByTagName("body")[0],c&&c.style?(b=y.createElement("div"),d=y.createElement("div"),d.style.cssText="position:absolute;border:0;width:0;height:0;top:0;left:-9999px",c.appendChild(d).appendChild(b),typeof b.style.zoom!==K&&(b.style.cssText="-webkit-box-sizing:content-box;-moz-box-sizing:content-box;box-sizing:content-box;display:block;margin:0;border:0;padding:1px;width:1px;zoom:1",b.appendChild(y.createElement("div")).style.width="5px",a=3!==b.offsetWidth),c.removeChild(d),a):void 0}}();var Gb=/^margin/,Hb=new RegExp("^("+S+")(?!px)[a-z%]+$","i"),Ib,Jb,Kb=/^(top|right|bottom|left)$/;a.getComputedStyle?(Ib=function(a){return a.ownerDocument.defaultView.getComputedStyle(a,null)},Jb=function(a,b,c){var d,e,f,g,h=a.style;return c=c||Ib(a),g=c?c.getPropertyValue(b)||c[b]:void 0,c&&(""!==g||m.contains(a.ownerDocument,a)||(g=m.style(a,b)),Hb.test(g)&&Gb.test(b)&&(d=h.width,e=h.minWidth,f=h.maxWidth,h.minWidth=h.maxWidth=h.width=g,g=c.width,h.width=d,h.minWidth=e,h.maxWidth=f)),void 0===g?g:g+""}):y.documentElement.currentStyle&&(Ib=function(a){return a.currentStyle},Jb=function(a,b,c){var d,e,f,g,h=a.style;return c=c||Ib(a),g=c?c[b]:void 0,null==g&&h&&h[b]&&(g=h[b]),Hb.test(g)&&!Kb.test(b)&&(d=h.left,e=a.runtimeStyle,f=e&&e.left,f&&(e.left=a.currentStyle.left),h.left="fontSize"===b?"1em":g,g=h.pixelLeft+"px",h.left=d,f&&(e.left=f)),void 0===g?g:g+""||"auto"});function Lb(a,b){return{get:function(){var c=a();if(null!=c)return c?void delete this.get:(this.get=b).apply(this,arguments)}}}!function(){var b,c,d,e,f,g,h;if(b=y.createElement("div"),b.innerHTML=" <link/><table></table><a href='/a'>a</a><input type='checkbox'/>",d=b.getElementsByTagName("a")[0],c=d&&d.style){c.cssText="float:left;opacity:.5",k.opacity="0.5"===c.opacity,k.cssFloat=!!c.cssFloat,b.style.backgroundClip="content-box",b.cloneNode(!0).style.backgroundClip="",k.clearCloneStyle="content-box"===b.style.backgroundClip,k.boxSizing=""===c.boxSizing||""===c.MozBoxSizing||""===c.WebkitBoxSizing,m.extend(k,{reliableHiddenOffsets:function(){return null==g&&i(),g},boxSizingReliable:function(){return null==f&&i(),f},pixelPosition:function(){return null==e&&i(),e},reliableMarginRight:function(){return null==h&&i(),h}});function i(){var b,c,d,i;c=y.getElementsByTagName("body")[0],c&&c.style&&(b=y.createElement("div"),d=y.createElement("div"),d.style.cssText="position:absolute;border:0;width:0;height:0;top:0;left:-9999px",c.appendChild(d).appendChild(b),b.style.cssText="-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box;display:block;margin-top:1%;top:1%;border:1px;padding:1px;width:4px;position:absolute",e=f=!1,h=!0,a.getComputedStyle&&(e="1%"!==(a.getComputedStyle(b,null)||{}).top,f="4px"===(a.getComputedStyle(b,null)||{width:"4px"}).width,i=b.appendChild(y.createElement("div")),i.style.cssText=b.style.cssText="-webkit-box-sizing:content-box;-moz-box-sizing:content-box;box-sizing:content-box;display:block;margin:0;border:0;padding:0",i.style.marginRight=i.style.width="0",b.style.width="1px",h=!parseFloat((a.getComputedStyle(i,null)||{}).marginRight)),b.innerHTML="<table><tr><td></td><td>t</td></tr></table>",i=b.getElementsByTagName("td"),i[0].style.cssText="margin:0;border:0;padding:0;display:none",g=0===i[0].offsetHeight,g&&(i[0].style.display="",i[1].style.display="none",g=0===i[0].offsetHeight),c.removeChild(d))}}}(),m.swap=function(a,b,c,d){var e,f,g={};for(f in b)g[f]=a.style[f],a.style[f]=b[f];e=c.apply(a,d||[]);for(f in b)a.style[f]=g[f];return e};var Mb=/alpha\([^)]*\)/i,Nb=/opacity\s*=\s*([^)]*)/,Ob=/^(none|table(?!-c[ea]).+)/,Pb=new RegExp("^("+S+")(.*)$","i"),Qb=new RegExp("^([+-])=("+S+")","i"),Rb={position:"absolute",visibility:"hidden",display:"block"},Sb={letterSpacing:"0",fontWeight:"400"},Tb=["Webkit","O","Moz","ms"];function Ub(a,b){if(b in a)return b;var c=b.charAt(0).toUpperCase()+b.slice(1),d=b,e=Tb.length;while(e--)if(b=Tb[e]+c,b in a)return b;return d}function Vb(a,b){for(var c,d,e,f=[],g=0,h=a.length;h>g;g++)d=a[g],d.style&&(f[g]=m._data(d,"olddisplay"),c=d.style.display,b?(f[g]||"none"!==c||(d.style.display=""),""===d.style.display&&U(d)&&(f[g]=m._data(d,"olddisplay",Fb(d.nodeName)))):(e=U(d),(c&&"none"!==c||!e)&&m._data(d,"olddisplay",e?c:m.css(d,"display"))));for(g=0;h>g;g++)d=a[g],d.style&&(b&&"none"!==d.style.display&&""!==d.style.display||(d.style.display=b?f[g]||"":"none"));return a}function Wb(a,b,c){var d=Pb.exec(b);return d?Math.max(0,d[1]-(c||0))+(d[2]||"px"):b}function Xb(a,b,c,d,e){for(var f=c===(d?"border":"content")?4:"width"===b?1:0,g=0;4>f;f+=2)"margin"===c&&(g+=m.css(a,c+T[f],!0,e)),d?("content"===c&&(g-=m.css(a,"padding"+T[f],!0,e)),"margin"!==c&&(g-=m.css(a,"border"+T[f]+"Width",!0,e))):(g+=m.css(a,"padding"+T[f],!0,e),"padding"!==c&&(g+=m.css(a,"border"+T[f]+"Width",!0,e)));return g}function Yb(a,b,c){var d=!0,e="width"===b?a.offsetWidth:a.offsetHeight,f=Ib(a),g=k.boxSizing&&"border-box"===m.css(a,"boxSizing",!1,f);if(0>=e||null==e){if(e=Jb(a,b,f),(0>e||null==e)&&(e=a.style[b]),Hb.test(e))return e;d=g&&(k.boxSizingReliable()||e===a.style[b]),e=parseFloat(e)||0}return e+Xb(a,b,c||(g?"border":"content"),d,f)+"px"}m.extend({cssHooks:{opacity:{get:function(a,b){if(b){var c=Jb(a,"opacity");return""===c?"1":c}}}},cssNumber:{columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{"float":k.cssFloat?"cssFloat":"styleFloat"},style:function(a,b,c,d){if(a&&3!==a.nodeType&&8!==a.nodeType&&a.style){var e,f,g,h=m.camelCase(b),i=a.style;if(b=m.cssProps[h]||(m.cssProps[h]=Ub(i,h)),g=m.cssHooks[b]||m.cssHooks[h],void 0===c)return g&&"get"in g&&void 0!==(e=g.get(a,!1,d))?e:i[b];if(f=typeof c,"string"===f&&(e=Qb.exec(c))&&(c=(e[1]+1)*e[2]+parseFloat(m.css(a,b)),f="number"),null!=c&&c===c&&("number"!==f||m.cssNumber[h]||(c+="px"),k.clearCloneStyle||""!==c||0!==b.indexOf("background")||(i[b]="inherit"),!(g&&"set"in g&&void 0===(c=g.set(a,c,d)))))try{i[b]=c}catch(j){}}},css:function(a,b,c,d){var e,f,g,h=m.camelCase(b);return b=m.cssProps[h]||(m.cssProps[h]=Ub(a.style,h)),g=m.cssHooks[b]||m.cssHooks[h],g&&"get"in g&&(f=g.get(a,!0,c)),void 0===f&&(f=Jb(a,b,d)),"normal"===f&&b in Sb&&(f=Sb[b]),""===c||c?(e=parseFloat(f),c===!0||m.isNumeric(e)?e||0:f):f}}),m.each(["height","width"],function(a,b){m.cssHooks[b]={get:function(a,c,d){return c?Ob.test(m.css(a,"display"))&&0===a.offsetWidth?m.swap(a,Rb,function(){return Yb(a,b,d)}):Yb(a,b,d):void 0},set:function(a,c,d){var e=d&&Ib(a);return Wb(a,c,d?Xb(a,b,d,k.boxSizing&&"border-box"===m.css(a,"boxSizing",!1,e),e):0)}}}),k.opacity||(m.cssHooks.opacity={get:function(a,b){return Nb.test((b&&a.currentStyle?a.currentStyle.filter:a.style.filter)||"")?.01*parseFloat(RegExp.$1)+"":b?"1":""},set:function(a,b){var c=a.style,d=a.currentStyle,e=m.isNumeric(b)?"alpha(opacity="+100*b+")":"",f=d&&d.filter||c.filter||"";c.zoom=1,(b>=1||""===b)&&""===m.trim(f.replace(Mb,""))&&c.removeAttribute&&(c.removeAttribute("filter"),""===b||d&&!d.filter)||(c.filter=Mb.test(f)?f.replace(Mb,e):f+" "+e)}}),m.cssHooks.marginRight=Lb(k.reliableMarginRight,function(a,b){return b?m.swap(a,{display:"inline-block"},Jb,[a,"marginRight"]):void 0}),m.each({margin:"",padding:"",border:"Width"},function(a,b){m.cssHooks[a+b]={expand:function(c){for(var d=0,e={},f="string"==typeof c?c.split(" "):[c];4>d;d++)e[a+T[d]+b]=f[d]||f[d-2]||f[0];return e}},Gb.test(a)||(m.cssHooks[a+b].set=Wb)}),m.fn.extend({css:function(a,b){return V(this,function(a,b,c){var d,e,f={},g=0;if(m.isArray(b)){for(d=Ib(a),e=b.length;e>g;g++)f[b[g]]=m.css(a,b[g],!1,d);return f}return void 0!==c?m.style(a,b,c):m.css(a,b)},a,b,arguments.length>1)},show:function(){return Vb(this,!0)},hide:function(){return Vb(this)},toggle:function(a){return"boolean"==typeof a?a?this.show():this.hide():this.each(function(){U(this)?m(this).show():m(this).hide()})}});function Zb(a,b,c,d,e){return new Zb.prototype.init(a,b,c,d,e)}m.Tween=Zb,Zb.prototype={constructor:Zb,init:function(a,b,c,d,e,f){this.elem=a,this.prop=c,this.easing=e||"swing",this.options=b,this.start=this.now=this.cur(),this.end=d,this.unit=f||(m.cssNumber[c]?"":"px")
+},cur:function(){var a=Zb.propHooks[this.prop];return a&&a.get?a.get(this):Zb.propHooks._default.get(this)},run:function(a){var b,c=Zb.propHooks[this.prop];return this.pos=b=this.options.duration?m.easing[this.easing](a,this.options.duration*a,0,1,this.options.duration):a,this.now=(this.end-this.start)*b+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),c&&c.set?c.set(this):Zb.propHooks._default.set(this),this}},Zb.prototype.init.prototype=Zb.prototype,Zb.propHooks={_default:{get:function(a){var b;return null==a.elem[a.prop]||a.elem.style&&null!=a.elem.style[a.prop]?(b=m.css(a.elem,a.prop,""),b&&"auto"!==b?b:0):a.elem[a.prop]},set:function(a){m.fx.step[a.prop]?m.fx.step[a.prop](a):a.elem.style&&(null!=a.elem.style[m.cssProps[a.prop]]||m.cssHooks[a.prop])?m.style(a.elem,a.prop,a.now+a.unit):a.elem[a.prop]=a.now}}},Zb.propHooks.scrollTop=Zb.propHooks.scrollLeft={set:function(a){a.elem.nodeType&&a.elem.parentNode&&(a.elem[a.prop]=a.now)}},m.easing={linear:function(a){return a},swing:function(a){return.5-Math.cos(a*Math.PI)/2}},m.fx=Zb.prototype.init,m.fx.step={};var $b,_b,ac=/^(?:toggle|show|hide)$/,bc=new RegExp("^(?:([+-])=|)("+S+")([a-z%]*)$","i"),cc=/queueHooks$/,dc=[ic],ec={"*":[function(a,b){var c=this.createTween(a,b),d=c.cur(),e=bc.exec(b),f=e&&e[3]||(m.cssNumber[a]?"":"px"),g=(m.cssNumber[a]||"px"!==f&&+d)&&bc.exec(m.css(c.elem,a)),h=1,i=20;if(g&&g[3]!==f){f=f||g[3],e=e||[],g=+d||1;do h=h||".5",g/=h,m.style(c.elem,a,g+f);while(h!==(h=c.cur()/d)&&1!==h&&--i)}return e&&(g=c.start=+g||+d||0,c.unit=f,c.end=e[1]?g+(e[1]+1)*e[2]:+e[2]),c}]};function fc(){return setTimeout(function(){$b=void 0}),$b=m.now()}function gc(a,b){var c,d={height:a},e=0;for(b=b?1:0;4>e;e+=2-b)c=T[e],d["margin"+c]=d["padding"+c]=a;return b&&(d.opacity=d.width=a),d}function hc(a,b,c){for(var d,e=(ec[b]||[]).concat(ec["*"]),f=0,g=e.length;g>f;f++)if(d=e[f].call(c,b,a))return d}function ic(a,b,c){var d,e,f,g,h,i,j,l,n=this,o={},p=a.style,q=a.nodeType&&U(a),r=m._data(a,"fxshow");c.queue||(h=m._queueHooks(a,"fx"),null==h.unqueued&&(h.unqueued=0,i=h.empty.fire,h.empty.fire=function(){h.unqueued||i()}),h.unqueued++,n.always(function(){n.always(function(){h.unqueued--,m.queue(a,"fx").length||h.empty.fire()})})),1===a.nodeType&&("height"in b||"width"in b)&&(c.overflow=[p.overflow,p.overflowX,p.overflowY],j=m.css(a,"display"),l="none"===j?m._data(a,"olddisplay")||Fb(a.nodeName):j,"inline"===l&&"none"===m.css(a,"float")&&(k.inlineBlockNeedsLayout&&"inline"!==Fb(a.nodeName)?p.zoom=1:p.display="inline-block")),c.overflow&&(p.overflow="hidden",k.shrinkWrapBlocks()||n.always(function(){p.overflow=c.overflow[0],p.overflowX=c.overflow[1],p.overflowY=c.overflow[2]}));for(d in b)if(e=b[d],ac.exec(e)){if(delete b[d],f=f||"toggle"===e,e===(q?"hide":"show")){if("show"!==e||!r||void 0===r[d])continue;q=!0}o[d]=r&&r[d]||m.style(a,d)}else j=void 0;if(m.isEmptyObject(o))"inline"===("none"===j?Fb(a.nodeName):j)&&(p.display=j);else{r?"hidden"in r&&(q=r.hidden):r=m._data(a,"fxshow",{}),f&&(r.hidden=!q),q?m(a).show():n.done(function(){m(a).hide()}),n.done(function(){var b;m._removeData(a,"fxshow");for(b in o)m.style(a,b,o[b])});for(d in o)g=hc(q?r[d]:0,d,n),d in r||(r[d]=g.start,q&&(g.end=g.start,g.start="width"===d||"height"===d?1:0))}}function jc(a,b){var c,d,e,f,g;for(c in a)if(d=m.camelCase(c),e=b[d],f=a[c],m.isArray(f)&&(e=f[1],f=a[c]=f[0]),c!==d&&(a[d]=f,delete a[c]),g=m.cssHooks[d],g&&"expand"in g){f=g.expand(f),delete a[d];for(c in f)c in a||(a[c]=f[c],b[c]=e)}else b[d]=e}function kc(a,b,c){var d,e,f=0,g=dc.length,h=m.Deferred().always(function(){delete i.elem}),i=function(){if(e)return!1;for(var b=$b||fc(),c=Math.max(0,j.startTime+j.duration-b),d=c/j.duration||0,f=1-d,g=0,i=j.tweens.length;i>g;g++)j.tweens[g].run(f);return h.notifyWith(a,[j,f,c]),1>f&&i?c:(h.resolveWith(a,[j]),!1)},j=h.promise({elem:a,props:m.extend({},b),opts:m.extend(!0,{specialEasing:{}},c),originalProperties:b,originalOptions:c,startTime:$b||fc(),duration:c.duration,tweens:[],createTween:function(b,c){var d=m.Tween(a,j.opts,b,c,j.opts.specialEasing[b]||j.opts.easing);return j.tweens.push(d),d},stop:function(b){var c=0,d=b?j.tweens.length:0;if(e)return this;for(e=!0;d>c;c++)j.tweens[c].run(1);return b?h.resolveWith(a,[j,b]):h.rejectWith(a,[j,b]),this}}),k=j.props;for(jc(k,j.opts.specialEasing);g>f;f++)if(d=dc[f].call(j,a,k,j.opts))return d;return m.map(k,hc,j),m.isFunction(j.opts.start)&&j.opts.start.call(a,j),m.fx.timer(m.extend(i,{elem:a,anim:j,queue:j.opts.queue})),j.progress(j.opts.progress).done(j.opts.done,j.opts.complete).fail(j.opts.fail).always(j.opts.always)}m.Animation=m.extend(kc,{tweener:function(a,b){m.isFunction(a)?(b=a,a=["*"]):a=a.split(" ");for(var c,d=0,e=a.length;e>d;d++)c=a[d],ec[c]=ec[c]||[],ec[c].unshift(b)},prefilter:function(a,b){b?dc.unshift(a):dc.push(a)}}),m.speed=function(a,b,c){var d=a&&"object"==typeof a?m.extend({},a):{complete:c||!c&&b||m.isFunction(a)&&a,duration:a,easing:c&&b||b&&!m.isFunction(b)&&b};return d.duration=m.fx.off?0:"number"==typeof d.duration?d.duration:d.duration in m.fx.speeds?m.fx.speeds[d.duration]:m.fx.speeds._default,(null==d.queue||d.queue===!0)&&(d.queue="fx"),d.old=d.complete,d.complete=function(){m.isFunction(d.old)&&d.old.call(this),d.queue&&m.dequeue(this,d.queue)},d},m.fn.extend({fadeTo:function(a,b,c,d){return this.filter(U).css("opacity",0).show().end().animate({opacity:b},a,c,d)},animate:function(a,b,c,d){var e=m.isEmptyObject(a),f=m.speed(b,c,d),g=function(){var b=kc(this,m.extend({},a),f);(e||m._data(this,"finish"))&&b.stop(!0)};return g.finish=g,e||f.queue===!1?this.each(g):this.queue(f.queue,g)},stop:function(a,b,c){var d=function(a){var b=a.stop;delete a.stop,b(c)};return"string"!=typeof a&&(c=b,b=a,a=void 0),b&&a!==!1&&this.queue(a||"fx",[]),this.each(function(){var b=!0,e=null!=a&&a+"queueHooks",f=m.timers,g=m._data(this);if(e)g[e]&&g[e].stop&&d(g[e]);else for(e in g)g[e]&&g[e].stop&&cc.test(e)&&d(g[e]);for(e=f.length;e--;)f[e].elem!==this||null!=a&&f[e].queue!==a||(f[e].anim.stop(c),b=!1,f.splice(e,1));(b||!c)&&m.dequeue(this,a)})},finish:function(a){return a!==!1&&(a=a||"fx"),this.each(function(){var b,c=m._data(this),d=c[a+"queue"],e=c[a+"queueHooks"],f=m.timers,g=d?d.length:0;for(c.finish=!0,m.queue(this,a,[]),e&&e.stop&&e.stop.call(this,!0),b=f.length;b--;)f[b].elem===this&&f[b].queue===a&&(f[b].anim.stop(!0),f.splice(b,1));for(b=0;g>b;b++)d[b]&&d[b].finish&&d[b].finish.call(this);delete c.finish})}}),m.each(["toggle","show","hide"],function(a,b){var c=m.fn[b];m.fn[b]=function(a,d,e){return null==a||"boolean"==typeof a?c.apply(this,arguments):this.animate(gc(b,!0),a,d,e)}}),m.each({slideDown:gc("show"),slideUp:gc("hide"),slideToggle:gc("toggle"),fadeIn:{opacity:"show"},fadeOut:{opacity:"hide"},fadeToggle:{opacity:"toggle"}},function(a,b){m.fn[a]=function(a,c,d){return this.animate(b,a,c,d)}}),m.timers=[],m.fx.tick=function(){var a,b=m.timers,c=0;for($b=m.now();c<b.length;c++)a=b[c],a()||b[c]!==a||b.splice(c--,1);b.length||m.fx.stop(),$b=void 0},m.fx.timer=function(a){m.timers.push(a),a()?m.fx.start():m.timers.pop()},m.fx.interval=13,m.fx.start=function(){_b||(_b=setInterval(m.fx.tick,m.fx.interval))},m.fx.stop=function(){clearInterval(_b),_b=null},m.fx.speeds={slow:600,fast:200,_default:400},m.fn.delay=function(a,b){return a=m.fx?m.fx.speeds[a]||a:a,b=b||"fx",this.queue(b,function(b,c){var d=setTimeout(b,a);c.stop=function(){clearTimeout(d)}})},function(){var a,b,c,d,e;b=y.createElement("div"),b.setAttribute("className","t"),b.innerHTML=" <link/><table></table><a href='/a'>a</a><input type='checkbox'/>",d=b.getElementsByTagName("a")[0],c=y.createElement("select"),e=c.appendChild(y.createElement("option")),a=b.getElementsByTagName("input")[0],d.style.cssText="top:1px",k.getSetAttribute="t"!==b.className,k.style=/top/.test(d.getAttribute("style")),k.hrefNormalized="/a"===d.getAttribute("href"),k.checkOn=!!a.value,k.optSelected=e.selected,k.enctype=!!y.createElement("form").enctype,c.disabled=!0,k.optDisabled=!e.disabled,a=y.createElement("input"),a.setAttribute("value",""),k.input=""===a.getAttribute("value"),a.value="t",a.setAttribute("type","radio"),k.radioValue="t"===a.value}();var lc=/\r/g;m.fn.extend({val:function(a){var b,c,d,e=this[0];{if(arguments.length)return d=m.isFunction(a),this.each(function(c){var e;1===this.nodeType&&(e=d?a.call(this,c,m(this).val()):a,null==e?e="":"number"==typeof e?e+="":m.isArray(e)&&(e=m.map(e,function(a){return null==a?"":a+""})),b=m.valHooks[this.type]||m.valHooks[this.nodeName.toLowerCase()],b&&"set"in b&&void 0!==b.set(this,e,"value")||(this.value=e))});if(e)return b=m.valHooks[e.type]||m.valHooks[e.nodeName.toLowerCase()],b&&"get"in b&&void 0!==(c=b.get(e,"value"))?c:(c=e.value,"string"==typeof c?c.replace(lc,""):null==c?"":c)}}}),m.extend({valHooks:{option:{get:function(a){var b=m.find.attr(a,"value");return null!=b?b:m.trim(m.text(a))}},select:{get:function(a){for(var b,c,d=a.options,e=a.selectedIndex,f="select-one"===a.type||0>e,g=f?null:[],h=f?e+1:d.length,i=0>e?h:f?e:0;h>i;i++)if(c=d[i],!(!c.selected&&i!==e||(k.optDisabled?c.disabled:null!==c.getAttribute("disabled"))||c.parentNode.disabled&&m.nodeName(c.parentNode,"optgroup"))){if(b=m(c).val(),f)return b;g.push(b)}return g},set:function(a,b){var c,d,e=a.options,f=m.makeArray(b),g=e.length;while(g--)if(d=e[g],m.inArray(m.valHooks.option.get(d),f)>=0)try{d.selected=c=!0}catch(h){d.scrollHeight}else d.selected=!1;return c||(a.selectedIndex=-1),e}}}}),m.each(["radio","checkbox"],function(){m.valHooks[this]={set:function(a,b){return m.isArray(b)?a.checked=m.inArray(m(a).val(),b)>=0:void 0}},k.checkOn||(m.valHooks[this].get=function(a){return null===a.getAttribute("value")?"on":a.value})});var mc,nc,oc=m.expr.attrHandle,pc=/^(?:checked|selected)$/i,qc=k.getSetAttribute,rc=k.input;m.fn.extend({attr:function(a,b){return V(this,m.attr,a,b,arguments.length>1)},removeAttr:function(a){return this.each(function(){m.removeAttr(this,a)})}}),m.extend({attr:function(a,b,c){var d,e,f=a.nodeType;if(a&&3!==f&&8!==f&&2!==f)return typeof a.getAttribute===K?m.prop(a,b,c):(1===f&&m.isXMLDoc(a)||(b=b.toLowerCase(),d=m.attrHooks[b]||(m.expr.match.bool.test(b)?nc:mc)),void 0===c?d&&"get"in d&&null!==(e=d.get(a,b))?e:(e=m.find.attr(a,b),null==e?void 0:e):null!==c?d&&"set"in d&&void 0!==(e=d.set(a,c,b))?e:(a.setAttribute(b,c+""),c):void m.removeAttr(a,b))},removeAttr:function(a,b){var c,d,e=0,f=b&&b.match(E);if(f&&1===a.nodeType)while(c=f[e++])d=m.propFix[c]||c,m.expr.match.bool.test(c)?rc&&qc||!pc.test(c)?a[d]=!1:a[m.camelCase("default-"+c)]=a[d]=!1:m.attr(a,c,""),a.removeAttribute(qc?c:d)},attrHooks:{type:{set:function(a,b){if(!k.radioValue&&"radio"===b&&m.nodeName(a,"input")){var c=a.value;return a.setAttribute("type",b),c&&(a.value=c),b}}}}}),nc={set:function(a,b,c){return b===!1?m.removeAttr(a,c):rc&&qc||!pc.test(c)?a.setAttribute(!qc&&m.propFix[c]||c,c):a[m.camelCase("default-"+c)]=a[c]=!0,c}},m.each(m.expr.match.bool.source.match(/\w+/g),function(a,b){var c=oc[b]||m.find.attr;oc[b]=rc&&qc||!pc.test(b)?function(a,b,d){var e,f;return d||(f=oc[b],oc[b]=e,e=null!=c(a,b,d)?b.toLowerCase():null,oc[b]=f),e}:function(a,b,c){return c?void 0:a[m.camelCase("default-"+b)]?b.toLowerCase():null}}),rc&&qc||(m.attrHooks.value={set:function(a,b,c){return m.nodeName(a,"input")?void(a.defaultValue=b):mc&&mc.set(a,b,c)}}),qc||(mc={set:function(a,b,c){var d=a.getAttributeNode(c);return d||a.setAttributeNode(d=a.ownerDocument.createAttribute(c)),d.value=b+="","value"===c||b===a.getAttribute(c)?b:void 0}},oc.id=oc.name=oc.coords=function(a,b,c){var d;return c?void 0:(d=a.getAttributeNode(b))&&""!==d.value?d.value:null},m.valHooks.button={get:function(a,b){var c=a.getAttributeNode(b);return c&&c.specified?c.value:void 0},set:mc.set},m.attrHooks.contenteditable={set:function(a,b,c){mc.set(a,""===b?!1:b,c)}},m.each(["width","height"],function(a,b){m.attrHooks[b]={set:function(a,c){return""===c?(a.setAttribute(b,"auto"),c):void 0}}})),k.style||(m.attrHooks.style={get:function(a){return a.style.cssText||void 0},set:function(a,b){return a.style.cssText=b+""}});var sc=/^(?:input|select|textarea|button|object)$/i,tc=/^(?:a|area)$/i;m.fn.extend({prop:function(a,b){return V(this,m.prop,a,b,arguments.length>1)},removeProp:function(a){return a=m.propFix[a]||a,this.each(function(){try{this[a]=void 0,delete this[a]}catch(b){}})}}),m.extend({propFix:{"for":"htmlFor","class":"className"},prop:function(a,b,c){var d,e,f,g=a.nodeType;if(a&&3!==g&&8!==g&&2!==g)return f=1!==g||!m.isXMLDoc(a),f&&(b=m.propFix[b]||b,e=m.propHooks[b]),void 0!==c?e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:a[b]=c:e&&"get"in e&&null!==(d=e.get(a,b))?d:a[b]},propHooks:{tabIndex:{get:function(a){var b=m.find.attr(a,"tabindex");return b?parseInt(b,10):sc.test(a.nodeName)||tc.test(a.nodeName)&&a.href?0:-1}}}}),k.hrefNormalized||m.each(["href","src"],function(a,b){m.propHooks[b]={get:function(a){return a.getAttribute(b,4)}}}),k.optSelected||(m.propHooks.selected={get:function(a){var b=a.parentNode;return b&&(b.selectedIndex,b.parentNode&&b.parentNode.selectedIndex),null}}),m.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){m.propFix[this.toLowerCase()]=this}),k.enctype||(m.propFix.enctype="encoding");var uc=/[\t\r\n\f]/g;m.fn.extend({addClass:function(a){var b,c,d,e,f,g,h=0,i=this.length,j="string"==typeof a&&a;if(m.isFunction(a))return this.each(function(b){m(this).addClass(a.call(this,b,this.className))});if(j)for(b=(a||"").match(E)||[];i>h;h++)if(c=this[h],d=1===c.nodeType&&(c.className?(" "+c.className+" ").replace(uc," "):" ")){f=0;while(e=b[f++])d.indexOf(" "+e+" ")<0&&(d+=e+" ");g=m.trim(d),c.className!==g&&(c.className=g)}return this},removeClass:function(a){var b,c,d,e,f,g,h=0,i=this.length,j=0===arguments.length||"string"==typeof a&&a;if(m.isFunction(a))return this.each(function(b){m(this).removeClass(a.call(this,b,this.className))});if(j)for(b=(a||"").match(E)||[];i>h;h++)if(c=this[h],d=1===c.nodeType&&(c.className?(" "+c.className+" ").replace(uc," "):"")){f=0;while(e=b[f++])while(d.indexOf(" "+e+" ")>=0)d=d.replace(" "+e+" "," ");g=a?m.trim(d):"",c.className!==g&&(c.className=g)}return this},toggleClass:function(a,b){var c=typeof a;return"boolean"==typeof b&&"string"===c?b?this.addClass(a):this.removeClass(a):this.each(m.isFunction(a)?function(c){m(this).toggleClass(a.call(this,c,this.className,b),b)}:function(){if("string"===c){var b,d=0,e=m(this),f=a.match(E)||[];while(b=f[d++])e.hasClass(b)?e.removeClass(b):e.addClass(b)}else(c===K||"boolean"===c)&&(this.className&&m._data(this,"__className__",this.className),this.className=this.className||a===!1?"":m._data(this,"__className__")||"")})},hasClass:function(a){for(var b=" "+a+" ",c=0,d=this.length;d>c;c++)if(1===this[c].nodeType&&(" "+this[c].className+" ").replace(uc," ").indexOf(b)>=0)return!0;return!1}}),m.each("blur focus focusin focusout load resize scroll unload click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup error contextmenu".split(" "),function(a,b){m.fn[b]=function(a,c){return arguments.length>0?this.on(b,null,a,c):this.trigger(b)}}),m.fn.extend({hover:function(a,b){return this.mouseenter(a).mouseleave(b||a)},bind:function(a,b,c){return this.on(a,null,b,c)},unbind:function(a,b){return this.off(a,null,b)},delegate:function(a,b,c,d){return this.on(b,a,c,d)},undelegate:function(a,b,c){return 1===arguments.length?this.off(a,"**"):this.off(b,a||"**",c)}});var vc=m.now(),wc=/\?/,xc=/(,)|(\[|{)|(}|])|"(?:[^"\\\r\n]|\\["\\\/bfnrt]|\\u[\da-fA-F]{4})*"\s*:?|true|false|null|-?(?!0\d)\d+(?:\.\d+|)(?:[eE][+-]?\d+|)/g;m.parseJSON=function(b){if(a.JSON&&a.JSON.parse)return a.JSON.parse(b+"");var c,d=null,e=m.trim(b+"");return e&&!m.trim(e.replace(xc,function(a,b,e,f){return c&&b&&(d=0),0===d?a:(c=e||b,d+=!f-!e,"")}))?Function("return "+e)():m.error("Invalid JSON: "+b)},m.parseXML=function(b){var c,d;if(!b||"string"!=typeof b)return null;try{a.DOMParser?(d=new DOMParser,c=d.parseFromString(b,"text/xml")):(c=new ActiveXObject("Microsoft.XMLDOM"),c.async="false",c.loadXML(b))}catch(e){c=void 0}return c&&c.documentElement&&!c.getElementsByTagName("parsererror").length||m.error("Invalid XML: "+b),c};var yc,zc,Ac=/#.*$/,Bc=/([?&])_=[^&]*/,Cc=/^(.*?):[ \t]*([^\r\n]*)\r?$/gm,Dc=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Ec=/^(?:GET|HEAD)$/,Fc=/^\/\//,Gc=/^([\w.+-]+:)(?:\/\/(?:[^\/?#]*@|)([^\/?#:]*)(?::(\d+)|)|)/,Hc={},Ic={},Jc="*/".concat("*");try{zc=location.href}catch(Kc){zc=y.createElement("a"),zc.href="",zc=zc.href}yc=Gc.exec(zc.toLowerCase())||[];function Lc(a){return function(b,c){"string"!=typeof b&&(c=b,b="*");var d,e=0,f=b.toLowerCase().match(E)||[];if(m.isFunction(c))while(d=f[e++])"+"===d.charAt(0)?(d=d.slice(1)||"*",(a[d]=a[d]||[]).unshift(c)):(a[d]=a[d]||[]).push(c)}}function Mc(a,b,c,d){var e={},f=a===Ic;function g(h){var i;return e[h]=!0,m.each(a[h]||[],function(a,h){var j=h(b,c,d);return"string"!=typeof j||f||e[j]?f?!(i=j):void 0:(b.dataTypes.unshift(j),g(j),!1)}),i}return g(b.dataTypes[0])||!e["*"]&&g("*")}function Nc(a,b){var c,d,e=m.ajaxSettings.flatOptions||{};for(d in b)void 0!==b[d]&&((e[d]?a:c||(c={}))[d]=b[d]);return c&&m.extend(!0,a,c),a}function Oc(a,b,c){var d,e,f,g,h=a.contents,i=a.dataTypes;while("*"===i[0])i.shift(),void 0===e&&(e=a.mimeType||b.getResponseHeader("Content-Type"));if(e)for(g in h)if(h[g]&&h[g].test(e)){i.unshift(g);break}if(i[0]in c)f=i[0];else{for(g in c){if(!i[0]||a.converters[g+" "+i[0]]){f=g;break}d||(d=g)}f=f||d}return f?(f!==i[0]&&i.unshift(f),c[f]):void 0}function Pc(a,b,c,d){var e,f,g,h,i,j={},k=a.dataTypes.slice();if(k[1])for(g in a.converters)j[g.toLowerCase()]=a.converters[g];f=k.shift();while(f)if(a.responseFields[f]&&(c[a.responseFields[f]]=b),!i&&d&&a.dataFilter&&(b=a.dataFilter(b,a.dataType)),i=f,f=k.shift())if("*"===f)f=i;else if("*"!==i&&i!==f){if(g=j[i+" "+f]||j["* "+f],!g)for(e in j)if(h=e.split(" "),h[1]===f&&(g=j[i+" "+h[0]]||j["* "+h[0]])){g===!0?g=j[e]:j[e]!==!0&&(f=h[0],k.unshift(h[1]));break}if(g!==!0)if(g&&a["throws"])b=g(b);else try{b=g(b)}catch(l){return{state:"parsererror",error:g?l:"No conversion from "+i+" to "+f}}}return{state:"success",data:b}}m.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:zc,type:"GET",isLocal:Dc.test(yc[1]),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":Jc,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/xml/,html:/html/,json:/json/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":m.parseJSON,"text xml":m.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(a,b){return b?Nc(Nc(a,m.ajaxSettings),b):Nc(m.ajaxSettings,a)},ajaxPrefilter:Lc(Hc),ajaxTransport:Lc(Ic),ajax:function(a,b){"object"==typeof a&&(b=a,a=void 0),b=b||{};var c,d,e,f,g,h,i,j,k=m.ajaxSetup({},b),l=k.context||k,n=k.context&&(l.nodeType||l.jquery)?m(l):m.event,o=m.Deferred(),p=m.Callbacks("once memory"),q=k.statusCode||{},r={},s={},t=0,u="canceled",v={readyState:0,getResponseHeader:function(a){var b;if(2===t){if(!j){j={};while(b=Cc.exec(f))j[b[1].toLowerCase()]=b[2]}b=j[a.toLowerCase()]}return null==b?null:b},getAllResponseHeaders:function(){return 2===t?f:null},setRequestHeader:function(a,b){var c=a.toLowerCase();return t||(a=s[c]=s[c]||a,r[a]=b),this},overrideMimeType:function(a){return t||(k.mimeType=a),this},statusCode:function(a){var b;if(a)if(2>t)for(b in a)q[b]=[q[b],a[b]];else v.always(a[v.status]);return this},abort:function(a){var b=a||u;return i&&i.abort(b),x(0,b),this}};if(o.promise(v).complete=p.add,v.success=v.done,v.error=v.fail,k.url=((a||k.url||zc)+"").replace(Ac,"").replace(Fc,yc[1]+"//"),k.type=b.method||b.type||k.method||k.type,k.dataTypes=m.trim(k.dataType||"*").toLowerCase().match(E)||[""],null==k.crossDomain&&(c=Gc.exec(k.url.toLowerCase()),k.crossDomain=!(!c||c[1]===yc[1]&&c[2]===yc[2]&&(c[3]||("http:"===c[1]?"80":"443"))===(yc[3]||("http:"===yc[1]?"80":"443")))),k.data&&k.processData&&"string"!=typeof k.data&&(k.data=m.param(k.data,k.traditional)),Mc(Hc,k,b,v),2===t)return v;h=k.global,h&&0===m.active++&&m.event.trigger("ajaxStart"),k.type=k.type.toUpperCase(),k.hasContent=!Ec.test(k.type),e=k.url,k.hasContent||(k.data&&(e=k.url+=(wc.test(e)?"&":"?")+k.data,delete k.data),k.cache===!1&&(k.url=Bc.test(e)?e.replace(Bc,"$1_="+vc++):e+(wc.test(e)?"&":"?")+"_="+vc++)),k.ifModified&&(m.lastModified[e]&&v.setRequestHeader("If-Modified-Since",m.lastModified[e]),m.etag[e]&&v.setRequestHeader("If-None-Match",m.etag[e])),(k.data&&k.hasContent&&k.contentType!==!1||b.contentType)&&v.setRequestHeader("Content-Type",k.contentType),v.setRequestHeader("Accept",k.dataTypes[0]&&k.accepts[k.dataTypes[0]]?k.accepts[k.dataTypes[0]]+("*"!==k.dataTypes[0]?", "+Jc+"; q=0.01":""):k.accepts["*"]);for(d in k.headers)v.setRequestHeader(d,k.headers[d]);if(k.beforeSend&&(k.beforeSend.call(l,v,k)===!1||2===t))return v.abort();u="abort";for(d in{success:1,error:1,complete:1})v[d](k[d]);if(i=Mc(Ic,k,b,v)){v.readyState=1,h&&n.trigger("ajaxSend",[v,k]),k.async&&k.timeout>0&&(g=setTimeout(function(){v.abort("timeout")},k.timeout));try{t=1,i.send(r,x)}catch(w){if(!(2>t))throw w;x(-1,w)}}else x(-1,"No Transport");function x(a,b,c,d){var j,r,s,u,w,x=b;2!==t&&(t=2,g&&clearTimeout(g),i=void 0,f=d||"",v.readyState=a>0?4:0,j=a>=200&&300>a||304===a,c&&(u=Oc(k,v,c)),u=Pc(k,u,v,j),j?(k.ifModified&&(w=v.getResponseHeader("Last-Modified"),w&&(m.lastModified[e]=w),w=v.getResponseHeader("etag"),w&&(m.etag[e]=w)),204===a||"HEAD"===k.type?x="nocontent":304===a?x="notmodified":(x=u.state,r=u.data,s=u.error,j=!s)):(s=x,(a||!x)&&(x="error",0>a&&(a=0))),v.status=a,v.statusText=(b||x)+"",j?o.resolveWith(l,[r,x,v]):o.rejectWith(l,[v,x,s]),v.statusCode(q),q=void 0,h&&n.trigger(j?"ajaxSuccess":"ajaxError",[v,k,j?r:s]),p.fireWith(l,[v,x]),h&&(n.trigger("ajaxComplete",[v,k]),--m.active||m.event.trigger("ajaxStop")))}return v},getJSON:function(a,b,c){return m.get(a,b,c,"json")},getScript:function(a,b){return m.get(a,void 0,b,"script")}}),m.each(["get","post"],function(a,b){m[b]=function(a,c,d,e){return m.isFunction(c)&&(e=e||d,d=c,c=void 0),m.ajax({url:a,type:b,dataType:e,data:c,success:d})}}),m.each(["ajaxStart","ajaxStop","ajaxComplete","ajaxError","ajaxSuccess","ajaxSend"],function(a,b){m.fn[b]=function(a){return this.on(b,a)}}),m._evalUrl=function(a){return m.ajax({url:a,type:"GET",dataType:"script",async:!1,global:!1,"throws":!0})},m.fn.extend({wrapAll:function(a){if(m.isFunction(a))return this.each(function(b){m(this).wrapAll(a.call(this,b))});if(this[0]){var b=m(a,this[0].ownerDocument).eq(0).clone(!0);this[0].parentNode&&b.insertBefore(this[0]),b.map(function(){var a=this;while(a.firstChild&&1===a.firstChild.nodeType)a=a.firstChild;return a}).append(this)}return this},wrapInner:function(a){return this.each(m.isFunction(a)?function(b){m(this).wrapInner(a.call(this,b))}:function(){var b=m(this),c=b.contents();c.length?c.wrapAll(a):b.append(a)})},wrap:function(a){var b=m.isFunction(a);return this.each(function(c){m(this).wrapAll(b?a.call(this,c):a)})},unwrap:function(){return this.parent().each(function(){m.nodeName(this,"body")||m(this).replaceWith(this.childNodes)}).end()}}),m.expr.filters.hidden=function(a){return a.offsetWidth<=0&&a.offsetHeight<=0||!k.reliableHiddenOffsets()&&"none"===(a.style&&a.style.display||m.css(a,"display"))},m.expr.filters.visible=function(a){return!m.expr.filters.hidden(a)};var Qc=/%20/g,Rc=/\[\]$/,Sc=/\r?\n/g,Tc=/^(?:submit|button|image|reset|file)$/i,Uc=/^(?:input|select|textarea|keygen)/i;function Vc(a,b,c,d){var e;if(m.isArray(b))m.each(b,function(b,e){c||Rc.test(a)?d(a,e):Vc(a+"["+("object"==typeof e?b:"")+"]",e,c,d)});else if(c||"object"!==m.type(b))d(a,b);else for(e in b)Vc(a+"["+e+"]",b[e],c,d)}m.param=function(a,b){var c,d=[],e=function(a,b){b=m.isFunction(b)?b():null==b?"":b,d[d.length]=encodeURIComponent(a)+"="+encodeURIComponent(b)};if(void 0===b&&(b=m.ajaxSettings&&m.ajaxSettings.traditional),m.isArray(a)||a.jquery&&!m.isPlainObject(a))m.each(a,function(){e(this.name,this.value)});else for(c in a)Vc(c,a[c],b,e);return d.join("&").replace(Qc,"+")},m.fn.extend({serialize:function(){return m.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var a=m.prop(this,"elements");return a?m.makeArray(a):this}).filter(function(){var a=this.type;return this.name&&!m(this).is(":disabled")&&Uc.test(this.nodeName)&&!Tc.test(a)&&(this.checked||!W.test(a))}).map(function(a,b){var c=m(this).val();return null==c?null:m.isArray(c)?m.map(c,function(a){return{name:b.name,value:a.replace(Sc,"\r\n")}}):{name:b.name,value:c.replace(Sc,"\r\n")}}).get()}}),m.ajaxSettings.xhr=void 0!==a.ActiveXObject?function(){return!this.isLocal&&/^(get|post|head|put|delete|options)$/i.test(this.type)&&Zc()||$c()}:Zc;var Wc=0,Xc={},Yc=m.ajaxSettings.xhr();a.ActiveXObject&&m(a).on("unload",function(){for(var a in Xc)Xc[a](void 0,!0)}),k.cors=!!Yc&&"withCredentials"in Yc,Yc=k.ajax=!!Yc,Yc&&m.ajaxTransport(function(a){if(!a.crossDomain||k.cors){var b;return{send:function(c,d){var e,f=a.xhr(),g=++Wc;if(f.open(a.type,a.url,a.async,a.username,a.password),a.xhrFields)for(e in a.xhrFields)f[e]=a.xhrFields[e];a.mimeType&&f.overrideMimeType&&f.overrideMimeType(a.mimeType),a.crossDomain||c["X-Requested-With"]||(c["X-Requested-With"]="XMLHttpRequest");for(e in c)void 0!==c[e]&&f.setRequestHeader(e,c[e]+"");f.send(a.hasContent&&a.data||null),b=function(c,e){var h,i,j;if(b&&(e||4===f.readyState))if(delete Xc[g],b=void 0,f.onreadystatechange=m.noop,e)4!==f.readyState&&f.abort();else{j={},h=f.status,"string"==typeof f.responseText&&(j.text=f.responseText);try{i=f.statusText}catch(k){i=""}h||!a.isLocal||a.crossDomain?1223===h&&(h=204):h=j.text?200:404}j&&d(h,i,j,f.getAllResponseHeaders())},a.async?4===f.readyState?setTimeout(b):f.onreadystatechange=Xc[g]=b:b()},abort:function(){b&&b(void 0,!0)}}}});function Zc(){try{return new a.XMLHttpRequest}catch(b){}}function $c(){try{return new a.ActiveXObject("Microsoft.XMLHTTP")}catch(b){}}m.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/(?:java|ecma)script/},converters:{"text script":function(a){return m.globalEval(a),a}}}),m.ajaxPrefilter("script",function(a){void 0===a.cache&&(a.cache=!1),a.crossDomain&&(a.type="GET",a.global=!1)}),m.ajaxTransport("script",function(a){if(a.crossDomain){var b,c=y.head||m("head")[0]||y.documentElement;return{send:function(d,e){b=y.createElement("script"),b.async=!0,a.scriptCharset&&(b.charset=a.scriptCharset),b.src=a.url,b.onload=b.onreadystatechange=function(a,c){(c||!b.readyState||/loaded|complete/.test(b.readyState))&&(b.onload=b.onreadystatechange=null,b.parentNode&&b.parentNode.removeChild(b),b=null,c||e(200,"success"))},c.insertBefore(b,c.firstChild)},abort:function(){b&&b.onload(void 0,!0)}}}});var _c=[],ad=/(=)\?(?=&|$)|\?\?/;m.ajaxSetup({jsonp:"callback",jsonpCallback:function(){var a=_c.pop()||m.expando+"_"+vc++;return this[a]=!0,a}}),m.ajaxPrefilter("json jsonp",function(b,c,d){var e,f,g,h=b.jsonp!==!1&&(ad.test(b.url)?"url":"string"==typeof b.data&&!(b.contentType||"").indexOf("application/x-www-form-urlencoded")&&ad.test(b.data)&&"data");return h||"jsonp"===b.dataTypes[0]?(e=b.jsonpCallback=m.isFunction(b.jsonpCallback)?b.jsonpCallback():b.jsonpCallback,h?b[h]=b[h].replace(ad,"$1"+e):b.jsonp!==!1&&(b.url+=(wc.test(b.url)?"&":"?")+b.jsonp+"="+e),b.converters["script json"]=function(){return g||m.error(e+" was not called"),g[0]},b.dataTypes[0]="json",f=a[e],a[e]=function(){g=arguments},d.always(function(){a[e]=f,b[e]&&(b.jsonpCallback=c.jsonpCallback,_c.push(e)),g&&m.isFunction(f)&&f(g[0]),g=f=void 0}),"script"):void 0}),m.parseHTML=function(a,b,c){if(!a||"string"!=typeof a)return null;"boolean"==typeof b&&(c=b,b=!1),b=b||y;var d=u.exec(a),e=!c&&[];return d?[b.createElement(d[1])]:(d=m.buildFragment([a],b,e),e&&e.length&&m(e).remove(),m.merge([],d.childNodes))};var bd=m.fn.load;m.fn.load=function(a,b,c){if("string"!=typeof a&&bd)return bd.apply(this,arguments);var d,e,f,g=this,h=a.indexOf(" ");return h>=0&&(d=m.trim(a.slice(h,a.length)),a=a.slice(0,h)),m.isFunction(b)?(c=b,b=void 0):b&&"object"==typeof b&&(f="POST"),g.length>0&&m.ajax({url:a,type:f,dataType:"html",data:b}).done(function(a){e=arguments,g.html(d?m("<div>").append(m.parseHTML(a)).find(d):a)}).complete(c&&function(a,b){g.each(c,e||[a.responseText,b,a])}),this},m.expr.filters.animated=function(a){return m.grep(m.timers,function(b){return a===b.elem}).length};var cd=a.document.documentElement;function dd(a){return m.isWindow(a)?a:9===a.nodeType?a.defaultView||a.parentWindow:!1}m.offset={setOffset:function(a,b,c){var d,e,f,g,h,i,j,k=m.css(a,"position"),l=m(a),n={};"static"===k&&(a.style.position="relative"),h=l.offset(),f=m.css(a,"top"),i=m.css(a,"left"),j=("absolute"===k||"fixed"===k)&&m.inArray("auto",[f,i])>-1,j?(d=l.position(),g=d.top,e=d.left):(g=parseFloat(f)||0,e=parseFloat(i)||0),m.isFunction(b)&&(b=b.call(a,c,h)),null!=b.top&&(n.top=b.top-h.top+g),null!=b.left&&(n.left=b.left-h.left+e),"using"in b?b.using.call(a,n):l.css(n)}},m.fn.extend({offset:function(a){if(arguments.length)return void 0===a?this:this.each(function(b){m.offset.setOffset(this,a,b)});var b,c,d={top:0,left:0},e=this[0],f=e&&e.ownerDocument;if(f)return b=f.documentElement,m.contains(b,e)?(typeof e.getBoundingClientRect!==K&&(d=e.getBoundingClientRect()),c=dd(f),{top:d.top+(c.pageYOffset||b.scrollTop)-(b.clientTop||0),left:d.left+(c.pageXOffset||b.scrollLeft)-(b.clientLeft||0)}):d},position:function(){if(this[0]){var a,b,c={top:0,left:0},d=this[0];return"fixed"===m.css(d,"position")?b=d.getBoundingClientRect():(a=this.offsetParent(),b=this.offset(),m.nodeName(a[0],"html")||(c=a.offset()),c.top+=m.css(a[0],"borderTopWidth",!0),c.left+=m.css(a[0],"borderLeftWidth",!0)),{top:b.top-c.top-m.css(d,"marginTop",!0),left:b.left-c.left-m.css(d,"marginLeft",!0)}}},offsetParent:function(){return this.map(function(){var a=this.offsetParent||cd;while(a&&!m.nodeName(a,"html")&&"static"===m.css(a,"position"))a=a.offsetParent;return a||cd})}}),m.each({scrollLeft:"pageXOffset",scrollTop:"pageYOffset"},function(a,b){var c=/Y/.test(b);m.fn[a]=function(d){return V(this,function(a,d,e){var f=dd(a);return void 0===e?f?b in f?f[b]:f.document.documentElement[d]:a[d]:void(f?f.scrollTo(c?m(f).scrollLeft():e,c?e:m(f).scrollTop()):a[d]=e)},a,d,arguments.length,null)}}),m.each(["top","left"],function(a,b){m.cssHooks[b]=Lb(k.pixelPosition,function(a,c){return c?(c=Jb(a,b),Hb.test(c)?m(a).position()[b]+"px":c):void 0})}),m.each({Height:"height",Width:"width"},function(a,b){m.each({padding:"inner"+a,content:b,"":"outer"+a},function(c,d){m.fn[d]=function(d,e){var f=arguments.length&&(c||"boolean"!=typeof d),g=c||(d===!0||e===!0?"margin":"border");return V(this,function(b,c,d){var e;return m.isWindow(b)?b.document.documentElement["client"+a]:9===b.nodeType?(e=b.documentElement,Math.max(b.body["scroll"+a],e["scroll"+a],b.body["offset"+a],e["offset"+a],e["client"+a])):void 0===d?m.css(b,c,g):m.style(b,c,d,g)},b,f?d:void 0,f,null)}})}),m.fn.size=function(){return this.length},m.fn.andSelf=m.fn.addBack,"function"==typeof define&&define.amd&&define("jquery",[],function(){return m});var ed=a.jQuery,fd=a.$;return m.noConflict=function(b){return a.$===m&&(a.$=fd),b&&a.jQuery===m&&(a.jQuery=ed),m},typeof b===K&&(a.jQuery=a.$=m),m});
+
+
+
+/** d3js **/
+!function(){function n(n,t){return t>n?-1:n>t?1:n>=t?0:0/0}function t(n){return null!=n&&!isNaN(n)}function e(n){return{left:function(t,e,r,u){for(arguments.length<3&&(r=0),arguments.length<4&&(u=t.length);u>r;){var i=r+u>>>1;n(t[i],e)<0?r=i+1:u=i}return r},right:function(t,e,r,u){for(arguments.length<3&&(r=0),arguments.length<4&&(u=t.length);u>r;){var i=r+u>>>1;n(t[i],e)>0?u=i:r=i+1}return r}}}function r(n){return n.length}function u(n){for(var t=1;n*t%1;)t*=10;return t}function i(n,t){try{for(var e in t)Object.defineProperty(n.prototype,e,{value:t[e],enumerable:!1})}catch(r){n.prototype=t}}function o(){}function a(n){return ia+n in this}function c(n){return n=ia+n,n in this&&delete this[n]}function s(){var n=[];return this.forEach(function(t){n.push(t)}),n}function l(){var n=0;for(var t in this)t.charCodeAt(0)===oa&&++n;return n}function f(){for(var n in this)if(n.charCodeAt(0)===oa)return!1;return!0}function h(){}function g(n,t,e){return function(){var r=e.apply(t,arguments);return r===t?n:r}}function p(n,t){if(t in n)return t;t=t.charAt(0).toUpperCase()+t.substring(1);for(var e=0,r=aa.length;r>e;++e){var u=aa[e]+t;if(u in n)return u}}function v(){}function d(){}function m(n){function t(){for(var t,r=e,u=-1,i=r.length;++u<i;)(t=r[u].on)&&t.apply(this,arguments);return n}var e=[],r=new o;return t.on=function(t,u){var i,o=r.get(t);return arguments.length<2?o&&o.on:(o&&(o.on=null,e=e.slice(0,i=e.indexOf(o)).concat(e.slice(i+1)),r.remove(t)),u&&e.push(r.set(t,{on:u})),n)},t}function y(){Zo.event.preventDefault()}function x(){for(var n,t=Zo.event;n=t.sourceEvent;)t=n;return t}function M(n){for(var t=new d,e=0,r=arguments.length;++e<r;)t[arguments[e]]=m(t);return t.of=function(e,r){return function(u){try{var i=u.sourceEvent=Zo.event;u.target=n,Zo.event=u,t[u.type].apply(e,r)}finally{Zo.event=i}}},t}function _(n){return sa(n,pa),n}function b(n){return"function"==typeof n?n:function(){return la(n,this)}}function w(n){return"function"==typeof n?n:function(){return fa(n,this)}}function S(n,t){function e(){this.removeAttribute(n)}function r(){this.removeAttributeNS(n.space,n.local)}function u(){this.setAttribute(n,t)}function i(){this.setAttributeNS(n.space,n.local,t)}function o(){var e=t.apply(this,arguments);null==e?this.removeAttribute(n):this.setAttribute(n,e)}function a(){var e=t.apply(this,arguments);null==e?this.removeAttributeNS(n.space,n.local):this.setAttributeNS(n.space,n.local,e)}return n=Zo.ns.qualify(n),null==t?n.local?r:e:"function"==typeof t?n.local?a:o:n.local?i:u}function k(n){return n.trim().replace(/\s+/g," ")}function E(n){return new RegExp("(?:^|\\s+)"+Zo.requote(n)+"(?:\\s+|$)","g")}function A(n){return(n+"").trim().split(/^|\s+/)}function C(n,t){function e(){for(var e=-1;++e<u;)n[e](this,t)}function r(){for(var e=-1,r=t.apply(this,arguments);++e<u;)n[e](this,r)}n=A(n).map(N);var u=n.length;return"function"==typeof t?r:e}function N(n){var t=E(n);return function(e,r){if(u=e.classList)return r?u.add(n):u.remove(n);var u=e.getAttribute("class")||"";r?(t.lastIndex=0,t.test(u)||e.setAttribute("class",k(u+" "+n))):e.setAttribute("class",k(u.replace(t," ")))}}function z(n,t,e){function r(){this.style.removeProperty(n)}function u(){this.style.setProperty(n,t,e)}function i(){var r=t.apply(this,arguments);null==r?this.style.removeProperty(n):this.style.setProperty(n,r,e)}return null==t?r:"function"==typeof t?i:u}function L(n,t){function e(){delete this[n]}function r(){this[n]=t}function u(){var e=t.apply(this,arguments);null==e?delete this[n]:this[n]=e}return null==t?e:"function"==typeof t?u:r}function T(n){return"function"==typeof n?n:(n=Zo.ns.qualify(n)).local?function(){return this.ownerDocument.createElementNS(n.space,n.local)}:function(){return this.ownerDocument.createElementNS(this.namespaceURI,n)}}function q(n){return{__data__:n}}function R(n){return function(){return ga(this,n)}}function D(t){return arguments.length||(t=n),function(n,e){return n&&e?t(n.__data__,e.__data__):!n-!e}}function P(n,t){for(var e=0,r=n.length;r>e;e++)for(var u,i=n[e],o=0,a=i.length;a>o;o++)(u=i[o])&&t(u,o,e);return n}function U(n){return sa(n,da),n}function j(n){var t,e;return function(r,u,i){var o,a=n[i].update,c=a.length;for(i!=e&&(e=i,t=0),u>=t&&(t=u+1);!(o=a[t])&&++t<c;);return o}}function H(){var n=this.__transition__;n&&++n.active}function F(n,t,e){function r(){var t=this[o];t&&(this.removeEventListener(n,t,t.$),delete this[o])}function u(){var u=c(t,Xo(arguments));r.call(this),this.addEventListener(n,this[o]=u,u.$=e),u._=t}function i(){var t,e=new RegExp("^__on([^.]+)"+Zo.requote(n)+"$");for(var r in this)if(t=r.match(e)){var u=this[r];this.removeEventListener(t[1],u,u.$),delete this[r]}}var o="__on"+n,a=n.indexOf("."),c=O;a>0&&(n=n.substring(0,a));var s=ya.get(n);return s&&(n=s,c=Y),a?t?u:r:t?v:i}function O(n,t){return function(e){var r=Zo.event;Zo.event=e,t[0]=this.__data__;try{n.apply(this,t)}finally{Zo.event=r}}}function Y(n,t){var e=O(n,t);return function(n){var t=this,r=n.relatedTarget;r&&(r===t||8&r.compareDocumentPosition(t))||e.call(t,n)}}function I(){var n=".dragsuppress-"+ ++Ma,t="click"+n,e=Zo.select(Wo).on("touchmove"+n,y).on("dragstart"+n,y).on("selectstart"+n,y);if(xa){var r=Bo.style,u=r[xa];r[xa]="none"}return function(i){function o(){e.on(t,null)}e.on(n,null),xa&&(r[xa]=u),i&&(e.on(t,function(){y(),o()},!0),setTimeout(o,0))}}function Z(n,t){t.changedTouches&&(t=t.changedTouches[0]);var e=n.ownerSVGElement||n;if(e.createSVGPoint){var r=e.createSVGPoint();if(0>_a&&(Wo.scrollX||Wo.scrollY)){e=Zo.select("body").append("svg").style({position:"absolute",top:0,left:0,margin:0,padding:0,border:"none"},"important");var u=e[0][0].getScreenCTM();_a=!(u.f||u.e),e.remove()}return _a?(r.x=t.pageX,r.y=t.pageY):(r.x=t.clientX,r.y=t.clientY),r=r.matrixTransform(n.getScreenCTM().inverse()),[r.x,r.y]}var i=n.getBoundingClientRect();return[t.clientX-i.left-n.clientLeft,t.clientY-i.top-n.clientTop]}function V(){return Zo.event.changedTouches[0].identifier}function X(){return Zo.event.target}function $(){return Wo}function B(n){return n>0?1:0>n?-1:0}function W(n,t,e){return(t[0]-n[0])*(e[1]-n[1])-(t[1]-n[1])*(e[0]-n[0])}function J(n){return n>1?0:-1>n?ba:Math.acos(n)}function G(n){return n>1?Sa:-1>n?-Sa:Math.asin(n)}function K(n){return((n=Math.exp(n))-1/n)/2}function Q(n){return((n=Math.exp(n))+1/n)/2}function nt(n){return((n=Math.exp(2*n))-1)/(n+1)}function tt(n){return(n=Math.sin(n/2))*n}function et(){}function rt(n,t,e){return this instanceof rt?(this.h=+n,this.s=+t,void(this.l=+e)):arguments.length<2?n instanceof rt?new rt(n.h,n.s,n.l):mt(""+n,yt,rt):new rt(n,t,e)}function ut(n,t,e){function r(n){return n>360?n-=360:0>n&&(n+=360),60>n?i+(o-i)*n/60:180>n?o:240>n?i+(o-i)*(240-n)/60:i}function u(n){return Math.round(255*r(n))}var i,o;return n=isNaN(n)?0:(n%=360)<0?n+360:n,t=isNaN(t)?0:0>t?0:t>1?1:t,e=0>e?0:e>1?1:e,o=.5>=e?e*(1+t):e+t-e*t,i=2*e-o,new gt(u(n+120),u(n),u(n-120))}function it(n,t,e){return this instanceof it?(this.h=+n,this.c=+t,void(this.l=+e)):arguments.length<2?n instanceof it?new it(n.h,n.c,n.l):n instanceof at?st(n.l,n.a,n.b):st((n=xt((n=Zo.rgb(n)).r,n.g,n.b)).l,n.a,n.b):new it(n,t,e)}function ot(n,t,e){return isNaN(n)&&(n=0),isNaN(t)&&(t=0),new at(e,Math.cos(n*=Aa)*t,Math.sin(n)*t)}function at(n,t,e){return this instanceof at?(this.l=+n,this.a=+t,void(this.b=+e)):arguments.length<2?n instanceof at?new at(n.l,n.a,n.b):n instanceof it?ot(n.l,n.c,n.h):xt((n=gt(n)).r,n.g,n.b):new at(n,t,e)}function ct(n,t,e){var r=(n+16)/116,u=r+t/500,i=r-e/200;return u=lt(u)*ja,r=lt(r)*Ha,i=lt(i)*Fa,new gt(ht(3.2404542*u-1.5371385*r-.4985314*i),ht(-.969266*u+1.8760108*r+.041556*i),ht(.0556434*u-.2040259*r+1.0572252*i))}function st(n,t,e){return n>0?new it(Math.atan2(e,t)*Ca,Math.sqrt(t*t+e*e),n):new it(0/0,0/0,n)}function lt(n){return n>.206893034?n*n*n:(n-4/29)/7.787037}function ft(n){return n>.008856?Math.pow(n,1/3):7.787037*n+4/29}function ht(n){return Math.round(255*(.00304>=n?12.92*n:1.055*Math.pow(n,1/2.4)-.055))}function gt(n,t,e){return this instanceof gt?(this.r=~~n,this.g=~~t,void(this.b=~~e)):arguments.length<2?n instanceof gt?new gt(n.r,n.g,n.b):mt(""+n,gt,ut):new gt(n,t,e)}function pt(n){return new gt(n>>16,255&n>>8,255&n)}function vt(n){return pt(n)+""}function dt(n){return 16>n?"0"+Math.max(0,n).toString(16):Math.min(255,n).toString(16)}function mt(n,t,e){var r,u,i,o=0,a=0,c=0;if(r=/([a-z]+)\((.*)\)/i.exec(n))switch(u=r[2].split(","),r[1]){case"hsl":return e(parseFloat(u[0]),parseFloat(u[1])/100,parseFloat(u[2])/100);case"rgb":return t(_t(u[0]),_t(u[1]),_t(u[2]))}return(i=Ia.get(n))?t(i.r,i.g,i.b):(null==n||"#"!==n.charAt(0)||isNaN(i=parseInt(n.substring(1),16))||(4===n.length?(o=(3840&i)>>4,o=o>>4|o,a=240&i,a=a>>4|a,c=15&i,c=c<<4|c):7===n.length&&(o=(16711680&i)>>16,a=(65280&i)>>8,c=255&i)),t(o,a,c))}function yt(n,t,e){var r,u,i=Math.min(n/=255,t/=255,e/=255),o=Math.max(n,t,e),a=o-i,c=(o+i)/2;return a?(u=.5>c?a/(o+i):a/(2-o-i),r=n==o?(t-e)/a+(e>t?6:0):t==o?(e-n)/a+2:(n-t)/a+4,r*=60):(r=0/0,u=c>0&&1>c?0:r),new rt(r,u,c)}function xt(n,t,e){n=Mt(n),t=Mt(t),e=Mt(e);var r=ft((.4124564*n+.3575761*t+.1804375*e)/ja),u=ft((.2126729*n+.7151522*t+.072175*e)/Ha),i=ft((.0193339*n+.119192*t+.9503041*e)/Fa);return at(116*u-16,500*(r-u),200*(u-i))}function Mt(n){return(n/=255)<=.04045?n/12.92:Math.pow((n+.055)/1.055,2.4)}function _t(n){var t=parseFloat(n);return"%"===n.charAt(n.length-1)?Math.round(2.55*t):t}function bt(n){return"function"==typeof n?n:function(){return n}}function wt(n){return n}function St(n){return function(t,e,r){return 2===arguments.length&&"function"==typeof e&&(r=e,e=null),kt(t,e,n,r)}}function kt(n,t,e,r){function u(){var n,t=c.status;if(!t&&c.responseText||t>=200&&300>t||304===t){try{n=e.call(i,c)}catch(r){return o.error.call(i,r),void 0}o.load.call(i,n)}else o.error.call(i,c)}var i={},o=Zo.dispatch("beforesend","progress","load","error"),a={},c=new XMLHttpRequest,s=null;return!Wo.XDomainRequest||"withCredentials"in c||!/^(http(s)?:)?\/\//.test(n)||(c=new XDomainRequest),"onload"in c?c.onload=c.onerror=u:c.onreadystatechange=function(){c.readyState>3&&u()},c.onprogress=function(n){var t=Zo.event;Zo.event=n;try{o.progress.call(i,c)}finally{Zo.event=t}},i.header=function(n,t){return n=(n+"").toLowerCase(),arguments.length<2?a[n]:(null==t?delete a[n]:a[n]=t+"",i)},i.mimeType=function(n){return arguments.length?(t=null==n?null:n+"",i):t},i.responseType=function(n){return arguments.length?(s=n,i):s},i.response=function(n){return e=n,i},["get","post"].forEach(function(n){i[n]=function(){return i.send.apply(i,[n].concat(Xo(arguments)))}}),i.send=function(e,r,u){if(2===arguments.length&&"function"==typeof r&&(u=r,r=null),c.open(e,n,!0),null==t||"accept"in a||(a.accept=t+",*/*"),c.setRequestHeader)for(var l in a)c.setRequestHeader(l,a[l]);return null!=t&&c.overrideMimeType&&c.overrideMimeType(t),null!=s&&(c.responseType=s),null!=u&&i.on("error",u).on("load",function(n){u(null,n)}),o.beforesend.call(i,c),c.send(null==r?null:r),i},i.abort=function(){return c.abort(),i},Zo.rebind(i,o,"on"),null==r?i:i.get(Et(r))}function Et(n){return 1===n.length?function(t,e){n(null==t?e:null)}:n}function At(){var n=Ct(),t=Nt()-n;t>24?(isFinite(t)&&(clearTimeout($a),$a=setTimeout(At,t)),Xa=0):(Xa=1,Wa(At))}function Ct(){var n=Date.now();for(Ba=Za;Ba;)n>=Ba.t&&(Ba.f=Ba.c(n-Ba.t)),Ba=Ba.n;return n}function Nt(){for(var n,t=Za,e=1/0;t;)t.f?t=n?n.n=t.n:Za=t.n:(t.t<e&&(e=t.t),t=(n=t).n);return Va=n,e}function zt(n,t){return t-(n?Math.ceil(Math.log(n)/Math.LN10):1)}function Lt(n,t){var e=Math.pow(10,3*ua(8-t));return{scale:t>8?function(n){return n/e}:function(n){return n*e},symbol:n}}function Tt(n){var t=n.decimal,e=n.thousands,r=n.grouping,u=n.currency,i=r?function(n){for(var t=n.length,u=[],i=0,o=r[0];t>0&&o>0;)u.push(n.substring(t-=o,t+o)),o=r[i=(i+1)%r.length];return u.reverse().join(e)}:wt;return function(n){var e=Ga.exec(n),r=e[1]||" ",o=e[2]||">",a=e[3]||"",c=e[4]||"",s=e[5],l=+e[6],f=e[7],h=e[8],g=e[9],p=1,v="",d="",m=!1;switch(h&&(h=+h.substring(1)),(s||"0"===r&&"="===o)&&(s=r="0",o="=",f&&(l-=Math.floor((l-1)/4))),g){case"n":f=!0,g="g";break;case"%":p=100,d="%",g="f";break;case"p":p=100,d="%",g="r";break;case"b":case"o":case"x":case"X":"#"===c&&(v="0"+g.toLowerCase());case"c":case"d":m=!0,h=0;break;case"s":p=-1,g="r"}"$"===c&&(v=u[0],d=u[1]),"r"!=g||h||(g="g"),null!=h&&("g"==g?h=Math.max(1,Math.min(21,h)):("e"==g||"f"==g)&&(h=Math.max(0,Math.min(20,h)))),g=Ka.get(g)||qt;var y=s&&f;return function(n){var e=d;if(m&&n%1)return"";var u=0>n||0===n&&0>1/n?(n=-n,"-"):a;if(0>p){var c=Zo.formatPrefix(n,h);n=c.scale(n),e=c.symbol+d}else n*=p;n=g(n,h);var x=n.lastIndexOf("."),M=0>x?n:n.substring(0,x),_=0>x?"":t+n.substring(x+1);!s&&f&&(M=i(M));var b=v.length+M.length+_.length+(y?0:u.length),w=l>b?new Array(b=l-b+1).join(r):"";return y&&(M=i(w+M)),u+=v,n=M+_,("<"===o?u+n+w:">"===o?w+u+n:"^"===o?w.substring(0,b>>=1)+u+n+w.substring(b):u+(y?n:w+n))+e}}}function qt(n){return n+""}function Rt(){this._=new Date(arguments.length>1?Date.UTC.apply(this,arguments):arguments[0])}function Dt(n,t,e){function r(t){var e=n(t),r=i(e,1);return r-t>t-e?e:r}function u(e){return t(e=n(new nc(e-1)),1),e}function i(n,e){return t(n=new nc(+n),e),n}function o(n,r,i){var o=u(n),a=[];if(i>1)for(;r>o;)e(o)%i||a.push(new Date(+o)),t(o,1);else for(;r>o;)a.push(new Date(+o)),t(o,1);return a}function a(n,t,e){try{nc=Rt;var r=new Rt;return r._=n,o(r,t,e)}finally{nc=Date}}n.floor=n,n.round=r,n.ceil=u,n.offset=i,n.range=o;var c=n.utc=Pt(n);return c.floor=c,c.round=Pt(r),c.ceil=Pt(u),c.offset=Pt(i),c.range=a,n}function Pt(n){return function(t,e){try{nc=Rt;var r=new Rt;return r._=t,n(r,e)._}finally{nc=Date}}}function Ut(n){function t(n){function t(t){for(var e,u,i,o=[],a=-1,c=0;++a<r;)37===n.charCodeAt(a)&&(o.push(n.substring(c,a)),null!=(u=ec[e=n.charAt(++a)])&&(e=n.charAt(++a)),(i=C[e])&&(e=i(t,null==u?"e"===e?" ":"0":u)),o.push(e),c=a+1);return o.push(n.substring(c,a)),o.join("")}var r=n.length;return t.parse=function(t){var r={y:1900,m:0,d:1,H:0,M:0,S:0,L:0,Z:null},u=e(r,n,t,0);if(u!=t.length)return null;"p"in r&&(r.H=r.H%12+12*r.p);var i=null!=r.Z&&nc!==Rt,o=new(i?Rt:nc);return"j"in r?o.setFullYear(r.y,0,r.j):"w"in r&&("W"in r||"U"in r)?(o.setFullYear(r.y,0,1),o.setFullYear(r.y,0,"W"in r?(r.w+6)%7+7*r.W-(o.getDay()+5)%7:r.w+7*r.U-(o.getDay()+6)%7)):o.setFullYear(r.y,r.m,r.d),o.setHours(r.H+Math.floor(r.Z/100),r.M+r.Z%100,r.S,r.L),i?o._:o},t.toString=function(){return n},t}function e(n,t,e,r){for(var u,i,o,a=0,c=t.length,s=e.length;c>a;){if(r>=s)return-1;if(u=t.charCodeAt(a++),37===u){if(o=t.charAt(a++),i=N[o in ec?t.charAt(a++):o],!i||(r=i(n,e,r))<0)return-1}else if(u!=e.charCodeAt(r++))return-1}return r}function r(n,t,e){b.lastIndex=0;var r=b.exec(t.substring(e));return r?(n.w=w.get(r[0].toLowerCase()),e+r[0].length):-1}function u(n,t,e){M.lastIndex=0;var r=M.exec(t.substring(e));return r?(n.w=_.get(r[0].toLowerCase()),e+r[0].length):-1}function i(n,t,e){E.lastIndex=0;var r=E.exec(t.substring(e));return r?(n.m=A.get(r[0].toLowerCase()),e+r[0].length):-1}function o(n,t,e){S.lastIndex=0;var r=S.exec(t.substring(e));return r?(n.m=k.get(r[0].toLowerCase()),e+r[0].length):-1}function a(n,t,r){return e(n,C.c.toString(),t,r)}function c(n,t,r){return e(n,C.x.toString(),t,r)}function s(n,t,r){return e(n,C.X.toString(),t,r)}function l(n,t,e){var r=x.get(t.substring(e,e+=2).toLowerCase());return null==r?-1:(n.p=r,e)}var f=n.dateTime,h=n.date,g=n.time,p=n.periods,v=n.days,d=n.shortDays,m=n.months,y=n.shortMonths;t.utc=function(n){function e(n){try{nc=Rt;var t=new nc;return t._=n,r(t)}finally{nc=Date}}var r=t(n);return e.parse=function(n){try{nc=Rt;var t=r.parse(n);return t&&t._}finally{nc=Date}},e.toString=r.toString,e},t.multi=t.utc.multi=re;var x=Zo.map(),M=Ht(v),_=Ft(v),b=Ht(d),w=Ft(d),S=Ht(m),k=Ft(m),E=Ht(y),A=Ft(y);p.forEach(function(n,t){x.set(n.toLowerCase(),t)});var C={a:function(n){return d[n.getDay()]},A:function(n){return v[n.getDay()]},b:function(n){return y[n.getMonth()]},B:function(n){return m[n.getMonth()]},c:t(f),d:function(n,t){return jt(n.getDate(),t,2)},e:function(n,t){return jt(n.getDate(),t,2)},H:function(n,t){return jt(n.getHours(),t,2)},I:function(n,t){return jt(n.getHours()%12||12,t,2)},j:function(n,t){return jt(1+Qa.dayOfYear(n),t,3)},L:function(n,t){return jt(n.getMilliseconds(),t,3)},m:function(n,t){return jt(n.getMonth()+1,t,2)},M:function(n,t){return jt(n.getMinutes(),t,2)},p:function(n){return p[+(n.getHours()>=12)]},S:function(n,t){return jt(n.getSeconds(),t,2)},U:function(n,t){return jt(Qa.sundayOfYear(n),t,2)},w:function(n){return n.getDay()},W:function(n,t){return jt(Qa.mondayOfYear(n),t,2)},x:t(h),X:t(g),y:function(n,t){return jt(n.getFullYear()%100,t,2)},Y:function(n,t){return jt(n.getFullYear()%1e4,t,4)},Z:te,"%":function(){return"%"}},N={a:r,A:u,b:i,B:o,c:a,d:Wt,e:Wt,H:Gt,I:Gt,j:Jt,L:ne,m:Bt,M:Kt,p:l,S:Qt,U:Yt,w:Ot,W:It,x:c,X:s,y:Vt,Y:Zt,Z:Xt,"%":ee};return t}function jt(n,t,e){var r=0>n?"-":"",u=(r?-n:n)+"",i=u.length;return r+(e>i?new Array(e-i+1).join(t)+u:u)}function Ht(n){return new RegExp("^(?:"+n.map(Zo.requote).join("|")+")","i")}function Ft(n){for(var t=new o,e=-1,r=n.length;++e<r;)t.set(n[e].toLowerCase(),e);return t}function Ot(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+1));return r?(n.w=+r[0],e+r[0].length):-1}function Yt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e));return r?(n.U=+r[0],e+r[0].length):-1}function It(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e));return r?(n.W=+r[0],e+r[0].length):-1}function Zt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+4));return r?(n.y=+r[0],e+r[0].length):-1}function Vt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+2));return r?(n.y=$t(+r[0]),e+r[0].length):-1}function Xt(n,t,e){return/^[+-]\d{4}$/.test(t=t.substring(e,e+5))?(n.Z=-t,e+5):-1}function $t(n){return n+(n>68?1900:2e3)}function Bt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+2));return r?(n.m=r[0]-1,e+r[0].length):-1}function Wt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+2));return r?(n.d=+r[0],e+r[0].length):-1}function Jt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+3));return r?(n.j=+r[0],e+r[0].length):-1}function Gt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+2));return r?(n.H=+r[0],e+r[0].length):-1}function Kt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+2));return r?(n.M=+r[0],e+r[0].length):-1}function Qt(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+2));return r?(n.S=+r[0],e+r[0].length):-1}function ne(n,t,e){rc.lastIndex=0;var r=rc.exec(t.substring(e,e+3));return r?(n.L=+r[0],e+r[0].length):-1}function te(n){var t=n.getTimezoneOffset(),e=t>0?"-":"+",r=~~(ua(t)/60),u=ua(t)%60;return e+jt(r,"0",2)+jt(u,"0",2)}function ee(n,t,e){uc.lastIndex=0;var r=uc.exec(t.substring(e,e+1));return r?e+r[0].length:-1}function re(n){for(var t=n.length,e=-1;++e<t;)n[e][0]=this(n[e][0]);return function(t){for(var e=0,r=n[e];!r[1](t);)r=n[++e];return r[0](t)}}function ue(){}function ie(n,t,e){var r=e.s=n+t,u=r-n,i=r-u;e.t=n-i+(t-u)}function oe(n,t){n&&cc.hasOwnProperty(n.type)&&cc[n.type](n,t)}function ae(n,t,e){var r,u=-1,i=n.length-e;for(t.lineStart();++u<i;)r=n[u],t.point(r[0],r[1],r[2]);t.lineEnd()}function ce(n,t){var e=-1,r=n.length;for(t.polygonStart();++e<r;)ae(n[e],t,1);t.polygonEnd()}function se(){function n(n,t){n*=Aa,t=t*Aa/2+ba/4;var e=n-r,o=e>=0?1:-1,a=o*e,c=Math.cos(t),s=Math.sin(t),l=i*s,f=u*c+l*Math.cos(a),h=l*o*Math.sin(a);lc.add(Math.atan2(h,f)),r=n,u=c,i=s}var t,e,r,u,i;fc.point=function(o,a){fc.point=n,r=(t=o)*Aa,u=Math.cos(a=(e=a)*Aa/2+ba/4),i=Math.sin(a)},fc.lineEnd=function(){n(t,e)}}function le(n){var t=n[0],e=n[1],r=Math.cos(e);return[r*Math.cos(t),r*Math.sin(t),Math.sin(e)]}function fe(n,t){return n[0]*t[0]+n[1]*t[1]+n[2]*t[2]}function he(n,t){return[n[1]*t[2]-n[2]*t[1],n[2]*t[0]-n[0]*t[2],n[0]*t[1]-n[1]*t[0]]}function ge(n,t){n[0]+=t[0],n[1]+=t[1],n[2]+=t[2]}function pe(n,t){return[n[0]*t,n[1]*t,n[2]*t]}function ve(n){var t=Math.sqrt(n[0]*n[0]+n[1]*n[1]+n[2]*n[2]);n[0]/=t,n[1]/=t,n[2]/=t}function de(n){return[Math.atan2(n[1],n[0]),G(n[2])]}function me(n,t){return ua(n[0]-t[0])<ka&&ua(n[1]-t[1])<ka}function ye(n,t){n*=Aa;var e=Math.cos(t*=Aa);xe(e*Math.cos(n),e*Math.sin(n),Math.sin(t))}function xe(n,t,e){++hc,pc+=(n-pc)/hc,vc+=(t-vc)/hc,dc+=(e-dc)/hc}function Me(){function n(n,u){n*=Aa;var i=Math.cos(u*=Aa),o=i*Math.cos(n),a=i*Math.sin(n),c=Math.sin(u),s=Math.atan2(Math.sqrt((s=e*c-r*a)*s+(s=r*o-t*c)*s+(s=t*a-e*o)*s),t*o+e*a+r*c);gc+=s,mc+=s*(t+(t=o)),yc+=s*(e+(e=a)),xc+=s*(r+(r=c)),xe(t,e,r)}var t,e,r;wc.point=function(u,i){u*=Aa;var o=Math.cos(i*=Aa);t=o*Math.cos(u),e=o*Math.sin(u),r=Math.sin(i),wc.point=n,xe(t,e,r)}}function _e(){wc.point=ye}function be(){function n(n,t){n*=Aa;var e=Math.cos(t*=Aa),o=e*Math.cos(n),a=e*Math.sin(n),c=Math.sin(t),s=u*c-i*a,l=i*o-r*c,f=r*a-u*o,h=Math.sqrt(s*s+l*l+f*f),g=r*o+u*a+i*c,p=h&&-J(g)/h,v=Math.atan2(h,g);Mc+=p*s,_c+=p*l,bc+=p*f,gc+=v,mc+=v*(r+(r=o)),yc+=v*(u+(u=a)),xc+=v*(i+(i=c)),xe(r,u,i)}var t,e,r,u,i;wc.point=function(o,a){t=o,e=a,wc.point=n,o*=Aa;var c=Math.cos(a*=Aa);r=c*Math.cos(o),u=c*Math.sin(o),i=Math.sin(a),xe(r,u,i)},wc.lineEnd=function(){n(t,e),wc.lineEnd=_e,wc.point=ye}}function we(){return!0}function Se(n,t,e,r,u){var i=[],o=[];if(n.forEach(function(n){if(!((t=n.length-1)<=0)){var t,e=n[0],r=n[t];if(me(e,r)){u.lineStart();for(var a=0;t>a;++a)u.point((e=n[a])[0],e[1]);return u.lineEnd(),void 0}var c=new Ee(e,n,null,!0),s=new Ee(e,null,c,!1);c.o=s,i.push(c),o.push(s),c=new Ee(r,n,null,!1),s=new Ee(r,null,c,!0),c.o=s,i.push(c),o.push(s)}}),o.sort(t),ke(i),ke(o),i.length){for(var a=0,c=e,s=o.length;s>a;++a)o[a].e=c=!c;for(var l,f,h=i[0];;){for(var g=h,p=!0;g.v;)if((g=g.n)===h)return;l=g.z,u.lineStart();do{if(g.v=g.o.v=!0,g.e){if(p)for(var a=0,s=l.length;s>a;++a)u.point((f=l[a])[0],f[1]);else r(g.x,g.n.x,1,u);g=g.n}else{if(p){l=g.p.z;for(var a=l.length-1;a>=0;--a)u.point((f=l[a])[0],f[1])}else r(g.x,g.p.x,-1,u);g=g.p}g=g.o,l=g.z,p=!p}while(!g.v);u.lineEnd()}}}function ke(n){if(t=n.length){for(var t,e,r=0,u=n[0];++r<t;)u.n=e=n[r],e.p=u,u=e;u.n=e=n[0],e.p=u}}function Ee(n,t,e,r){this.x=n,this.z=t,this.o=e,this.e=r,this.v=!1,this.n=this.p=null}function Ae(n,t,e,r){return function(u,i){function o(t,e){var r=u(t,e);n(t=r[0],e=r[1])&&i.point(t,e)}function a(n,t){var e=u(n,t);d.point(e[0],e[1])}function c(){y.point=a,d.lineStart()}function s(){y.point=o,d.lineEnd()}function l(n,t){v.push([n,t]);var e=u(n,t);M.point(e[0],e[1])}function f(){M.lineStart(),v=[]}function h(){l(v[0][0],v[0][1]),M.lineEnd();var n,t=M.clean(),e=x.buffer(),r=e.length;if(v.pop(),p.push(v),v=null,r)if(1&t){n=e[0];var u,r=n.length-1,o=-1;if(r>0){for(_||(i.polygonStart(),_=!0),i.lineStart();++o<r;)i.point((u=n[o])[0],u[1]);i.lineEnd()}}else r>1&&2&t&&e.push(e.pop().concat(e.shift())),g.push(e.filter(Ce))}var g,p,v,d=t(i),m=u.invert(r[0],r[1]),y={point:o,lineStart:c,lineEnd:s,polygonStart:function(){y.point=l,y.lineStart=f,y.lineEnd=h,g=[],p=[]},polygonEnd:function(){y.point=o,y.lineStart=c,y.lineEnd=s,g=Zo.merge(g);var n=Le(m,p);g.length?(_||(i.polygonStart(),_=!0),Se(g,ze,n,e,i)):n&&(_||(i.polygonStart(),_=!0),i.lineStart(),e(null,null,1,i),i.lineEnd()),_&&(i.polygonEnd(),_=!1),g=p=null},sphere:function(){i.polygonStart(),i.lineStart(),e(null,null,1,i),i.lineEnd(),i.polygonEnd()}},x=Ne(),M=t(x),_=!1;return y}}function Ce(n){return n.length>1}function Ne(){var n,t=[];return{lineStart:function(){t.push(n=[])},point:function(t,e){n.push([t,e])},lineEnd:v,buffer:function(){var e=t;return t=[],n=null,e},rejoin:function(){t.length>1&&t.push(t.pop().concat(t.shift()))}}}function ze(n,t){return((n=n.x)[0]<0?n[1]-Sa-ka:Sa-n[1])-((t=t.x)[0]<0?t[1]-Sa-ka:Sa-t[1])}function Le(n,t){var e=n[0],r=n[1],u=[Math.sin(e),-Math.cos(e),0],i=0,o=0;lc.reset();for(var a=0,c=t.length;c>a;++a){var s=t[a],l=s.length;if(l)for(var f=s[0],h=f[0],g=f[1]/2+ba/4,p=Math.sin(g),v=Math.cos(g),d=1;;){d===l&&(d=0),n=s[d];var m=n[0],y=n[1]/2+ba/4,x=Math.sin(y),M=Math.cos(y),_=m-h,b=_>=0?1:-1,w=b*_,S=w>ba,k=p*x;if(lc.add(Math.atan2(k*b*Math.sin(w),v*M+k*Math.cos(w))),i+=S?_+b*wa:_,S^h>=e^m>=e){var E=he(le(f),le(n));ve(E);var A=he(u,E);ve(A);var C=(S^_>=0?-1:1)*G(A[2]);(r>C||r===C&&(E[0]||E[1]))&&(o+=S^_>=0?1:-1)}if(!d++)break;h=m,p=x,v=M,f=n}}return(-ka>i||ka>i&&0>lc)^1&o}function Te(n){var t,e=0/0,r=0/0,u=0/0;return{lineStart:function(){n.lineStart(),t=1},point:function(i,o){var a=i>0?ba:-ba,c=ua(i-e);ua(c-ba)<ka?(n.point(e,r=(r+o)/2>0?Sa:-Sa),n.point(u,r),n.lineEnd(),n.lineStart(),n.point(a,r),n.point(i,r),t=0):u!==a&&c>=ba&&(ua(e-u)<ka&&(e-=u*ka),ua(i-a)<ka&&(i-=a*ka),r=qe(e,r,i,o),n.point(u,r),n.lineEnd(),n.lineStart(),n.point(a,r),t=0),n.point(e=i,r=o),u=a},lineEnd:function(){n.lineEnd(),e=r=0/0},clean:function(){return 2-t}}}function qe(n,t,e,r){var u,i,o=Math.sin(n-e);return ua(o)>ka?Math.atan((Math.sin(t)*(i=Math.cos(r))*Math.sin(e)-Math.sin(r)*(u=Math.cos(t))*Math.sin(n))/(u*i*o)):(t+r)/2}function Re(n,t,e,r){var u;if(null==n)u=e*Sa,r.point(-ba,u),r.point(0,u),r.point(ba,u),r.point(ba,0),r.point(ba,-u),r.point(0,-u),r.point(-ba,-u),r.point(-ba,0),r.point(-ba,u);else if(ua(n[0]-t[0])>ka){var i=n[0]<t[0]?ba:-ba;u=e*i/2,r.point(-i,u),r.point(0,u),r.point(i,u)}else r.point(t[0],t[1])}function De(n){function t(n,t){return Math.cos(n)*Math.cos(t)>i}function e(n){var e,i,c,s,l;return{lineStart:function(){s=c=!1,l=1},point:function(f,h){var g,p=[f,h],v=t(f,h),d=o?v?0:u(f,h):v?u(f+(0>f?ba:-ba),h):0;if(!e&&(s=c=v)&&n.lineStart(),v!==c&&(g=r(e,p),(me(e,g)||me(p,g))&&(p[0]+=ka,p[1]+=ka,v=t(p[0],p[1]))),v!==c)l=0,v?(n.lineStart(),g=r(p,e),n.point(g[0],g[1])):(g=r(e,p),n.point(g[0],g[1]),n.lineEnd()),e=g;else if(a&&e&&o^v){var m;d&i||!(m=r(p,e,!0))||(l=0,o?(n.lineStart(),n.point(m[0][0],m[0][1]),n.point(m[1][0],m[1][1]),n.lineEnd()):(n.point(m[1][0],m[1][1]),n.lineEnd(),n.lineStart(),n.point(m[0][0],m[0][1])))}!v||e&&me(e,p)||n.point(p[0],p[1]),e=p,c=v,i=d},lineEnd:function(){c&&n.lineEnd(),e=null},clean:function(){return l|(s&&c)<<1}}}function r(n,t,e){var r=le(n),u=le(t),o=[1,0,0],a=he(r,u),c=fe(a,a),s=a[0],l=c-s*s;if(!l)return!e&&n;var f=i*c/l,h=-i*s/l,g=he(o,a),p=pe(o,f),v=pe(a,h);ge(p,v);var d=g,m=fe(p,d),y=fe(d,d),x=m*m-y*(fe(p,p)-1);if(!(0>x)){var M=Math.sqrt(x),_=pe(d,(-m-M)/y);if(ge(_,p),_=de(_),!e)return _;var b,w=n[0],S=t[0],k=n[1],E=t[1];w>S&&(b=w,w=S,S=b);var A=S-w,C=ua(A-ba)<ka,N=C||ka>A;if(!C&&k>E&&(b=k,k=E,E=b),N?C?k+E>0^_[1]<(ua(_[0]-w)<ka?k:E):k<=_[1]&&_[1]<=E:A>ba^(w<=_[0]&&_[0]<=S)){var z=pe(d,(-m+M)/y);return ge(z,p),[_,de(z)]}}}function u(t,e){var r=o?n:ba-n,u=0;return-r>t?u|=1:t>r&&(u|=2),-r>e?u|=4:e>r&&(u|=8),u}var i=Math.cos(n),o=i>0,a=ua(i)>ka,c=sr(n,6*Aa);return Ae(t,e,c,o?[0,-n]:[-ba,n-ba])}function Pe(n,t,e,r){return function(u){var i,o=u.a,a=u.b,c=o.x,s=o.y,l=a.x,f=a.y,h=0,g=1,p=l-c,v=f-s;if(i=n-c,p||!(i>0)){if(i/=p,0>p){if(h>i)return;g>i&&(g=i)}else if(p>0){if(i>g)return;i>h&&(h=i)}if(i=e-c,p||!(0>i)){if(i/=p,0>p){if(i>g)return;i>h&&(h=i)}else if(p>0){if(h>i)return;g>i&&(g=i)}if(i=t-s,v||!(i>0)){if(i/=v,0>v){if(h>i)return;g>i&&(g=i)}else if(v>0){if(i>g)return;i>h&&(h=i)}if(i=r-s,v||!(0>i)){if(i/=v,0>v){if(i>g)return;i>h&&(h=i)}else if(v>0){if(h>i)return;g>i&&(g=i)}return h>0&&(u.a={x:c+h*p,y:s+h*v}),1>g&&(u.b={x:c+g*p,y:s+g*v}),u}}}}}}function Ue(n,t,e,r){function u(r,u){return ua(r[0]-n)<ka?u>0?0:3:ua(r[0]-e)<ka?u>0?2:1:ua(r[1]-t)<ka?u>0?1:0:u>0?3:2}function i(n,t){return o(n.x,t.x)}function o(n,t){var e=u(n,1),r=u(t,1);return e!==r?e-r:0===e?t[1]-n[1]:1===e?n[0]-t[0]:2===e?n[1]-t[1]:t[0]-n[0]}return function(a){function c(n){for(var t=0,e=d.length,r=n[1],u=0;e>u;++u)for(var i,o=1,a=d[u],c=a.length,s=a[0];c>o;++o)i=a[o],s[1]<=r?i[1]>r&&W(s,i,n)>0&&++t:i[1]<=r&&W(s,i,n)<0&&--t,s=i;return 0!==t}function s(i,a,c,s){var l=0,f=0;if(null==i||(l=u(i,c))!==(f=u(a,c))||o(i,a)<0^c>0){do s.point(0===l||3===l?n:e,l>1?r:t);while((l=(l+c+4)%4)!==f)}else s.point(a[0],a[1])}function l(u,i){return u>=n&&e>=u&&i>=t&&r>=i}function f(n,t){l(n,t)&&a.point(n,t)}function h(){N.point=p,d&&d.push(m=[]),S=!0,w=!1,_=b=0/0}function g(){v&&(p(y,x),M&&w&&A.rejoin(),v.push(A.buffer())),N.point=f,w&&a.lineEnd()}function p(n,t){n=Math.max(-kc,Math.min(kc,n)),t=Math.max(-kc,Math.min(kc,t));var e=l(n,t);if(d&&m.push([n,t]),S)y=n,x=t,M=e,S=!1,e&&(a.lineStart(),a.point(n,t));else if(e&&w)a.point(n,t);else{var r={a:{x:_,y:b},b:{x:n,y:t}};C(r)?(w||(a.lineStart(),a.point(r.a.x,r.a.y)),a.point(r.b.x,r.b.y),e||a.lineEnd(),k=!1):e&&(a.lineStart(),a.point(n,t),k=!1)}_=n,b=t,w=e}var v,d,m,y,x,M,_,b,w,S,k,E=a,A=Ne(),C=Pe(n,t,e,r),N={point:f,lineStart:h,lineEnd:g,polygonStart:function(){a=A,v=[],d=[],k=!0},polygonEnd:function(){a=E,v=Zo.merge(v);var t=c([n,r]),e=k&&t,u=v.length;(e||u)&&(a.polygonStart(),e&&(a.lineStart(),s(null,null,1,a),a.lineEnd()),u&&Se(v,i,t,s,a),a.polygonEnd()),v=d=m=null}};return N}}function je(n,t){function e(e,r){return e=n(e,r),t(e[0],e[1])}return n.invert&&t.invert&&(e.invert=function(e,r){return e=t.invert(e,r),e&&n.invert(e[0],e[1])}),e}function He(n){var t=0,e=ba/3,r=tr(n),u=r(t,e);return u.parallels=function(n){return arguments.length?r(t=n[0]*ba/180,e=n[1]*ba/180):[180*(t/ba),180*(e/ba)]},u}function Fe(n,t){function e(n,t){var e=Math.sqrt(i-2*u*Math.sin(t))/u;return[e*Math.sin(n*=u),o-e*Math.cos(n)]}var r=Math.sin(n),u=(r+Math.sin(t))/2,i=1+r*(2*u-r),o=Math.sqrt(i)/u;return e.invert=function(n,t){var e=o-t;return[Math.atan2(n,e)/u,G((i-(n*n+e*e)*u*u)/(2*u))]},e}function Oe(){function n(n,t){Ac+=u*n-r*t,r=n,u=t}var t,e,r,u;Tc.point=function(i,o){Tc.point=n,t=r=i,e=u=o},Tc.lineEnd=function(){n(t,e)}}function Ye(n,t){Cc>n&&(Cc=n),n>zc&&(zc=n),Nc>t&&(Nc=t),t>Lc&&(Lc=t)}function Ie(){function n(n,t){o.push("M",n,",",t,i)}function t(n,t){o.push("M",n,",",t),a.point=e}function e(n,t){o.push("L",n,",",t)}function r(){a.point=n}function u(){o.push("Z")}var i=Ze(4.5),o=[],a={point:n,lineStart:function(){a.point=t},lineEnd:r,polygonStart:function(){a.lineEnd=u},polygonEnd:function(){a.lineEnd=r,a.point=n},pointRadius:function(n){return i=Ze(n),a},result:function(){if(o.length){var n=o.join("");return o=[],n}}};return a}function Ze(n){return"m0,"+n+"a"+n+","+n+" 0 1,1 0,"+-2*n+"a"+n+","+n+" 0 1,1 0,"+2*n+"z"}function Ve(n,t){pc+=n,vc+=t,++dc}function Xe(){function n(n,r){var u=n-t,i=r-e,o=Math.sqrt(u*u+i*i);mc+=o*(t+n)/2,yc+=o*(e+r)/2,xc+=o,Ve(t=n,e=r)}var t,e;Rc.point=function(r,u){Rc.point=n,Ve(t=r,e=u)}}function $e(){Rc.point=Ve}function Be(){function n(n,t){var e=n-r,i=t-u,o=Math.sqrt(e*e+i*i);mc+=o*(r+n)/2,yc+=o*(u+t)/2,xc+=o,o=u*n-r*t,Mc+=o*(r+n),_c+=o*(u+t),bc+=3*o,Ve(r=n,u=t)}var t,e,r,u;Rc.point=function(i,o){Rc.point=n,Ve(t=r=i,e=u=o)},Rc.lineEnd=function(){n(t,e)}}function We(n){function t(t,e){n.moveTo(t,e),n.arc(t,e,o,0,wa)}function e(t,e){n.moveTo(t,e),a.point=r}function r(t,e){n.lineTo(t,e)}function u(){a.point=t}function i(){n.closePath()}var o=4.5,a={point:t,lineStart:function(){a.point=e},lineEnd:u,polygonStart:function(){a.lineEnd=i},polygonEnd:function(){a.lineEnd=u,a.point=t},pointRadius:function(n){return o=n,a},result:v};return a}function Je(n){function t(n){return(a?r:e)(n)}function e(t){return Qe(t,function(e,r){e=n(e,r),t.point(e[0],e[1])})}function r(t){function e(e,r){e=n(e,r),t.point(e[0],e[1])}function r(){x=0/0,S.point=i,t.lineStart()}function i(e,r){var i=le([e,r]),o=n(e,r);u(x,M,y,_,b,w,x=o[0],M=o[1],y=e,_=i[0],b=i[1],w=i[2],a,t),t.point(x,M)}function o(){S.point=e,t.lineEnd()}function c(){r(),S.point=s,S.lineEnd=l}function s(n,t){i(f=n,h=t),g=x,p=M,v=_,d=b,m=w,S.point=i}function l(){u(x,M,y,_,b,w,g,p,f,v,d,m,a,t),S.lineEnd=o,o()}var f,h,g,p,v,d,m,y,x,M,_,b,w,S={point:e,lineStart:r,lineEnd:o,polygonStart:function(){t.polygonStart(),S.lineStart=c},polygonEnd:function(){t.polygonEnd(),S.lineStart=r}};return S}function u(t,e,r,a,c,s,l,f,h,g,p,v,d,m){var y=l-t,x=f-e,M=y*y+x*x;if(M>4*i&&d--){var _=a+g,b=c+p,w=s+v,S=Math.sqrt(_*_+b*b+w*w),k=Math.asin(w/=S),E=ua(ua(w)-1)<ka||ua(r-h)<ka?(r+h)/2:Math.atan2(b,_),A=n(E,k),C=A[0],N=A[1],z=C-t,L=N-e,T=x*z-y*L;(T*T/M>i||ua((y*z+x*L)/M-.5)>.3||o>a*g+c*p+s*v)&&(u(t,e,r,a,c,s,C,N,E,_/=S,b/=S,w,d,m),m.point(C,N),u(C,N,E,_,b,w,l,f,h,g,p,v,d,m))}}var i=.5,o=Math.cos(30*Aa),a=16;
+return t.precision=function(n){return arguments.length?(a=(i=n*n)>0&&16,t):Math.sqrt(i)},t}function Ge(n){var t=Je(function(t,e){return n([t*Ca,e*Ca])});return function(n){return er(t(n))}}function Ke(n){this.stream=n}function Qe(n,t){return{point:t,sphere:function(){n.sphere()},lineStart:function(){n.lineStart()},lineEnd:function(){n.lineEnd()},polygonStart:function(){n.polygonStart()},polygonEnd:function(){n.polygonEnd()}}}function nr(n){return tr(function(){return n})()}function tr(n){function t(n){return n=a(n[0]*Aa,n[1]*Aa),[n[0]*h+c,s-n[1]*h]}function e(n){return n=a.invert((n[0]-c)/h,(s-n[1])/h),n&&[n[0]*Ca,n[1]*Ca]}function r(){a=je(o=ir(m,y,x),i);var n=i(v,d);return c=g-n[0]*h,s=p+n[1]*h,u()}function u(){return l&&(l.valid=!1,l=null),t}var i,o,a,c,s,l,f=Je(function(n,t){return n=i(n,t),[n[0]*h+c,s-n[1]*h]}),h=150,g=480,p=250,v=0,d=0,m=0,y=0,x=0,M=Sc,_=wt,b=null,w=null;return t.stream=function(n){return l&&(l.valid=!1),l=er(M(o,f(_(n)))),l.valid=!0,l},t.clipAngle=function(n){return arguments.length?(M=null==n?(b=n,Sc):De((b=+n)*Aa),u()):b},t.clipExtent=function(n){return arguments.length?(w=n,_=n?Ue(n[0][0],n[0][1],n[1][0],n[1][1]):wt,u()):w},t.scale=function(n){return arguments.length?(h=+n,r()):h},t.translate=function(n){return arguments.length?(g=+n[0],p=+n[1],r()):[g,p]},t.center=function(n){return arguments.length?(v=n[0]%360*Aa,d=n[1]%360*Aa,r()):[v*Ca,d*Ca]},t.rotate=function(n){return arguments.length?(m=n[0]%360*Aa,y=n[1]%360*Aa,x=n.length>2?n[2]%360*Aa:0,r()):[m*Ca,y*Ca,x*Ca]},Zo.rebind(t,f,"precision"),function(){return i=n.apply(this,arguments),t.invert=i.invert&&e,r()}}function er(n){return Qe(n,function(t,e){n.point(t*Aa,e*Aa)})}function rr(n,t){return[n,t]}function ur(n,t){return[n>ba?n-wa:-ba>n?n+wa:n,t]}function ir(n,t,e){return n?t||e?je(ar(n),cr(t,e)):ar(n):t||e?cr(t,e):ur}function or(n){return function(t,e){return t+=n,[t>ba?t-wa:-ba>t?t+wa:t,e]}}function ar(n){var t=or(n);return t.invert=or(-n),t}function cr(n,t){function e(n,t){var e=Math.cos(t),a=Math.cos(n)*e,c=Math.sin(n)*e,s=Math.sin(t),l=s*r+a*u;return[Math.atan2(c*i-l*o,a*r-s*u),G(l*i+c*o)]}var r=Math.cos(n),u=Math.sin(n),i=Math.cos(t),o=Math.sin(t);return e.invert=function(n,t){var e=Math.cos(t),a=Math.cos(n)*e,c=Math.sin(n)*e,s=Math.sin(t),l=s*i-c*o;return[Math.atan2(c*i+s*o,a*r+l*u),G(l*r-a*u)]},e}function sr(n,t){var e=Math.cos(n),r=Math.sin(n);return function(u,i,o,a){var c=o*t;null!=u?(u=lr(e,u),i=lr(e,i),(o>0?i>u:u>i)&&(u+=o*wa)):(u=n+o*wa,i=n-.5*c);for(var s,l=u;o>0?l>i:i>l;l-=c)a.point((s=de([e,-r*Math.cos(l),-r*Math.sin(l)]))[0],s[1])}}function lr(n,t){var e=le(t);e[0]-=n,ve(e);var r=J(-e[1]);return((-e[2]<0?-r:r)+2*Math.PI-ka)%(2*Math.PI)}function fr(n,t,e){var r=Zo.range(n,t-ka,e).concat(t);return function(n){return r.map(function(t){return[n,t]})}}function hr(n,t,e){var r=Zo.range(n,t-ka,e).concat(t);return function(n){return r.map(function(t){return[t,n]})}}function gr(n){return n.source}function pr(n){return n.target}function vr(n,t,e,r){var u=Math.cos(t),i=Math.sin(t),o=Math.cos(r),a=Math.sin(r),c=u*Math.cos(n),s=u*Math.sin(n),l=o*Math.cos(e),f=o*Math.sin(e),h=2*Math.asin(Math.sqrt(tt(r-t)+u*o*tt(e-n))),g=1/Math.sin(h),p=h?function(n){var t=Math.sin(n*=h)*g,e=Math.sin(h-n)*g,r=e*c+t*l,u=e*s+t*f,o=e*i+t*a;return[Math.atan2(u,r)*Ca,Math.atan2(o,Math.sqrt(r*r+u*u))*Ca]}:function(){return[n*Ca,t*Ca]};return p.distance=h,p}function dr(){function n(n,u){var i=Math.sin(u*=Aa),o=Math.cos(u),a=ua((n*=Aa)-t),c=Math.cos(a);Dc+=Math.atan2(Math.sqrt((a=o*Math.sin(a))*a+(a=r*i-e*o*c)*a),e*i+r*o*c),t=n,e=i,r=o}var t,e,r;Pc.point=function(u,i){t=u*Aa,e=Math.sin(i*=Aa),r=Math.cos(i),Pc.point=n},Pc.lineEnd=function(){Pc.point=Pc.lineEnd=v}}function mr(n,t){function e(t,e){var r=Math.cos(t),u=Math.cos(e),i=n(r*u);return[i*u*Math.sin(t),i*Math.sin(e)]}return e.invert=function(n,e){var r=Math.sqrt(n*n+e*e),u=t(r),i=Math.sin(u),o=Math.cos(u);return[Math.atan2(n*i,r*o),Math.asin(r&&e*i/r)]},e}function yr(n,t){function e(n,t){o>0?-Sa+ka>t&&(t=-Sa+ka):t>Sa-ka&&(t=Sa-ka);var e=o/Math.pow(u(t),i);return[e*Math.sin(i*n),o-e*Math.cos(i*n)]}var r=Math.cos(n),u=function(n){return Math.tan(ba/4+n/2)},i=n===t?Math.sin(n):Math.log(r/Math.cos(t))/Math.log(u(t)/u(n)),o=r*Math.pow(u(n),i)/i;return i?(e.invert=function(n,t){var e=o-t,r=B(i)*Math.sqrt(n*n+e*e);return[Math.atan2(n,e)/i,2*Math.atan(Math.pow(o/r,1/i))-Sa]},e):Mr}function xr(n,t){function e(n,t){var e=i-t;return[e*Math.sin(u*n),i-e*Math.cos(u*n)]}var r=Math.cos(n),u=n===t?Math.sin(n):(r-Math.cos(t))/(t-n),i=r/u+n;return ua(u)<ka?rr:(e.invert=function(n,t){var e=i-t;return[Math.atan2(n,e)/u,i-B(u)*Math.sqrt(n*n+e*e)]},e)}function Mr(n,t){return[n,Math.log(Math.tan(ba/4+t/2))]}function _r(n){var t,e=nr(n),r=e.scale,u=e.translate,i=e.clipExtent;return e.scale=function(){var n=r.apply(e,arguments);return n===e?t?e.clipExtent(null):e:n},e.translate=function(){var n=u.apply(e,arguments);return n===e?t?e.clipExtent(null):e:n},e.clipExtent=function(n){var o=i.apply(e,arguments);if(o===e){if(t=null==n){var a=ba*r(),c=u();i([[c[0]-a,c[1]-a],[c[0]+a,c[1]+a]])}}else t&&(o=null);return o},e.clipExtent(null)}function br(n,t){return[Math.log(Math.tan(ba/4+t/2)),-n]}function wr(n){return n[0]}function Sr(n){return n[1]}function kr(n){for(var t=n.length,e=[0,1],r=2,u=2;t>u;u++){for(;r>1&&W(n[e[r-2]],n[e[r-1]],n[u])<=0;)--r;e[r++]=u}return e.slice(0,r)}function Er(n,t){return n[0]-t[0]||n[1]-t[1]}function Ar(n,t,e){return(e[0]-t[0])*(n[1]-t[1])<(e[1]-t[1])*(n[0]-t[0])}function Cr(n,t,e,r){var u=n[0],i=e[0],o=t[0]-u,a=r[0]-i,c=n[1],s=e[1],l=t[1]-c,f=r[1]-s,h=(a*(c-s)-f*(u-i))/(f*o-a*l);return[u+h*o,c+h*l]}function Nr(n){var t=n[0],e=n[n.length-1];return!(t[0]-e[0]||t[1]-e[1])}function zr(){Gr(this),this.edge=this.site=this.circle=null}function Lr(n){var t=Bc.pop()||new zr;return t.site=n,t}function Tr(n){Yr(n),Vc.remove(n),Bc.push(n),Gr(n)}function qr(n){var t=n.circle,e=t.x,r=t.cy,u={x:e,y:r},i=n.P,o=n.N,a=[n];Tr(n);for(var c=i;c.circle&&ua(e-c.circle.x)<ka&&ua(r-c.circle.cy)<ka;)i=c.P,a.unshift(c),Tr(c),c=i;a.unshift(c),Yr(c);for(var s=o;s.circle&&ua(e-s.circle.x)<ka&&ua(r-s.circle.cy)<ka;)o=s.N,a.push(s),Tr(s),s=o;a.push(s),Yr(s);var l,f=a.length;for(l=1;f>l;++l)s=a[l],c=a[l-1],Br(s.edge,c.site,s.site,u);c=a[0],s=a[f-1],s.edge=Xr(c.site,s.site,null,u),Or(c),Or(s)}function Rr(n){for(var t,e,r,u,i=n.x,o=n.y,a=Vc._;a;)if(r=Dr(a,o)-i,r>ka)a=a.L;else{if(u=i-Pr(a,o),!(u>ka)){r>-ka?(t=a.P,e=a):u>-ka?(t=a,e=a.N):t=e=a;break}if(!a.R){t=a;break}a=a.R}var c=Lr(n);if(Vc.insert(t,c),t||e){if(t===e)return Yr(t),e=Lr(t.site),Vc.insert(c,e),c.edge=e.edge=Xr(t.site,c.site),Or(t),Or(e),void 0;if(!e)return c.edge=Xr(t.site,c.site),void 0;Yr(t),Yr(e);var s=t.site,l=s.x,f=s.y,h=n.x-l,g=n.y-f,p=e.site,v=p.x-l,d=p.y-f,m=2*(h*d-g*v),y=h*h+g*g,x=v*v+d*d,M={x:(d*y-g*x)/m+l,y:(h*x-v*y)/m+f};Br(e.edge,s,p,M),c.edge=Xr(s,n,null,M),e.edge=Xr(n,p,null,M),Or(t),Or(e)}}function Dr(n,t){var e=n.site,r=e.x,u=e.y,i=u-t;if(!i)return r;var o=n.P;if(!o)return-1/0;e=o.site;var a=e.x,c=e.y,s=c-t;if(!s)return a;var l=a-r,f=1/i-1/s,h=l/s;return f?(-h+Math.sqrt(h*h-2*f*(l*l/(-2*s)-c+s/2+u-i/2)))/f+r:(r+a)/2}function Pr(n,t){var e=n.N;if(e)return Dr(e,t);var r=n.site;return r.y===t?r.x:1/0}function Ur(n){this.site=n,this.edges=[]}function jr(n){for(var t,e,r,u,i,o,a,c,s,l,f=n[0][0],h=n[1][0],g=n[0][1],p=n[1][1],v=Zc,d=v.length;d--;)if(i=v[d],i&&i.prepare())for(a=i.edges,c=a.length,o=0;c>o;)l=a[o].end(),r=l.x,u=l.y,s=a[++o%c].start(),t=s.x,e=s.y,(ua(r-t)>ka||ua(u-e)>ka)&&(a.splice(o,0,new Wr($r(i.site,l,ua(r-f)<ka&&p-u>ka?{x:f,y:ua(t-f)<ka?e:p}:ua(u-p)<ka&&h-r>ka?{x:ua(e-p)<ka?t:h,y:p}:ua(r-h)<ka&&u-g>ka?{x:h,y:ua(t-h)<ka?e:g}:ua(u-g)<ka&&r-f>ka?{x:ua(e-g)<ka?t:f,y:g}:null),i.site,null)),++c)}function Hr(n,t){return t.angle-n.angle}function Fr(){Gr(this),this.x=this.y=this.arc=this.site=this.cy=null}function Or(n){var t=n.P,e=n.N;if(t&&e){var r=t.site,u=n.site,i=e.site;if(r!==i){var o=u.x,a=u.y,c=r.x-o,s=r.y-a,l=i.x-o,f=i.y-a,h=2*(c*f-s*l);if(!(h>=-Ea)){var g=c*c+s*s,p=l*l+f*f,v=(f*g-s*p)/h,d=(c*p-l*g)/h,f=d+a,m=Wc.pop()||new Fr;m.arc=n,m.site=u,m.x=v+o,m.y=f+Math.sqrt(v*v+d*d),m.cy=f,n.circle=m;for(var y=null,x=$c._;x;)if(m.y<x.y||m.y===x.y&&m.x<=x.x){if(!x.L){y=x.P;break}x=x.L}else{if(!x.R){y=x;break}x=x.R}$c.insert(y,m),y||(Xc=m)}}}}function Yr(n){var t=n.circle;t&&(t.P||(Xc=t.N),$c.remove(t),Wc.push(t),Gr(t),n.circle=null)}function Ir(n){for(var t,e=Ic,r=Pe(n[0][0],n[0][1],n[1][0],n[1][1]),u=e.length;u--;)t=e[u],(!Zr(t,n)||!r(t)||ua(t.a.x-t.b.x)<ka&&ua(t.a.y-t.b.y)<ka)&&(t.a=t.b=null,e.splice(u,1))}function Zr(n,t){var e=n.b;if(e)return!0;var r,u,i=n.a,o=t[0][0],a=t[1][0],c=t[0][1],s=t[1][1],l=n.l,f=n.r,h=l.x,g=l.y,p=f.x,v=f.y,d=(h+p)/2,m=(g+v)/2;if(v===g){if(o>d||d>=a)return;if(h>p){if(i){if(i.y>=s)return}else i={x:d,y:c};e={x:d,y:s}}else{if(i){if(i.y<c)return}else i={x:d,y:s};e={x:d,y:c}}}else if(r=(h-p)/(v-g),u=m-r*d,-1>r||r>1)if(h>p){if(i){if(i.y>=s)return}else i={x:(c-u)/r,y:c};e={x:(s-u)/r,y:s}}else{if(i){if(i.y<c)return}else i={x:(s-u)/r,y:s};e={x:(c-u)/r,y:c}}else if(v>g){if(i){if(i.x>=a)return}else i={x:o,y:r*o+u};e={x:a,y:r*a+u}}else{if(i){if(i.x<o)return}else i={x:a,y:r*a+u};e={x:o,y:r*o+u}}return n.a=i,n.b=e,!0}function Vr(n,t){this.l=n,this.r=t,this.a=this.b=null}function Xr(n,t,e,r){var u=new Vr(n,t);return Ic.push(u),e&&Br(u,n,t,e),r&&Br(u,t,n,r),Zc[n.i].edges.push(new Wr(u,n,t)),Zc[t.i].edges.push(new Wr(u,t,n)),u}function $r(n,t,e){var r=new Vr(n,null);return r.a=t,r.b=e,Ic.push(r),r}function Br(n,t,e,r){n.a||n.b?n.l===e?n.b=r:n.a=r:(n.a=r,n.l=t,n.r=e)}function Wr(n,t,e){var r=n.a,u=n.b;this.edge=n,this.site=t,this.angle=e?Math.atan2(e.y-t.y,e.x-t.x):n.l===t?Math.atan2(u.x-r.x,r.y-u.y):Math.atan2(r.x-u.x,u.y-r.y)}function Jr(){this._=null}function Gr(n){n.U=n.C=n.L=n.R=n.P=n.N=null}function Kr(n,t){var e=t,r=t.R,u=e.U;u?u.L===e?u.L=r:u.R=r:n._=r,r.U=u,e.U=r,e.R=r.L,e.R&&(e.R.U=e),r.L=e}function Qr(n,t){var e=t,r=t.L,u=e.U;u?u.L===e?u.L=r:u.R=r:n._=r,r.U=u,e.U=r,e.L=r.R,e.L&&(e.L.U=e),r.R=e}function nu(n){for(;n.L;)n=n.L;return n}function tu(n,t){var e,r,u,i=n.sort(eu).pop();for(Ic=[],Zc=new Array(n.length),Vc=new Jr,$c=new Jr;;)if(u=Xc,i&&(!u||i.y<u.y||i.y===u.y&&i.x<u.x))(i.x!==e||i.y!==r)&&(Zc[i.i]=new Ur(i),Rr(i),e=i.x,r=i.y),i=n.pop();else{if(!u)break;qr(u.arc)}t&&(Ir(t),jr(t));var o={cells:Zc,edges:Ic};return Vc=$c=Ic=Zc=null,o}function eu(n,t){return t.y-n.y||t.x-n.x}function ru(n,t,e){return(n.x-e.x)*(t.y-n.y)-(n.x-t.x)*(e.y-n.y)}function uu(n){return n.x}function iu(n){return n.y}function ou(){return{leaf:!0,nodes:[],point:null,x:null,y:null}}function au(n,t,e,r,u,i){if(!n(t,e,r,u,i)){var o=.5*(e+u),a=.5*(r+i),c=t.nodes;c[0]&&au(n,c[0],e,r,o,a),c[1]&&au(n,c[1],o,r,u,a),c[2]&&au(n,c[2],e,a,o,i),c[3]&&au(n,c[3],o,a,u,i)}}function cu(n,t){n=Zo.rgb(n),t=Zo.rgb(t);var e=n.r,r=n.g,u=n.b,i=t.r-e,o=t.g-r,a=t.b-u;return function(n){return"#"+dt(Math.round(e+i*n))+dt(Math.round(r+o*n))+dt(Math.round(u+a*n))}}function su(n,t){var e,r={},u={};for(e in n)e in t?r[e]=hu(n[e],t[e]):u[e]=n[e];for(e in t)e in n||(u[e]=t[e]);return function(n){for(e in r)u[e]=r[e](n);return u}}function lu(n,t){return t-=n=+n,function(e){return n+t*e}}function fu(n,t){var e,r,u,i=Gc.lastIndex=Kc.lastIndex=0,o=-1,a=[],c=[];for(n+="",t+="";(e=Gc.exec(n))&&(r=Kc.exec(t));)(u=r.index)>i&&(u=t.substring(i,u),a[o]?a[o]+=u:a[++o]=u),(e=e[0])===(r=r[0])?a[o]?a[o]+=r:a[++o]=r:(a[++o]=null,c.push({i:o,x:lu(e,r)})),i=Kc.lastIndex;return i<t.length&&(u=t.substring(i),a[o]?a[o]+=u:a[++o]=u),a.length<2?c[0]?(t=c[0].x,function(n){return t(n)+""}):function(){return t}:(t=c.length,function(n){for(var e,r=0;t>r;++r)a[(e=c[r]).i]=e.x(n);return a.join("")})}function hu(n,t){for(var e,r=Zo.interpolators.length;--r>=0&&!(e=Zo.interpolators[r](n,t)););return e}function gu(n,t){var e,r=[],u=[],i=n.length,o=t.length,a=Math.min(n.length,t.length);for(e=0;a>e;++e)r.push(hu(n[e],t[e]));for(;i>e;++e)u[e]=n[e];for(;o>e;++e)u[e]=t[e];return function(n){for(e=0;a>e;++e)u[e]=r[e](n);return u}}function pu(n){return function(t){return 0>=t?0:t>=1?1:n(t)}}function vu(n){return function(t){return 1-n(1-t)}}function du(n){return function(t){return.5*(.5>t?n(2*t):2-n(2-2*t))}}function mu(n){return n*n}function yu(n){return n*n*n}function xu(n){if(0>=n)return 0;if(n>=1)return 1;var t=n*n,e=t*n;return 4*(.5>n?e:3*(n-t)+e-.75)}function Mu(n){return function(t){return Math.pow(t,n)}}function _u(n){return 1-Math.cos(n*Sa)}function bu(n){return Math.pow(2,10*(n-1))}function wu(n){return 1-Math.sqrt(1-n*n)}function Su(n,t){var e;return arguments.length<2&&(t=.45),arguments.length?e=t/wa*Math.asin(1/n):(n=1,e=t/4),function(r){return 1+n*Math.pow(2,-10*r)*Math.sin((r-e)*wa/t)}}function ku(n){return n||(n=1.70158),function(t){return t*t*((n+1)*t-n)}}function Eu(n){return 1/2.75>n?7.5625*n*n:2/2.75>n?7.5625*(n-=1.5/2.75)*n+.75:2.5/2.75>n?7.5625*(n-=2.25/2.75)*n+.9375:7.5625*(n-=2.625/2.75)*n+.984375}function Au(n,t){n=Zo.hcl(n),t=Zo.hcl(t);var e=n.h,r=n.c,u=n.l,i=t.h-e,o=t.c-r,a=t.l-u;return isNaN(o)&&(o=0,r=isNaN(r)?t.c:r),isNaN(i)?(i=0,e=isNaN(e)?t.h:e):i>180?i-=360:-180>i&&(i+=360),function(n){return ot(e+i*n,r+o*n,u+a*n)+""}}function Cu(n,t){n=Zo.hsl(n),t=Zo.hsl(t);var e=n.h,r=n.s,u=n.l,i=t.h-e,o=t.s-r,a=t.l-u;return isNaN(o)&&(o=0,r=isNaN(r)?t.s:r),isNaN(i)?(i=0,e=isNaN(e)?t.h:e):i>180?i-=360:-180>i&&(i+=360),function(n){return ut(e+i*n,r+o*n,u+a*n)+""}}function Nu(n,t){n=Zo.lab(n),t=Zo.lab(t);var e=n.l,r=n.a,u=n.b,i=t.l-e,o=t.a-r,a=t.b-u;return function(n){return ct(e+i*n,r+o*n,u+a*n)+""}}function zu(n,t){return t-=n,function(e){return Math.round(n+t*e)}}function Lu(n){var t=[n.a,n.b],e=[n.c,n.d],r=qu(t),u=Tu(t,e),i=qu(Ru(e,t,-u))||0;t[0]*e[1]<e[0]*t[1]&&(t[0]*=-1,t[1]*=-1,r*=-1,u*=-1),this.rotate=(r?Math.atan2(t[1],t[0]):Math.atan2(-e[0],e[1]))*Ca,this.translate=[n.e,n.f],this.scale=[r,i],this.skew=i?Math.atan2(u,i)*Ca:0}function Tu(n,t){return n[0]*t[0]+n[1]*t[1]}function qu(n){var t=Math.sqrt(Tu(n,n));return t&&(n[0]/=t,n[1]/=t),t}function Ru(n,t,e){return n[0]+=e*t[0],n[1]+=e*t[1],n}function Du(n,t){var e,r=[],u=[],i=Zo.transform(n),o=Zo.transform(t),a=i.translate,c=o.translate,s=i.rotate,l=o.rotate,f=i.skew,h=o.skew,g=i.scale,p=o.scale;return a[0]!=c[0]||a[1]!=c[1]?(r.push("translate(",null,",",null,")"),u.push({i:1,x:lu(a[0],c[0])},{i:3,x:lu(a[1],c[1])})):c[0]||c[1]?r.push("translate("+c+")"):r.push(""),s!=l?(s-l>180?l+=360:l-s>180&&(s+=360),u.push({i:r.push(r.pop()+"rotate(",null,")")-2,x:lu(s,l)})):l&&r.push(r.pop()+"rotate("+l+")"),f!=h?u.push({i:r.push(r.pop()+"skewX(",null,")")-2,x:lu(f,h)}):h&&r.push(r.pop()+"skewX("+h+")"),g[0]!=p[0]||g[1]!=p[1]?(e=r.push(r.pop()+"scale(",null,",",null,")"),u.push({i:e-4,x:lu(g[0],p[0])},{i:e-2,x:lu(g[1],p[1])})):(1!=p[0]||1!=p[1])&&r.push(r.pop()+"scale("+p+")"),e=u.length,function(n){for(var t,i=-1;++i<e;)r[(t=u[i]).i]=t.x(n);return r.join("")}}function Pu(n,t){return t=t-(n=+n)?1/(t-n):0,function(e){return(e-n)*t}}function Uu(n,t){return t=t-(n=+n)?1/(t-n):0,function(e){return Math.max(0,Math.min(1,(e-n)*t))}}function ju(n){for(var t=n.source,e=n.target,r=Fu(t,e),u=[t];t!==r;)t=t.parent,u.push(t);for(var i=u.length;e!==r;)u.splice(i,0,e),e=e.parent;return u}function Hu(n){for(var t=[],e=n.parent;null!=e;)t.push(n),n=e,e=e.parent;return t.push(n),t}function Fu(n,t){if(n===t)return n;for(var e=Hu(n),r=Hu(t),u=e.pop(),i=r.pop(),o=null;u===i;)o=u,u=e.pop(),i=r.pop();return o}function Ou(n){n.fixed|=2}function Yu(n){n.fixed&=-7}function Iu(n){n.fixed|=4,n.px=n.x,n.py=n.y}function Zu(n){n.fixed&=-5}function Vu(n,t,e){var r=0,u=0;if(n.charge=0,!n.leaf)for(var i,o=n.nodes,a=o.length,c=-1;++c<a;)i=o[c],null!=i&&(Vu(i,t,e),n.charge+=i.charge,r+=i.charge*i.cx,u+=i.charge*i.cy);if(n.point){n.leaf||(n.point.x+=Math.random()-.5,n.point.y+=Math.random()-.5);var s=t*e[n.point.index];n.charge+=n.pointCharge=s,r+=s*n.point.x,u+=s*n.point.y}n.cx=r/n.charge,n.cy=u/n.charge}function Xu(n,t){return Zo.rebind(n,t,"sort","children","value"),n.nodes=n,n.links=Ku,n}function $u(n,t){for(var e=[n];null!=(n=e.pop());)if(t(n),(u=n.children)&&(r=u.length))for(var r,u;--r>=0;)e.push(u[r])}function Bu(n,t){for(var e=[n],r=[];null!=(n=e.pop());)if(r.push(n),(i=n.children)&&(u=i.length))for(var u,i,o=-1;++o<u;)e.push(i[o]);for(;null!=(n=r.pop());)t(n)}function Wu(n){return n.children}function Ju(n){return n.value}function Gu(n,t){return t.value-n.value}function Ku(n){return Zo.merge(n.map(function(n){return(n.children||[]).map(function(t){return{source:n,target:t}})}))}function Qu(n){return n.x}function ni(n){return n.y}function ti(n,t,e){n.y0=t,n.y=e}function ei(n){return Zo.range(n.length)}function ri(n){for(var t=-1,e=n[0].length,r=[];++t<e;)r[t]=0;return r}function ui(n){for(var t,e=1,r=0,u=n[0][1],i=n.length;i>e;++e)(t=n[e][1])>u&&(r=e,u=t);return r}function ii(n){return n.reduce(oi,0)}function oi(n,t){return n+t[1]}function ai(n,t){return ci(n,Math.ceil(Math.log(t.length)/Math.LN2+1))}function ci(n,t){for(var e=-1,r=+n[0],u=(n[1]-r)/t,i=[];++e<=t;)i[e]=u*e+r;return i}function si(n){return[Zo.min(n),Zo.max(n)]}function li(n,t){return n.value-t.value}function fi(n,t){var e=n._pack_next;n._pack_next=t,t._pack_prev=n,t._pack_next=e,e._pack_prev=t}function hi(n,t){n._pack_next=t,t._pack_prev=n}function gi(n,t){var e=t.x-n.x,r=t.y-n.y,u=n.r+t.r;return.999*u*u>e*e+r*r}function pi(n){function t(n){l=Math.min(n.x-n.r,l),f=Math.max(n.x+n.r,f),h=Math.min(n.y-n.r,h),g=Math.max(n.y+n.r,g)}if((e=n.children)&&(s=e.length)){var e,r,u,i,o,a,c,s,l=1/0,f=-1/0,h=1/0,g=-1/0;if(e.forEach(vi),r=e[0],r.x=-r.r,r.y=0,t(r),s>1&&(u=e[1],u.x=u.r,u.y=0,t(u),s>2))for(i=e[2],yi(r,u,i),t(i),fi(r,i),r._pack_prev=i,fi(i,u),u=r._pack_next,o=3;s>o;o++){yi(r,u,i=e[o]);var p=0,v=1,d=1;for(a=u._pack_next;a!==u;a=a._pack_next,v++)if(gi(a,i)){p=1;break}if(1==p)for(c=r._pack_prev;c!==a._pack_prev&&!gi(c,i);c=c._pack_prev,d++);p?(d>v||v==d&&u.r<r.r?hi(r,u=a):hi(r=c,u),o--):(fi(r,i),u=i,t(i))}var m=(l+f)/2,y=(h+g)/2,x=0;for(o=0;s>o;o++)i=e[o],i.x-=m,i.y-=y,x=Math.max(x,i.r+Math.sqrt(i.x*i.x+i.y*i.y));n.r=x,e.forEach(di)}}function vi(n){n._pack_next=n._pack_prev=n}function di(n){delete n._pack_next,delete n._pack_prev}function mi(n,t,e,r){var u=n.children;if(n.x=t+=r*n.x,n.y=e+=r*n.y,n.r*=r,u)for(var i=-1,o=u.length;++i<o;)mi(u[i],t,e,r)}function yi(n,t,e){var r=n.r+e.r,u=t.x-n.x,i=t.y-n.y;if(r&&(u||i)){var o=t.r+e.r,a=u*u+i*i;o*=o,r*=r;var c=.5+(r-o)/(2*a),s=Math.sqrt(Math.max(0,2*o*(r+a)-(r-=a)*r-o*o))/(2*a);e.x=n.x+c*u+s*i,e.y=n.y+c*i-s*u}else e.x=n.x+r,e.y=n.y}function xi(n,t){return n.parent==t.parent?1:2}function Mi(n){var t=n.children;return t.length?t[0]:n.t}function _i(n){var t,e=n.children;return(t=e.length)?e[t-1]:n.t}function bi(n,t,e){var r=e/(t.i-n.i);t.c-=r,t.s+=e,n.c+=r,t.z+=e,t.m+=e}function wi(n){for(var t,e=0,r=0,u=n.children,i=u.length;--i>=0;)t=u[i],t.z+=e,t.m+=e,e+=t.s+(r+=t.c)}function Si(n,t,e){return n.a.parent===t.parent?n.a:e}function ki(n){return 1+Zo.max(n,function(n){return n.y})}function Ei(n){return n.reduce(function(n,t){return n+t.x},0)/n.length}function Ai(n){var t=n.children;return t&&t.length?Ai(t[0]):n}function Ci(n){var t,e=n.children;return e&&(t=e.length)?Ci(e[t-1]):n}function Ni(n){return{x:n.x,y:n.y,dx:n.dx,dy:n.dy}}function zi(n,t){var e=n.x+t[3],r=n.y+t[0],u=n.dx-t[1]-t[3],i=n.dy-t[0]-t[2];return 0>u&&(e+=u/2,u=0),0>i&&(r+=i/2,i=0),{x:e,y:r,dx:u,dy:i}}function Li(n){var t=n[0],e=n[n.length-1];return e>t?[t,e]:[e,t]}function Ti(n){return n.rangeExtent?n.rangeExtent():Li(n.range())}function qi(n,t,e,r){var u=e(n[0],n[1]),i=r(t[0],t[1]);return function(n){return i(u(n))}}function Ri(n,t){var e,r=0,u=n.length-1,i=n[r],o=n[u];return i>o&&(e=r,r=u,u=e,e=i,i=o,o=e),n[r]=t.floor(i),n[u]=t.ceil(o),n}function Di(n){return n?{floor:function(t){return Math.floor(t/n)*n},ceil:function(t){return Math.ceil(t/n)*n}}:ss}function Pi(n,t,e,r){var u=[],i=[],o=0,a=Math.min(n.length,t.length)-1;for(n[a]<n[0]&&(n=n.slice().reverse(),t=t.slice().reverse());++o<=a;)u.push(e(n[o-1],n[o])),i.push(r(t[o-1],t[o]));return function(t){var e=Zo.bisect(n,t,1,a)-1;return i[e](u[e](t))}}function Ui(n,t,e,r){function u(){var u=Math.min(n.length,t.length)>2?Pi:qi,c=r?Uu:Pu;return o=u(n,t,c,e),a=u(t,n,c,hu),i}function i(n){return o(n)}var o,a;return i.invert=function(n){return a(n)},i.domain=function(t){return arguments.length?(n=t.map(Number),u()):n},i.range=function(n){return arguments.length?(t=n,u()):t},i.rangeRound=function(n){return i.range(n).interpolate(zu)},i.clamp=function(n){return arguments.length?(r=n,u()):r},i.interpolate=function(n){return arguments.length?(e=n,u()):e},i.ticks=function(t){return Oi(n,t)},i.tickFormat=function(t,e){return Yi(n,t,e)},i.nice=function(t){return Hi(n,t),u()},i.copy=function(){return Ui(n,t,e,r)},u()}function ji(n,t){return Zo.rebind(n,t,"range","rangeRound","interpolate","clamp")}function Hi(n,t){return Ri(n,Di(Fi(n,t)[2]))}function Fi(n,t){null==t&&(t=10);var e=Li(n),r=e[1]-e[0],u=Math.pow(10,Math.floor(Math.log(r/t)/Math.LN10)),i=t/r*u;return.15>=i?u*=10:.35>=i?u*=5:.75>=i&&(u*=2),e[0]=Math.ceil(e[0]/u)*u,e[1]=Math.floor(e[1]/u)*u+.5*u,e[2]=u,e}function Oi(n,t){return Zo.range.apply(Zo,Fi(n,t))}function Yi(n,t,e){var r=Fi(n,t);if(e){var u=Ga.exec(e);if(u.shift(),"s"===u[8]){var i=Zo.formatPrefix(Math.max(ua(r[0]),ua(r[1])));return u[7]||(u[7]="."+Ii(i.scale(r[2]))),u[8]="f",e=Zo.format(u.join("")),function(n){return e(i.scale(n))+i.symbol}}u[7]||(u[7]="."+Zi(u[8],r)),e=u.join("")}else e=",."+Ii(r[2])+"f";return Zo.format(e)}function Ii(n){return-Math.floor(Math.log(n)/Math.LN10+.01)}function Zi(n,t){var e=Ii(t[2]);return n in ls?Math.abs(e-Ii(Math.max(ua(t[0]),ua(t[1]))))+ +("e"!==n):e-2*("%"===n)}function Vi(n,t,e,r){function u(n){return(e?Math.log(0>n?0:n):-Math.log(n>0?0:-n))/Math.log(t)}function i(n){return e?Math.pow(t,n):-Math.pow(t,-n)}function o(t){return n(u(t))}return o.invert=function(t){return i(n.invert(t))},o.domain=function(t){return arguments.length?(e=t[0]>=0,n.domain((r=t.map(Number)).map(u)),o):r},o.base=function(e){return arguments.length?(t=+e,n.domain(r.map(u)),o):t},o.nice=function(){var t=Ri(r.map(u),e?Math:hs);return n.domain(t),r=t.map(i),o},o.ticks=function(){var n=Li(r),o=[],a=n[0],c=n[1],s=Math.floor(u(a)),l=Math.ceil(u(c)),f=t%1?2:t;if(isFinite(l-s)){if(e){for(;l>s;s++)for(var h=1;f>h;h++)o.push(i(s)*h);o.push(i(s))}else for(o.push(i(s));s++<l;)for(var h=f-1;h>0;h--)o.push(i(s)*h);for(s=0;o[s]<a;s++);for(l=o.length;o[l-1]>c;l--);o=o.slice(s,l)}return o},o.tickFormat=function(n,t){if(!arguments.length)return fs;arguments.length<2?t=fs:"function"!=typeof t&&(t=Zo.format(t));var r,a=Math.max(.1,n/o.ticks().length),c=e?(r=1e-12,Math.ceil):(r=-1e-12,Math.floor);return function(n){return n/i(c(u(n)+r))<=a?t(n):""}},o.copy=function(){return Vi(n.copy(),t,e,r)},ji(o,n)}function Xi(n,t,e){function r(t){return n(u(t))}var u=$i(t),i=$i(1/t);return r.invert=function(t){return i(n.invert(t))},r.domain=function(t){return arguments.length?(n.domain((e=t.map(Number)).map(u)),r):e},r.ticks=function(n){return Oi(e,n)},r.tickFormat=function(n,t){return Yi(e,n,t)},r.nice=function(n){return r.domain(Hi(e,n))},r.exponent=function(o){return arguments.length?(u=$i(t=o),i=$i(1/t),n.domain(e.map(u)),r):t},r.copy=function(){return Xi(n.copy(),t,e)},ji(r,n)}function $i(n){return function(t){return 0>t?-Math.pow(-t,n):Math.pow(t,n)}}function Bi(n,t){function e(e){return i[((u.get(e)||("range"===t.t?u.set(e,n.push(e)):0/0))-1)%i.length]}function r(t,e){return Zo.range(n.length).map(function(n){return t+e*n})}var u,i,a;return e.domain=function(r){if(!arguments.length)return n;n=[],u=new o;for(var i,a=-1,c=r.length;++a<c;)u.has(i=r[a])||u.set(i,n.push(i));return e[t.t].apply(e,t.a)},e.range=function(n){return arguments.length?(i=n,a=0,t={t:"range",a:arguments},e):i},e.rangePoints=function(u,o){arguments.length<2&&(o=0);var c=u[0],s=u[1],l=(s-c)/(Math.max(1,n.length-1)+o);return i=r(n.length<2?(c+s)/2:c+l*o/2,l),a=0,t={t:"rangePoints",a:arguments},e},e.rangeBands=function(u,o,c){arguments.length<2&&(o=0),arguments.length<3&&(c=o);var s=u[1]<u[0],l=u[s-0],f=u[1-s],h=(f-l)/(n.length-o+2*c);return i=r(l+h*c,h),s&&i.reverse(),a=h*(1-o),t={t:"rangeBands",a:arguments},e},e.rangeRoundBands=function(u,o,c){arguments.length<2&&(o=0),arguments.length<3&&(c=o);var s=u[1]<u[0],l=u[s-0],f=u[1-s],h=Math.floor((f-l)/(n.length-o+2*c)),g=f-l-(n.length-o)*h;return i=r(l+Math.round(g/2),h),s&&i.reverse(),a=Math.round(h*(1-o)),t={t:"rangeRoundBands",a:arguments},e},e.rangeBand=function(){return a},e.rangeExtent=function(){return Li(t.a[0])},e.copy=function(){return Bi(n,t)},e.domain(n)}function Wi(e,r){function u(){var n=0,t=r.length;for(o=[];++n<t;)o[n-1]=Zo.quantile(e,n/t);return i}function i(n){return isNaN(n=+n)?void 0:r[Zo.bisect(o,n)]}var o;return i.domain=function(r){return arguments.length?(e=r.filter(t).sort(n),u()):e},i.range=function(n){return arguments.length?(r=n,u()):r},i.quantiles=function(){return o},i.invertExtent=function(n){return n=r.indexOf(n),0>n?[0/0,0/0]:[n>0?o[n-1]:e[0],n<o.length?o[n]:e[e.length-1]]},i.copy=function(){return Wi(e,r)},u()}function Ji(n,t,e){function r(t){return e[Math.max(0,Math.min(o,Math.floor(i*(t-n))))]}function u(){return i=e.length/(t-n),o=e.length-1,r}var i,o;return r.domain=function(e){return arguments.length?(n=+e[0],t=+e[e.length-1],u()):[n,t]},r.range=function(n){return arguments.length?(e=n,u()):e},r.invertExtent=function(t){return t=e.indexOf(t),t=0>t?0/0:t/i+n,[t,t+1/i]},r.copy=function(){return Ji(n,t,e)},u()}function Gi(n,t){function e(e){return e>=e?t[Zo.bisect(n,e)]:void 0}return e.domain=function(t){return arguments.length?(n=t,e):n},e.range=function(n){return arguments.length?(t=n,e):t},e.invertExtent=function(e){return e=t.indexOf(e),[n[e-1],n[e]]},e.copy=function(){return Gi(n,t)},e}function Ki(n){function t(n){return+n}return t.invert=t,t.domain=t.range=function(e){return arguments.length?(n=e.map(t),t):n},t.ticks=function(t){return Oi(n,t)},t.tickFormat=function(t,e){return Yi(n,t,e)},t.copy=function(){return Ki(n)},t}function Qi(n){return n.innerRadius}function no(n){return n.outerRadius}function to(n){return n.startAngle}function eo(n){return n.endAngle}function ro(n){function t(t){function o(){s.push("M",i(n(l),a))}for(var c,s=[],l=[],f=-1,h=t.length,g=bt(e),p=bt(r);++f<h;)u.call(this,c=t[f],f)?l.push([+g.call(this,c,f),+p.call(this,c,f)]):l.length&&(o(),l=[]);return l.length&&o(),s.length?s.join(""):null}var e=wr,r=Sr,u=we,i=uo,o=i.key,a=.7;return t.x=function(n){return arguments.length?(e=n,t):e},t.y=function(n){return arguments.length?(r=n,t):r},t.defined=function(n){return arguments.length?(u=n,t):u},t.interpolate=function(n){return arguments.length?(o="function"==typeof n?i=n:(i=xs.get(n)||uo).key,t):o},t.tension=function(n){return arguments.length?(a=n,t):a},t}function uo(n){return n.join("L")}function io(n){return uo(n)+"Z"}function oo(n){for(var t=0,e=n.length,r=n[0],u=[r[0],",",r[1]];++t<e;)u.push("H",(r[0]+(r=n[t])[0])/2,"V",r[1]);return e>1&&u.push("H",r[0]),u.join("")}function ao(n){for(var t=0,e=n.length,r=n[0],u=[r[0],",",r[1]];++t<e;)u.push("V",(r=n[t])[1],"H",r[0]);return u.join("")}function co(n){for(var t=0,e=n.length,r=n[0],u=[r[0],",",r[1]];++t<e;)u.push("H",(r=n[t])[0],"V",r[1]);return u.join("")}function so(n,t){return n.length<4?uo(n):n[1]+ho(n.slice(1,n.length-1),go(n,t))}function lo(n,t){return n.length<3?uo(n):n[0]+ho((n.push(n[0]),n),go([n[n.length-2]].concat(n,[n[1]]),t))}function fo(n,t){return n.length<3?uo(n):n[0]+ho(n,go(n,t))}function ho(n,t){if(t.length<1||n.length!=t.length&&n.length!=t.length+2)return uo(n);var e=n.length!=t.length,r="",u=n[0],i=n[1],o=t[0],a=o,c=1;if(e&&(r+="Q"+(i[0]-2*o[0]/3)+","+(i[1]-2*o[1]/3)+","+i[0]+","+i[1],u=n[1],c=2),t.length>1){a=t[1],i=n[c],c++,r+="C"+(u[0]+o[0])+","+(u[1]+o[1])+","+(i[0]-a[0])+","+(i[1]-a[1])+","+i[0]+","+i[1];for(var s=2;s<t.length;s++,c++)i=n[c],a=t[s],r+="S"+(i[0]-a[0])+","+(i[1]-a[1])+","+i[0]+","+i[1]}if(e){var l=n[c];r+="Q"+(i[0]+2*a[0]/3)+","+(i[1]+2*a[1]/3)+","+l[0]+","+l[1]}return r}function go(n,t){for(var e,r=[],u=(1-t)/2,i=n[0],o=n[1],a=1,c=n.length;++a<c;)e=i,i=o,o=n[a],r.push([u*(o[0]-e[0]),u*(o[1]-e[1])]);return r}function po(n){if(n.length<3)return uo(n);var t=1,e=n.length,r=n[0],u=r[0],i=r[1],o=[u,u,u,(r=n[1])[0]],a=[i,i,i,r[1]],c=[u,",",i,"L",xo(bs,o),",",xo(bs,a)];for(n.push(n[e-1]);++t<=e;)r=n[t],o.shift(),o.push(r[0]),a.shift(),a.push(r[1]),Mo(c,o,a);return n.pop(),c.push("L",r),c.join("")}function vo(n){if(n.length<4)return uo(n);for(var t,e=[],r=-1,u=n.length,i=[0],o=[0];++r<3;)t=n[r],i.push(t[0]),o.push(t[1]);for(e.push(xo(bs,i)+","+xo(bs,o)),--r;++r<u;)t=n[r],i.shift(),i.push(t[0]),o.shift(),o.push(t[1]),Mo(e,i,o);return e.join("")}function mo(n){for(var t,e,r=-1,u=n.length,i=u+4,o=[],a=[];++r<4;)e=n[r%u],o.push(e[0]),a.push(e[1]);for(t=[xo(bs,o),",",xo(bs,a)],--r;++r<i;)e=n[r%u],o.shift(),o.push(e[0]),a.shift(),a.push(e[1]),Mo(t,o,a);return t.join("")}function yo(n,t){var e=n.length-1;if(e)for(var r,u,i=n[0][0],o=n[0][1],a=n[e][0]-i,c=n[e][1]-o,s=-1;++s<=e;)r=n[s],u=s/e,r[0]=t*r[0]+(1-t)*(i+u*a),r[1]=t*r[1]+(1-t)*(o+u*c);return po(n)}function xo(n,t){return n[0]*t[0]+n[1]*t[1]+n[2]*t[2]+n[3]*t[3]}function Mo(n,t,e){n.push("C",xo(Ms,t),",",xo(Ms,e),",",xo(_s,t),",",xo(_s,e),",",xo(bs,t),",",xo(bs,e))}function _o(n,t){return(t[1]-n[1])/(t[0]-n[0])}function bo(n){for(var t=0,e=n.length-1,r=[],u=n[0],i=n[1],o=r[0]=_o(u,i);++t<e;)r[t]=(o+(o=_o(u=i,i=n[t+1])))/2;return r[t]=o,r}function wo(n){for(var t,e,r,u,i=[],o=bo(n),a=-1,c=n.length-1;++a<c;)t=_o(n[a],n[a+1]),ua(t)<ka?o[a]=o[a+1]=0:(e=o[a]/t,r=o[a+1]/t,u=e*e+r*r,u>9&&(u=3*t/Math.sqrt(u),o[a]=u*e,o[a+1]=u*r));for(a=-1;++a<=c;)u=(n[Math.min(c,a+1)][0]-n[Math.max(0,a-1)][0])/(6*(1+o[a]*o[a])),i.push([u||0,o[a]*u||0]);return i}function So(n){return n.length<3?uo(n):n[0]+ho(n,wo(n))}function ko(n){for(var t,e,r,u=-1,i=n.length;++u<i;)t=n[u],e=t[0],r=t[1]+ms,t[0]=e*Math.cos(r),t[1]=e*Math.sin(r);return n}function Eo(n){function t(t){function c(){v.push("M",a(n(m),f),l,s(n(d.reverse()),f),"Z")}for(var h,g,p,v=[],d=[],m=[],y=-1,x=t.length,M=bt(e),_=bt(u),b=e===r?function(){return g}:bt(r),w=u===i?function(){return p}:bt(i);++y<x;)o.call(this,h=t[y],y)?(d.push([g=+M.call(this,h,y),p=+_.call(this,h,y)]),m.push([+b.call(this,h,y),+w.call(this,h,y)])):d.length&&(c(),d=[],m=[]);return d.length&&c(),v.length?v.join(""):null}var e=wr,r=wr,u=0,i=Sr,o=we,a=uo,c=a.key,s=a,l="L",f=.7;return t.x=function(n){return arguments.length?(e=r=n,t):r},t.x0=function(n){return arguments.length?(e=n,t):e},t.x1=function(n){return arguments.length?(r=n,t):r},t.y=function(n){return arguments.length?(u=i=n,t):i},t.y0=function(n){return arguments.length?(u=n,t):u},t.y1=function(n){return arguments.length?(i=n,t):i},t.defined=function(n){return arguments.length?(o=n,t):o},t.interpolate=function(n){return arguments.length?(c="function"==typeof n?a=n:(a=xs.get(n)||uo).key,s=a.reverse||a,l=a.closed?"M":"L",t):c},t.tension=function(n){return arguments.length?(f=n,t):f},t}function Ao(n){return n.radius}function Co(n){return[n.x,n.y]}function No(n){return function(){var t=n.apply(this,arguments),e=t[0],r=t[1]+ms;return[e*Math.cos(r),e*Math.sin(r)]}}function zo(){return 64}function Lo(){return"circle"}function To(n){var t=Math.sqrt(n/ba);return"M0,"+t+"A"+t+","+t+" 0 1,1 0,"+-t+"A"+t+","+t+" 0 1,1 0,"+t+"Z"}function qo(n,t){return sa(n,Cs),n.id=t,n}function Ro(n,t,e,r){var u=n.id;return P(n,"function"==typeof e?function(n,i,o){n.__transition__[u].tween.set(t,r(e.call(n,n.__data__,i,o)))}:(e=r(e),function(n){n.__transition__[u].tween.set(t,e)}))}function Do(n){return null==n&&(n=""),function(){this.textContent=n}}function Po(n,t,e,r){var u=n.__transition__||(n.__transition__={active:0,count:0}),i=u[e];if(!i){var a=r.time;i=u[e]={tween:new o,time:a,ease:r.ease,delay:r.delay,duration:r.duration},++u.count,Zo.timer(function(r){function o(r){return u.active>e?s():(u.active=e,i.event&&i.event.start.call(n,l,t),i.tween.forEach(function(e,r){(r=r.call(n,l,t))&&v.push(r)}),Zo.timer(function(){return p.c=c(r||1)?we:c,1},0,a),void 0)}function c(r){if(u.active!==e)return s();for(var o=r/g,a=f(o),c=v.length;c>0;)v[--c].call(n,a);
+return o>=1?(i.event&&i.event.end.call(n,l,t),s()):void 0}function s(){return--u.count?delete u[e]:delete n.__transition__,1}var l=n.__data__,f=i.ease,h=i.delay,g=i.duration,p=Ba,v=[];return p.t=h+a,r>=h?o(r-h):(p.c=o,void 0)},0,a)}}function Uo(n,t){n.attr("transform",function(n){return"translate("+t(n)+",0)"})}function jo(n,t){n.attr("transform",function(n){return"translate(0,"+t(n)+")"})}function Ho(n){return n.toISOString()}function Fo(n,t,e){function r(t){return n(t)}function u(n,e){var r=n[1]-n[0],u=r/e,i=Zo.bisect(Us,u);return i==Us.length?[t.year,Fi(n.map(function(n){return n/31536e6}),e)[2]]:i?t[u/Us[i-1]<Us[i]/u?i-1:i]:[Fs,Fi(n,e)[2]]}return r.invert=function(t){return Oo(n.invert(t))},r.domain=function(t){return arguments.length?(n.domain(t),r):n.domain().map(Oo)},r.nice=function(n,t){function e(e){return!isNaN(e)&&!n.range(e,Oo(+e+1),t).length}var i=r.domain(),o=Li(i),a=null==n?u(o,10):"number"==typeof n&&u(o,n);return a&&(n=a[0],t=a[1]),r.domain(Ri(i,t>1?{floor:function(t){for(;e(t=n.floor(t));)t=Oo(t-1);return t},ceil:function(t){for(;e(t=n.ceil(t));)t=Oo(+t+1);return t}}:n))},r.ticks=function(n,t){var e=Li(r.domain()),i=null==n?u(e,10):"number"==typeof n?u(e,n):!n.range&&[{range:n},t];return i&&(n=i[0],t=i[1]),n.range(e[0],Oo(+e[1]+1),1>t?1:t)},r.tickFormat=function(){return e},r.copy=function(){return Fo(n.copy(),t,e)},ji(r,n)}function Oo(n){return new Date(n)}function Yo(n){return JSON.parse(n.responseText)}function Io(n){var t=$o.createRange();return t.selectNode($o.body),t.createContextualFragment(n.responseText)}var Zo={version:"3.4.11"};Date.now||(Date.now=function(){return+new Date});var Vo=[].slice,Xo=function(n){return Vo.call(n)},$o=document,Bo=$o.documentElement,Wo=window;try{Xo(Bo.childNodes)[0].nodeType}catch(Jo){Xo=function(n){for(var t=n.length,e=new Array(t);t--;)e[t]=n[t];return e}}try{$o.createElement("div").style.setProperty("opacity",0,"")}catch(Go){var Ko=Wo.Element.prototype,Qo=Ko.setAttribute,na=Ko.setAttributeNS,ta=Wo.CSSStyleDeclaration.prototype,ea=ta.setProperty;Ko.setAttribute=function(n,t){Qo.call(this,n,t+"")},Ko.setAttributeNS=function(n,t,e){na.call(this,n,t,e+"")},ta.setProperty=function(n,t,e){ea.call(this,n,t+"",e)}}Zo.ascending=n,Zo.descending=function(n,t){return n>t?-1:t>n?1:t>=n?0:0/0},Zo.min=function(n,t){var e,r,u=-1,i=n.length;if(1===arguments.length){for(;++u<i&&!(null!=(e=n[u])&&e>=e);)e=void 0;for(;++u<i;)null!=(r=n[u])&&e>r&&(e=r)}else{for(;++u<i&&!(null!=(e=t.call(n,n[u],u))&&e>=e);)e=void 0;for(;++u<i;)null!=(r=t.call(n,n[u],u))&&e>r&&(e=r)}return e},Zo.max=function(n,t){var e,r,u=-1,i=n.length;if(1===arguments.length){for(;++u<i&&!(null!=(e=n[u])&&e>=e);)e=void 0;for(;++u<i;)null!=(r=n[u])&&r>e&&(e=r)}else{for(;++u<i&&!(null!=(e=t.call(n,n[u],u))&&e>=e);)e=void 0;for(;++u<i;)null!=(r=t.call(n,n[u],u))&&r>e&&(e=r)}return e},Zo.extent=function(n,t){var e,r,u,i=-1,o=n.length;if(1===arguments.length){for(;++i<o&&!(null!=(e=u=n[i])&&e>=e);)e=u=void 0;for(;++i<o;)null!=(r=n[i])&&(e>r&&(e=r),r>u&&(u=r))}else{for(;++i<o&&!(null!=(e=u=t.call(n,n[i],i))&&e>=e);)e=void 0;for(;++i<o;)null!=(r=t.call(n,n[i],i))&&(e>r&&(e=r),r>u&&(u=r))}return[e,u]},Zo.sum=function(n,t){var e,r=0,u=n.length,i=-1;if(1===arguments.length)for(;++i<u;)isNaN(e=+n[i])||(r+=e);else for(;++i<u;)isNaN(e=+t.call(n,n[i],i))||(r+=e);return r},Zo.mean=function(n,e){var r,u=0,i=n.length,o=-1,a=i;if(1===arguments.length)for(;++o<i;)t(r=n[o])?u+=r:--a;else for(;++o<i;)t(r=e.call(n,n[o],o))?u+=r:--a;return a?u/a:void 0},Zo.quantile=function(n,t){var e=(n.length-1)*t+1,r=Math.floor(e),u=+n[r-1],i=e-r;return i?u+i*(n[r]-u):u},Zo.median=function(e,r){return arguments.length>1&&(e=e.map(r)),e=e.filter(t),e.length?Zo.quantile(e.sort(n),.5):void 0};var ra=e(n);Zo.bisectLeft=ra.left,Zo.bisect=Zo.bisectRight=ra.right,Zo.bisector=function(t){return e(1===t.length?function(e,r){return n(t(e),r)}:t)},Zo.shuffle=function(n){for(var t,e,r=n.length;r;)e=0|Math.random()*r--,t=n[r],n[r]=n[e],n[e]=t;return n},Zo.permute=function(n,t){for(var e=t.length,r=new Array(e);e--;)r[e]=n[t[e]];return r},Zo.pairs=function(n){for(var t,e=0,r=n.length-1,u=n[0],i=new Array(0>r?0:r);r>e;)i[e]=[t=u,u=n[++e]];return i},Zo.zip=function(){if(!(u=arguments.length))return[];for(var n=-1,t=Zo.min(arguments,r),e=new Array(t);++n<t;)for(var u,i=-1,o=e[n]=new Array(u);++i<u;)o[i]=arguments[i][n];return e},Zo.transpose=function(n){return Zo.zip.apply(Zo,n)},Zo.keys=function(n){var t=[];for(var e in n)t.push(e);return t},Zo.values=function(n){var t=[];for(var e in n)t.push(n[e]);return t},Zo.entries=function(n){var t=[];for(var e in n)t.push({key:e,value:n[e]});return t},Zo.merge=function(n){for(var t,e,r,u=n.length,i=-1,o=0;++i<u;)o+=n[i].length;for(e=new Array(o);--u>=0;)for(r=n[u],t=r.length;--t>=0;)e[--o]=r[t];return e};var ua=Math.abs;Zo.range=function(n,t,e){if(arguments.length<3&&(e=1,arguments.length<2&&(t=n,n=0)),1/0===(t-n)/e)throw new Error("infinite range");var r,i=[],o=u(ua(e)),a=-1;if(n*=o,t*=o,e*=o,0>e)for(;(r=n+e*++a)>t;)i.push(r/o);else for(;(r=n+e*++a)<t;)i.push(r/o);return i},Zo.map=function(n){var t=new o;if(n instanceof o)n.forEach(function(n,e){t.set(n,e)});else for(var e in n)t.set(e,n[e]);return t},i(o,{has:a,get:function(n){return this[ia+n]},set:function(n,t){return this[ia+n]=t},remove:c,keys:s,values:function(){var n=[];return this.forEach(function(t,e){n.push(e)}),n},entries:function(){var n=[];return this.forEach(function(t,e){n.push({key:t,value:e})}),n},size:l,empty:f,forEach:function(n){for(var t in this)t.charCodeAt(0)===oa&&n.call(this,t.substring(1),this[t])}});var ia="\x00",oa=ia.charCodeAt(0);Zo.nest=function(){function n(t,a,c){if(c>=i.length)return r?r.call(u,a):e?a.sort(e):a;for(var s,l,f,h,g=-1,p=a.length,v=i[c++],d=new o;++g<p;)(h=d.get(s=v(l=a[g])))?h.push(l):d.set(s,[l]);return t?(l=t(),f=function(e,r){l.set(e,n(t,r,c))}):(l={},f=function(e,r){l[e]=n(t,r,c)}),d.forEach(f),l}function t(n,e){if(e>=i.length)return n;var r=[],u=a[e++];return n.forEach(function(n,u){r.push({key:n,values:t(u,e)})}),u?r.sort(function(n,t){return u(n.key,t.key)}):r}var e,r,u={},i=[],a=[];return u.map=function(t,e){return n(e,t,0)},u.entries=function(e){return t(n(Zo.map,e,0),0)},u.key=function(n){return i.push(n),u},u.sortKeys=function(n){return a[i.length-1]=n,u},u.sortValues=function(n){return e=n,u},u.rollup=function(n){return r=n,u},u},Zo.set=function(n){var t=new h;if(n)for(var e=0,r=n.length;r>e;++e)t.add(n[e]);return t},i(h,{has:a,add:function(n){return this[ia+n]=!0,n},remove:function(n){return n=ia+n,n in this&&delete this[n]},values:s,size:l,empty:f,forEach:function(n){for(var t in this)t.charCodeAt(0)===oa&&n.call(this,t.substring(1))}}),Zo.behavior={},Zo.rebind=function(n,t){for(var e,r=1,u=arguments.length;++r<u;)n[e=arguments[r]]=g(n,t,t[e]);return n};var aa=["webkit","ms","moz","Moz","o","O"];Zo.dispatch=function(){for(var n=new d,t=-1,e=arguments.length;++t<e;)n[arguments[t]]=m(n);return n},d.prototype.on=function(n,t){var e=n.indexOf("."),r="";if(e>=0&&(r=n.substring(e+1),n=n.substring(0,e)),n)return arguments.length<2?this[n].on(r):this[n].on(r,t);if(2===arguments.length){if(null==t)for(n in this)this.hasOwnProperty(n)&&this[n].on(r,null);return this}},Zo.event=null,Zo.requote=function(n){return n.replace(ca,"\\$&")};var ca=/[\\\^\$\*\+\?\|\[\]\(\)\.\{\}]/g,sa={}.__proto__?function(n,t){n.__proto__=t}:function(n,t){for(var e in t)n[e]=t[e]},la=function(n,t){return t.querySelector(n)},fa=function(n,t){return t.querySelectorAll(n)},ha=Bo.matches||Bo[p(Bo,"matchesSelector")],ga=function(n,t){return ha.call(n,t)};"function"==typeof Sizzle&&(la=function(n,t){return Sizzle(n,t)[0]||null},fa=Sizzle,ga=Sizzle.matchesSelector),Zo.selection=function(){return ma};var pa=Zo.selection.prototype=[];pa.select=function(n){var t,e,r,u,i=[];n=b(n);for(var o=-1,a=this.length;++o<a;){i.push(t=[]),t.parentNode=(r=this[o]).parentNode;for(var c=-1,s=r.length;++c<s;)(u=r[c])?(t.push(e=n.call(u,u.__data__,c,o)),e&&"__data__"in u&&(e.__data__=u.__data__)):t.push(null)}return _(i)},pa.selectAll=function(n){var t,e,r=[];n=w(n);for(var u=-1,i=this.length;++u<i;)for(var o=this[u],a=-1,c=o.length;++a<c;)(e=o[a])&&(r.push(t=Xo(n.call(e,e.__data__,a,u))),t.parentNode=e);return _(r)};var va={svg:"http://www.w3.org/2000/svg",xhtml:"http://www.w3.org/1999/xhtml",xlink:"http://www.w3.org/1999/xlink",xml:"http://www.w3.org/XML/1998/namespace",xmlns:"http://www.w3.org/2000/xmlns/"};Zo.ns={prefix:va,qualify:function(n){var t=n.indexOf(":"),e=n;return t>=0&&(e=n.substring(0,t),n=n.substring(t+1)),va.hasOwnProperty(e)?{space:va[e],local:n}:n}},pa.attr=function(n,t){if(arguments.length<2){if("string"==typeof n){var e=this.node();return n=Zo.ns.qualify(n),n.local?e.getAttributeNS(n.space,n.local):e.getAttribute(n)}for(t in n)this.each(S(t,n[t]));return this}return this.each(S(n,t))},pa.classed=function(n,t){if(arguments.length<2){if("string"==typeof n){var e=this.node(),r=(n=A(n)).length,u=-1;if(t=e.classList){for(;++u<r;)if(!t.contains(n[u]))return!1}else for(t=e.getAttribute("class");++u<r;)if(!E(n[u]).test(t))return!1;return!0}for(t in n)this.each(C(t,n[t]));return this}return this.each(C(n,t))},pa.style=function(n,t,e){var r=arguments.length;if(3>r){if("string"!=typeof n){2>r&&(t="");for(e in n)this.each(z(e,n[e],t));return this}if(2>r)return Wo.getComputedStyle(this.node(),null).getPropertyValue(n);e=""}return this.each(z(n,t,e))},pa.property=function(n,t){if(arguments.length<2){if("string"==typeof n)return this.node()[n];for(t in n)this.each(L(t,n[t]));return this}return this.each(L(n,t))},pa.text=function(n){return arguments.length?this.each("function"==typeof n?function(){var t=n.apply(this,arguments);this.textContent=null==t?"":t}:null==n?function(){this.textContent=""}:function(){this.textContent=n}):this.node().textContent},pa.html=function(n){return arguments.length?this.each("function"==typeof n?function(){var t=n.apply(this,arguments);this.innerHTML=null==t?"":t}:null==n?function(){this.innerHTML=""}:function(){this.innerHTML=n}):this.node().innerHTML},pa.append=function(n){return n=T(n),this.select(function(){return this.appendChild(n.apply(this,arguments))})},pa.insert=function(n,t){return n=T(n),t=b(t),this.select(function(){return this.insertBefore(n.apply(this,arguments),t.apply(this,arguments)||null)})},pa.remove=function(){return this.each(function(){var n=this.parentNode;n&&n.removeChild(this)})},pa.data=function(n,t){function e(n,e){var r,u,i,a=n.length,f=e.length,h=Math.min(a,f),g=new Array(f),p=new Array(f),v=new Array(a);if(t){var d,m=new o,y=new o,x=[];for(r=-1;++r<a;)d=t.call(u=n[r],u.__data__,r),m.has(d)?v[r]=u:m.set(d,u),x.push(d);for(r=-1;++r<f;)d=t.call(e,i=e[r],r),(u=m.get(d))?(g[r]=u,u.__data__=i):y.has(d)||(p[r]=q(i)),y.set(d,i),m.remove(d);for(r=-1;++r<a;)m.has(x[r])&&(v[r]=n[r])}else{for(r=-1;++r<h;)u=n[r],i=e[r],u?(u.__data__=i,g[r]=u):p[r]=q(i);for(;f>r;++r)p[r]=q(e[r]);for(;a>r;++r)v[r]=n[r]}p.update=g,p.parentNode=g.parentNode=v.parentNode=n.parentNode,c.push(p),s.push(g),l.push(v)}var r,u,i=-1,a=this.length;if(!arguments.length){for(n=new Array(a=(r=this[0]).length);++i<a;)(u=r[i])&&(n[i]=u.__data__);return n}var c=U([]),s=_([]),l=_([]);if("function"==typeof n)for(;++i<a;)e(r=this[i],n.call(r,r.parentNode.__data__,i));else for(;++i<a;)e(r=this[i],n);return s.enter=function(){return c},s.exit=function(){return l},s},pa.datum=function(n){return arguments.length?this.property("__data__",n):this.property("__data__")},pa.filter=function(n){var t,e,r,u=[];"function"!=typeof n&&(n=R(n));for(var i=0,o=this.length;o>i;i++){u.push(t=[]),t.parentNode=(e=this[i]).parentNode;for(var a=0,c=e.length;c>a;a++)(r=e[a])&&n.call(r,r.__data__,a,i)&&t.push(r)}return _(u)},pa.order=function(){for(var n=-1,t=this.length;++n<t;)for(var e,r=this[n],u=r.length-1,i=r[u];--u>=0;)(e=r[u])&&(i&&i!==e.nextSibling&&i.parentNode.insertBefore(e,i),i=e);return this},pa.sort=function(n){n=D.apply(this,arguments);for(var t=-1,e=this.length;++t<e;)this[t].sort(n);return this.order()},pa.each=function(n){return P(this,function(t,e,r){n.call(t,t.__data__,e,r)})},pa.call=function(n){var t=Xo(arguments);return n.apply(t[0]=this,t),this},pa.empty=function(){return!this.node()},pa.node=function(){for(var n=0,t=this.length;t>n;n++)for(var e=this[n],r=0,u=e.length;u>r;r++){var i=e[r];if(i)return i}return null},pa.size=function(){var n=0;return this.each(function(){++n}),n};var da=[];Zo.selection.enter=U,Zo.selection.enter.prototype=da,da.append=pa.append,da.empty=pa.empty,da.node=pa.node,da.call=pa.call,da.size=pa.size,da.select=function(n){for(var t,e,r,u,i,o=[],a=-1,c=this.length;++a<c;){r=(u=this[a]).update,o.push(t=[]),t.parentNode=u.parentNode;for(var s=-1,l=u.length;++s<l;)(i=u[s])?(t.push(r[s]=e=n.call(u.parentNode,i.__data__,s,a)),e.__data__=i.__data__):t.push(null)}return _(o)},da.insert=function(n,t){return arguments.length<2&&(t=j(this)),pa.insert.call(this,n,t)},pa.transition=function(){for(var n,t,e=Ss||++Ns,r=[],u=ks||{time:Date.now(),ease:xu,delay:0,duration:250},i=-1,o=this.length;++i<o;){r.push(n=[]);for(var a=this[i],c=-1,s=a.length;++c<s;)(t=a[c])&&Po(t,c,e,u),n.push(t)}return qo(r,e)},pa.interrupt=function(){return this.each(H)},Zo.select=function(n){var t=["string"==typeof n?la(n,$o):n];return t.parentNode=Bo,_([t])},Zo.selectAll=function(n){var t=Xo("string"==typeof n?fa(n,$o):n);return t.parentNode=Bo,_([t])};var ma=Zo.select(Bo);pa.on=function(n,t,e){var r=arguments.length;if(3>r){if("string"!=typeof n){2>r&&(t=!1);for(e in n)this.each(F(e,n[e],t));return this}if(2>r)return(r=this.node()["__on"+n])&&r._;e=!1}return this.each(F(n,t,e))};var ya=Zo.map({mouseenter:"mouseover",mouseleave:"mouseout"});ya.forEach(function(n){"on"+n in $o&&ya.remove(n)});var xa="onselectstart"in $o?null:p(Bo.style,"userSelect"),Ma=0;Zo.mouse=function(n){return Z(n,x())};var _a=/WebKit/.test(Wo.navigator.userAgent)?-1:0;Zo.touches=function(n,t){return arguments.length<2&&(t=x().touches),t?Xo(t).map(function(t){var e=Z(n,t);return e.identifier=t.identifier,e}):[]},Zo.behavior.drag=function(){function n(){this.on("mousedown.drag",u).on("touchstart.drag",i)}function t(n,t,u,i,o){return function(){function a(){var n,e,r=t(h,v);r&&(n=r[0]-x[0],e=r[1]-x[1],p|=n|e,x=r,g({type:"drag",x:r[0]+s[0],y:r[1]+s[1],dx:n,dy:e}))}function c(){t(h,v)&&(m.on(i+d,null).on(o+d,null),y(p&&Zo.event.target===f),g({type:"dragend"}))}var s,l=this,f=Zo.event.target,h=l.parentNode,g=e.of(l,arguments),p=0,v=n(),d=".drag"+(null==v?"":"-"+v),m=Zo.select(u()).on(i+d,a).on(o+d,c),y=I(),x=t(h,v);r?(s=r.apply(l,arguments),s=[s.x-x[0],s.y-x[1]]):s=[0,0],g({type:"dragstart"})}}var e=M(n,"drag","dragstart","dragend"),r=null,u=t(v,Zo.mouse,$,"mousemove","mouseup"),i=t(V,Zo.touch,X,"touchmove","touchend");return n.origin=function(t){return arguments.length?(r=t,n):r},Zo.rebind(n,e,"on")};var ba=Math.PI,wa=2*ba,Sa=ba/2,ka=1e-6,Ea=ka*ka,Aa=ba/180,Ca=180/ba,Na=Math.SQRT2,za=2,La=4;Zo.interpolateZoom=function(n,t){function e(n){var t=n*y;if(m){var e=Q(v),o=i/(za*h)*(e*nt(Na*t+v)-K(v));return[r+o*s,u+o*l,i*e/Q(Na*t+v)]}return[r+n*s,u+n*l,i*Math.exp(Na*t)]}var r=n[0],u=n[1],i=n[2],o=t[0],a=t[1],c=t[2],s=o-r,l=a-u,f=s*s+l*l,h=Math.sqrt(f),g=(c*c-i*i+La*f)/(2*i*za*h),p=(c*c-i*i-La*f)/(2*c*za*h),v=Math.log(Math.sqrt(g*g+1)-g),d=Math.log(Math.sqrt(p*p+1)-p),m=d-v,y=(m||Math.log(c/i))/Na;return e.duration=1e3*y,e},Zo.behavior.zoom=function(){function n(n){n.on(A,s).on(Ra+".zoom",f).on("dblclick.zoom",h).on(z,l)}function t(n){return[(n[0]-S.x)/S.k,(n[1]-S.y)/S.k]}function e(n){return[n[0]*S.k+S.x,n[1]*S.k+S.y]}function r(n){S.k=Math.max(E[0],Math.min(E[1],n))}function u(n,t){t=e(t),S.x+=n[0]-t[0],S.y+=n[1]-t[1]}function i(){_&&_.domain(x.range().map(function(n){return(n-S.x)/S.k}).map(x.invert)),w&&w.domain(b.range().map(function(n){return(n-S.y)/S.k}).map(b.invert))}function o(n){n({type:"zoomstart"})}function a(n){i(),n({type:"zoom",scale:S.k,translate:[S.x,S.y]})}function c(n){n({type:"zoomend"})}function s(){function n(){l=1,u(Zo.mouse(r),h),a(s)}function e(){f.on(C,null).on(N,null),g(l&&Zo.event.target===i),c(s)}var r=this,i=Zo.event.target,s=L.of(r,arguments),l=0,f=Zo.select(Wo).on(C,n).on(N,e),h=t(Zo.mouse(r)),g=I();H.call(r),o(s)}function l(){function n(){var n=Zo.touches(g);return h=S.k,n.forEach(function(n){n.identifier in v&&(v[n.identifier]=t(n))}),n}function e(){var t=Zo.event.target;Zo.select(t).on(M,i).on(_,f),b.push(t);for(var e=Zo.event.changedTouches,o=0,c=e.length;c>o;++o)v[e[o].identifier]=null;var s=n(),l=Date.now();if(1===s.length){if(500>l-m){var h=s[0],g=v[h.identifier];r(2*S.k),u(h,g),y(),a(p)}m=l}else if(s.length>1){var h=s[0],x=s[1],w=h[0]-x[0],k=h[1]-x[1];d=w*w+k*k}}function i(){for(var n,t,e,i,o=Zo.touches(g),c=0,s=o.length;s>c;++c,i=null)if(e=o[c],i=v[e.identifier]){if(t)break;n=e,t=i}if(i){var l=(l=e[0]-n[0])*l+(l=e[1]-n[1])*l,f=d&&Math.sqrt(l/d);n=[(n[0]+e[0])/2,(n[1]+e[1])/2],t=[(t[0]+i[0])/2,(t[1]+i[1])/2],r(f*h)}m=null,u(n,t),a(p)}function f(){if(Zo.event.touches.length){for(var t=Zo.event.changedTouches,e=0,r=t.length;r>e;++e)delete v[t[e].identifier];for(var u in v)return void n()}Zo.selectAll(b).on(x,null),w.on(A,s).on(z,l),k(),c(p)}var h,g=this,p=L.of(g,arguments),v={},d=0,x=".zoom-"+Zo.event.changedTouches[0].identifier,M="touchmove"+x,_="touchend"+x,b=[],w=Zo.select(g).on(A,null).on(z,e),k=I();H.call(g),e(),o(p)}function f(){var n=L.of(this,arguments);d?clearTimeout(d):(g=t(p=v||Zo.mouse(this)),H.call(this),o(n)),d=setTimeout(function(){d=null,c(n)},50),y(),r(Math.pow(2,.002*Ta())*S.k),u(p,g),a(n)}function h(){var n=L.of(this,arguments),e=Zo.mouse(this),i=t(e),s=Math.log(S.k)/Math.LN2;o(n),r(Math.pow(2,Zo.event.shiftKey?Math.ceil(s)-1:Math.floor(s)+1)),u(e,i),a(n),c(n)}var g,p,v,d,m,x,_,b,w,S={x:0,y:0,k:1},k=[960,500],E=qa,A="mousedown.zoom",C="mousemove.zoom",N="mouseup.zoom",z="touchstart.zoom",L=M(n,"zoomstart","zoom","zoomend");return n.event=function(n){n.each(function(){var n=L.of(this,arguments),t=S;Ss?Zo.select(this).transition().each("start.zoom",function(){S=this.__chart__||{x:0,y:0,k:1},o(n)}).tween("zoom:zoom",function(){var e=k[0],r=k[1],u=e/2,i=r/2,o=Zo.interpolateZoom([(u-S.x)/S.k,(i-S.y)/S.k,e/S.k],[(u-t.x)/t.k,(i-t.y)/t.k,e/t.k]);return function(t){var r=o(t),c=e/r[2];this.__chart__=S={x:u-r[0]*c,y:i-r[1]*c,k:c},a(n)}}).each("end.zoom",function(){c(n)}):(this.__chart__=S,o(n),a(n),c(n))})},n.translate=function(t){return arguments.length?(S={x:+t[0],y:+t[1],k:S.k},i(),n):[S.x,S.y]},n.scale=function(t){return arguments.length?(S={x:S.x,y:S.y,k:+t},i(),n):S.k},n.scaleExtent=function(t){return arguments.length?(E=null==t?qa:[+t[0],+t[1]],n):E},n.center=function(t){return arguments.length?(v=t&&[+t[0],+t[1]],n):v},n.size=function(t){return arguments.length?(k=t&&[+t[0],+t[1]],n):k},n.x=function(t){return arguments.length?(_=t,x=t.copy(),S={x:0,y:0,k:1},n):_},n.y=function(t){return arguments.length?(w=t,b=t.copy(),S={x:0,y:0,k:1},n):w},Zo.rebind(n,L,"on")};var Ta,qa=[0,1/0],Ra="onwheel"in $o?(Ta=function(){return-Zo.event.deltaY*(Zo.event.deltaMode?120:1)},"wheel"):"onmousewheel"in $o?(Ta=function(){return Zo.event.wheelDelta},"mousewheel"):(Ta=function(){return-Zo.event.detail},"MozMousePixelScroll");Zo.color=et,et.prototype.toString=function(){return this.rgb()+""},Zo.hsl=rt;var Da=rt.prototype=new et;Da.brighter=function(n){return n=Math.pow(.7,arguments.length?n:1),new rt(this.h,this.s,this.l/n)},Da.darker=function(n){return n=Math.pow(.7,arguments.length?n:1),new rt(this.h,this.s,n*this.l)},Da.rgb=function(){return ut(this.h,this.s,this.l)},Zo.hcl=it;var Pa=it.prototype=new et;Pa.brighter=function(n){return new it(this.h,this.c,Math.min(100,this.l+Ua*(arguments.length?n:1)))},Pa.darker=function(n){return new it(this.h,this.c,Math.max(0,this.l-Ua*(arguments.length?n:1)))},Pa.rgb=function(){return ot(this.h,this.c,this.l).rgb()},Zo.lab=at;var Ua=18,ja=.95047,Ha=1,Fa=1.08883,Oa=at.prototype=new et;Oa.brighter=function(n){return new at(Math.min(100,this.l+Ua*(arguments.length?n:1)),this.a,this.b)},Oa.darker=function(n){return new at(Math.max(0,this.l-Ua*(arguments.length?n:1)),this.a,this.b)},Oa.rgb=function(){return ct(this.l,this.a,this.b)},Zo.rgb=gt;var Ya=gt.prototype=new et;Ya.brighter=function(n){n=Math.pow(.7,arguments.length?n:1);var t=this.r,e=this.g,r=this.b,u=30;return t||e||r?(t&&u>t&&(t=u),e&&u>e&&(e=u),r&&u>r&&(r=u),new gt(Math.min(255,t/n),Math.min(255,e/n),Math.min(255,r/n))):new gt(u,u,u)},Ya.darker=function(n){return n=Math.pow(.7,arguments.length?n:1),new gt(n*this.r,n*this.g,n*this.b)},Ya.hsl=function(){return yt(this.r,this.g,this.b)},Ya.toString=function(){return"#"+dt(this.r)+dt(this.g)+dt(this.b)};var Ia=Zo.map({aliceblue:15792383,antiquewhite:16444375,aqua:65535,aquamarine:8388564,azure:15794175,beige:16119260,bisque:16770244,black:0,blanchedalmond:16772045,blue:255,blueviolet:9055202,brown:10824234,burlywood:14596231,cadetblue:6266528,chartreuse:8388352,chocolate:13789470,coral:16744272,cornflowerblue:6591981,cornsilk:16775388,crimson:14423100,cyan:65535,darkblue:139,darkcyan:35723,darkgoldenrod:12092939,darkgray:11119017,darkgreen:25600,darkgrey:11119017,darkkhaki:12433259,darkmagenta:9109643,darkolivegreen:5597999,darkorange:16747520,darkorchid:10040012,darkred:9109504,darksalmon:15308410,darkseagreen:9419919,darkslateblue:4734347,darkslategray:3100495,darkslategrey:3100495,darkturquoise:52945,darkviolet:9699539,deeppink:16716947,deepskyblue:49151,dimgray:6908265,dimgrey:6908265,dodgerblue:2003199,firebrick:11674146,floralwhite:16775920,forestgreen:2263842,fuchsia:16711935,gainsboro:14474460,ghostwhite:16316671,gold:16766720,goldenrod:14329120,gray:8421504,green:32768,greenyellow:11403055,grey:8421504,honeydew:15794160,hotpink:16738740,indianred:13458524,indigo:4915330,ivory:16777200,khaki:15787660,lavender:15132410,lavenderblush:16773365,lawngreen:8190976,lemonchiffon:16775885,lightblue:11393254,lightcoral:15761536,lightcyan:14745599,lightgoldenrodyellow:16448210,lightgray:13882323,lightgreen:9498256,lightgrey:13882323,lightpink:16758465,lightsalmon:16752762,lightseagreen:2142890,lightskyblue:8900346,lightslategray:7833753,lightslategrey:7833753,lightsteelblue:11584734,lightyellow:16777184,lime:65280,limegreen:3329330,linen:16445670,magenta:16711935,maroon:8388608,mediumaquamarine:6737322,mediumblue:205,mediumorchid:12211667,mediumpurple:9662683,mediumseagreen:3978097,mediumslateblue:8087790,mediumspringgreen:64154,mediumturquoise:4772300,mediumvioletred:13047173,midnightblue:1644912,mintcream:16121850,mistyrose:16770273,moccasin:16770229,navajowhite:16768685,navy:128,oldlace:16643558,olive:8421376,olivedrab:7048739,orange:16753920,orangered:16729344,orchid:14315734,palegoldenrod:15657130,palegreen:10025880,paleturquoise:11529966,palevioletred:14381203,papayawhip:16773077,peachpuff:16767673,peru:13468991,pink:16761035,plum:14524637,powderblue:11591910,purple:8388736,red:16711680,rosybrown:12357519,royalblue:4286945,saddlebrown:9127187,salmon:16416882,sandybrown:16032864,seagreen:3050327,seashell:16774638,sienna:10506797,silver:12632256,skyblue:8900331,slateblue:6970061,slategray:7372944,slategrey:7372944,snow:16775930,springgreen:65407,steelblue:4620980,tan:13808780,teal:32896,thistle:14204888,tomato:16737095,turquoise:4251856,violet:15631086,wheat:16113331,white:16777215,whitesmoke:16119285,yellow:16776960,yellowgreen:10145074});Ia.forEach(function(n,t){Ia.set(n,pt(t))}),Zo.functor=bt,Zo.xhr=St(wt),Zo.dsv=function(n,t){function e(n,e,i){arguments.length<3&&(i=e,e=null);var o=kt(n,t,null==e?r:u(e),i);return o.row=function(n){return arguments.length?o.response(null==(e=n)?r:u(n)):e},o}function r(n){return e.parse(n.responseText)}function u(n){return function(t){return e.parse(t.responseText,n)}}function i(t){return t.map(o).join(n)}function o(n){return a.test(n)?'"'+n.replace(/\"/g,'""')+'"':n}var a=new RegExp('["'+n+"\n]"),c=n.charCodeAt(0);return e.parse=function(n,t){var r;return e.parseRows(n,function(n,e){if(r)return r(n,e-1);var u=new Function("d","return {"+n.map(function(n,t){return JSON.stringify(n)+": d["+t+"]"}).join(",")+"}");r=t?function(n,e){return t(u(n),e)}:u})},e.parseRows=function(n,t){function e(){if(l>=s)return o;if(u)return u=!1,i;var t=l;if(34===n.charCodeAt(t)){for(var e=t;e++<s;)if(34===n.charCodeAt(e)){if(34!==n.charCodeAt(e+1))break;++e}l=e+2;var r=n.charCodeAt(e+1);return 13===r?(u=!0,10===n.charCodeAt(e+2)&&++l):10===r&&(u=!0),n.substring(t+1,e).replace(/""/g,'"')}for(;s>l;){var r=n.charCodeAt(l++),a=1;if(10===r)u=!0;else if(13===r)u=!0,10===n.charCodeAt(l)&&(++l,++a);else if(r!==c)continue;return n.substring(t,l-a)}return n.substring(t)}for(var r,u,i={},o={},a=[],s=n.length,l=0,f=0;(r=e())!==o;){for(var h=[];r!==i&&r!==o;)h.push(r),r=e();(!t||(h=t(h,f++)))&&a.push(h)}return a},e.format=function(t){if(Array.isArray(t[0]))return e.formatRows(t);var r=new h,u=[];return t.forEach(function(n){for(var t in n)r.has(t)||u.push(r.add(t))}),[u.map(o).join(n)].concat(t.map(function(t){return u.map(function(n){return o(t[n])}).join(n)})).join("\n")},e.formatRows=function(n){return n.map(i).join("\n")},e},Zo.csv=Zo.dsv(",","text/csv"),Zo.tsv=Zo.dsv(" ","text/tab-separated-values"),Zo.touch=function(n,t,e){if(arguments.length<3&&(e=t,t=x().changedTouches),t)for(var r,u=0,i=t.length;i>u;++u)if((r=t[u]).identifier===e)return Z(n,r)};var Za,Va,Xa,$a,Ba,Wa=Wo[p(Wo,"requestAnimationFrame")]||function(n){setTimeout(n,17)};Zo.timer=function(n,t,e){var r=arguments.length;2>r&&(t=0),3>r&&(e=Date.now());var u=e+t,i={c:n,t:u,f:!1,n:null};Va?Va.n=i:Za=i,Va=i,Xa||($a=clearTimeout($a),Xa=1,Wa(At))},Zo.timer.flush=function(){Ct(),Nt()},Zo.round=function(n,t){return t?Math.round(n*(t=Math.pow(10,t)))/t:Math.round(n)};var Ja=["y","z","a","f","p","n","\xb5","m","","k","M","G","T","P","E","Z","Y"].map(Lt);Zo.formatPrefix=function(n,t){var e=0;return n&&(0>n&&(n*=-1),t&&(n=Zo.round(n,zt(n,t))),e=1+Math.floor(1e-12+Math.log(n)/Math.LN10),e=Math.max(-24,Math.min(24,3*Math.floor((e-1)/3)))),Ja[8+e/3]};var Ga=/(?:([^{])?([<>=^]))?([+\- ])?([$#])?(0)?(\d+)?(,)?(\.-?\d+)?([a-z%])?/i,Ka=Zo.map({b:function(n){return n.toString(2)},c:function(n){return String.fromCharCode(n)},o:function(n){return n.toString(8)},x:function(n){return n.toString(16)},X:function(n){return n.toString(16).toUpperCase()},g:function(n,t){return n.toPrecision(t)},e:function(n,t){return n.toExponential(t)},f:function(n,t){return n.toFixed(t)},r:function(n,t){return(n=Zo.round(n,zt(n,t))).toFixed(Math.max(0,Math.min(20,zt(n*(1+1e-15),t))))}}),Qa=Zo.time={},nc=Date;Rt.prototype={getDate:function(){return this._.getUTCDate()},getDay:function(){return this._.getUTCDay()},getFullYear:function(){return this._.getUTCFullYear()},getHours:function(){return this._.getUTCHours()},getMilliseconds:function(){return this._.getUTCMilliseconds()},getMinutes:function(){return this._.getUTCMinutes()},getMonth:function(){return this._.getUTCMonth()},getSeconds:function(){return this._.getUTCSeconds()},getTime:function(){return this._.getTime()},getTimezoneOffset:function(){return 0},valueOf:function(){return this._.valueOf()},setDate:function(){tc.setUTCDate.apply(this._,arguments)},setDay:function(){tc.setUTCDay.apply(this._,arguments)},setFullYear:function(){tc.setUTCFullYear.apply(this._,arguments)},setHours:function(){tc.setUTCHours.apply(this._,arguments)},setMilliseconds:function(){tc.setUTCMilliseconds.apply(this._,arguments)},setMinutes:function(){tc.setUTCMinutes.apply(this._,arguments)},setMonth:function(){tc.setUTCMonth.apply(this._,arguments)},setSeconds:function(){tc.setUTCSeconds.apply(this._,arguments)},setTime:function(){tc.setTime.apply(this._,arguments)}};var tc=Date.prototype;Qa.year=Dt(function(n){return n=Qa.day(n),n.setMonth(0,1),n},function(n,t){n.setFullYear(n.getFullYear()+t)},function(n){return n.getFullYear()}),Qa.years=Qa.year.range,Qa.years.utc=Qa.year.utc.range,Qa.day=Dt(function(n){var t=new nc(2e3,0);return t.setFullYear(n.getFullYear(),n.getMonth(),n.getDate()),t},function(n,t){n.setDate(n.getDate()+t)},function(n){return n.getDate()-1}),Qa.days=Qa.day.range,Qa.days.utc=Qa.day.utc.range,Qa.dayOfYear=function(n){var t=Qa.year(n);return Math.floor((n-t-6e4*(n.getTimezoneOffset()-t.getTimezoneOffset()))/864e5)},["sunday","monday","tuesday","wednesday","thursday","friday","saturday"].forEach(function(n,t){t=7-t;var e=Qa[n]=Dt(function(n){return(n=Qa.day(n)).setDate(n.getDate()-(n.getDay()+t)%7),n},function(n,t){n.setDate(n.getDate()+7*Math.floor(t))},function(n){var e=Qa.year(n).getDay();return Math.floor((Qa.dayOfYear(n)+(e+t)%7)/7)-(e!==t)});Qa[n+"s"]=e.range,Qa[n+"s"].utc=e.utc.range,Qa[n+"OfYear"]=function(n){var e=Qa.year(n).getDay();return Math.floor((Qa.dayOfYear(n)+(e+t)%7)/7)}}),Qa.week=Qa.sunday,Qa.weeks=Qa.sunday.range,Qa.weeks.utc=Qa.sunday.utc.range,Qa.weekOfYear=Qa.sundayOfYear;var ec={"-":"",_:" ",0:"0"},rc=/^\s*\d+/,uc=/^%/;Zo.locale=function(n){return{numberFormat:Tt(n),timeFormat:Ut(n)}};var ic=Zo.locale({decimal:".",thousands:",",grouping:[3],currency:["$",""],dateTime:"%a %b %e %X %Y",date:"%m/%d/%Y",time:"%H:%M:%S",periods:["AM","PM"],days:["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"],shortDays:["Sun","Mon","Tue","Wed","Thu","Fri","Sat"],months:["January","February","March","April","May","June","July","August","September","October","November","December"],shortMonths:["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]});Zo.format=ic.numberFormat,Zo.geo={},ue.prototype={s:0,t:0,add:function(n){ie(n,this.t,oc),ie(oc.s,this.s,this),this.s?this.t+=oc.t:this.s=oc.t},reset:function(){this.s=this.t=0},valueOf:function(){return this.s}};var oc=new ue;Zo.geo.stream=function(n,t){n&&ac.hasOwnProperty(n.type)?ac[n.type](n,t):oe(n,t)};var ac={Feature:function(n,t){oe(n.geometry,t)},FeatureCollection:function(n,t){for(var e=n.features,r=-1,u=e.length;++r<u;)oe(e[r].geometry,t)}},cc={Sphere:function(n,t){t.sphere()},Point:function(n,t){n=n.coordinates,t.point(n[0],n[1],n[2])},MultiPoint:function(n,t){for(var e=n.coordinates,r=-1,u=e.length;++r<u;)n=e[r],t.point(n[0],n[1],n[2])},LineString:function(n,t){ae(n.coordinates,t,0)},MultiLineString:function(n,t){for(var e=n.coordinates,r=-1,u=e.length;++r<u;)ae(e[r],t,0)},Polygon:function(n,t){ce(n.coordinates,t)},MultiPolygon:function(n,t){for(var e=n.coordinates,r=-1,u=e.length;++r<u;)ce(e[r],t)},GeometryCollection:function(n,t){for(var e=n.geometries,r=-1,u=e.length;++r<u;)oe(e[r],t)}};Zo.geo.area=function(n){return sc=0,Zo.geo.stream(n,fc),sc};var sc,lc=new ue,fc={sphere:function(){sc+=4*ba},point:v,lineStart:v,lineEnd:v,polygonStart:function(){lc.reset(),fc.lineStart=se},polygonEnd:function(){var n=2*lc;sc+=0>n?4*ba+n:n,fc.lineStart=fc.lineEnd=fc.point=v}};Zo.geo.bounds=function(){function n(n,t){x.push(M=[l=n,h=n]),f>t&&(f=t),t>g&&(g=t)}function t(t,e){var r=le([t*Aa,e*Aa]);if(m){var u=he(m,r),i=[u[1],-u[0],0],o=he(i,u);ve(o),o=de(o);var c=t-p,s=c>0?1:-1,v=o[0]*Ca*s,d=ua(c)>180;if(d^(v>s*p&&s*t>v)){var y=o[1]*Ca;y>g&&(g=y)}else if(v=(v+360)%360-180,d^(v>s*p&&s*t>v)){var y=-o[1]*Ca;f>y&&(f=y)}else f>e&&(f=e),e>g&&(g=e);d?p>t?a(l,t)>a(l,h)&&(h=t):a(t,h)>a(l,h)&&(l=t):h>=l?(l>t&&(l=t),t>h&&(h=t)):t>p?a(l,t)>a(l,h)&&(h=t):a(t,h)>a(l,h)&&(l=t)}else n(t,e);m=r,p=t}function e(){_.point=t}function r(){M[0]=l,M[1]=h,_.point=n,m=null}function u(n,e){if(m){var r=n-p;y+=ua(r)>180?r+(r>0?360:-360):r}else v=n,d=e;fc.point(n,e),t(n,e)}function i(){fc.lineStart()}function o(){u(v,d),fc.lineEnd(),ua(y)>ka&&(l=-(h=180)),M[0]=l,M[1]=h,m=null}function a(n,t){return(t-=n)<0?t+360:t}function c(n,t){return n[0]-t[0]}function s(n,t){return t[0]<=t[1]?t[0]<=n&&n<=t[1]:n<t[0]||t[1]<n}var l,f,h,g,p,v,d,m,y,x,M,_={point:n,lineStart:e,lineEnd:r,polygonStart:function(){_.point=u,_.lineStart=i,_.lineEnd=o,y=0,fc.polygonStart()},polygonEnd:function(){fc.polygonEnd(),_.point=n,_.lineStart=e,_.lineEnd=r,0>lc?(l=-(h=180),f=-(g=90)):y>ka?g=90:-ka>y&&(f=-90),M[0]=l,M[1]=h}};return function(n){g=h=-(l=f=1/0),x=[],Zo.geo.stream(n,_);var t=x.length;if(t){x.sort(c);for(var e,r=1,u=x[0],i=[u];t>r;++r)e=x[r],s(e[0],u)||s(e[1],u)?(a(u[0],e[1])>a(u[0],u[1])&&(u[1]=e[1]),a(e[0],u[1])>a(u[0],u[1])&&(u[0]=e[0])):i.push(u=e);
+for(var o,e,p=-1/0,t=i.length-1,r=0,u=i[t];t>=r;u=e,++r)e=i[r],(o=a(u[1],e[0]))>p&&(p=o,l=e[0],h=u[1])}return x=M=null,1/0===l||1/0===f?[[0/0,0/0],[0/0,0/0]]:[[l,f],[h,g]]}}(),Zo.geo.centroid=function(n){hc=gc=pc=vc=dc=mc=yc=xc=Mc=_c=bc=0,Zo.geo.stream(n,wc);var t=Mc,e=_c,r=bc,u=t*t+e*e+r*r;return Ea>u&&(t=mc,e=yc,r=xc,ka>gc&&(t=pc,e=vc,r=dc),u=t*t+e*e+r*r,Ea>u)?[0/0,0/0]:[Math.atan2(e,t)*Ca,G(r/Math.sqrt(u))*Ca]};var hc,gc,pc,vc,dc,mc,yc,xc,Mc,_c,bc,wc={sphere:v,point:ye,lineStart:Me,lineEnd:_e,polygonStart:function(){wc.lineStart=be},polygonEnd:function(){wc.lineStart=Me}},Sc=Ae(we,Te,Re,[-ba,-ba/2]),kc=1e9;Zo.geo.clipExtent=function(){var n,t,e,r,u,i,o={stream:function(n){return u&&(u.valid=!1),u=i(n),u.valid=!0,u},extent:function(a){return arguments.length?(i=Ue(n=+a[0][0],t=+a[0][1],e=+a[1][0],r=+a[1][1]),u&&(u.valid=!1,u=null),o):[[n,t],[e,r]]}};return o.extent([[0,0],[960,500]])},(Zo.geo.conicEqualArea=function(){return He(Fe)}).raw=Fe,Zo.geo.albers=function(){return Zo.geo.conicEqualArea().rotate([96,0]).center([-.6,38.7]).parallels([29.5,45.5]).scale(1070)},Zo.geo.albersUsa=function(){function n(n){var i=n[0],o=n[1];return t=null,e(i,o),t||(r(i,o),t)||u(i,o),t}var t,e,r,u,i=Zo.geo.albers(),o=Zo.geo.conicEqualArea().rotate([154,0]).center([-2,58.5]).parallels([55,65]),a=Zo.geo.conicEqualArea().rotate([157,0]).center([-3,19.9]).parallels([8,18]),c={point:function(n,e){t=[n,e]}};return n.invert=function(n){var t=i.scale(),e=i.translate(),r=(n[0]-e[0])/t,u=(n[1]-e[1])/t;return(u>=.12&&.234>u&&r>=-.425&&-.214>r?o:u>=.166&&.234>u&&r>=-.214&&-.115>r?a:i).invert(n)},n.stream=function(n){var t=i.stream(n),e=o.stream(n),r=a.stream(n);return{point:function(n,u){t.point(n,u),e.point(n,u),r.point(n,u)},sphere:function(){t.sphere(),e.sphere(),r.sphere()},lineStart:function(){t.lineStart(),e.lineStart(),r.lineStart()},lineEnd:function(){t.lineEnd(),e.lineEnd(),r.lineEnd()},polygonStart:function(){t.polygonStart(),e.polygonStart(),r.polygonStart()},polygonEnd:function(){t.polygonEnd(),e.polygonEnd(),r.polygonEnd()}}},n.precision=function(t){return arguments.length?(i.precision(t),o.precision(t),a.precision(t),n):i.precision()},n.scale=function(t){return arguments.length?(i.scale(t),o.scale(.35*t),a.scale(t),n.translate(i.translate())):i.scale()},n.translate=function(t){if(!arguments.length)return i.translate();var s=i.scale(),l=+t[0],f=+t[1];return e=i.translate(t).clipExtent([[l-.455*s,f-.238*s],[l+.455*s,f+.238*s]]).stream(c).point,r=o.translate([l-.307*s,f+.201*s]).clipExtent([[l-.425*s+ka,f+.12*s+ka],[l-.214*s-ka,f+.234*s-ka]]).stream(c).point,u=a.translate([l-.205*s,f+.212*s]).clipExtent([[l-.214*s+ka,f+.166*s+ka],[l-.115*s-ka,f+.234*s-ka]]).stream(c).point,n},n.scale(1070)};var Ec,Ac,Cc,Nc,zc,Lc,Tc={point:v,lineStart:v,lineEnd:v,polygonStart:function(){Ac=0,Tc.lineStart=Oe},polygonEnd:function(){Tc.lineStart=Tc.lineEnd=Tc.point=v,Ec+=ua(Ac/2)}},qc={point:Ye,lineStart:v,lineEnd:v,polygonStart:v,polygonEnd:v},Rc={point:Ve,lineStart:Xe,lineEnd:$e,polygonStart:function(){Rc.lineStart=Be},polygonEnd:function(){Rc.point=Ve,Rc.lineStart=Xe,Rc.lineEnd=$e}};Zo.geo.path=function(){function n(n){return n&&("function"==typeof a&&i.pointRadius(+a.apply(this,arguments)),o&&o.valid||(o=u(i)),Zo.geo.stream(n,o)),i.result()}function t(){return o=null,n}var e,r,u,i,o,a=4.5;return n.area=function(n){return Ec=0,Zo.geo.stream(n,u(Tc)),Ec},n.centroid=function(n){return pc=vc=dc=mc=yc=xc=Mc=_c=bc=0,Zo.geo.stream(n,u(Rc)),bc?[Mc/bc,_c/bc]:xc?[mc/xc,yc/xc]:dc?[pc/dc,vc/dc]:[0/0,0/0]},n.bounds=function(n){return zc=Lc=-(Cc=Nc=1/0),Zo.geo.stream(n,u(qc)),[[Cc,Nc],[zc,Lc]]},n.projection=function(n){return arguments.length?(u=(e=n)?n.stream||Ge(n):wt,t()):e},n.context=function(n){return arguments.length?(i=null==(r=n)?new Ie:new We(n),"function"!=typeof a&&i.pointRadius(a),t()):r},n.pointRadius=function(t){return arguments.length?(a="function"==typeof t?t:(i.pointRadius(+t),+t),n):a},n.projection(Zo.geo.albersUsa()).context(null)},Zo.geo.transform=function(n){return{stream:function(t){var e=new Ke(t);for(var r in n)e[r]=n[r];return e}}},Ke.prototype={point:function(n,t){this.stream.point(n,t)},sphere:function(){this.stream.sphere()},lineStart:function(){this.stream.lineStart()},lineEnd:function(){this.stream.lineEnd()},polygonStart:function(){this.stream.polygonStart()},polygonEnd:function(){this.stream.polygonEnd()}},Zo.geo.projection=nr,Zo.geo.projectionMutator=tr,(Zo.geo.equirectangular=function(){return nr(rr)}).raw=rr.invert=rr,Zo.geo.rotation=function(n){function t(t){return t=n(t[0]*Aa,t[1]*Aa),t[0]*=Ca,t[1]*=Ca,t}return n=ir(n[0]%360*Aa,n[1]*Aa,n.length>2?n[2]*Aa:0),t.invert=function(t){return t=n.invert(t[0]*Aa,t[1]*Aa),t[0]*=Ca,t[1]*=Ca,t},t},ur.invert=rr,Zo.geo.circle=function(){function n(){var n="function"==typeof r?r.apply(this,arguments):r,t=ir(-n[0]*Aa,-n[1]*Aa,0).invert,u=[];return e(null,null,1,{point:function(n,e){u.push(n=t(n,e)),n[0]*=Ca,n[1]*=Ca}}),{type:"Polygon",coordinates:[u]}}var t,e,r=[0,0],u=6;return n.origin=function(t){return arguments.length?(r=t,n):r},n.angle=function(r){return arguments.length?(e=sr((t=+r)*Aa,u*Aa),n):t},n.precision=function(r){return arguments.length?(e=sr(t*Aa,(u=+r)*Aa),n):u},n.angle(90)},Zo.geo.distance=function(n,t){var e,r=(t[0]-n[0])*Aa,u=n[1]*Aa,i=t[1]*Aa,o=Math.sin(r),a=Math.cos(r),c=Math.sin(u),s=Math.cos(u),l=Math.sin(i),f=Math.cos(i);return Math.atan2(Math.sqrt((e=f*o)*e+(e=s*l-c*f*a)*e),c*l+s*f*a)},Zo.geo.graticule=function(){function n(){return{type:"MultiLineString",coordinates:t()}}function t(){return Zo.range(Math.ceil(i/d)*d,u,d).map(h).concat(Zo.range(Math.ceil(s/m)*m,c,m).map(g)).concat(Zo.range(Math.ceil(r/p)*p,e,p).filter(function(n){return ua(n%d)>ka}).map(l)).concat(Zo.range(Math.ceil(a/v)*v,o,v).filter(function(n){return ua(n%m)>ka}).map(f))}var e,r,u,i,o,a,c,s,l,f,h,g,p=10,v=p,d=90,m=360,y=2.5;return n.lines=function(){return t().map(function(n){return{type:"LineString",coordinates:n}})},n.outline=function(){return{type:"Polygon",coordinates:[h(i).concat(g(c).slice(1),h(u).reverse().slice(1),g(s).reverse().slice(1))]}},n.extent=function(t){return arguments.length?n.majorExtent(t).minorExtent(t):n.minorExtent()},n.majorExtent=function(t){return arguments.length?(i=+t[0][0],u=+t[1][0],s=+t[0][1],c=+t[1][1],i>u&&(t=i,i=u,u=t),s>c&&(t=s,s=c,c=t),n.precision(y)):[[i,s],[u,c]]},n.minorExtent=function(t){return arguments.length?(r=+t[0][0],e=+t[1][0],a=+t[0][1],o=+t[1][1],r>e&&(t=r,r=e,e=t),a>o&&(t=a,a=o,o=t),n.precision(y)):[[r,a],[e,o]]},n.step=function(t){return arguments.length?n.majorStep(t).minorStep(t):n.minorStep()},n.majorStep=function(t){return arguments.length?(d=+t[0],m=+t[1],n):[d,m]},n.minorStep=function(t){return arguments.length?(p=+t[0],v=+t[1],n):[p,v]},n.precision=function(t){return arguments.length?(y=+t,l=fr(a,o,90),f=hr(r,e,y),h=fr(s,c,90),g=hr(i,u,y),n):y},n.majorExtent([[-180,-90+ka],[180,90-ka]]).minorExtent([[-180,-80-ka],[180,80+ka]])},Zo.geo.greatArc=function(){function n(){return{type:"LineString",coordinates:[t||r.apply(this,arguments),e||u.apply(this,arguments)]}}var t,e,r=gr,u=pr;return n.distance=function(){return Zo.geo.distance(t||r.apply(this,arguments),e||u.apply(this,arguments))},n.source=function(e){return arguments.length?(r=e,t="function"==typeof e?null:e,n):r},n.target=function(t){return arguments.length?(u=t,e="function"==typeof t?null:t,n):u},n.precision=function(){return arguments.length?n:0},n},Zo.geo.interpolate=function(n,t){return vr(n[0]*Aa,n[1]*Aa,t[0]*Aa,t[1]*Aa)},Zo.geo.length=function(n){return Dc=0,Zo.geo.stream(n,Pc),Dc};var Dc,Pc={sphere:v,point:v,lineStart:dr,lineEnd:v,polygonStart:v,polygonEnd:v},Uc=mr(function(n){return Math.sqrt(2/(1+n))},function(n){return 2*Math.asin(n/2)});(Zo.geo.azimuthalEqualArea=function(){return nr(Uc)}).raw=Uc;var jc=mr(function(n){var t=Math.acos(n);return t&&t/Math.sin(t)},wt);(Zo.geo.azimuthalEquidistant=function(){return nr(jc)}).raw=jc,(Zo.geo.conicConformal=function(){return He(yr)}).raw=yr,(Zo.geo.conicEquidistant=function(){return He(xr)}).raw=xr;var Hc=mr(function(n){return 1/n},Math.atan);(Zo.geo.gnomonic=function(){return nr(Hc)}).raw=Hc,Mr.invert=function(n,t){return[n,2*Math.atan(Math.exp(t))-Sa]},(Zo.geo.mercator=function(){return _r(Mr)}).raw=Mr;var Fc=mr(function(){return 1},Math.asin);(Zo.geo.orthographic=function(){return nr(Fc)}).raw=Fc;var Oc=mr(function(n){return 1/(1+n)},function(n){return 2*Math.atan(n)});(Zo.geo.stereographic=function(){return nr(Oc)}).raw=Oc,br.invert=function(n,t){return[-t,2*Math.atan(Math.exp(n))-Sa]},(Zo.geo.transverseMercator=function(){var n=_r(br),t=n.center,e=n.rotate;return n.center=function(n){return n?t([-n[1],n[0]]):(n=t(),[n[1],-n[0]])},n.rotate=function(n){return n?e([n[0],n[1],n.length>2?n[2]+90:90]):(n=e(),[n[0],n[1],n[2]-90])},e([0,0,90])}).raw=br,Zo.geom={},Zo.geom.hull=function(n){function t(n){if(n.length<3)return[];var t,u=bt(e),i=bt(r),o=n.length,a=[],c=[];for(t=0;o>t;t++)a.push([+u.call(this,n[t],t),+i.call(this,n[t],t),t]);for(a.sort(Er),t=0;o>t;t++)c.push([a[t][0],-a[t][1]]);var s=kr(a),l=kr(c),f=l[0]===s[0],h=l[l.length-1]===s[s.length-1],g=[];for(t=s.length-1;t>=0;--t)g.push(n[a[s[t]][2]]);for(t=+f;t<l.length-h;++t)g.push(n[a[l[t]][2]]);return g}var e=wr,r=Sr;return arguments.length?t(n):(t.x=function(n){return arguments.length?(e=n,t):e},t.y=function(n){return arguments.length?(r=n,t):r},t)},Zo.geom.polygon=function(n){return sa(n,Yc),n};var Yc=Zo.geom.polygon.prototype=[];Yc.area=function(){for(var n,t=-1,e=this.length,r=this[e-1],u=0;++t<e;)n=r,r=this[t],u+=n[1]*r[0]-n[0]*r[1];return.5*u},Yc.centroid=function(n){var t,e,r=-1,u=this.length,i=0,o=0,a=this[u-1];for(arguments.length||(n=-1/(6*this.area()));++r<u;)t=a,a=this[r],e=t[0]*a[1]-a[0]*t[1],i+=(t[0]+a[0])*e,o+=(t[1]+a[1])*e;return[i*n,o*n]},Yc.clip=function(n){for(var t,e,r,u,i,o,a=Nr(n),c=-1,s=this.length-Nr(this),l=this[s-1];++c<s;){for(t=n.slice(),n.length=0,u=this[c],i=t[(r=t.length-a)-1],e=-1;++e<r;)o=t[e],Ar(o,l,u)?(Ar(i,l,u)||n.push(Cr(i,o,l,u)),n.push(o)):Ar(i,l,u)&&n.push(Cr(i,o,l,u)),i=o;a&&n.push(n[0]),l=u}return n};var Ic,Zc,Vc,Xc,$c,Bc=[],Wc=[];Ur.prototype.prepare=function(){for(var n,t=this.edges,e=t.length;e--;)n=t[e].edge,n.b&&n.a||t.splice(e,1);return t.sort(Hr),t.length},Wr.prototype={start:function(){return this.edge.l===this.site?this.edge.a:this.edge.b},end:function(){return this.edge.l===this.site?this.edge.b:this.edge.a}},Jr.prototype={insert:function(n,t){var e,r,u;if(n){if(t.P=n,t.N=n.N,n.N&&(n.N.P=t),n.N=t,n.R){for(n=n.R;n.L;)n=n.L;n.L=t}else n.R=t;e=n}else this._?(n=nu(this._),t.P=null,t.N=n,n.P=n.L=t,e=n):(t.P=t.N=null,this._=t,e=null);for(t.L=t.R=null,t.U=e,t.C=!0,n=t;e&&e.C;)r=e.U,e===r.L?(u=r.R,u&&u.C?(e.C=u.C=!1,r.C=!0,n=r):(n===e.R&&(Kr(this,e),n=e,e=n.U),e.C=!1,r.C=!0,Qr(this,r))):(u=r.L,u&&u.C?(e.C=u.C=!1,r.C=!0,n=r):(n===e.L&&(Qr(this,e),n=e,e=n.U),e.C=!1,r.C=!0,Kr(this,r))),e=n.U;this._.C=!1},remove:function(n){n.N&&(n.N.P=n.P),n.P&&(n.P.N=n.N),n.N=n.P=null;var t,e,r,u=n.U,i=n.L,o=n.R;if(e=i?o?nu(o):i:o,u?u.L===n?u.L=e:u.R=e:this._=e,i&&o?(r=e.C,e.C=n.C,e.L=i,i.U=e,e!==o?(u=e.U,e.U=n.U,n=e.R,u.L=n,e.R=o,o.U=e):(e.U=u,u=e,n=e.R)):(r=n.C,n=e),n&&(n.U=u),!r){if(n&&n.C)return n.C=!1,void 0;do{if(n===this._)break;if(n===u.L){if(t=u.R,t.C&&(t.C=!1,u.C=!0,Kr(this,u),t=u.R),t.L&&t.L.C||t.R&&t.R.C){t.R&&t.R.C||(t.L.C=!1,t.C=!0,Qr(this,t),t=u.R),t.C=u.C,u.C=t.R.C=!1,Kr(this,u),n=this._;break}}else if(t=u.L,t.C&&(t.C=!1,u.C=!0,Qr(this,u),t=u.L),t.L&&t.L.C||t.R&&t.R.C){t.L&&t.L.C||(t.R.C=!1,t.C=!0,Kr(this,t),t=u.L),t.C=u.C,u.C=t.L.C=!1,Qr(this,u),n=this._;break}t.C=!0,n=u,u=u.U}while(!n.C);n&&(n.C=!1)}}},Zo.geom.voronoi=function(n){function t(n){var t=new Array(n.length),r=a[0][0],u=a[0][1],i=a[1][0],o=a[1][1];return tu(e(n),a).cells.forEach(function(e,a){var c=e.edges,s=e.site,l=t[a]=c.length?c.map(function(n){var t=n.start();return[t.x,t.y]}):s.x>=r&&s.x<=i&&s.y>=u&&s.y<=o?[[r,o],[i,o],[i,u],[r,u]]:[];l.point=n[a]}),t}function e(n){return n.map(function(n,t){return{x:Math.round(i(n,t)/ka)*ka,y:Math.round(o(n,t)/ka)*ka,i:t}})}var r=wr,u=Sr,i=r,o=u,a=Jc;return n?t(n):(t.links=function(n){return tu(e(n)).edges.filter(function(n){return n.l&&n.r}).map(function(t){return{source:n[t.l.i],target:n[t.r.i]}})},t.triangles=function(n){var t=[];return tu(e(n)).cells.forEach(function(e,r){for(var u,i,o=e.site,a=e.edges.sort(Hr),c=-1,s=a.length,l=a[s-1].edge,f=l.l===o?l.r:l.l;++c<s;)u=l,i=f,l=a[c].edge,f=l.l===o?l.r:l.l,r<i.i&&r<f.i&&ru(o,i,f)<0&&t.push([n[r],n[i.i],n[f.i]])}),t},t.x=function(n){return arguments.length?(i=bt(r=n),t):r},t.y=function(n){return arguments.length?(o=bt(u=n),t):u},t.clipExtent=function(n){return arguments.length?(a=null==n?Jc:n,t):a===Jc?null:a},t.size=function(n){return arguments.length?t.clipExtent(n&&[[0,0],n]):a===Jc?null:a&&a[1]},t)};var Jc=[[-1e6,-1e6],[1e6,1e6]];Zo.geom.delaunay=function(n){return Zo.geom.voronoi().triangles(n)},Zo.geom.quadtree=function(n,t,e,r,u){function i(n){function i(n,t,e,r,u,i,o,a){if(!isNaN(e)&&!isNaN(r))if(n.leaf){var c=n.x,l=n.y;if(null!=c)if(ua(c-e)+ua(l-r)<.01)s(n,t,e,r,u,i,o,a);else{var f=n.point;n.x=n.y=n.point=null,s(n,f,c,l,u,i,o,a),s(n,t,e,r,u,i,o,a)}else n.x=e,n.y=r,n.point=t}else s(n,t,e,r,u,i,o,a)}function s(n,t,e,r,u,o,a,c){var s=.5*(u+a),l=.5*(o+c),f=e>=s,h=r>=l,g=(h<<1)+f;n.leaf=!1,n=n.nodes[g]||(n.nodes[g]=ou()),f?u=s:a=s,h?o=l:c=l,i(n,t,e,r,u,o,a,c)}var l,f,h,g,p,v,d,m,y,x=bt(a),M=bt(c);if(null!=t)v=t,d=e,m=r,y=u;else if(m=y=-(v=d=1/0),f=[],h=[],p=n.length,o)for(g=0;p>g;++g)l=n[g],l.x<v&&(v=l.x),l.y<d&&(d=l.y),l.x>m&&(m=l.x),l.y>y&&(y=l.y),f.push(l.x),h.push(l.y);else for(g=0;p>g;++g){var _=+x(l=n[g],g),b=+M(l,g);v>_&&(v=_),d>b&&(d=b),_>m&&(m=_),b>y&&(y=b),f.push(_),h.push(b)}var w=m-v,S=y-d;w>S?y=d+w:m=v+S;var k=ou();if(k.add=function(n){i(k,n,+x(n,++g),+M(n,g),v,d,m,y)},k.visit=function(n){au(n,k,v,d,m,y)},g=-1,null==t){for(;++g<p;)i(k,n[g],f[g],h[g],v,d,m,y);--g}else n.forEach(k.add);return f=h=n=l=null,k}var o,a=wr,c=Sr;return(o=arguments.length)?(a=uu,c=iu,3===o&&(u=e,r=t,e=t=0),i(n)):(i.x=function(n){return arguments.length?(a=n,i):a},i.y=function(n){return arguments.length?(c=n,i):c},i.extent=function(n){return arguments.length?(null==n?t=e=r=u=null:(t=+n[0][0],e=+n[0][1],r=+n[1][0],u=+n[1][1]),i):null==t?null:[[t,e],[r,u]]},i.size=function(n){return arguments.length?(null==n?t=e=r=u=null:(t=e=0,r=+n[0],u=+n[1]),i):null==t?null:[r-t,u-e]},i)},Zo.interpolateRgb=cu,Zo.interpolateObject=su,Zo.interpolateNumber=lu,Zo.interpolateString=fu;var Gc=/[-+]?(?:\d+\.?\d*|\.?\d+)(?:[eE][-+]?\d+)?/g,Kc=new RegExp(Gc.source,"g");Zo.interpolate=hu,Zo.interpolators=[function(n,t){var e=typeof t;return("string"===e?Ia.has(t)||/^(#|rgb\(|hsl\()/.test(t)?cu:fu:t instanceof et?cu:Array.isArray(t)?gu:"object"===e&&isNaN(t)?su:lu)(n,t)}],Zo.interpolateArray=gu;var Qc=function(){return wt},ns=Zo.map({linear:Qc,poly:Mu,quad:function(){return mu},cubic:function(){return yu},sin:function(){return _u},exp:function(){return bu},circle:function(){return wu},elastic:Su,back:ku,bounce:function(){return Eu}}),ts=Zo.map({"in":wt,out:vu,"in-out":du,"out-in":function(n){return du(vu(n))}});Zo.ease=function(n){var t=n.indexOf("-"),e=t>=0?n.substring(0,t):n,r=t>=0?n.substring(t+1):"in";return e=ns.get(e)||Qc,r=ts.get(r)||wt,pu(r(e.apply(null,Vo.call(arguments,1))))},Zo.interpolateHcl=Au,Zo.interpolateHsl=Cu,Zo.interpolateLab=Nu,Zo.interpolateRound=zu,Zo.transform=function(n){var t=$o.createElementNS(Zo.ns.prefix.svg,"g");return(Zo.transform=function(n){if(null!=n){t.setAttribute("transform",n);var e=t.transform.baseVal.consolidate()}return new Lu(e?e.matrix:es)})(n)},Lu.prototype.toString=function(){return"translate("+this.translate+")rotate("+this.rotate+")skewX("+this.skew+")scale("+this.scale+")"};var es={a:1,b:0,c:0,d:1,e:0,f:0};Zo.interpolateTransform=Du,Zo.layout={},Zo.layout.bundle=function(){return function(n){for(var t=[],e=-1,r=n.length;++e<r;)t.push(ju(n[e]));return t}},Zo.layout.chord=function(){function n(){var n,s,f,h,g,p={},v=[],d=Zo.range(i),m=[];for(e=[],r=[],n=0,h=-1;++h<i;){for(s=0,g=-1;++g<i;)s+=u[h][g];v.push(s),m.push(Zo.range(i)),n+=s}for(o&&d.sort(function(n,t){return o(v[n],v[t])}),a&&m.forEach(function(n,t){n.sort(function(n,e){return a(u[t][n],u[t][e])})}),n=(wa-l*i)/n,s=0,h=-1;++h<i;){for(f=s,g=-1;++g<i;){var y=d[h],x=m[y][g],M=u[y][x],_=s,b=s+=M*n;p[y+"-"+x]={index:y,subindex:x,startAngle:_,endAngle:b,value:M}}r[y]={index:y,startAngle:f,endAngle:s,value:(s-f)/n},s+=l}for(h=-1;++h<i;)for(g=h-1;++g<i;){var w=p[h+"-"+g],S=p[g+"-"+h];(w.value||S.value)&&e.push(w.value<S.value?{source:S,target:w}:{source:w,target:S})}c&&t()}function t(){e.sort(function(n,t){return c((n.source.value+n.target.value)/2,(t.source.value+t.target.value)/2)})}var e,r,u,i,o,a,c,s={},l=0;return s.matrix=function(n){return arguments.length?(i=(u=n)&&u.length,e=r=null,s):u},s.padding=function(n){return arguments.length?(l=n,e=r=null,s):l},s.sortGroups=function(n){return arguments.length?(o=n,e=r=null,s):o},s.sortSubgroups=function(n){return arguments.length?(a=n,e=null,s):a},s.sortChords=function(n){return arguments.length?(c=n,e&&t(),s):c},s.chords=function(){return e||n(),e},s.groups=function(){return r||n(),r},s},Zo.layout.force=function(){function n(n){return function(t,e,r,u){if(t.point!==n){var i=t.cx-n.x,o=t.cy-n.y,a=u-e,c=i*i+o*o;if(c>a*a/d){if(p>c){var s=t.charge/c;n.px-=i*s,n.py-=o*s}return!0}if(t.point&&c&&p>c){var s=t.pointCharge/c;n.px-=i*s,n.py-=o*s}}return!t.charge}}function t(n){n.px=Zo.event.x,n.py=Zo.event.y,a.resume()}var e,r,u,i,o,a={},c=Zo.dispatch("start","tick","end"),s=[1,1],l=.9,f=rs,h=us,g=-30,p=is,v=.1,d=.64,m=[],y=[];return a.tick=function(){if((r*=.99)<.005)return c.end({type:"end",alpha:r=0}),!0;var t,e,a,f,h,p,d,x,M,_=m.length,b=y.length;for(e=0;b>e;++e)a=y[e],f=a.source,h=a.target,x=h.x-f.x,M=h.y-f.y,(p=x*x+M*M)&&(p=r*i[e]*((p=Math.sqrt(p))-u[e])/p,x*=p,M*=p,h.x-=x*(d=f.weight/(h.weight+f.weight)),h.y-=M*d,f.x+=x*(d=1-d),f.y+=M*d);if((d=r*v)&&(x=s[0]/2,M=s[1]/2,e=-1,d))for(;++e<_;)a=m[e],a.x+=(x-a.x)*d,a.y+=(M-a.y)*d;if(g)for(Vu(t=Zo.geom.quadtree(m),r,o),e=-1;++e<_;)(a=m[e]).fixed||t.visit(n(a));for(e=-1;++e<_;)a=m[e],a.fixed?(a.x=a.px,a.y=a.py):(a.x-=(a.px-(a.px=a.x))*l,a.y-=(a.py-(a.py=a.y))*l);c.tick({type:"tick",alpha:r})},a.nodes=function(n){return arguments.length?(m=n,a):m},a.links=function(n){return arguments.length?(y=n,a):y},a.size=function(n){return arguments.length?(s=n,a):s},a.linkDistance=function(n){return arguments.length?(f="function"==typeof n?n:+n,a):f},a.distance=a.linkDistance,a.linkStrength=function(n){return arguments.length?(h="function"==typeof n?n:+n,a):h},a.friction=function(n){return arguments.length?(l=+n,a):l},a.charge=function(n){return arguments.length?(g="function"==typeof n?n:+n,a):g},a.chargeDistance=function(n){return arguments.length?(p=n*n,a):Math.sqrt(p)},a.gravity=function(n){return arguments.length?(v=+n,a):v},a.theta=function(n){return arguments.length?(d=n*n,a):Math.sqrt(d)},a.alpha=function(n){return arguments.length?(n=+n,r?r=n>0?n:0:n>0&&(c.start({type:"start",alpha:r=n}),Zo.timer(a.tick)),a):r},a.start=function(){function n(n,r){if(!e){for(e=new Array(c),a=0;c>a;++a)e[a]=[];for(a=0;s>a;++a){var u=y[a];e[u.source.index].push(u.target),e[u.target.index].push(u.source)}}for(var i,o=e[t],a=-1,s=o.length;++a<s;)if(!isNaN(i=o[a][n]))return i;return Math.random()*r}var t,e,r,c=m.length,l=y.length,p=s[0],v=s[1];for(t=0;c>t;++t)(r=m[t]).index=t,r.weight=0;for(t=0;l>t;++t)r=y[t],"number"==typeof r.source&&(r.source=m[r.source]),"number"==typeof r.target&&(r.target=m[r.target]),++r.source.weight,++r.target.weight;for(t=0;c>t;++t)r=m[t],isNaN(r.x)&&(r.x=n("x",p)),isNaN(r.y)&&(r.y=n("y",v)),isNaN(r.px)&&(r.px=r.x),isNaN(r.py)&&(r.py=r.y);if(u=[],"function"==typeof f)for(t=0;l>t;++t)u[t]=+f.call(this,y[t],t);else for(t=0;l>t;++t)u[t]=f;if(i=[],"function"==typeof h)for(t=0;l>t;++t)i[t]=+h.call(this,y[t],t);else for(t=0;l>t;++t)i[t]=h;if(o=[],"function"==typeof g)for(t=0;c>t;++t)o[t]=+g.call(this,m[t],t);else for(t=0;c>t;++t)o[t]=g;return a.resume()},a.resume=function(){return a.alpha(.1)},a.stop=function(){return a.alpha(0)},a.drag=function(){return e||(e=Zo.behavior.drag().origin(wt).on("dragstart.force",Ou).on("drag.force",t).on("dragend.force",Yu)),arguments.length?(this.on("mouseover.force",Iu).on("mouseout.force",Zu).call(e),void 0):e},Zo.rebind(a,c,"on")};var rs=20,us=1,is=1/0;Zo.layout.hierarchy=function(){function n(u){var i,o=[u],a=[];for(u.depth=0;null!=(i=o.pop());)if(a.push(i),(s=e.call(n,i,i.depth))&&(c=s.length)){for(var c,s,l;--c>=0;)o.push(l=s[c]),l.parent=i,l.depth=i.depth+1;r&&(i.value=0),i.children=s}else r&&(i.value=+r.call(n,i,i.depth)||0),delete i.children;return Bu(u,function(n){var e,u;t&&(e=n.children)&&e.sort(t),r&&(u=n.parent)&&(u.value+=n.value)}),a}var t=Gu,e=Wu,r=Ju;return n.sort=function(e){return arguments.length?(t=e,n):t},n.children=function(t){return arguments.length?(e=t,n):e},n.value=function(t){return arguments.length?(r=t,n):r},n.revalue=function(t){return r&&($u(t,function(n){n.children&&(n.value=0)}),Bu(t,function(t){var e;t.children||(t.value=+r.call(n,t,t.depth)||0),(e=t.parent)&&(e.value+=t.value)})),t},n},Zo.layout.partition=function(){function n(t,e,r,u){var i=t.children;if(t.x=e,t.y=t.depth*u,t.dx=r,t.dy=u,i&&(o=i.length)){var o,a,c,s=-1;for(r=t.value?r/t.value:0;++s<o;)n(a=i[s],e,c=a.value*r,u),e+=c}}function t(n){var e=n.children,r=0;if(e&&(u=e.length))for(var u,i=-1;++i<u;)r=Math.max(r,t(e[i]));return 1+r}function e(e,i){var o=r.call(this,e,i);return n(o[0],0,u[0],u[1]/t(o[0])),o}var r=Zo.layout.hierarchy(),u=[1,1];return e.size=function(n){return arguments.length?(u=n,e):u},Xu(e,r)},Zo.layout.pie=function(){function n(i){var o=i.map(function(e,r){return+t.call(n,e,r)}),a=+("function"==typeof r?r.apply(this,arguments):r),c=(("function"==typeof u?u.apply(this,arguments):u)-a)/Zo.sum(o),s=Zo.range(i.length);null!=e&&s.sort(e===os?function(n,t){return o[t]-o[n]}:function(n,t){return e(i[n],i[t])});var l=[];return s.forEach(function(n){var t;l[n]={data:i[n],value:t=o[n],startAngle:a,endAngle:a+=t*c}}),l}var t=Number,e=os,r=0,u=wa;return n.value=function(e){return arguments.length?(t=e,n):t},n.sort=function(t){return arguments.length?(e=t,n):e},n.startAngle=function(t){return arguments.length?(r=t,n):r},n.endAngle=function(t){return arguments.length?(u=t,n):u},n};var os={};Zo.layout.stack=function(){function n(a,c){var s=a.map(function(e,r){return t.call(n,e,r)}),l=s.map(function(t){return t.map(function(t,e){return[i.call(n,t,e),o.call(n,t,e)]})}),f=e.call(n,l,c);s=Zo.permute(s,f),l=Zo.permute(l,f);var h,g,p,v=r.call(n,l,c),d=s.length,m=s[0].length;for(g=0;m>g;++g)for(u.call(n,s[0][g],p=v[g],l[0][g][1]),h=1;d>h;++h)u.call(n,s[h][g],p+=l[h-1][g][1],l[h][g][1]);return a}var t=wt,e=ei,r=ri,u=ti,i=Qu,o=ni;return n.values=function(e){return arguments.length?(t=e,n):t},n.order=function(t){return arguments.length?(e="function"==typeof t?t:as.get(t)||ei,n):e},n.offset=function(t){return arguments.length?(r="function"==typeof t?t:cs.get(t)||ri,n):r},n.x=function(t){return arguments.length?(i=t,n):i},n.y=function(t){return arguments.length?(o=t,n):o},n.out=function(t){return arguments.length?(u=t,n):u},n};var as=Zo.map({"inside-out":function(n){var t,e,r=n.length,u=n.map(ui),i=n.map(ii),o=Zo.range(r).sort(function(n,t){return u[n]-u[t]}),a=0,c=0,s=[],l=[];for(t=0;r>t;++t)e=o[t],c>a?(a+=i[e],s.push(e)):(c+=i[e],l.push(e));return l.reverse().concat(s)},reverse:function(n){return Zo.range(n.length).reverse()},"default":ei}),cs=Zo.map({silhouette:function(n){var t,e,r,u=n.length,i=n[0].length,o=[],a=0,c=[];for(e=0;i>e;++e){for(t=0,r=0;u>t;t++)r+=n[t][e][1];r>a&&(a=r),o.push(r)}for(e=0;i>e;++e)c[e]=(a-o[e])/2;return c},wiggle:function(n){var t,e,r,u,i,o,a,c,s,l=n.length,f=n[0],h=f.length,g=[];for(g[0]=c=s=0,e=1;h>e;++e){for(t=0,u=0;l>t;++t)u+=n[t][e][1];for(t=0,i=0,a=f[e][0]-f[e-1][0];l>t;++t){for(r=0,o=(n[t][e][1]-n[t][e-1][1])/(2*a);t>r;++r)o+=(n[r][e][1]-n[r][e-1][1])/a;i+=o*n[t][e][1]}g[e]=c-=u?i/u*a:0,s>c&&(s=c)}for(e=0;h>e;++e)g[e]-=s;return g},expand:function(n){var t,e,r,u=n.length,i=n[0].length,o=1/u,a=[];for(e=0;i>e;++e){for(t=0,r=0;u>t;t++)r+=n[t][e][1];if(r)for(t=0;u>t;t++)n[t][e][1]/=r;else for(t=0;u>t;t++)n[t][e][1]=o}for(e=0;i>e;++e)a[e]=0;return a},zero:ri});Zo.layout.histogram=function(){function n(n,i){for(var o,a,c=[],s=n.map(e,this),l=r.call(this,s,i),f=u.call(this,l,s,i),i=-1,h=s.length,g=f.length-1,p=t?1:1/h;++i<g;)o=c[i]=[],o.dx=f[i+1]-(o.x=f[i]),o.y=0;if(g>0)for(i=-1;++i<h;)a=s[i],a>=l[0]&&a<=l[1]&&(o=c[Zo.bisect(f,a,1,g)-1],o.y+=p,o.push(n[i]));return c}var t=!0,e=Number,r=si,u=ai;return n.value=function(t){return arguments.length?(e=t,n):e},n.range=function(t){return arguments.length?(r=bt(t),n):r},n.bins=function(t){return arguments.length?(u="number"==typeof t?function(n){return ci(n,t)}:bt(t),n):u},n.frequency=function(e){return arguments.length?(t=!!e,n):t},n},Zo.layout.pack=function(){function n(n,i){var o=e.call(this,n,i),a=o[0],c=u[0],s=u[1],l=null==t?Math.sqrt:"function"==typeof t?t:function(){return t};if(a.x=a.y=0,Bu(a,function(n){n.r=+l(n.value)}),Bu(a,pi),r){var f=r*(t?1:Math.max(2*a.r/c,2*a.r/s))/2;Bu(a,function(n){n.r+=f}),Bu(a,pi),Bu(a,function(n){n.r-=f})}return mi(a,c/2,s/2,t?1:1/Math.max(2*a.r/c,2*a.r/s)),o}var t,e=Zo.layout.hierarchy().sort(li),r=0,u=[1,1];return n.size=function(t){return arguments.length?(u=t,n):u},n.radius=function(e){return arguments.length?(t=null==e||"function"==typeof e?e:+e,n):t},n.padding=function(t){return arguments.length?(r=+t,n):r},Xu(n,e)},Zo.layout.tree=function(){function n(n,u){var l=o.call(this,n,u),f=l[0],h=t(f);if(Bu(h,e),h.parent.m=-h.z,$u(h,r),s)$u(f,i);else{var g=f,p=f,v=f;$u(f,function(n){n.x<g.x&&(g=n),n.x>p.x&&(p=n),n.depth>v.depth&&(v=n)});var d=a(g,p)/2-g.x,m=c[0]/(p.x+a(p,g)/2+d),y=c[1]/(v.depth||1);$u(f,function(n){n.x=(n.x+d)*m,n.y=n.depth*y})}return l}function t(n){for(var t,e={A:null,children:[n]},r=[e];null!=(t=r.pop());)for(var u,i=t.children,o=0,a=i.length;a>o;++o)r.push((i[o]=u={_:i[o],parent:t,children:(u=i[o].children)&&u.slice()||[],A:null,a:null,z:0,m:0,c:0,s:0,t:null,i:o}).a=u);return e.children[0]}function e(n){var t=n.children,e=n.parent.children,r=n.i?e[n.i-1]:null;if(t.length){wi(n);var i=(t[0].z+t[t.length-1].z)/2;r?(n.z=r.z+a(n._,r._),n.m=n.z-i):n.z=i}else r&&(n.z=r.z+a(n._,r._));n.parent.A=u(n,r,n.parent.A||e[0])}function r(n){n._.x=n.z+n.parent.m,n.m+=n.parent.m}function u(n,t,e){if(t){for(var r,u=n,i=n,o=t,c=u.parent.children[0],s=u.m,l=i.m,f=o.m,h=c.m;o=_i(o),u=Mi(u),o&&u;)c=Mi(c),i=_i(i),i.a=n,r=o.z+f-u.z-s+a(o._,u._),r>0&&(bi(Si(o,n,e),n,r),s+=r,l+=r),f+=o.m,s+=u.m,h+=c.m,l+=i.m;o&&!_i(i)&&(i.t=o,i.m+=f-l),u&&!Mi(c)&&(c.t=u,c.m+=s-h,e=n)}return e}function i(n){n.x*=c[0],n.y=n.depth*c[1]}var o=Zo.layout.hierarchy().sort(null).value(null),a=xi,c=[1,1],s=null;return n.separation=function(t){return arguments.length?(a=t,n):a},n.size=function(t){return arguments.length?(s=null==(c=t)?i:null,n):s?null:c},n.nodeSize=function(t){return arguments.length?(s=null==(c=t)?null:i,n):s?c:null},Xu(n,o)},Zo.layout.cluster=function(){function n(n,i){var o,a=t.call(this,n,i),c=a[0],s=0;Bu(c,function(n){var t=n.children;t&&t.length?(n.x=Ei(t),n.y=ki(t)):(n.x=o?s+=e(n,o):0,n.y=0,o=n)});var l=Ai(c),f=Ci(c),h=l.x-e(l,f)/2,g=f.x+e(f,l)/2;return Bu(c,u?function(n){n.x=(n.x-c.x)*r[0],n.y=(c.y-n.y)*r[1]}:function(n){n.x=(n.x-h)/(g-h)*r[0],n.y=(1-(c.y?n.y/c.y:1))*r[1]}),a}var t=Zo.layout.hierarchy().sort(null).value(null),e=xi,r=[1,1],u=!1;return n.separation=function(t){return arguments.length?(e=t,n):e},n.size=function(t){return arguments.length?(u=null==(r=t),n):u?null:r},n.nodeSize=function(t){return arguments.length?(u=null!=(r=t),n):u?r:null},Xu(n,t)},Zo.layout.treemap=function(){function n(n,t){for(var e,r,u=-1,i=n.length;++u<i;)r=(e=n[u]).value*(0>t?0:t),e.area=isNaN(r)||0>=r?0:r}function t(e){var i=e.children;if(i&&i.length){var o,a,c,s=f(e),l=[],h=i.slice(),p=1/0,v="slice"===g?s.dx:"dice"===g?s.dy:"slice-dice"===g?1&e.depth?s.dy:s.dx:Math.min(s.dx,s.dy);for(n(h,s.dx*s.dy/e.value),l.area=0;(c=h.length)>0;)l.push(o=h[c-1]),l.area+=o.area,"squarify"!==g||(a=r(l,v))<=p?(h.pop(),p=a):(l.area-=l.pop().area,u(l,v,s,!1),v=Math.min(s.dx,s.dy),l.length=l.area=0,p=1/0);l.length&&(u(l,v,s,!0),l.length=l.area=0),i.forEach(t)}}function e(t){var r=t.children;if(r&&r.length){var i,o=f(t),a=r.slice(),c=[];for(n(a,o.dx*o.dy/t.value),c.area=0;i=a.pop();)c.push(i),c.area+=i.area,null!=i.z&&(u(c,i.z?o.dx:o.dy,o,!a.length),c.length=c.area=0);r.forEach(e)}}function r(n,t){for(var e,r=n.area,u=0,i=1/0,o=-1,a=n.length;++o<a;)(e=n[o].area)&&(i>e&&(i=e),e>u&&(u=e));return r*=r,t*=t,r?Math.max(t*u*p/r,r/(t*i*p)):1/0}function u(n,t,e,r){var u,i=-1,o=n.length,a=e.x,s=e.y,l=t?c(n.area/t):0;if(t==e.dx){for((r||l>e.dy)&&(l=e.dy);++i<o;)u=n[i],u.x=a,u.y=s,u.dy=l,a+=u.dx=Math.min(e.x+e.dx-a,l?c(u.area/l):0);u.z=!0,u.dx+=e.x+e.dx-a,e.y+=l,e.dy-=l}else{for((r||l>e.dx)&&(l=e.dx);++i<o;)u=n[i],u.x=a,u.y=s,u.dx=l,s+=u.dy=Math.min(e.y+e.dy-s,l?c(u.area/l):0);u.z=!1,u.dy+=e.y+e.dy-s,e.x+=l,e.dx-=l}}function i(r){var u=o||a(r),i=u[0];return i.x=0,i.y=0,i.dx=s[0],i.dy=s[1],o&&a.revalue(i),n([i],i.dx*i.dy/i.value),(o?e:t)(i),h&&(o=u),u}var o,a=Zo.layout.hierarchy(),c=Math.round,s=[1,1],l=null,f=Ni,h=!1,g="squarify",p=.5*(1+Math.sqrt(5));return i.size=function(n){return arguments.length?(s=n,i):s},i.padding=function(n){function t(t){var e=n.call(i,t,t.depth);return null==e?Ni(t):zi(t,"number"==typeof e?[e,e,e,e]:e)}function e(t){return zi(t,n)}if(!arguments.length)return l;var r;return f=null==(l=n)?Ni:"function"==(r=typeof n)?t:"number"===r?(n=[n,n,n,n],e):e,i},i.round=function(n){return arguments.length?(c=n?Math.round:Number,i):c!=Number},i.sticky=function(n){return arguments.length?(h=n,o=null,i):h},i.ratio=function(n){return arguments.length?(p=n,i):p},i.mode=function(n){return arguments.length?(g=n+"",i):g},Xu(i,a)},Zo.random={normal:function(n,t){var e=arguments.length;return 2>e&&(t=1),1>e&&(n=0),function(){var e,r,u;do e=2*Math.random()-1,r=2*Math.random()-1,u=e*e+r*r;while(!u||u>1);return n+t*e*Math.sqrt(-2*Math.log(u)/u)}},logNormal:function(){var n=Zo.random.normal.apply(Zo,arguments);return function(){return Math.exp(n())}},bates:function(n){var t=Zo.random.irwinHall(n);return function(){return t()/n}},irwinHall:function(n){return function(){for(var t=0,e=0;n>e;e++)t+=Math.random();return t}}},Zo.scale={};var ss={floor:wt,ceil:wt};Zo.scale.linear=function(){return Ui([0,1],[0,1],hu,!1)};var ls={s:1,g:1,p:1,r:1,e:1};Zo.scale.log=function(){return Vi(Zo.scale.linear().domain([0,1]),10,!0,[1,10])};var fs=Zo.format(".0e"),hs={floor:function(n){return-Math.ceil(-n)},ceil:function(n){return-Math.floor(-n)}};Zo.scale.pow=function(){return Xi(Zo.scale.linear(),1,[0,1])},Zo.scale.sqrt=function(){return Zo.scale.pow().exponent(.5)},Zo.scale.ordinal=function(){return Bi([],{t:"range",a:[[]]})},Zo.scale.category10=function(){return Zo.scale.ordinal().range(gs)},Zo.scale.category20=function(){return Zo.scale.ordinal().range(ps)},Zo.scale.category20b=function(){return Zo.scale.ordinal().range(vs)},Zo.scale.category20c=function(){return Zo.scale.ordinal().range(ds)};var gs=[2062260,16744206,2924588,14034728,9725885,9197131,14907330,8355711,12369186,1556175].map(vt),ps=[2062260,11454440,16744206,16759672,2924588,10018698,14034728,16750742,9725885,12955861,9197131,12885140,14907330,16234194,8355711,13092807,12369186,14408589,1556175,10410725].map(vt),vs=[3750777,5395619,7040719,10264286,6519097,9216594,11915115,13556636,9202993,12426809,15186514,15190932,8666169,11356490,14049643,15177372,8077683,10834324,13528509,14589654].map(vt),ds=[3244733,7057110,10406625,13032431,15095053,16616764,16625259,16634018,3253076,7652470,10607003,13101504,7695281,10394312,12369372,14342891,6513507,9868950,12434877,14277081].map(vt);Zo.scale.quantile=function(){return Wi([],[])},Zo.scale.quantize=function(){return Ji(0,1,[0,1])},Zo.scale.threshold=function(){return Gi([.5],[0,1])},Zo.scale.identity=function(){return Ki([0,1])},Zo.svg={},Zo.svg.arc=function(){function n(){var n=t.apply(this,arguments),i=e.apply(this,arguments),o=r.apply(this,arguments)+ms,a=u.apply(this,arguments)+ms,c=(o>a&&(c=o,o=a,a=c),a-o),s=ba>c?"0":"1",l=Math.cos(o),f=Math.sin(o),h=Math.cos(a),g=Math.sin(a);
+return c>=ys?n?"M0,"+i+"A"+i+","+i+" 0 1,1 0,"+-i+"A"+i+","+i+" 0 1,1 0,"+i+"M0,"+n+"A"+n+","+n+" 0 1,0 0,"+-n+"A"+n+","+n+" 0 1,0 0,"+n+"Z":"M0,"+i+"A"+i+","+i+" 0 1,1 0,"+-i+"A"+i+","+i+" 0 1,1 0,"+i+"Z":n?"M"+i*l+","+i*f+"A"+i+","+i+" 0 "+s+",1 "+i*h+","+i*g+"L"+n*h+","+n*g+"A"+n+","+n+" 0 "+s+",0 "+n*l+","+n*f+"Z":"M"+i*l+","+i*f+"A"+i+","+i+" 0 "+s+",1 "+i*h+","+i*g+"L0,0"+"Z"}var t=Qi,e=no,r=to,u=eo;return n.innerRadius=function(e){return arguments.length?(t=bt(e),n):t},n.outerRadius=function(t){return arguments.length?(e=bt(t),n):e},n.startAngle=function(t){return arguments.length?(r=bt(t),n):r},n.endAngle=function(t){return arguments.length?(u=bt(t),n):u},n.centroid=function(){var n=(t.apply(this,arguments)+e.apply(this,arguments))/2,i=(r.apply(this,arguments)+u.apply(this,arguments))/2+ms;return[Math.cos(i)*n,Math.sin(i)*n]},n};var ms=-Sa,ys=wa-ka;Zo.svg.line=function(){return ro(wt)};var xs=Zo.map({linear:uo,"linear-closed":io,step:oo,"step-before":ao,"step-after":co,basis:po,"basis-open":vo,"basis-closed":mo,bundle:yo,cardinal:fo,"cardinal-open":so,"cardinal-closed":lo,monotone:So});xs.forEach(function(n,t){t.key=n,t.closed=/-closed$/.test(n)});var Ms=[0,2/3,1/3,0],_s=[0,1/3,2/3,0],bs=[0,1/6,2/3,1/6];Zo.svg.line.radial=function(){var n=ro(ko);return n.radius=n.x,delete n.x,n.angle=n.y,delete n.y,n},ao.reverse=co,co.reverse=ao,Zo.svg.area=function(){return Eo(wt)},Zo.svg.area.radial=function(){var n=Eo(ko);return n.radius=n.x,delete n.x,n.innerRadius=n.x0,delete n.x0,n.outerRadius=n.x1,delete n.x1,n.angle=n.y,delete n.y,n.startAngle=n.y0,delete n.y0,n.endAngle=n.y1,delete n.y1,n},Zo.svg.chord=function(){function n(n,a){var c=t(this,i,n,a),s=t(this,o,n,a);return"M"+c.p0+r(c.r,c.p1,c.a1-c.a0)+(e(c,s)?u(c.r,c.p1,c.r,c.p0):u(c.r,c.p1,s.r,s.p0)+r(s.r,s.p1,s.a1-s.a0)+u(s.r,s.p1,c.r,c.p0))+"Z"}function t(n,t,e,r){var u=t.call(n,e,r),i=a.call(n,u,r),o=c.call(n,u,r)+ms,l=s.call(n,u,r)+ms;return{r:i,a0:o,a1:l,p0:[i*Math.cos(o),i*Math.sin(o)],p1:[i*Math.cos(l),i*Math.sin(l)]}}function e(n,t){return n.a0==t.a0&&n.a1==t.a1}function r(n,t,e){return"A"+n+","+n+" 0 "+ +(e>ba)+",1 "+t}function u(n,t,e,r){return"Q 0,0 "+r}var i=gr,o=pr,a=Ao,c=to,s=eo;return n.radius=function(t){return arguments.length?(a=bt(t),n):a},n.source=function(t){return arguments.length?(i=bt(t),n):i},n.target=function(t){return arguments.length?(o=bt(t),n):o},n.startAngle=function(t){return arguments.length?(c=bt(t),n):c},n.endAngle=function(t){return arguments.length?(s=bt(t),n):s},n},Zo.svg.diagonal=function(){function n(n,u){var i=t.call(this,n,u),o=e.call(this,n,u),a=(i.y+o.y)/2,c=[i,{x:i.x,y:a},{x:o.x,y:a},o];return c=c.map(r),"M"+c[0]+"C"+c[1]+" "+c[2]+" "+c[3]}var t=gr,e=pr,r=Co;return n.source=function(e){return arguments.length?(t=bt(e),n):t},n.target=function(t){return arguments.length?(e=bt(t),n):e},n.projection=function(t){return arguments.length?(r=t,n):r},n},Zo.svg.diagonal.radial=function(){var n=Zo.svg.diagonal(),t=Co,e=n.projection;return n.projection=function(n){return arguments.length?e(No(t=n)):t},n},Zo.svg.symbol=function(){function n(n,r){return(ws.get(t.call(this,n,r))||To)(e.call(this,n,r))}var t=Lo,e=zo;return n.type=function(e){return arguments.length?(t=bt(e),n):t},n.size=function(t){return arguments.length?(e=bt(t),n):e},n};var ws=Zo.map({circle:To,cross:function(n){var t=Math.sqrt(n/5)/2;return"M"+-3*t+","+-t+"H"+-t+"V"+-3*t+"H"+t+"V"+-t+"H"+3*t+"V"+t+"H"+t+"V"+3*t+"H"+-t+"V"+t+"H"+-3*t+"Z"},diamond:function(n){var t=Math.sqrt(n/(2*As)),e=t*As;return"M0,"+-t+"L"+e+",0"+" 0,"+t+" "+-e+",0"+"Z"},square:function(n){var t=Math.sqrt(n)/2;return"M"+-t+","+-t+"L"+t+","+-t+" "+t+","+t+" "+-t+","+t+"Z"},"triangle-down":function(n){var t=Math.sqrt(n/Es),e=t*Es/2;return"M0,"+e+"L"+t+","+-e+" "+-t+","+-e+"Z"},"triangle-up":function(n){var t=Math.sqrt(n/Es),e=t*Es/2;return"M0,"+-e+"L"+t+","+e+" "+-t+","+e+"Z"}});Zo.svg.symbolTypes=ws.keys();var Ss,ks,Es=Math.sqrt(3),As=Math.tan(30*Aa),Cs=[],Ns=0;Cs.call=pa.call,Cs.empty=pa.empty,Cs.node=pa.node,Cs.size=pa.size,Zo.transition=function(n){return arguments.length?Ss?n.transition():n:ma.transition()},Zo.transition.prototype=Cs,Cs.select=function(n){var t,e,r,u=this.id,i=[];n=b(n);for(var o=-1,a=this.length;++o<a;){i.push(t=[]);for(var c=this[o],s=-1,l=c.length;++s<l;)(r=c[s])&&(e=n.call(r,r.__data__,s,o))?("__data__"in r&&(e.__data__=r.__data__),Po(e,s,u,r.__transition__[u]),t.push(e)):t.push(null)}return qo(i,u)},Cs.selectAll=function(n){var t,e,r,u,i,o=this.id,a=[];n=w(n);for(var c=-1,s=this.length;++c<s;)for(var l=this[c],f=-1,h=l.length;++f<h;)if(r=l[f]){i=r.__transition__[o],e=n.call(r,r.__data__,f,c),a.push(t=[]);for(var g=-1,p=e.length;++g<p;)(u=e[g])&&Po(u,g,o,i),t.push(u)}return qo(a,o)},Cs.filter=function(n){var t,e,r,u=[];"function"!=typeof n&&(n=R(n));for(var i=0,o=this.length;o>i;i++){u.push(t=[]);for(var e=this[i],a=0,c=e.length;c>a;a++)(r=e[a])&&n.call(r,r.__data__,a,i)&&t.push(r)}return qo(u,this.id)},Cs.tween=function(n,t){var e=this.id;return arguments.length<2?this.node().__transition__[e].tween.get(n):P(this,null==t?function(t){t.__transition__[e].tween.remove(n)}:function(r){r.__transition__[e].tween.set(n,t)})},Cs.attr=function(n,t){function e(){this.removeAttribute(a)}function r(){this.removeAttributeNS(a.space,a.local)}function u(n){return null==n?e:(n+="",function(){var t,e=this.getAttribute(a);return e!==n&&(t=o(e,n),function(n){this.setAttribute(a,t(n))})})}function i(n){return null==n?r:(n+="",function(){var t,e=this.getAttributeNS(a.space,a.local);return e!==n&&(t=o(e,n),function(n){this.setAttributeNS(a.space,a.local,t(n))})})}if(arguments.length<2){for(t in n)this.attr(t,n[t]);return this}var o="transform"==n?Du:hu,a=Zo.ns.qualify(n);return Ro(this,"attr."+n,t,a.local?i:u)},Cs.attrTween=function(n,t){function e(n,e){var r=t.call(this,n,e,this.getAttribute(u));return r&&function(n){this.setAttribute(u,r(n))}}function r(n,e){var r=t.call(this,n,e,this.getAttributeNS(u.space,u.local));return r&&function(n){this.setAttributeNS(u.space,u.local,r(n))}}var u=Zo.ns.qualify(n);return this.tween("attr."+n,u.local?r:e)},Cs.style=function(n,t,e){function r(){this.style.removeProperty(n)}function u(t){return null==t?r:(t+="",function(){var r,u=Wo.getComputedStyle(this,null).getPropertyValue(n);return u!==t&&(r=hu(u,t),function(t){this.style.setProperty(n,r(t),e)})})}var i=arguments.length;if(3>i){if("string"!=typeof n){2>i&&(t="");for(e in n)this.style(e,n[e],t);return this}e=""}return Ro(this,"style."+n,t,u)},Cs.styleTween=function(n,t,e){function r(r,u){var i=t.call(this,r,u,Wo.getComputedStyle(this,null).getPropertyValue(n));return i&&function(t){this.style.setProperty(n,i(t),e)}}return arguments.length<3&&(e=""),this.tween("style."+n,r)},Cs.text=function(n){return Ro(this,"text",n,Do)},Cs.remove=function(){return this.each("end.transition",function(){var n;this.__transition__.count<2&&(n=this.parentNode)&&n.removeChild(this)})},Cs.ease=function(n){var t=this.id;return arguments.length<1?this.node().__transition__[t].ease:("function"!=typeof n&&(n=Zo.ease.apply(Zo,arguments)),P(this,function(e){e.__transition__[t].ease=n}))},Cs.delay=function(n){var t=this.id;return arguments.length<1?this.node().__transition__[t].delay:P(this,"function"==typeof n?function(e,r,u){e.__transition__[t].delay=+n.call(e,e.__data__,r,u)}:(n=+n,function(e){e.__transition__[t].delay=n}))},Cs.duration=function(n){var t=this.id;return arguments.length<1?this.node().__transition__[t].duration:P(this,"function"==typeof n?function(e,r,u){e.__transition__[t].duration=Math.max(1,n.call(e,e.__data__,r,u))}:(n=Math.max(1,n),function(e){e.__transition__[t].duration=n}))},Cs.each=function(n,t){var e=this.id;if(arguments.length<2){var r=ks,u=Ss;Ss=e,P(this,function(t,r,u){ks=t.__transition__[e],n.call(t,t.__data__,r,u)}),ks=r,Ss=u}else P(this,function(r){var u=r.__transition__[e];(u.event||(u.event=Zo.dispatch("start","end"))).on(n,t)});return this},Cs.transition=function(){for(var n,t,e,r,u=this.id,i=++Ns,o=[],a=0,c=this.length;c>a;a++){o.push(n=[]);for(var t=this[a],s=0,l=t.length;l>s;s++)(e=t[s])&&(r=Object.create(e.__transition__[u]),r.delay+=r.duration,Po(e,s,i,r)),n.push(e)}return qo(o,i)},Zo.svg.axis=function(){function n(n){n.each(function(){var n,s=Zo.select(this),l=this.__chart__||e,f=this.__chart__=e.copy(),h=null==c?f.ticks?f.ticks.apply(f,a):f.domain():c,g=null==t?f.tickFormat?f.tickFormat.apply(f,a):wt:t,p=s.selectAll(".tick").data(h,f),v=p.enter().insert("g",".domain").attr("class","tick").style("opacity",ka),d=Zo.transition(p.exit()).style("opacity",ka).remove(),m=Zo.transition(p.order()).style("opacity",1),y=Ti(f),x=s.selectAll(".domain").data([0]),M=(x.enter().append("path").attr("class","domain"),Zo.transition(x));v.append("line"),v.append("text");var _=v.select("line"),b=m.select("line"),w=p.select("text").text(g),S=v.select("text"),k=m.select("text");switch(r){case"bottom":n=Uo,_.attr("y2",u),S.attr("y",Math.max(u,0)+o),b.attr("x2",0).attr("y2",u),k.attr("x",0).attr("y",Math.max(u,0)+o),w.attr("dy",".71em").style("text-anchor","middle"),M.attr("d","M"+y[0]+","+i+"V0H"+y[1]+"V"+i);break;case"top":n=Uo,_.attr("y2",-u),S.attr("y",-(Math.max(u,0)+o)),b.attr("x2",0).attr("y2",-u),k.attr("x",0).attr("y",-(Math.max(u,0)+o)),w.attr("dy","0em").style("text-anchor","middle"),M.attr("d","M"+y[0]+","+-i+"V0H"+y[1]+"V"+-i);break;case"left":n=jo,_.attr("x2",-u),S.attr("x",-(Math.max(u,0)+o)),b.attr("x2",-u).attr("y2",0),k.attr("x",-(Math.max(u,0)+o)).attr("y",0),w.attr("dy",".32em").style("text-anchor","end"),M.attr("d","M"+-i+","+y[0]+"H0V"+y[1]+"H"+-i);break;case"right":n=jo,_.attr("x2",u),S.attr("x",Math.max(u,0)+o),b.attr("x2",u).attr("y2",0),k.attr("x",Math.max(u,0)+o).attr("y",0),w.attr("dy",".32em").style("text-anchor","start"),M.attr("d","M"+i+","+y[0]+"H0V"+y[1]+"H"+i)}if(f.rangeBand){var E=f,A=E.rangeBand()/2;l=f=function(n){return E(n)+A}}else l.rangeBand?l=f:d.call(n,f);v.call(n,l),m.call(n,f)})}var t,e=Zo.scale.linear(),r=zs,u=6,i=6,o=3,a=[10],c=null;return n.scale=function(t){return arguments.length?(e=t,n):e},n.orient=function(t){return arguments.length?(r=t in Ls?t+"":zs,n):r},n.ticks=function(){return arguments.length?(a=arguments,n):a},n.tickValues=function(t){return arguments.length?(c=t,n):c},n.tickFormat=function(e){return arguments.length?(t=e,n):t},n.tickSize=function(t){var e=arguments.length;return e?(u=+t,i=+arguments[e-1],n):u},n.innerTickSize=function(t){return arguments.length?(u=+t,n):u},n.outerTickSize=function(t){return arguments.length?(i=+t,n):i},n.tickPadding=function(t){return arguments.length?(o=+t,n):o},n.tickSubdivide=function(){return arguments.length&&n},n};var zs="bottom",Ls={top:1,right:1,bottom:1,left:1};Zo.svg.brush=function(){function n(i){i.each(function(){var i=Zo.select(this).style("pointer-events","all").style("-webkit-tap-highlight-color","rgba(0,0,0,0)").on("mousedown.brush",u).on("touchstart.brush",u),o=i.selectAll(".background").data([0]);o.enter().append("rect").attr("class","background").style("visibility","hidden").style("cursor","crosshair"),i.selectAll(".extent").data([0]).enter().append("rect").attr("class","extent").style("cursor","move");var a=i.selectAll(".resize").data(p,wt);a.exit().remove(),a.enter().append("g").attr("class",function(n){return"resize "+n}).style("cursor",function(n){return Ts[n]}).append("rect").attr("x",function(n){return/[ew]$/.test(n)?-3:null}).attr("y",function(n){return/^[ns]/.test(n)?-3:null}).attr("width",6).attr("height",6).style("visibility","hidden"),a.style("display",n.empty()?"none":null);var l,f=Zo.transition(i),h=Zo.transition(o);c&&(l=Ti(c),h.attr("x",l[0]).attr("width",l[1]-l[0]),e(f)),s&&(l=Ti(s),h.attr("y",l[0]).attr("height",l[1]-l[0]),r(f)),t(f)})}function t(n){n.selectAll(".resize").attr("transform",function(n){return"translate("+l[+/e$/.test(n)]+","+f[+/^s/.test(n)]+")"})}function e(n){n.select(".extent").attr("x",l[0]),n.selectAll(".extent,.n>rect,.s>rect").attr("width",l[1]-l[0])}function r(n){n.select(".extent").attr("y",f[0]),n.selectAll(".extent,.e>rect,.w>rect").attr("height",f[1]-f[0])}function u(){function u(){32==Zo.event.keyCode&&(C||(x=null,z[0]-=l[1],z[1]-=f[1],C=2),y())}function p(){32==Zo.event.keyCode&&2==C&&(z[0]+=l[1],z[1]+=f[1],C=0,y())}function v(){var n=Zo.mouse(_),u=!1;M&&(n[0]+=M[0],n[1]+=M[1]),C||(Zo.event.altKey?(x||(x=[(l[0]+l[1])/2,(f[0]+f[1])/2]),z[0]=l[+(n[0]<x[0])],z[1]=f[+(n[1]<x[1])]):x=null),E&&d(n,c,0)&&(e(S),u=!0),A&&d(n,s,1)&&(r(S),u=!0),u&&(t(S),w({type:"brush",mode:C?"move":"resize"}))}function d(n,t,e){var r,u,a=Ti(t),c=a[0],s=a[1],p=z[e],v=e?f:l,d=v[1]-v[0];return C&&(c-=p,s-=d+p),r=(e?g:h)?Math.max(c,Math.min(s,n[e])):n[e],C?u=(r+=p)+d:(x&&(p=Math.max(c,Math.min(s,2*x[e]-r))),r>p?(u=r,r=p):u=p),v[0]!=r||v[1]!=u?(e?o=null:i=null,v[0]=r,v[1]=u,!0):void 0}function m(){v(),S.style("pointer-events","all").selectAll(".resize").style("display",n.empty()?"none":null),Zo.select("body").style("cursor",null),L.on("mousemove.brush",null).on("mouseup.brush",null).on("touchmove.brush",null).on("touchend.brush",null).on("keydown.brush",null).on("keyup.brush",null),N(),w({type:"brushend"})}var x,M,_=this,b=Zo.select(Zo.event.target),w=a.of(_,arguments),S=Zo.select(_),k=b.datum(),E=!/^(n|s)$/.test(k)&&c,A=!/^(e|w)$/.test(k)&&s,C=b.classed("extent"),N=I(),z=Zo.mouse(_),L=Zo.select(Wo).on("keydown.brush",u).on("keyup.brush",p);if(Zo.event.changedTouches?L.on("touchmove.brush",v).on("touchend.brush",m):L.on("mousemove.brush",v).on("mouseup.brush",m),S.interrupt().selectAll("*").interrupt(),C)z[0]=l[0]-z[0],z[1]=f[0]-z[1];else if(k){var T=+/w$/.test(k),q=+/^n/.test(k);M=[l[1-T]-z[0],f[1-q]-z[1]],z[0]=l[T],z[1]=f[q]}else Zo.event.altKey&&(x=z.slice());S.style("pointer-events","none").selectAll(".resize").style("display",null),Zo.select("body").style("cursor",b.style("cursor")),w({type:"brushstart"}),v()}var i,o,a=M(n,"brushstart","brush","brushend"),c=null,s=null,l=[0,0],f=[0,0],h=!0,g=!0,p=qs[0];return n.event=function(n){n.each(function(){var n=a.of(this,arguments),t={x:l,y:f,i:i,j:o},e=this.__chart__||t;this.__chart__=t,Ss?Zo.select(this).transition().each("start.brush",function(){i=e.i,o=e.j,l=e.x,f=e.y,n({type:"brushstart"})}).tween("brush:brush",function(){var e=gu(l,t.x),r=gu(f,t.y);return i=o=null,function(u){l=t.x=e(u),f=t.y=r(u),n({type:"brush",mode:"resize"})}}).each("end.brush",function(){i=t.i,o=t.j,n({type:"brush",mode:"resize"}),n({type:"brushend"})}):(n({type:"brushstart"}),n({type:"brush",mode:"resize"}),n({type:"brushend"}))})},n.x=function(t){return arguments.length?(c=t,p=qs[!c<<1|!s],n):c},n.y=function(t){return arguments.length?(s=t,p=qs[!c<<1|!s],n):s},n.clamp=function(t){return arguments.length?(c&&s?(h=!!t[0],g=!!t[1]):c?h=!!t:s&&(g=!!t),n):c&&s?[h,g]:c?h:s?g:null},n.extent=function(t){var e,r,u,a,h;return arguments.length?(c&&(e=t[0],r=t[1],s&&(e=e[0],r=r[0]),i=[e,r],c.invert&&(e=c(e),r=c(r)),e>r&&(h=e,e=r,r=h),(e!=l[0]||r!=l[1])&&(l=[e,r])),s&&(u=t[0],a=t[1],c&&(u=u[1],a=a[1]),o=[u,a],s.invert&&(u=s(u),a=s(a)),u>a&&(h=u,u=a,a=h),(u!=f[0]||a!=f[1])&&(f=[u,a])),n):(c&&(i?(e=i[0],r=i[1]):(e=l[0],r=l[1],c.invert&&(e=c.invert(e),r=c.invert(r)),e>r&&(h=e,e=r,r=h))),s&&(o?(u=o[0],a=o[1]):(u=f[0],a=f[1],s.invert&&(u=s.invert(u),a=s.invert(a)),u>a&&(h=u,u=a,a=h))),c&&s?[[e,u],[r,a]]:c?[e,r]:s&&[u,a])},n.clear=function(){return n.empty()||(l=[0,0],f=[0,0],i=o=null),n},n.empty=function(){return!!c&&l[0]==l[1]||!!s&&f[0]==f[1]},Zo.rebind(n,a,"on")};var Ts={n:"ns-resize",e:"ew-resize",s:"ns-resize",w:"ew-resize",nw:"nwse-resize",ne:"nesw-resize",se:"nwse-resize",sw:"nesw-resize"},qs=[["n","e","s","w","nw","ne","se","sw"],["e","w"],["n","s"],[]],Rs=Qa.format=ic.timeFormat,Ds=Rs.utc,Ps=Ds("%Y-%m-%dT%H:%M:%S.%LZ");Rs.iso=Date.prototype.toISOString&&+new Date("2000-01-01T00:00:00.000Z")?Ho:Ps,Ho.parse=function(n){var t=new Date(n);return isNaN(t)?null:t},Ho.toString=Ps.toString,Qa.second=Dt(function(n){return new nc(1e3*Math.floor(n/1e3))},function(n,t){n.setTime(n.getTime()+1e3*Math.floor(t))},function(n){return n.getSeconds()}),Qa.seconds=Qa.second.range,Qa.seconds.utc=Qa.second.utc.range,Qa.minute=Dt(function(n){return new nc(6e4*Math.floor(n/6e4))},function(n,t){n.setTime(n.getTime()+6e4*Math.floor(t))},function(n){return n.getMinutes()}),Qa.minutes=Qa.minute.range,Qa.minutes.utc=Qa.minute.utc.range,Qa.hour=Dt(function(n){var t=n.getTimezoneOffset()/60;return new nc(36e5*(Math.floor(n/36e5-t)+t))},function(n,t){n.setTime(n.getTime()+36e5*Math.floor(t))},function(n){return n.getHours()}),Qa.hours=Qa.hour.range,Qa.hours.utc=Qa.hour.utc.range,Qa.month=Dt(function(n){return n=Qa.day(n),n.setDate(1),n},function(n,t){n.setMonth(n.getMonth()+t)},function(n){return n.getMonth()}),Qa.months=Qa.month.range,Qa.months.utc=Qa.month.utc.range;var Us=[1e3,5e3,15e3,3e4,6e4,3e5,9e5,18e5,36e5,108e5,216e5,432e5,864e5,1728e5,6048e5,2592e6,7776e6,31536e6],js=[[Qa.second,1],[Qa.second,5],[Qa.second,15],[Qa.second,30],[Qa.minute,1],[Qa.minute,5],[Qa.minute,15],[Qa.minute,30],[Qa.hour,1],[Qa.hour,3],[Qa.hour,6],[Qa.hour,12],[Qa.day,1],[Qa.day,2],[Qa.week,1],[Qa.month,1],[Qa.month,3],[Qa.year,1]],Hs=Rs.multi([[".%L",function(n){return n.getMilliseconds()}],[":%S",function(n){return n.getSeconds()}],["%I:%M",function(n){return n.getMinutes()}],["%I %p",function(n){return n.getHours()}],["%a %d",function(n){return n.getDay()&&1!=n.getDate()}],["%b %d",function(n){return 1!=n.getDate()}],["%B",function(n){return n.getMonth()}],["%Y",we]]),Fs={range:function(n,t,e){return Zo.range(Math.ceil(n/e)*e,+t,e).map(Oo)},floor:wt,ceil:wt};js.year=Qa.year,Qa.scale=function(){return Fo(Zo.scale.linear(),js,Hs)};var Os=js.map(function(n){return[n[0].utc,n[1]]}),Ys=Ds.multi([[".%L",function(n){return n.getUTCMilliseconds()}],[":%S",function(n){return n.getUTCSeconds()}],["%I:%M",function(n){return n.getUTCMinutes()}],["%I %p",function(n){return n.getUTCHours()}],["%a %d",function(n){return n.getUTCDay()&&1!=n.getUTCDate()}],["%b %d",function(n){return 1!=n.getUTCDate()}],["%B",function(n){return n.getUTCMonth()}],["%Y",we]]);Os.year=Qa.year.utc,Qa.scale.utc=function(){return Fo(Zo.scale.linear(),Os,Ys)},Zo.text=St(function(n){return n.responseText}),Zo.json=function(n,t){return kt(n,"application/json",Yo,t)},Zo.html=function(n,t){return kt(n,"text/html",Io,t)},Zo.xml=St(function(n){return n.responseXML}),"function"==typeof define&&define.amd?define(Zo):"object"==typeof module&&module.exports&&(module.exports=Zo),this.d3=Zo}();
+
+// parseUri 1.2.2
+// (c) Steven Levithan <stevenlevithan.com>
+// MIT License
+
+function parseUri (str) {
+ var o = parseUri.options,
+ m = o.parser[o.strictMode ? "strict" : "loose"].exec(str),
+ uri = {},
+ i = 14;
+
+ while (i--) uri[o.key[i]] = m[i] || "";
+
+ uri[o.q.name] = {};
+ uri[o.key[12]].replace(o.q.parser, function ($0, $1, $2) {
+ if ($1) uri[o.q.name][$1] = $2;
+ });
+
+ return uri;
+};
+
+parseUri.options = {
+ strictMode: false,
+ key: ["source","protocol","authority","userInfo","user","password","host","port","relative","path","directory","file","query","anchor"],
+ q: {
+ name: "queryKey",
+ parser: /(?:^|&)([^&=]*)=?([^&]*)/g
+ },
+ parser: {
+ strict: /^(?:([^:\/?#]+):)?(?:\/\/((?:(([^:@]*)(?::([^:@]*))?)?@)?([^:\/?#]*)(?::(\d*))?))?((((?:[^?#\/]*\/)*)([^?#]*))(?:\?([^#]*))?(?:#(.*))?)/,
+ loose: /^(?:(?![^:@]+:[^:@\/]*@)([^:\/?#.]+):)?(?:\/\/)?((?:(([^:@]*)(?::([^:@]*))?)?@)?([^:\/?#]*)(?::(\d*))?)(((\/(?:[^?#](?![^?#\/]*\.[^?#\/.]+(?:[?#]|$)))*\/?)?([^?#\/]*))(?:\?([^#]*))?(?:#(.*))?)/
+ }
+};
+
+
+// graph.js
+var drawGraph = function() {
+
+ $("svg").remove();
+
+ //Dataset and metric to draw is passed via query option:
+ query = parseUri(location).queryKey;
+ query.stats = unescape(query.stats);
+ stats_db = '/tests/artifacts/' + query.stats + '/stats';
+ var metric = query.metric;
+ var operation = query.operation;
+ var smoothing = query.smoothing;
+ var show_aggregates = query.show_aggregates;
+
+ xmin = query.xmin;
+ xmax = query.xmax;
+ ymin = query.ymin;
+ ymax = query.ymax;
+
+ //Pull metrics from the stats json:
+ stress_metrics = $.extend([], stats['stats'][0]['metrics']);
+ $.each(stress_metrics, function(i,v) {
+ stress_metrics[i] = v.replace(/\W/g,"_");
+ });
+ stress_metric_names = {};
+ $.each(stress_metrics, function(i,v) {
+ stress_metric_names[v] = stats['stats'][0]['metrics'][i];
+ });
+ //Replace names of shorthand metric names with longer ones:
+ //Replace names of shorthand metric names with longer ones:
+ $.extend(stress_metric_names, {
+ "mean": "latency mean",
+ "med" : "latency median",
+ "_95" : "latency 95th pct",
+ "_99" : "latency 99th pct",
+ "_999": "latency 99.9th pct",
+ "max" : "latency max",
+ "max_ms" : "gc max (ms)",
+ "sum_ms" : "gc sum (ms)",
+ "sdv_ms" : "gc sdv (ms)",
+ "mb" : "gc MB"
+ });
+
+ var updateURLBar = function() {
+ //Update the URL bar with the current parameters:
+ window.history.replaceState(null,null,parseUri(location).path + "?" + $.param(query));
+ };
+
+ //Check query parameters:
+ if (metric == undefined) {
+ metric = query.metric = 'op_s';
+ }
+ if (operation == undefined) {
+ operation = query.operation = stats['stats'][0]['test'];
+ }
+ if (smoothing == undefined) {
+ smoothing = query.smoothing = 1;
+ }
+ if (show_aggregates == undefined || query.show_aggregates == 'true') {
+ show_aggregates = query.show_aggregates = true;
+ } else {
+ show_aggregates = query.show_aggregates = false;
+ }
+ updateURLBar();
+
+ var metric_index = stress_metrics.indexOf(metric);
+ var time_index = stress_metrics.indexOf('time');
+
+ /// Add dropdown controls to select chart criteria / options:
+ var chart_controls = $('<div id="chart_controls"/>');
+ var chart_controls_tbl = $('<table/>');
+ chart_controls.append(chart_controls_tbl);
+ $('body').append(chart_controls);
+ var metric_selector = $('<select id="metric_selector"/>');
+ $.each(stress_metric_names, function(k,v) {
+ if (k == 'time') {
+ return; //Elapsed time makes no sense to graph, skip it.
+ }
+ var option = $('<option/>').attr('value', k).text(v);
+ if (metric == k) {
+ option.attr('selected','selected');
+ }
+ metric_selector.append(option);
+
+ });
+ chart_controls_tbl.append('<tr><td><label for="metric_selector"/>Choose metric:</label></td><td id="metric_selector_td"></td></tr>')
+ $('#metric_selector_td').append(metric_selector);
+
+ var operation_selector = $('<select id="operation_selector"/>')
+ chart_controls_tbl.append('<tr><td><label for="operation_selector"/>Choose operation:</label></td><td id="operation_selector_td"></td></tr>')
+ $('#operation_selector_td').append(operation_selector);
+
+
+ var smoothing_selector = $('<select id="smoothing_selector"/>')
+ $.each([1,2,3,4,5,6,7,8], function(i, v) {
+ var option = $('<option/>').attr('value', v).text(v);
+ if (smoothing == v) {
+ option.attr('selected','selected');
+ }
+ smoothing_selector.append(option);
+ });
+ chart_controls_tbl.append('<tr><td style="width:150px"><label for="smoothing_selector"/>Data smoothing:</label></td><td id="smoothing_selector_td"></td></tr>')
+ $("#smoothing_selector_td").append(smoothing_selector);
+
+ var show_aggregates_checkbox = $('<input type="checkbox" id="show_aggregates_checkbox"/>');
+ chart_controls_tbl.append('<tr><td style="padding-top:10px"><label for="show_aggregates_checkbox">Show aggregates</label></td><td id="show_aggregates_td"></td></tr>');
+ $("#show_aggregates_td").append(show_aggregates_checkbox);
+ show_aggregates_checkbox.attr("checked", show_aggregates);
+
+ chart_controls_tbl.append('<tr><td colspan="100%">Zoom: <a href="#" id="reset_zoom">reset</a><table id="zoom"><tr><td><label for="xmin"/>x min</label></td><td><input id="xmin"/></td><td><label for="xmax"/>x max</label></td><td><input id="xmax"/></td></tr><tr><td><label for="ymin"/>y min</label></td><td><input id="ymin"/></td><td><label for="ymax"/>y max</label></td><td><input id="ymax"/></td></tr></table></td></tr>');
+
+ chart_controls_tbl.append('<tr><td style="padding-top:10px" colspan="100%">To hide/show a dataset click on the associated colored box</td></tr>');
+
+ var raw_data;
+
+ //Callback to draw graph once we have json data.
+ var graph_callback = function() {
+ var data = [];
+ var trials = {};
+ var data_by_title = {};
+ //Keep track of what operations are availble from the test:
+ var operations = {};
+
+ raw_data.stats.forEach(function(d) {
+ // Make a copy of d so we never modify raw_data
+ d = $.extend({}, d);
+ operations[d.test] = true;
+ if (d.test!=operation) {
+ return;
+ }
+ d.title = d['label'] != undefined ? d['label'] : d['revision'];
+ data_by_title[d.title] = d;
+ data.push(d);
+ trials[d.title] = d;
+ //Clean up the intervals:
+ //Remove every other item, so as to smooth the line:
+ var new_intervals = [];
+ d.intervals.forEach(function(i, x) {
+ if (x % smoothing == 0) {
+ new_intervals.push(i);
+ }
+ });
+ d.intervals = new_intervals;
+ });
+
+ //Fill operations available from test:
+ operation_selector.children().remove();
+ $.each(operations, function(k) {
+ var option = $('<option/>').attr('value', k).text(k);
+ if (operation == k) {
+ option.attr('selected','selected');
+ }
+ operation_selector.append(option);
+ });
+
+ var getMetricValue = function(d) {
+ if (metric_index >= 0) {
+ //This is one of the metrics directly reported by stress:
+ return d[metric_index];
+ } else {
+ //This metric is not reported by stress, so compute it ourselves:
+ if (metric == 'num_timeouts') {
+ return d[stress_metrics.indexOf('interval_op_rate')] - d[stress_metrics.indexOf('interval_key_rate')];
+ }
+ }
+ };
+
+ //Parse the dates:
+ data.forEach(function(d) {
+ d.date = new Date(Date.parse(d.date));
+ });
+
+
+ $("svg").remove();
+ //Setup initial zoom level:
+ defaultZoom = function(initialize) {
+ if (!initialize) {
+ //Reset zoom query params:
+ query.xmin = xmin = undefined;
+ query.xmax = xmax = undefined;
+ query.ymin = ymin = undefined;
+ query.ymax = ymax = undefined;
+ }
+ query.xmin = xmin = query.xmin ? query.xmin : 0;
+ query.xmax = xmax = query.xmax ? query.xmax : Math.round(d3.max(data, function(d) {
+ if (d.intervals.length > 0) {
+ return d.intervals[d.intervals.length-1][time_index];
+ }
+ }) * 1.1 * 100) / 100;
+ query.ymin = ymin = query.ymin ? query.ymin : 0;
+ query.ymax = ymax = query.ymax ? query.ymax : Math.round(d3.max(data, function(d) {
+ return d3.max(d.intervals, function(i) {
+ return getMetricValue(i);
+ });
+ }) * 1.1 * 100) / 100;
+ $("#xmin").val(xmin);
+ $("#xmax").val(xmax);
+ $("#ymin").val(ymin);
+ $("#ymax").val(ymax);
+ var updateX = function() {
+ query.xmin = xmin = $("#xmin").val();
+ query.xmax = xmax = $("#xmax").val();
+ x.domain([xmin,xmax]);
+ updateURLBar();
+ };
+ var updateY = function() {
+ query.ymin = ymin = $("#ymin").val();
+ query.ymax = ymax = $("#ymax").val();
+ y.domain([ymin, ymax]);
+ updateURLBar();
+ };
+ $("#xmin,#xmax").unbind().change(function(e) {
+ updateX();
+ redrawLines();
+ });
+ $("#ymin,#ymax").unbind().change(function(e) {
+ updateY();
+ redrawLines();
+ });
+ // The first time defaultZoom is called, we pass
+ // initialize=true, and we do not call the change() method
+ // yet. On subsequent calls, without initialize, we do.
+ if (!initialize) {
+ updateX();
+ updateY();
+ redrawLines();
+ }
+ }
+ defaultZoom(true);
+
+ $("#reset_zoom").click(function(e) {
+ defaultZoom();
+ e.preventDefault();
+ });
+
+ //Setup chart:
+ var margin = {top: 20, right: 1180, bottom: 4240, left: 60};
+ var width = 2060 - margin.left - margin.right;
+ var height = 4700 - margin.top - margin.bottom;
+
+ var x = d3.scale.linear()
+ .domain([xmin, xmax])
+ .range([0, width]);
+
+ var y = d3.scale.linear()
+ .domain([ymin, ymax])
+ .range([height, 0]);
+
+ var color = d3.scale.category10();
+ color.domain(data.map(function(d){return d.title}));
+
+ var xAxis = d3.svg.axis()
+ .scale(x)
+ .orient("bottom");
+
+ var yAxis = d3.svg.axis()
+ .scale(y)
+ .orient("left");
+
+ var line = d3.svg.line()
+ .interpolate("basis")
+ .x(function(d) {
+ return x(d[time_index]); //time in seconds
+ })
+ .y(function(d) {
+ return y(getMetricValue(d));
+ });
+
+ $("body").append("<div id='svg_container'>");
+
+ var redrawLines = function() {
+ svg.select(".x.axis").call(xAxis);
+ svg.select(".y.axis").call(yAxis);
+ svg.selectAll(".line")
+ .attr("class","line")
+ .attr("d", function(d) {
+ return line(d.intervals);
+ })
+ $("#xmin").val(x.domain()[0]);
+ $("#xmax").val(x.domain()[1]);
+ $("#ymin").val(y.domain()[0]);
+ $("#ymax").val(y.domain()[1]);
+ }
+
+ var zoom = d3.behavior.zoom()
+ .x(x)
+ .y(y)
+ .on("zoom", redrawLines);
+
+ var svg = d3.select("div#svg_container").append("svg")
+ .attr("width", width + margin.left + margin.right + 250)
+ .attr("height", height + margin.top + margin.bottom)
+ .append("g")
+ .attr("transform", "translate(" + margin.left + "," + margin.top + ")")
+
+ // Clip Path
+ svg.append("svg:clipPath")
+ .attr("id", "chart_clip")
+ .append("svg:rect")
+ .attr("width", width)
+ .attr("height", height);
+
+ // Chart title
+ svg.append("text")
+ .attr("x", width / 2 )
+ .attr("y", 0 )
+ .style('font-size', '2em')
+ .style("text-anchor", "middle")
+ .text(raw_data.title + ' - ' + operation);
+
+ // Chart subtitle
+ svg.append("text")
+ .attr("x", width / 2 )
+ .attr("y", 15 )
+ .style('font-size', '1.2em')
+ .style("text-anchor", "middle")
+ .text((raw_data.subtitle ? raw_data.subtitle : ''));
+
+ // x-axis - time
+ svg.append("g")
+ .attr("class", "x axis")
+ .attr("transform", "translate(0," + height + ")")
+ .call(xAxis);
+
+ // x-axis label
+ svg.append("text")
+ .attr("x", width / 2 )
+ .attr("y", height + 30 )
+ .style("text-anchor", "middle")
+ .style("font-size", "1.2em")
+ .text(stress_metric_names['time']);
+
+ // y-axis
+ svg.append("g")
+ .attr("class", "y axis")
+ .call(yAxis)
+ .append("text")
+ .attr("transform", "rotate(-90)")
+ .attr("y", -60)
+ .attr("dy", ".91em")
+ .style("font-size", "1.2em")
+ .style("text-anchor", "end")
+ .text(stress_metric_names[metric]);
+
+ var trial = svg.selectAll(".trial")
+ .data(data)
+ .enter().append("g")
+ .attr("class", "trial")
+ .attr("title", function(d) {
+ return d.title;
+ });
+
+ // Draw benchmarked data:
+ trial.append("path")
+ .attr("class", "line")
+ .attr("clip-path", "url(#chart_clip)")
+ .attr("d", function(d) {
+ return line(d.intervals);
+ })
+ .style("stroke", function(d) { return color(d.title); });
+
+ var legend = svg.selectAll(".legend")
+ .data(color.domain())
+ .enter().append("g")
+ .attr("class", "legend")
+ .attr("transform", function(d, i) {
+ if (show_aggregates == true) {
+ var y_offset = 425 + (i*240) + 70;
+ } else {
+ var y_offset = 425 + (i*25) + 70;
+ }
+ var x_offset = -550;
+ return "translate(" + x_offset + "," + y_offset + ")";
+ });
+
+ var renderLegendText = function(linenum, getTextCallback) {
+ legend.append("text")
+ .attr("x", width - 24 - 250)
+ .attr("y", 12*linenum)
+ .attr("dy", ".35em")
+ .style("font-family", "monospace")
+ .style("font-size", "1.2em")
+ .style("text-anchor", "start")
+ .text(function(d) {
+ return getTextCallback(d);
+ });
+ };
+
+ var padTextEnd = function(text, length) {
+ for(var x=text.length; x<length; x++) {
+ text = text + '\u00A0';
+ }
+ return text;
+ };
+ var padTextStart = function(text, length) {
+ for(var x=text.length; x<length; x++) {
+ text = '\u00A0' + text;
+ }
+ return text;
+ };
+
+ renderLegendText(1, function(title) {
+ return padTextStart(title, title.length + 5);
+ });
+
+ if (show_aggregates === true) {
+ renderLegendText(2, function(title) {
+ return '---------------------------------------';
+ });
+
+ renderLegendText(3, function(title) {
+ return padTextEnd('op rate', 26) + " : " + data_by_title[title]['op rate'];
+ });
+
+ renderLegendText(4, function(title) {
+ return padTextEnd('partition rate', 26) + " : " + data_by_title[title]['partition rate'];
+ });
+
+ renderLegendText(5, function(title) {
+ return padTextEnd('row rate', 26) + ' : ' + data_by_title[title]['row rate'];
+ });
+
+ renderLegendText(6, function(title) {
+ return padTextEnd('latency mean', 26) + ' : ' + data_by_title[title]['latency mean'];
+ });
+
+ renderLegendText(7, function(title) {
+ return padTextEnd('latency median', 26) + ' : ' + data_by_title[title]['latency median'];
+ });
+
+ renderLegendText(8, function(title) {
+ return padTextEnd('latency 95th percentile', 26) + ' : ' + data_by_title[title]['latency 95th percentile'];
+ });
+
+ renderLegendText(9, function(title) {
+ return padTextEnd('latency 99th percentile', 26) + ' : ' + data_by_title[title]['latency 99th percentile'];
+ });
+
+ renderLegendText(10, function(title) {
+ return padTextEnd('latency 99.9th percentile', 26) + ' : ' + data_by_title[title]['latency 99.9th percentile'];
+ });
+
+ renderLegendText(11, function(title) {
+ return padTextEnd('latency max', 26) + ' : ' + data_by_title[title]['latency max'];
+ });
+
+ renderLegendText(12, function(title) {
+ return padTextEnd('total gc count', 26) + ' : ' + data_by_title[title]['total gc count'];
+ });
+
+ renderLegendText(13, function(title) {
+ return padTextEnd('total gc mb', 26) + ' : ' + data_by_title[title]['total gc mb'];
+ });
+
+ renderLegendText(14, function(title) {
+ return padTextEnd('total gc time (s)', 26) + ' : ' + data_by_title[title]['total gc time (s)'];
+ });
+
+ renderLegendText(15, function(title) {
+ return padTextEnd('avg gc time(ms)', 26) + ' : ' + data_by_title[title]['avg gc time(ms)'];
+ });
+
+ renderLegendText(16, function(title) {
+ return padTextEnd('stdev gc time(ms)', 26) + ' : ' + data_by_title[title]['stdev gc time(ms)'];
+ });
+
+ renderLegendText(17, function(title) {
+ return padTextEnd('Total operation time', 26) + ' : ' + data_by_title[title]['Total operation time'];
+ });
+
+ renderLegendText(18, function(title) {
+ var cmd = data_by_title[title]['command'];
+ return 'cmd: ' + cmd;
+ });
+ }
+ legend.append("rect")
+ .attr("x", width - 270)
+ .attr("width", 18)
+ .attr("height", 18)
+ .attr("class", "legend-rect")
+ .attr("title", function(title) {
+ return title;
+ })
+ .style("fill", color);
+
+ //Make trials hideable by double clicking on the colored legend box
+ $("rect.legend-rect").click(function() {
+ $("g.trial[title='" + $(this).attr('title') + "']").toggle();
+ });
+
+
+ // Chart control callbacks:
+ metric_selector.unbind().change(function(e) {
+ // change the metric in the url to reload the page:
+ metric = query.metric = this.value;
+ metric_index = stress_metrics.indexOf(metric);
+ graph_callback();
+ defaultZoom();
+ });
+ operation_selector.unbind().change(function(e) {
+ // change the metric in the url to reload the page:
+ operation = query.operation = this.value;
+ graph_callback();
+ defaultZoom();
+ });
+ smoothing_selector.unbind().change(function(e) {
+ // change the metric in the url to reload the page:
+ smoothing = query.smoothing = this.value;
+ graph_callback();
+ defaultZoom();
+ });
+ show_aggregates_checkbox.unbind().change(function(e) {
+ show_aggregates = query.show_aggregates = this.checked;
+ graph_callback();
+ });
+
+ updateURLBar();
+
+ $("#dl-test-data").attr("href",stats_db);
+
+ // Chart zoom/drag surface
+ // This should always be last, so it's on top of everything else
+ svg.append("svg:rect")
+ .attr("id", "zoom_drag_surface")
+ .attr("width", width)
+ .attr("height", height);
+ }
+
+ raw_data = stats;
+ graph_callback();
+
+}
+
+$(document).ready(function(){
+
+ drawGraph();
+
+});
+ -->
+ </script>
+ <style type="text/css">
+div#chart_controls {
+ margin-left: 900px;
+ position: absolute;
+}
+
+#chart_controls > table {
+ width: 640px;
+}
+
+#chart_controls td {
+ padding: 2px;
+}
+
+#chart_controls #zoom input {
+ width: 50px;
+}
+
+#chart_controls table#zoom {
+ padding-left: 20px;
+}
+
+#chart_controls table#zoom td {
+ padding-left: 20px;
+}
+
+#zoom_drag_surface {
+ fill: rgba(250, 250, 255, 0.0);
+ z-index: 100;
+}
+
+svg {
+ font: 10px sans-serif;
+}
+
+.axis path,
+.axis line {
+ fill: none;
+ stroke: #000;
+ shape-rendering: crispEdges;
+}
+
+.x.axis path {
+ display: none;
+}
+
+.line {
+ fill: none;
+ stroke: steelblue;
+ stroke-width: 1.5px;
+}
+
+ </style>
+</head>
+<body>
+</body>
+
