Apache cassandra diff

Clone this repo:
  1. 0e0d08b Probabilistic diff to sample partitions for diff testing based on probability by Jyothsna Konisa · 1 year, 8 months ago trunk
  2. 75b70c3 Cassandra diff should accept a provided job_id for retrying diffs by Jyothsna Konisa · 1 year, 11 months ago
  3. e3161b6 Improve metadata exception handling by Yifan Cai · 2 years, 1 month ago
  4. c3ce22c Always quote names in cql queries by Jyothsna Konisa · 2 years, 4 months ago
  5. 4c9bc4f Allow optional query retry by Yifan Cai · 2 years, 9 months ago

Apache Cassandra diff


See spark-job/localconfig.yaml for an example config.

See spark-job/localconfig-multi-keyspaces.yaml for an example config that compares tables under multiple keyspaces.

See spark-job/localconfig-auto-discover.yaml for an example config that auto discovers all user tables to compare. The auto discover mode excludes all system keyspaces and any keyspaces defined at disallowed_keyspaces in the yaml file.

Custom cluster providers

To make it easy to run in any environment the cluster providers are pluggable - there are two interfaces to implement. First, the ClusterProvider interface is used to create a connection to the clusters, and it is configured using JobConfiguration#clusterConfig (see below).


This section has 3 parts - source, target and metadata where source and target describes the clusters that should be compared and metadata describes where we store information about any mismatches and the progress the job has done. Metadata can be stored in one of the source/target clusters or in a separate cluster.

The fields under source/target/metadata are passed in to the ClusterProvider (described by impl) as a map, so any custom cluster providers can be configured here.

Setting up clusters for diff

One way of setting up clusters for diff is to restore a snapshot to two different clusters and then modifying one of the clusters to be able to make sure that the queries still return the same results. This could include upgrades/replacements/bounces/decommission/expansion.

Environment variables

Currently usernames and passwords are set as environment variables when running the diff tool and the api server:

  • diff.cluster.<identifier>.cql_user - the user name to use
  • diff.cluster.<identifier>.cql_password - password

where <identifier> should be source, target and metadata for the username/password combinations for the matching clusters in the configuration.


This example starts two cassandra single-node clusters in docker, runs stress to populate them and then runs diff to make sure the data matches;

You need to have docker and spark setup.

$ git clone <wherever>/cassandra-diff.git
$ cd cassandra-diff
$ mvn package
$ docker run --name cas-src -d  -p 9042:9042 cassandra:3.0.18
$ docker run --name cas-tgt -d  -p 9043:9042 cassandra:latest
# Create 1000 rows in table "standard1" under keyspace "keyspace1"
$ docker exec cas-src cassandra-stress write n=1k -schema keyspace="keyspace1"
$ docker exec cas-tgt cassandra-stress write n=1k -schema keyspace="keyspace1"
# Optionally, create another 1000 rows in table "standard1" under keyspace "keyspace2"
$ docker exec cas-src cassandra-stress write n=1k -schema keyspace="keyspace2"
$ docker exec cas-tgt cassandra-stress write n=1k -schema keyspace="keyspace2"
$ spark-submit --verbose --files ./spark-job/localconfig.yaml --class org.apache.cassandra.diff.DiffJob spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar localconfig.yaml
# If rows are created in "keyspace2", you can run pick up the localconfig-multi-keyspaces.yaml to compare data across multiple keyspaces! See the command below.
# $ spark-submit --verbose --files ./spark-job/localconfig-multi-keyspaces.yaml --class org.apache.cassandra.diff.DiffJob spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar localconfig-multi-keyspaces.yaml
# To use the auto discover mode, you can run the job with localconfig-auto-discover.yaml, which has the keyspace_tables field removed. 
# $ spark-submit --verbose --files ./spark-job/localconfig-auto-discover.yaml --class org.apache.cassandra.diff.DiffJob spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar localconfig-auto-discover.yaml
# ... logs
INFO  DiffJob:124 - FINISHED: {standard1=Matched Partitions - 1000, Mismatched Partitions - 0, Partition Errors - 0, Partitions Only In Source - 0, Partitions Only In Target - 0, Skipped Partitions - 0, Matched Rows - 1000, Matched Values - 6000, Mismatched Values - 0 }
## start api-server:
$ mvn install
$ cd api-server
$ mvn exec:java
$ curl -s localhost:8089/jobs/recent | python -mjson.tool
          "jobId": "99b8d556-07ed-4bfd-b978-7d9b7b2cc21a",
          "buckets": 100,
          "keyspace_tables": [
          "sourceClusterName": "local_test_1",
          "sourceClusterDesc": "ContactPoints Cluster: name=name, dc=datacenter1, contact points= []",
          "targetClusterName": "local_test_2",
          "targetClusterDesc": "ContactPoints Cluster: name=name, dc=datacenter1, contact points= []",
          "tasks": 10000,
          "start": "2019-08-16T11:47:36.123Z"
$ curl -s localhost:8089/jobs/99b8d556-07ed-4bfd-b978-7d9b7b2cc21a/results | python -mjson.tool
          "jobId": "99b8d556-07ed-4bfd-b978-7d9b7b2cc21a",
          "table": "keyspace1.standard1",
          "matchedPartitions": 1000,
          "mismatchedPartitions": 0,
          "matchedRows": 1000,
          "matchedValues": 6000,
          "mismatchedValues": 0,
          "onlyInSource": 0,
          "onlyInTarget": 0,
          "skippedPartitions": 0


We push maven artifacts to repository.apache.org. To create a release, follow the instructions [here|http://www.apache.org/dev/publishing-maven-artifacts.html], basically:

  1. make sure your ~/.m2/settings.xml has entries for the apache repositories:
            <!-- To publish a snapshot of some part of Maven -->
                <username> <!-- YOUR APACHE LDAP USERNAME --> </username>
                <password> <!-- YOUR APACHE LDAP PASSWORD (encrypted) --> </password>
            <!-- To stage a release of some part of Maven -->
                <username> <!-- YOUR APACHE LDAP USERNAME --> </username>
                <password> <!-- YOUR APACHE LDAP PASSWORD (encrypted) --> </password>
  2. Generate GPG keys:
$ gpg --gen-key
  1. Try a local install with the apache profile to make sure everything is setup correctly:
$ mvn clean install -Papache-release

Note, if you get an error like gpg: signing failed: Inappropriate ioctl for device you can run the command like this instead:

$ GPG_TTY=$(tty) mvn clean install -Papache-release
  1. Publish a snapshot:
$ mvn deploy
  1. Prepare the release:
$ mvn release:clean
$ mvn release:prepare
  1. Stage the release for a vote
$ mvn release:perform