Google Git
Sign in
apache / cassandra-spark-connector

DataStax Connector for Apache Spark to Apache Cassandra

Mirrored from https://github.com/apache/cassandra-spark-connector.git
Clone this repo:

Branches

More...

Tags

More...
  1. 3923141 Donation to Apache Cassandra and ASF by mck · 7 weeks ago trunk
  2. d1eb21d Merge pull request #1374 from msmygit/master by Madhavan · 3 months ago
  3. f4f0da6 Doc update for community by Madhavan Sridharan · 3 months ago
  4. 265b55d Merge pull request #1372 from msmygit/master by Madhavan · 3 months ago
  5. 2b520ce Document update to point to community support by Madhavan Sridharan · 3 months ago

Apache Cassandra Spark Connector

Lightning-fast cluster computing with Apache Spark™ and Apache Cassandra®.

CI

Quick Links

WhatWhere
CommunityChat with us at Apache Cassandra
Scala DocsMost Recent Release (3.5.1): Connector API docs, Connector Driver docs
Latest Production Release3.5.1

News

3.5.1

  • The latest release of the Spark-Cassandra-Connector introduces support for vector types, greatly enhancing its capabilities. This new feature allows developers to seamlessly integrate and work with Cassandra 5.0 and Astra vectors within the Spark ecosystem. By supporting vector types, the connector now provides insights into AI and Retrieval-Augmented Generation (RAG) data, enabling more advanced and efficient data processing and analysis.

Features

This library lets you expose Cassandra tables as Spark RDDs and Datasets/DataFrames, write Spark RDDs and Datasets/DataFrames to Cassandra tables, and execute arbitrary CQL queries in your Spark applications.

  • Compatible with Apache Cassandra version 2.1 or higher (see table below)
  • Compatible with Apache Spark 1.0 through 3.5 (see table below)
  • Compatible with Scala 2.11, 2.12 and 2.13
  • Exposes Cassandra tables as Spark RDDs and Datasets/DataFrames
  • Maps table rows to CassandraRow objects or tuples
  • Offers customizable object mapper for mapping rows to objects of user-defined classes
  • Saves RDDs back to Cassandra by implicit saveToCassandra call
  • Delete rows and columns from cassandra by implicit deleteFromCassandra call
  • Join with a subset of Cassandra data using joinWithCassandraTable call for RDDs, and optimizes join with data in Cassandra when using Datasets/DataFrames
  • Partition RDDs according to Cassandra replication using repartitionByCassandraReplica call
  • Converts data types between Cassandra and Scala
  • Supports all Cassandra data types including collections
  • Filters rows on the server side via the CQL WHERE clause
  • Allows for execution of arbitrary CQL statements
  • Plays nice with Cassandra Virtual Nodes
  • Could be used in all languages supporting Datasets/DataFrames API: Python, R, etc.

Version Compatibility

The connector project has several branches, each of which map into different supported versions of Spark and Cassandra. For previous releases the branch is named “bX.Y” where X.Y is the major+minor version; for example the “b1.6” branch corresponds to the 1.6 release. The “master” branch will normally contain development for the next connector release in progress.

Currently, the following branches are actively supported: 3.5.x (master), 3.4.x (b3.4), 3.3.x (b3.2), 3.2.x (b3.2), 3.1.x (b3.1), 3.0.x (b3.0) and 2.5.x (b2.5).

ConnectorSparkCassandraCassandra Java DriverMinimum Java VersionSupported Scala Versions
3.5.13.52.1.5*, 2.2, 3.x, 4.x, 5.04.18.182.12, 2.13
3.53.52.1.5*, 2.2, 3.x, 4.x4.1382.12, 2.13
3.43.42.1.5*, 2.2, 3.x, 4.x4.1382.12, 2.13
3.33.32.1.5*, 2.2, 3.x, 4.x4.1382.12
3.23.22.1.5*, 2.2, 3.x, 4.04.1382.12
3.13.12.1.5*, 2.2, 3.x, 4.04.1282.12
3.03.02.1.5*, 2.2, 3.x, 4.04.1282.12
2.52.42.1.5*, 2.2, 3.x, 4.04.1282.11, 2.12
2.4.22.42.1.5*, 2.2, 3.x3.082.11, 2.12
2.42.42.1.5*, 2.2, 3.x3.082.11
2.32.32.1.5*, 2.2, 3.x3.082.11
2.02.0, 2.1, 2.22.1.5*, 2.2, 3.x3.082.10, 2.11
1.61.62.1.5*, 2.2, 3.03.072.10, 2.11
1.51.5, 1.62.1.5*, 2.2, 3.03.072.10, 2.11
1.41.42.1.5*2.172.10, 2.11
1.31.32.1.5*2.172.10, 2.11
1.21.22.1, 2.02.172.10, 2.11
1.11.1, 1.02.1, 2.02.172.10, 2.11
1.01.0, 0.92.02.072.10, 2.11

*Compatible with 2.1.X where X >= 5

Hosted API Docs

API documentation for the Scala and Java interfaces are available online:

3.5.1

3.5.0

3.4.1

3.3.0

3.2.0

3.1.0

3.0.1

2.5.2

2.4.2

Download

This project is available on the Maven Central Repository. For SBT to download the connector binaries, sources and javadoc, put this in your project SBT config:

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "3.5.1"
  • The default Scala version for Spark 3.0+ is 2.12 please choose the appropriate build. See the FAQ for more information.

Building

See Building And Artifacts

Documentation

Online Training

In DS320: Analytics with Spark, you will learn how to effectively and efficiently solve analytical problems with Apache Spark, Apache Cassandra, and DataStax Enterprise. You will learn about Spark API, Spark-Cassandra Connector, Spark SQL, Spark Streaming, and crucial performance optimization techniques.

Community

Reporting Bugs

New issues may be reported using JIRA. Please include all relevant details including versions of Spark, Spark Cassandra Connector, Cassandra and/or DSE. A minimal reproducible case with sample code is ideal.

Mailing List

Questions and requests for help may be submitted to the user mailing list.

Q/A Exchange

For community help see https://cassandra.apache.org/_/community.html

Contributing

To protect the community, all contributors are required to sign the Apache Software Foundation's Contribution License Agreement.

Tips for Developing the Spark Cassandra Connector

Checklist for contributing changes to the project:

  • Create a SPARKC JIRA
  • Make sure that all unit tests and integration tests pass
  • Add an appropriate entry at the top of CHANGES.txt
  • If the change has any end-user impacts, also include changes to the ./doc files as needed
  • Prefix the pull request description with the JIRA number, for example: “SPARKC-123: Fix the ...”
  • Open a pull-request on GitHub and await review

Testing

To run unit and integration tests:

./sbt/sbt test
./sbt/sbt it:test

Note that the integration tests require CCM to be installed on your machine. See Tips for Developing the Spark Cassandra Connector for details.

By default, integration tests start up a separate, single Cassandra instance and run Spark in local mode. It is possible to run integration tests with your own Spark cluster. First, prepare a jar with testing code:

./sbt/sbt test:package

Then copy the generated test jar to your Spark nodes and run:

export IT_TEST_SPARK_MASTER=<Spark Master URL>
./sbt/sbt it:test

Generating Documents

To generate the Reference Document use

./sbt/sbt spark-cassandra-connector-unshaded/run (outputLocation)

outputLocation defaults to doc/reference.md

License

Copyright 2014, Apache Software Foundation

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Apache Cassandra, Apache Spark, Apache, Tomcat, Lucene, Solr, Hadoop, Spark, TinkerPop, and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.