!!! note Sedona Python currently only works with Shapely 1.x. If you use GeoPandas, please use <= GeoPandas 0.11.1. GeoPandas > 0.11.1 will automatically install Shapely 2.0. If you use Shapely, please use <= 1.8.4.

Sedona 1.5.3

Sedona 1.5.3 is compiled against Spark 3.3 / Spark 3.4 / Spark 3.5, Flink 1.12, Snowflake 7+, Java 8.

This release is a maintenance release that includes one bug fix on top of Sedona 1.5.2. No new features or major changes are added in this release.

Bug

Sedona 1.5.2

Sedona 1.5.2 is compiled against Spark 3.3 / Spark 3.4 / Spark 3.5, Flink 1.12, Snowflake 7+, Java 8.

This release is a maintenance release that includes bug fixes and minor improvements. No new features or major changes are added in this release.

New Contributors

Bug

Improvement

Test

Task

Sedona 1.5.1

Sedona 1.5.1 is compiled against Spark 3.3 / Spark 3.4 / Spark 3.5, Flink 1.12, Snowflake 7+, Java 8.

Highlights

  • [X] Sedona Snowflake Add support for Snowflake
  • [X] Sedona Spark Support Spark 3.5
  • [X] Sedona Spark Support Snowflake 7+
  • [X] Sedona Spark Added 20+ raster functions (or variants)
  • [X] Sedona Spark/Flink/Snowflake Added 7 vector functions (or variants)
  • [X] Sedona Spark GeoParquet reader and writer supports projjson in metadata
  • [X] Sedona Spark GeoParquet reader and writer conform to GeoParquet spec 1.0.0 instead of 1.0.0-beta1
  • [X] Sedona Spark Added a legacyMode in GeoParquet reader for 1.5.1+ users to read Parquet files written by Sedona 1.3.1 and earlier
  • [X] Sedona Spark Fixed a bug in GeoParquet writer so 1.3.1 and earlier users can read Parquet files written by 1.5.1+

Behavior change

  • All raster functions that take a geometry will implicitly transform the CRS of the geometry if needed.
  • The default CRS for these functions is 4326 for raster and geometry involved in raster functions, if not specified.
  • KeplerGL and DeckGL become optional dependencies for Sedona Spark Python.

New Contributors

Bug

New Feature

Improvement

Test

Task

Sedona 1.5.0

Sedona 1.5.0 is compiled against Spark 3.3 / Spark 3.4 / Flink 1.12, Java 8.

Highlights

API breaking changes:

  • The following functions in Sedona requires the input data must be in longitude/latitude order otherwise they might throw errors. You can use FlipCoordinates to swap X and Y.
    • ST_Transform
    • ST_DistanceSphere
    • ST_DistanceSpheroid
    • ST_GeoHash
    • All ST_H3 functions
    • All ST_S2 functions
    • All RS constructors
    • All RS predicates
    • Spark RDD: CRStransform
  • Rename RS_Count to RS_CountValue
  • Drop RS_HTML
  • Unshaded Sedona Spark code are all merged to a single jar sedona-spark

New features

  • Add 18 more ST functions for vector data processing in Sedona Spark and Sedona Flink
  • Add 36 more RS functions in Sedona Spark to support comprehensive raster data ETL and analytics
    • You can now directly join vector and raster datasets together
    • Flexible map algebra equations: SELECT RS_MapAlgebra(rast, 'D', 'out = (rast[3] - rast[0]) / (rast[3] + rast[0]);') as ndvi FROM raster_table
  • Add native support of Uber H3 functions in Sedona Spark and Sedona Flink.
  • Add SedonaKepler and SedonaPyDeck for interactive map visualization on Sedona Spark.

Bug

New Feature

Improvement

Task

Sedona 1.4.1

Sedona 1.4.1 is compiled against Spark 3.3 / Spark 3.4 / Flink 1.12, Java 8.

Highlights

  • [X] Sedona Spark More raster functions and bridge RasterUDT and Map Algebra operators. See Raster based operators and Raster to Map Algebra operators.
  • [X] Sedona Spark & Flink Added geodesic / geography functions:
    • ST_DistanceSphere
    • ST_DistanceSpheroid
    • ST_AreaSpheroid
    • ST_LengthSpheroid
  • [X] Sedona Spark & Flink Introduced SedonaContext to unify Sedona entry points.
  • [X] Sedona Spark Support Spark 3.4.
  • [X] Sedona Spark Added a number of new ST functions.
  • [X] Zeppelin Zeppelin helium plugin supports plotting geometries like linestring, polygon.

API change

  • Sedona Spark & Flink Introduced a new entry point called SedonaContext to unify all Sedona entry points in different compute engines and deprecate old Sedona register entry points. Users no longer have to register Sedona kryo serializer and import many tedious Python classes.
    • Sedona Spark:
      • Scala:
      import org.apache.sedona.spark.SedonaContext
val sedona = SedonaContext.create(SedonaContext.builder().master("local[*]").getOrCreate())
sedona.sql("SELECT ST_GeomFromWKT(XXX) FROM")
      • Python:
      from sedona.spark import *

config = SedonaContext.builder().\
   config('spark.jars.packages',
       'org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1,'
       'org.datasyslab:geotools-wrapper:1.4.0-28.2'). \
   getOrCreate()
sedona = SedonaContext.create(config)
sedona.sql("SELECT ST_GeomFromWKT(XXX) FROM")
    • Sedona Flink:
    import org.apache.sedona.flink.SedonaContext;
StreamTableEnvironment sedona = SedonaContext.create(env, tableEnv);
sedona.sqlQuery("SELECT ST_GeomFromWKT(XXX) FROM");

Bug

New Feature

Improvement

Sedona 1.4.0

Sedona 1.4.0 is compiled against, Spark 3.3 / Flink 1.12, Java 8.

Highlights

  • [X] Sedona Spark & Flink Serialize and deserialize geometries 3 - 7X faster
  • [X] Sedona Spark & Flink Google S2 based spatial join for fast approximate point-in-polygon join. See Join query in Spark and Join query in Flink
  • [X] Sedona Spark Pushdown spatial predicate on GeoParquet to reduce memory consumption by 10X: see explanation
  • [X] Sedona Spark Automatically use broadcast index spatial join for small datasets
  • [X] Sedona Spark New RasterUDT added to Sedona GeoTiff reader.
  • [X] Sedona Spark A number of bug fixes and improvement to the Sedona R module.

API change

  • Sedona Spark & Flink Packaging strategy changed. See Maven Coordinate. Please change your Sedona dependencies if needed. We recommend sedona-spark-shaded-3.0_2.12-1.4.0 and sedona-flink-shaded_2.12-1.4.0
  • Sedona Spark & Flink GeoTools-wrapper version upgraded. Please use geotools-wrapper-1.4.0-28.2.

Behavior change

  • Sedona Flink Sedona Flink no longer outputs any LinearRing type geometry. All LinearRing are changed to LineString.
  • Sedona Spark Join optimization strategy changed. Sedona no longer optimizes spatial join when use a spatial predicate together with an equijoin predicate. By default, it prefers equijoin whenever possible. SedonaConf adds a config option called sedona.join.optimizationmode, it can be configured as one of the following values:
    • all: optimize all joins having spatial predicate in join conditions. This was the behavior of Apache Sedona prior to 1.4.0.
    • none: disable spatial join optimization.
    • nonequi: only enable spatial join optimization on non-equi joins. This is the default mode.

When sedona.join.optimizationmode is configured as nonequi, it won't optimize join queries such as SELECT * FROM A, B WHERE A.x = B.x AND ST_Contains(A.geom, B.geom), since it is an equi-join with equi-condition A.x = B.x. Sedona will optimize for SELECT * FROM A, B WHERE ST_Contains(A.geom, B.geom)

Bug

New Feature

Improvement

Sedona 1.3.1

This version is a minor release on Sedona 1.3.0 line. It fixes a few critical bugs in 1.3.0. We suggest all 1.3.0 users to migrate to this version.

Bug fixes

  • SEDONA-204 - Init value in X/Y/Z max should be -Double.MAX
  • SEDONA-206 - Performance regression of ST_Transform in 1.3.0-incubating
  • SEDONA-210 - 1.3.0-incubating doesn't work with Scala 2.12 sbt projects
  • SEDONA-211 - Enforce release managers to use JDK 8
  • SEDONA-201 - Implement ST_MLineFromText and ST_MPolyFromText methods

New Feature

Improvement

  • SEDONA-194 - Merge org.datasyslab.sernetcdf into Sedona
  • SEDONA-208 - Use Spark RuntimeConfig in SedonaConf

!!!note Support of Spark 2.X and Scala 2.11 was removed in Sedona 1.3.0+ although some parts of the source code might still be compatible. Sedona 1.3.0+ releases binary for both Scala 2.12 and 2.13.

Sedona 1.3.0

This version is a major release on Sedona 1.3.0 line and consists of 50 PRs. It includes many new functions, optimization and bug fixes.

Highlights

  • [X] Sedona on Spark in this release is compiled against Spark 3.3.
  • [X] Sedona on Flink in this release is compiled against Flink 1.14.
  • [X] Scala 2.11 support is removed.
  • [X] Spark 2.X support is removed.
  • [X] Python 3.10 support is added.
  • [X] Aggregators in Flink are added
  • [X] Correctness fixes for corner cases in range join and distance join.
  • [X] Native GeoParquet read and write (../../tutorial/sql/#load-geoparquet).
    • df = spark.read.format("geoparquet").option("fieldGeometry", "myGeometryColumn").load("PATH/TO/MYFILE.parquet")
    • df.write.format("geoparquet").save("PATH/TO/MYFILE.parquet")
  • [X] DataFrame style API (../../tutorial/sql.md/#dataframe-style-api)
    • df.select(ST_Point(min_value, max_value).as("point"))
  • [X] Allow WKT format CRS in ST_Transform
    • ST_Transform(geom, "srcWktString", "tgtWktString")
GEOGCS["WGS 84",
  DATUM["WGS_1984",
  SPHEROID["WGS 84",6378137,298.257223563,
  AUTHORITY["EPSG","7030"]],
  AUTHORITY["EPSG","6326"]],
  PRIMEM["Greenwich",0,
  AUTHORITY["EPSG","8901"]],
  UNIT["degree",0.0174532925199433,
  AUTHORITY["EPSG","9122"]],
  AUTHORITY["EPSG","4326"]]

Bug fixes

  • SEDONA-119 - ST_Touches join query returns true for polygons whose interiors intersect
  • SEDONA-136 - Enable testAsEWKT for Flink
  • SEDONA-137 - Fix ST_Buffer for Flink to work
  • SEDONA-138 - Fix ST_GeoHash for Flink to work
  • SEDONA-153 - Python Serialization Fails with Nulls
  • SEDONA-158 - Fix wrong description about ST_GeometryN in the API docs
  • SEDONA-169 - Fix ST_RemovePoint in accordance with the API document
  • SEDONA-178 - Correctness issue in distance join queries
  • SEDONA-182 - ST_AsText should not return SRID
  • SEDONA-186 - collecting result rows of a spatial join query with SELECT * fails with serde error
  • SEDONA-188 - Python warns about missing jars even when some are found
  • SEDONA-193 - ST_AsBinary produces EWKB by mistake

New Features

Improvement

  • SEDONA-121 - Add equivalent constructors left over from Spark to Flink
  • SEDONA-132 - Create common module for SQL functions
  • SEDONA-133 - Allow user-defined schemas in Adapter.toDf()
  • SEDONA-139 - Fix wrong argument order in Flink unit tests
  • SEDONA-140 - Update Sedona Dependencies in R Package
  • SEDONA-143 - Add missing unit tests for the Flink predicates
  • SEDONA-144 - Add ST_AsGeoJSON to the Flink API
  • SEDONA-145 - Fix ST_AsEWKT to reserve the Z coordinate
  • SEDONA-146 - Add missing output functions to the Flink API
  • SEDONA-147 - Add SRID functions to the Flink API
  • SEDONA-148 - Add boolean functions to the Flink API
  • SEDONA-149 - Add Python 3.10 support
  • SEDONA-151 - Add ST aggregators to Sedona Flink
  • SEDONA-152 - Add reader/writer functions for GML and KML
  • SEDONA-154 - Add measurement functions to the Flink API
  • SEDONA-157 - Add coordinate accessors to the Flink API
  • SEDONA-159 - Add Nth accessor functions to the Flink API
  • SEDONA-160 - Fix geoparquetIOTests.scala to cleanup after test
  • SEDONA-161 - Add ST_Boundary to the Flink API
  • SEDONA-162 - Add ST_Envelope to the Flink API
  • SEDONA-163 - Better handle of unsupported types in shapefile reader
  • SEDONA-164 - Add geometry count functions to the Flink API
  • SEDONA-165 - Upgrade Apache Rat to 0.14
  • SEDONA-170 - Add ST_AddPoint and ST_RemovePoint to the Flink API
  • SEDONA-172 - Add ST_LineFromMultiPoint to Apache Sedona
  • SEDONA-176 - Make ST_Contains conform with OGC standard, and add ST_Covers and ST_CoveredBy functions.
  • SEDONA-177 - Support spatial predicates other than INTERSECTS and COVERS/COVERED_BY in RangeQuery.SpatialRangeQuery and JoinQuery.SpatialJoinQuery
  • SEDONA-181 - Build fails with java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
  • SEDONA-189 - Prepare geometries in broadcast join
  • SEDONA-192 - Null handling in predicates
  • SEDONA-195 - Add wkt validation and an optional srid to ST_GeomFromWKT/ST_GeomFromText

Task

  • SEDONA-150 - Drop Spark 2.4 and Scala 2.11 support

Sedona 1.2.1

This version is a maintenance release on Sedona 1.2.0 line. It includes bug fixes.

Sedona on Spark is now compiled against Spark 3.3, instead of Spark 3.2.

SQL (for Spark)

Bug fixes:

  • SEDONA-104: Bug in reading band values of GeoTiff images
  • SEDONA-118: Fix the wrong result in ST_Within
  • SEDONA-123: Fix the check for invalid lat/lon in ST_GeoHash

Improvement:

New features:

Flink

New features:

Sedona 1.2.0

This version is a major release on Sedona 1.2.0 line. It includes bug fixes and new features: Sedona with Apache Flink.

RDD

Bug fix:

  • SEDONA-18: Fix an error reading Shapefile
  • SEDONA-73: Exclude scala-library from scala-collection-compat

Improvement:

  • SEDONA-77: Refactor Format readers and spatial partitioning functions to be standalone libraries. So they can be used by Flink and others.

SQL

New features:

  • SEDONA-4: Handle nulls in SQL functions
  • SEDONA-65: Create ST_Difference function
  • SEDONA-68 Add St_Collect function.
  • SEDONA-82: Create ST_SymDifference function
  • SEDONA-75: Add support for “3D” geometries: Preserve Z coordinates on geometries when serializing, ST_AsText, ST_Z, ST_3DDistance
  • SEDONA-86: Support empty geometries in ST_AsBinary and ST_AsEWKB
  • SEDONA-90: Add ST_Union
  • SEDONA-100: Add st_multi function

Bug fix:

  • SEDONA-89: GeometryUDT equals should test equivalence of the other object

Flink

Major update:

  • SEDONA-80: Geospatial stream processing support in Flink Table API
  • SEDONA-85: ST_Geohash function in Flink
  • SEDONA-87: Support Flink Table and DataStream conversion
  • SEDONA-93: Add ST_GeomFromGeoJSON

Sedona 1.1.1

This version is a maintenance release on Sedona 1.1.X line. It includes bug fixes and a few new functions.

Global

New feature:

  • SEDONA-73: Scala source code supports Scala 2.13

SQL

Bug fix:

New features:

  • SEDONA-43: Add ST_GeoHash and ST_GeomFromGeoHash
  • SEDONA-45: Add ST_MakePolygon
  • SEDONA-71: Add ST_AsBinary, ST_AsEWKB, ST_SRID, ST_SetSRID

Sedona 1.1.0

This version is a major release on Sedona 1.1.0 line. It includes bug fixes and new features: R language API, Raster data and Map algebra support

Global

Dependency upgrade:

  • SEDONA-30: Use Geotools-wrapper 1.1.0-24.1 to include geotools GeoTiff libraries.

Improvement on join queries in core and SQL:

  • SEDONA-63: Skip empty partitions in NestedLoopJudgement
  • SEDONA-64: Broadcast dedupParams to improve performance

Behavior change:

  • SEDONA-62: Ignore HDF test in order to avoid NASA copyright issue

Core

Bug fix:

  • SEDONA-41: Fix rangeFilter bug when the leftCoveredByRight para is false
  • SEDONA-53: Fix SpatialKnnQuery NullPointerException

SQL

Major update:

  • SEDONA-30: Add GeoTiff raster I/O and Map Algebra function

New function:

  • SEDONA-27: Add ST_Subdivide and ST_SubdivideExplode functions

Bug fix:

  • SEDONA-56: Fix broadcast join with Adapter Query Engine enabled
  • SEDONA-22, SEDONA-60: Fix join queries in SparkSQL when one side has no rows or only one row

Viz

N/A

Python

Improvement:

  • SEDONA-59: Make pyspark dependency of Sedona Python optional

Bug fix:

  • SEDONA-50: Remove problematic logging conf that leads to errors on Databricks
  • Fix the issue: Spark dependency in setup.py was configured to be < v3.1.0 by mistake.

R

Major update:

Sedona 1.0.1

This version is a maintenance release on Sedona 1.0.0 line. It includes bug fixes, some new features, one ==API change==

Known issue

In Sedona v1.0.1 and earlier versions, the Spark dependency in setup.py was configured to be ==< v3.1.0== by mistake. When you install Sedona Python (apache-sedona v1.0.1) from PyPI, pip might uninstall PySpark 3.1.1 and install PySpark 3.0.2 on your machine.

Three ways to fix this:

  1. After install apache-sedona v1.0.1, uninstall PySpark 3.0.2 and reinstall PySpark 3.1.1

  2. Ask pip not to install Sedona dependencies: pip install --no-deps apache-sedona

  3. Install Sedona from the latest setup.py (on GitHub) manually.

Global

Dependency upgrade:

  • SEDONA-16: Use a GeoTools Maven Central wrapper to fix failed Jupyter notebook examples
  • SEDONA-29: upgrade to Spark 3.1.1
  • SEDONA-33: jts2geojson version from 0.14.3 to 0.16.1

Core

Bug fix:

  • SEDONA-35: Address user-data mutability issue with Adapter.toDF()

SQL

Bug fix:

  • SEDONA-14: Saving dataframe to CSV or Parquet fails due to unknown type
  • SEDONA-15: Add ST_MinimumBoundingRadius and ST_MinimumBoundingCircle functions
  • SEDONA-19: Global indexing does not work with SQL joins
  • SEDONA-20: Case object GeometryUDT and GeometryUDT instance not equal in Spark 3.0.2

New function:

  • SEDONA-21: allows Sedona to be used in pure SQL environment
  • SEDONA-24: Add ST_LineSubString and ST_LineInterpolatePoint
  • SEDONA-26: Add broadcast join support

Viz

Improvement:

API change:

  • SEDONA-29: Upgrade to Spark 3.1.1 and fix ST_Pixelize

Python

Bug fix:

  • SEDONA-19: Global indexing does not work with SQL joins

Sedona 1.0.0

This version is the first Sedona release since it joins the Apache Incubator. It includes new functions, bug fixes, and ==API changes==.

Global

Key dependency upgrade:

  • SEDONA-1: upgrade to JTS 1.18
  • upgrade to GeoTools 24.0
  • upgrade to jts2geojson 0.14.3

Key dependency packaging strategy change:

  • JTS, GeoTools, jts2geojson are no longer packaged in Sedona jars. End users need to add them manually. See here.

Key compilation target change:

  • SEDONA-3: Paths and class names have been changed to Apache Sedona
  • SEDONA-7: build the source code for Spark 2.4, 3.0, Scala 2.11, 2.12, Python 3.7, 3.8, 3.9. See here.

Sedona-core

Bug fix:

  • PR 443: read multiple Shape Files by multiPartitions
  • PR 451 (==API change==): modify CRSTransform to ignore datum shift

New function:

API / behavior change:

  • PR 488: JoinQuery.SpatialJoinQuery/DistanceJoinQuery now returns <Geometry, List> instead of <Geometry, HashSet> because we can no longer use HashSet in Sedona for duplicates removal. All original duplicates in both input RDDs will be preserved in the output.

Sedona-sql

Bug fix:

  • SEDONA-8 (==API change==): ST_Transform slow due to lock contention.
  • PR 427: ST_Point and ST_PolygonFromEnvelope now allows Double type

New function:

  • PR 499: ST_Azimuth, ST_X, ST_Y, ST_StartPoint, ST_Boundary, ST_EndPoint, ST_ExteriorRing, ST_GeometryN, ST_InteriorRingN, ST_Dump, ST_DumpPoints, ST_IsClosed, ST_NumInteriorRings, ST_AddPoint, ST_RemovePoint, ST_IsRing
  • PR 459: ST_LineMerge
  • PR 460: ST_NumGeometries
  • PR 469: ST_AsGeoJSON
  • SEDONA-8: ST_FlipCoordinates

Behavior change:

  • PR 480: Aggregate Functions rewrite for new Aggregator API. The functions can be used as typed functions in code and enable compilation-time type check.

API change:

  • SEDONA-11: Adapter.toDf() will directly generate a geometry type column. ST_GeomFromWKT is no longer needed.

Sedona-viz

API change: Drop the function which can generate SVG vector images because the required library has an incompatible license and the SVG image is not good at plotting big data

Sedona Python

API/Behavior change:

  • Python-to-Sedona adapter is moved to a separate module. To use Sedona Python, see here

New function:

  • PR 448: Add support for partition number in spatialPartitioning function spatial_rdd.spatialPartitioning(grid_type, NUM_PARTITION)