Merge pull request #169 from jiayuasu/master

Update README
tree: d71c08908a324984229d473f8f126d81e8c1ce16
  1. core/
  2. sql/
  3. viz/
  4. .gitignore
  5. .travis.yml
  6. _config.yml
  7. LICENSE
  8. pom.xml
  9. README.md
README.md

GeoSpark Logo

StatusStableLatestSource codeSpark compatibility
GeoSparkMaven Central with version prefix filterSonatype Nexus (Snapshots)Build StatusSpark 2.X, 1.X
GeoSparkSQLMaven Central with version prefix filterSonatype Nexus (Snapshots)Build StatusSpark SQL 2.1
GeoSparkVizMaven Central with version prefix filterSonatype Nexus (Snapshots)Build StatusSpark 2.X, 1.X

GeoSpark@Twitter||GeoSpark Discussion Board||Join the chat at https://gitter.im/geospark-datasys/Lobby

GeoSpark is listed as Infrastructure Project on Apache Spark Official Third Party Project Page

GeoSpark is a cluster computing system for processing large-scale spatial data. GeoSpark extends Apache Spark with a set of out-of-the-box Spatial Resilient Distributed Datasets (SRDDs) that efficiently load, process, and analyze large-scale spatial data across machines. GeoSpark provides APIs for Apache Spark programmer to easily develop their spatial analysis programs with Spatial Resilient Distributed Datasets (SRDDs) which have in house support for geometrical and Spatial Queries (Range, K Nearest Neighbors, Join).

GeoSpark artifacts are hosted in Maven Central: Maven Central Coordinates

Version release notes: click here

News!

  • GeoSpark 1.0 is released. This release mainly includes a complete version of GeoSparkSQL. Look at these exciting features! Documents are at GeoSpark Wiki
    • Supports SQL/MM-Part3, Spatial SQL standard
    • Supports pure Spark SQL statement. No DSL style any more!
    • Supports Spark query optimizer: the beloved GeoSpark Spatial Join / predicate pushdown!
    • Supports multiple GeoSpark parameters: take the control of your own program!
    • Supports constructors, functions, aggregate functions, and predicates!
  • GeoSparkSQL 1.0 is released. This module contains contributions from Jia Yu, Masha Basmanova, Mohamed Sarwat and Zongsi Zhang. Especially, we want to thank Masha for her great patch on designing spatial join strategy and optimization.
  • GeoSpark 0.9.1 is released (more details in Release notes)
  • Welcome GeoSpark new contributor, Masha Basmanova (@mbasmanova) from Facebook!
  • Welcome GeoSpark new contributor, Zongsi Zhang (@zongsizhang) from Arizona State University!

Important features (more)

Spatial SQL

GeoSparkSQL fully supports Apache Spark SQL. Features are as follows:

  • Supports SQL/MM-Part3, Spatial SQL standard
  • Supports pure Spark SQL statement. No DSL style any more!
  • Supports Spark query optimizer: the beloved GeoSpark Spatial Join / predicate pushdown!
  • Supports multiple GeoSpark parameters: take the control of your own program!
  • Supports constructors, functions, aggregate functions, and predicates!

Spatial Resilient Distributed Datasets (SRDDs)

Supported Spatial RDDs: PointRDD, RectangleRDD, PolygonRDD, LineStringRDD

The generic SpatialRDD supports heterogenous geometries:

  • Point
  • Polygon
  • Line string
  • Multi-point
  • Multi-polygon
  • Multi-line string
  • GeometryCollection
  • Circle

Supported input data format

Native input format support:

  • CSV
  • TSV
  • WKT
  • GeoJSON (single-line compact format)
  • NASA Earth Data NetCDF/HDF
  • ESRI ShapeFile(.shp, .shx, .dbf)

User-supplied input format mapper: Any single-line input formats

Spatial Partitioning

Supported Spatial Partitioning techniques: Quad-Tree (recommend), KDB-Tree (recommend), R-Tree, Voronoi diagram, Uniform grids (Experimental), Hilbert Curve (Experimental)

Spatial Index

Supported Spatial Indexes: Quad-Tree and R-Tree. R-Tree supports Spatial K Nearest Neighbors query.

Geometrical operation

DatasetBoundary, Minimum Bounding Rectangle, Polygon Union

Spatial Operation

Spatial Range Query, Distance Join Query, Spatial Join Query (Inside and Overlap), and Spatial K Nearest Neighbors Query.

Coordinate Reference System (CRS) Transformation (aka. Coordinate projection)

GeoSpark allows users to transform the original CRS (e.g., degree based coordinates such as EPSG:4326 and WGS84) to any other CRS (e.g., meter based coordinates such as EPSG:3857) so that it can accurately process both geographic data and geometrical data. Please specify your desired CRS in GeoSpark Spatial RDD constructor (Example).

Users

Companies that are using GeoSpark (incomplete list)

Please make a Pull Request to add yourself!

GeoSpark Tutorial (more)

GeoSpark full tutorial is available at GeoSpark GitHub Wiki: GeoSpark GitHub Wiki

GeoSpark Scala and Java template project is available here: Template Project

GeoSpark Function Use Cases: Scala Example, Java Example

GeoSpark Visualization Extension (GeoSparkViz)

GeoSparkViz is a large-scale in-memory geospatial visualization system.

GeoSparkViz provides native support for general cartographic design by extending GeoSpark to process large-scale spatial data. It can visulize Spatial RDD and Spatial Queries and render super high resolution image in parallel.

More details are available here: GeoSpark Visualization Extension

GeoSparkViz Gallery

Watch High Resolution on a real map

Publication

Jia Yu, Jinxuan Wu, Mohamed Sarwat. “A Demonstration of GeoSpark: A Cluster Computing Framework for Processing Big Spatial Data”. (demo paper) In Proceeding of IEEE International Conference on Data Engineering ICDE 2016, Helsinki, FI, May 2016

Jia Yu, Jinxuan Wu, Mohamed Sarwat. “GeoSpark: A Cluster Computing Framework for Processing Large-Scale Spatial Data”. (short paper) In Proceeding of the ACM International Conference on Advances in Geographic Information Systems ACM SIGSPATIAL GIS 2015, Seattle, WA, USA November 2015

Statement from our team

Benchmark

We welcome people to use GeoSpark for benchmark purpose. To achieve the best performance or enjoy all features of GeoSpark,

  • Please always use the latest version or state the version used in your benchmark so that we can trace back to the issues.
  • Please consider using GeoSpark core instead of GeoSparkSQL. Due to the limitation of SparkSQL (for instance, not support clustered index), we are not able to expose all features to SparkSQL.
  • Please open GeoSpark kryo serializer to reduce the memory footprint.

Paper citation

Currently, we have published two papers about GeoSpark. Only these two papers are associated with GeoSpark Development Team.

Contact

Questions

Contact

Project website

Please visit GeoSpark project wesbite for latest news and releases.

Data Systems Lab

GeoSpark is one of the projects initiated by Data Systems Lab at Arizona State University. The mission of Data Systems Lab is designing and developing experimental data management systems (e.g., database systems).