Google Git
Sign in
apache/spark/refs/heads/master/./python
tree: e66893e86c9f1f317311a9bfd313358cf4f4c17e
  1. benchmarks/
  2. conf_viztracer/
  3. conf_vscode/
  4. docs/
  5. lib/
  6. packaging/
  7. pyspark/
  8. test_coverage/
  9. test_support/
  10. .coveragerc
  11. .gitignore
  12. asv
  13. MANIFEST.in
  14. mypy.ini
  15. README.md
  16. run-tests
  17. run-tests-with-coverage
  18. run-tests.py
  19. run-with-viztracer
  20. run-with-vscode-breakpoints
python/README.md

Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R (deprecated), and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

PySpark is the Python distribution of Spark.

Project home page: https://spark.apache.org/

Main documentation: https://spark.apache.org/docs/latest/

PySpark documentation: https://spark.apache.org/docs/latest/api/python/index.html

PySpark on PyPI

There are a few PySpark packages published by the Apache Spark project to PyPI:

  • pyspark: Classic PySpark, includes Spark assembly JARs
  • pyspark-connect: Classic PySpark with Spark Connect configured as the default, includes pyspark
  • pyspark-client: Pure Python Spark Connect client, no JARs or JRE needed

For more information, see the installation guide. If you're building PySpark from source, see Building Spark and pyproject.toml.

Python vs. “Full” Distribution of Spark

PySpark is not intended to be a complete distribution of Spark. It's meant for local development or for interacting with an existing cluster (be it Spark standalone, YARN, or Kubernetes). Using PySpark to set up a new standalone Spark cluster is not supported. To set up a standalone cluster please use the full distribution of Spark.

When using PySpark with an existing Spark cluster you must ensure that the major and minor version (e.g. 4.3.*) match or you may experience odd errors.