tree: 30d4bf533b1d058fb98032d6c39ee5c4f0dd1503 [search]
  1. create-release/
  2. deps/
  3. spark-test-image/
  4. spark-test-image-util/
  5. sparktestsupport/
  6. .rat-excludes
  7. .scalafmt.conf
  8. agent_docs_lint.py
  9. change-scala-version.sh
  10. check-license
  11. check-protos.py
  12. check_pyspark_custom_errors.py
  13. checkstyle-suppressions.xml
  14. checkstyle.xml
  15. connect-gen-protos.sh
  16. connect-jvm-client-mima-check
  17. create_jira_and_branch.py
  18. create_spark_jira.py
  19. dispatch-if-changed.sh
  20. eslint.js
  21. free_disk_space
  22. free_disk_space_container
  23. gen-protos.sh
  24. generate_srs_registry.py
  25. is-changed.py
  26. java-file-header
  27. lint-java
  28. lint-js
  29. lint-python
  30. lint-r
  31. lint-r.R
  32. lint-scala
  33. make-distribution.sh
  34. merge_spark_pr.py
  35. mima
  36. next_version_candidates.py
  37. package-lock.json
  38. package.json
  39. pip-sanity-check.py
  40. pr_merge_status.py
  41. protobuf-breaking-changes-check.sh
  42. py-cleanup
  43. README.md
  44. reformat-python
  45. requirements.txt
  46. run-pip-tests
  47. run-tests
  48. run-tests.py
  49. sbt-checkstyle
  50. scalafmt
  51. scalastyle
  52. spark_jira_utils.py
  53. spark_merge_footer.py
  54. streaming-gen-protos.sh
  55. structured_logging_style.py
  56. test-dependencies.sh
dev/README.md

Spark Developer Scripts

This directory contains scripts useful to developers when packaging, testing, or committing to Spark.

Many of these scripts require Apache credentials to work correctly.

Testing sparktestsupport

The tests in sparktestsupport/tests/, which include the doctests of every sparktestsupport module, run automatically in CI and in dev/run-tests. To run them directly:

python3 -m unittest discover -s dev/sparktestsupport/tests -t dev

Keep short usage examples as doctests. Put regression cases, module-graph invariants, and tests requiring setup or mocks in the test files.