commit 991763c2cdf8a61c0724855d450928deeb87ff51
author Nicholas Chammas <nicholas.chammas@gmail.com> Wed May 01 09:55:39 2024 +0900
committer Hyukjin Kwon <gurwls223@apache.org> Wed May 01 09:55:39 2024 +0900
tree be631aebff42b0002af64f250204901f91da7037
parent c71d02ab7c809f0ac02fedddd3c18afe14326756

[SPARK-46894][PYTHON] Move PySpark error conditions into standalone JSON file

### What changes were proposed in this pull request?

Move PySpark error conditions into a standalone JSON file.

Adjust the instructions in `MANIFEST.in` so they package the new JSON file correctly.

### Why are the changes needed?

Having the JSON in its own file enables better IDE support for editing and managing the JSON. For example, VS Code will detect duplicate keys automatically and underline them.

This change also simplifies the logic to regenerate the JSON.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

1. Existing unit tests.
2. I confirmed that rewriting the JSON file works as intended:

    ```python
    from pyspark.errors.exceptions import _write_self
    _write_self()
    ```

3. I built a PySpark distribution, installed it, and confirmed the error class test still works.

    ```sh
    trash dist/
    ./dev/make-distribution.sh --pip -DskipTests -Phive

    # The JSON file is put in the dist/ directory.
    find dist -name 'error-conditions.json'
    # dist/python/pyspark/errors/error-conditions.json

    # It's also listed in SOURCES.txt.
    grep error-conditions.json dist/python/pyspark.egg-info/SOURCES.txt
    # pyspark/errors/error-conditions.json

    cd dist/python
    python -m venv venv
    source venv/bin/activate
    pip install .

    # It also gets installed into the newly created venv.
    find venv -name error-conditions.json
    # venv/lib/python3.11/site-packages/pyspark/errors/error-conditions.json

    # Running the test from inside the venv also succeeds.
    python pyspark/errors/tests/test_errors.py
    # Ran 3 tests in 0.000s
    #
    # OK
    ```

4. I repeated test 3, but this time used the generated ZIP to install PySpark in a fresh virtual environment:

    ```sh
    $ cd .../spark/
    $ trash dist
    $ ./dev/make-distribution.sh --pip -DskipTests -Phive
    $ cd .../Desktop/
    $ python -m venv venv
    $ source venv/bin/activate
    $ pip install .../spark/python/dist/pyspark-4.0.0.dev0.tar.gz
    $ find venv -name error-conditions.json
    venv/lib/python3.11/site-packages/pyspark/errors/error-conditions.json
    $ python
    Python 3.11.7 (main, Dec 11 2023, 02:35:11) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from pyspark.errors.error_classes import ERROR_CLASSES_MAP
    >>> len(ERROR_CLASSES_MAP)
    222
    ```

    You can see that the JSON file was packaged correctly and installed into the virtual environment, and `ERROR_CLASSES_MAP` loaded correctly as well.

5. I installed PySpark by including a different built ZIP in the `PYTHONPATH` as [described here][pyzip]:

    ```sh
    $ cd .../spark/
    $ trash dist
    $ ./dev/make-distribution.sh --pip -DskipTests -Phive
    $ cd .../Desktop/
    $ export SPARK_HOME=".../spark"
    $ export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
    $ python
    Python 3.11.7 (main, Dec 11 2023, 02:35:11) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from pyspark.errors.error_classes import ERROR_CLASSES_MAP
    >>> len(ERROR_CLASSES_MAP)
    222
    ```

[pyzip]: https://spark.apache.org/docs/3.5.1/api/python/getting_started/install.html#manually-downloading

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #44920 from nchammas/pyspark-error-json.

Authored-by: Nicholas Chammas <nicholas.chammas@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>