| ===================== |
| Run Hamilton at Scale |
| ===================== |
| |
| Hamilton enables a variety of tools for allowing you to scale your data processing by integrating with third-party libraries. |
| |
| Specifically, we have four examples that show how to scale Hamilton both by parallelizing transformations (ray and dask) and running |
| on larger, distributed datasets (pandas on spark, pyspark map UDFs). |
| |
| 1. Integrating hamilton with `pandas on spark <https://github.com/DAGWorks-Inc/hamilton/tree/main/examples/spark/pandas_on_spark>`_. |
| 2. Integrating hamilton with `ray <https://github.com/DAGWorks-Inc/hamilton/tree/main/examples/ray>`_. |
| 3. Integrating hamilton with `dask <https://github.com/DAGWorks-Inc/hamilton/tree/main/examples/dask>`_. |
| 4. Integrating hamilton with `pyspark <https://github.com/DagWorks-Inc/hamilton/tree/main/examples/spark/pyspark>`_. |
