Apache Hamilton

Apache Hamilton dataflow materialization plan

File structure

The Apache Hamilton refactor achieves the same data transformations. The project fits in essentially 2 files:

  • dataflow.py contains regular Python functions to define the dataflow (equivalent to Dagster tutorial/assets.py).
  • run.py contains the “driver code” to load and execute the dataflow (equivalent to Dagster tutorial/__init__.py).
  • notebook.ipynb contains code to execute the dataflow (similar code to run.py), but allows for an interactive exploration of results.

The directory also include mock_api.py which simulates a resource in the Dagster tutorial, and dataflow.png which is an artifact generated by the Apache Hamilton Driver.

Instructions

  1. Run the code
    python run.py
    

Going further