Here you'll find a website that curates a collection of Hamilton Dataflows that are ready to be used in your own projects. They are user-contributed and maintained, with the goal of making it easier for you to get started with Hamilton.
We expect this collection to grow over time, so check back often! As dataflows become mature we will move them into the official DAGWorks sub-package of this site and become maintained by DAGWorks's Hamilton team.
☝️ Use the search bar above to quickly find dataflows based on keyword search.
👈 On the left hand can find dataflows organized by User, DAGWorks, and tags.
There are two methods to get access to dataflows presented here.
Assumptions:
pip install sf-hamilton to get started. Come back here when you're ready to use Hamilton.For more extensive documentation, please see Hamilton User Contrib documentation.
Here we dynamically download the dataflow from the internet and execute it. This is useful for quickly iterating in a notebook and pulling in just the dataflow you need.
from hamilton import dataflow, driver # download into ~/.hamilton/dataflows and load the module -- WARNING: ensure you know what code you're importing! # NAME_OF_DATAFLOW = dataflow.import_module("NAME_OF_DATAFLOW") # if official dataflow NAME_OF_DATAFLOW = dataflow.import_module("NAME_OF_DATAFLOW", "NAME_OF_USER") dr = ( driver.Builder() .with_config({}) # replace with configuration as appropriate .with_modules(NAME_OF_DATAFLOW) .build() ) # execute the dataflow, specifying what you want back. Will return a dictionary. result = dr.execute( [NAME_OF_DATAFLOW.FUNCTION_NAME, ...], # this specifies what you want back inputs={...} # pass in inputs as appropriate )
See this notebook for an example.
This approach relies on you installing the package on your system. This is the recommended path for production purposes as you can version-lock your dependencies.
To install the package, run:
pip install sf-hamilton-contrib --upgrade
Once installed, you can import the dataflows as follows.
Things you need to know:
from hamilton import driver # from hamilton.contrib.dagworks import NAME_OF_DATAFLOW from hamilton.contrib.user.NAME_OF_USER import NAME_OF_DATAFLOW dr = ( driver.Builder() .with_config({}) # replace with configuration as appropriate .with_modules(NAME_OF_DATAFLOW) .build() ) # execute the dataflow, specifying what you want back. Will return a dictionary. result = dr.execute( [NAME_OF_DATAFLOW.FUNCTION_NAME, ...], # this specifies what you want back inputs={...} # pass in inputs as appropriate )
Getting started is one thing, but then modifying to your needs is another. So we have a prescribed flow to enable you to take a dataflow, and copy the code to a place of your choosing. This allows you to easily modify the dataflow as you see fit.
Run this in a notebook or python script to copy the dataflow to a directory of your choosing.
from hamilton import dataflows # dynamically pull and then copy NAME_OF_DATAFLOW = dataflows.import_module("NAME_OF_DATAFLOW", "NAME_OF_USER") dataflows.copy(NAME_OF_DATAFLOW, destination_path="PATH_TO_DIRECTORY") # copy from the installed library from hamilton.contrib.user.NAME_OF_USER import NAME_OF_DATAFLOW dataflows.copy(NAME_OF_DATAFLOW, destination_path="PATH_TO_DIRECTORY")
You can then modify/import the code as you see fit. See copy() for more details.
If you have a dataflow that you would like to share with the community, please submit a pull request to this repository. We will review your dataflow and if it meets our standards we will add it to the package. To submit a pull request please use this link as it'll take you to the specific PR template.
We want to ensure that the dataflows in this package are of high quality and are easy to use. To that end, we have a set of standards that we expect all dataflows to meet. If you have any questions, please reach out.
Standards:
Do you have the following?
Join our slack community to chat/ask Qs/etc.