GITBOOK-4: adds we've moved notice
diff --git a/README.md b/README.md index 7e6eefc..03b56e1 100644 --- a/README.md +++ b/README.md
@@ -1,79 +1,7 @@ --- -description: >- - The open source framework Hamilton (https://github.com/DAGWorks-Inc/hamilton), - originally built to manage and run Stitch Fix's data pipelines. +description: Please see the new docs at https://hamilton.readthedocs.io/en/latest/ --- -# Hamilton +# We've moved to https://hamilton.readthedocs.io/en/latest/ -## Getting Started - -If you want to jump in head first, we have a simple tutorial for getting started! To ask questions, please join our [slack community!](https://join.slack.com/t/hamilton-opensource/shared\_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg) - -{% content-ref url="less-than-15-minutes-to-mastery/" %} -[less-than-15-minutes-to-mastery](less-than-15-minutes-to-mastery/) -{% endcontent-ref %} - -## What is Hamilton? - -It's a general purpose micro-framework for creating [dataflows](https://en.wikipedia.org/wiki/Dataflow) from python functions! - -Specifically, Hamilton defines a novel paradigm, that allows you to specify a flow of (delayed) execution, that forms a Directed Acyclic Graph (DAG). It was original built to solve creating wide (1000+) column dataframes at Stitch Fix. Core to the design of Hamilton is a clear mapping of function name to dataflow output. That is, Hamilton forces a certain paradigm with writing functions, and aims for DAG clarity, easy modifications, _with always unit testable and naturally documentable code!_ - -For the backstory on how Hamilton came about, see the original Stitch Fix [blog post!](https://multithreaded.stitchfix.com/blog/2021/10/14/functions-dags-hamilton/). - -Hamilton's method of defining dataflows presents a new paradigm when it comes to creating, um, dataframes (let's use dataframes as an example, otherwise you can create _ANY_ python object). Rather than thinking about manipulating a central dataframe procedurally, and extracting the data you want, as is normal in some data engineering/data science work, you instead think about the column(s) (a.k.a. outputs) you want to create, and what inputs are required. There is no need for you to think about maintaining how to create this dataframe, meaning you do not need to think about any "glue" code; this is all taken care of by the Hamilton framework. Specifically, Hamilton enables you to run your dataflow with the following steps: - -1. Define your dataflow (a.k.a. pipeline), as a set of transforms using Hamilton's paradigm for writing python functions. -2. Specify the parameter values your dataflow requires -3. Specify which outputs you want to compute from your dataflow - -The rest is delegated to the framework, which handles the computation for you. - -Let's illustrate this with some code. If you were asked to write a simple transform (let's use pandas for the sake of argument), you may decide to write something simple like this: - -```python -df['col_c'] = df['col_a'] + df['col_b'] -``` - -To represent this in a way Hamilton can understand, you write: - -```python -def col_c(col_a: pd.Series, col_b: pd.Series) -> pd.Series: - """Creating column c from summing column a and column b.""" - return col_a + col_b -``` - - - -The Hamilton framework takes the above code, forms it into a computational DAG, and executes it for you! - -## Hamilton Open Source Community - -If you have questions, have ideas, or need help with Hamilton, join us on [slack](https://join.slack.com/t/hamilton-opensource/shared\_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg), and we'll try to help! - -## Installing Hamilton - -Installation should be quick and as simple as: - -``` -> pip install sf-hamilton -``` - -For more information please see [installing.md](less-than-15-minutes-to-mastery/installing.md "mention").  - -## License - -Hamilton is released under the [BSD 3-Clause Clear License](https://github.com/DAGWorks-Inc/hamilton/blob/main/LICENSE). - -## Contributing - -We take contributions, large and small. We operate via a Code of Conduct and expect anyone contributing to do the same. - -## User Guide - -Dive a little deeper and start exploring our API reference to get an idea of everything that's possible with the API: - -{% content-ref url="reference/api-reference/" %} -[api-reference](reference/api-reference/) -{% endcontent-ref %} +[https://hamilton.readthedocs.io/en/latest/](https://hamilton.readthedocs.io/en/latest/)
diff --git a/SUMMARY.md b/SUMMARY.md index f62e679..6fb4b3f 100644 --- a/SUMMARY.md +++ b/SUMMARY.md
@@ -1,53 +1,51 @@ # Table of contents -* [Hamilton](README.md) -* [<15 Minutes to Mastery](less-than-15-minutes-to-mastery/README.md) - * [Installing](less-than-15-minutes-to-mastery/installing.md) - * [Writing Your First Dataflow](less-than-15-minutes-to-mastery/writing-your-first-dataflow.md) - * [Running Your Dataflow](less-than-15-minutes-to-mastery/running-your-dataflow.md) - * [Digging in Further](less-than-15-minutes-to-mastery/digging-in-further.md) -* [Overview of Concepts](overview-of-concepts/README.md) - * [Lexicon](overview-of-concepts/lexicon.md) - * [Writing Transform Functions](overview-of-concepts/writing-transform-functions.md) - * [The Hamilton Driver](overview-of-concepts/the-hamilton-driver.md) - * [Decorators](overview-of-concepts/decorators.md) -* [Tutorials](tutorials/README.md) - * [Using Hamilton in a notebook](tutorials/using-hamilton-in-a-notebook.md) - * [Integrating Decorators](tutorials/integrating-decorators.md) - * [Data Quality Checks](tutorials/data-quality-checks.md) - * [⚡ Ray](tutorials/ray.md) - * [Model Training](tutorials/model-training.md) - * [Dask](tutorials/dask.md) - * [Spark](tutorials/spark.md) -* [Best Practices](best-practices/README.md) - * [Using within your ETL System](best-practices/using-within-your-etl-system.md) - * [Migrating to Hamilton](best-practices/migrating-to-hamilton.md) - * [Function Naming](best-practices/function-naming.md) - * [Code Organization](best-practices/code-organization.md) - * [Function Modifiers](best-practices/function-modifiers.md) - * [Common Indices](best-practices/common-indices.md) - * [Loading Data](best-practices/loading-data.md) - * [Output Immutability](best-practices/output-immutability.md) -* [Extensions](extensions.md) -* [Talks | Podcasts | Blogs | Papers](talks-or-podcasts-or-blogs-or-papers.md) -* [Hamilton Community](hamilton-community.md) -* [License](license.md) -* [Contributing](contributing.md) +* [We've moved to https://hamilton.readthedocs.io/en/latest/](README.md) -## Reference +## old-docs -* [API Reference](reference/api-reference/README.md) - * [Available Decorators](reference/api-reference/available-decorators.md) - * [Available Drivers](reference/api-reference/available-drivers.md) - * [Available Result Builders](reference/api-reference/available-result-builders.md) - * [Available Graph Adapters](reference/api-reference/available-graph-adapters.md) -* [API Extensions](reference/api-extensions/README.md) - * [Custom Result Builders](reference/api-extensions/custom-result-builders.md) - * [Custom Graph Adapters](reference/api-extensions/custom-graph-adapters.md) - * [Custom Drivers](reference/api-extensions/custom-drivers.md) - * [Custom Decorators](reference/api-extensions/custom-decorators.md) - -## unorganized-docs - -* [Docs to be organized](unorganized-docs/docs-to-be-organized/README.md) - * [Writing Drivers](unorganized-docs/docs-to-be-organized/writing-drivers.md) +* [Hamilton](old-docs/readme.md) +* [<15 Minutes to Mastery](old-docs/less-than-15-minutes-to-mastery/README.md) + * [Installing](old-docs/less-than-15-minutes-to-mastery/installing.md) + * [Writing Your First Dataflow](old-docs/less-than-15-minutes-to-mastery/writing-your-first-dataflow.md) + * [Running Your Dataflow](old-docs/less-than-15-minutes-to-mastery/running-your-dataflow.md) + * [Digging in Further](old-docs/less-than-15-minutes-to-mastery/digging-in-further.md) +* [Tutorials](old-docs/tutorials/README.md) + * [Using Hamilton in a notebook](old-docs/tutorials/using-hamilton-in-a-notebook.md) + * [Integrating Decorators](old-docs/tutorials/integrating-decorators.md) + * [Data Quality Checks](old-docs/tutorials/data-quality-checks.md) + * [⚡ Ray](old-docs/tutorials/ray.md) + * [Model Training](old-docs/tutorials/model-training.md) + * [Dask](old-docs/tutorials/dask.md) + * [Spark](old-docs/tutorials/spark.md) +* [Best Practices](old-docs/best-practices/README.md) + * [Using within your ETL System](old-docs/best-practices/using-within-your-etl-system.md) + * [Migrating to Hamilton](old-docs/best-practices/migrating-to-hamilton.md) + * [Function Naming](old-docs/best-practices/function-naming.md) + * [Code Organization](old-docs/best-practices/code-organization.md) + * [Function Modifiers](old-docs/best-practices/function-modifiers.md) + * [Common Indices](old-docs/best-practices/common-indices.md) + * [Loading Data](old-docs/best-practices/loading-data.md) + * [Output Immutability](old-docs/best-practices/output-immutability.md) +* [Extensions](old-docs/extensions.md) +* [Talks | Podcasts | Blogs | Papers](old-docs/talks-or-podcasts-or-blogs-or-papers.md) +* [Hamilton Community](old-docs/hamilton-community.md) +* [License](old-docs/license.md) +* [Contributing](old-docs/contributing.md) +* [API Reference](old-docs/api-reference/README.md) + * [Available Decorators](old-docs/api-reference/available-decorators.md) + * [Available Drivers](old-docs/api-reference/available-drivers.md) + * [Available Result Builders](old-docs/api-reference/available-result-builders.md) + * [Available Graph Adapters](old-docs/api-reference/available-graph-adapters.md) +* [API Extensions](old-docs/api-extensions/README.md) + * [Custom Result Builders](old-docs/api-extensions/custom-result-builders.md) + * [Custom Graph Adapters](old-docs/api-extensions/custom-graph-adapters.md) + * [Custom Drivers](old-docs/api-extensions/custom-drivers.md) + * [Custom Decorators](old-docs/api-extensions/custom-decorators.md) +* [Docs to be organized](old-docs/docs-to-be-organized/README.md) + * [Writing Drivers](old-docs/docs-to-be-organized/writing-drivers.md) +* [Overview of Concepts](old-docs/overview-of-concepts/README.md) + * [Lexicon](old-docs/overview-of-concepts/lexicon.md) + * [Writing Transform Functions](old-docs/overview-of-concepts/writing-transform-functions.md) + * [The Hamilton Driver](old-docs/overview-of-concepts/the-hamilton-driver.md) + * [Decorators](old-docs/overview-of-concepts/decorators.md)
diff --git a/reference/api-extensions/README.md b/old-docs/api-extensions/README.md similarity index 100% rename from reference/api-extensions/README.md rename to old-docs/api-extensions/README.md
diff --git a/reference/api-extensions/custom-decorators.md b/old-docs/api-extensions/custom-decorators.md similarity index 100% rename from reference/api-extensions/custom-decorators.md rename to old-docs/api-extensions/custom-decorators.md
diff --git a/reference/api-extensions/custom-drivers.md b/old-docs/api-extensions/custom-drivers.md similarity index 100% rename from reference/api-extensions/custom-drivers.md rename to old-docs/api-extensions/custom-drivers.md
diff --git a/reference/api-extensions/custom-graph-adapters.md b/old-docs/api-extensions/custom-graph-adapters.md similarity index 100% rename from reference/api-extensions/custom-graph-adapters.md rename to old-docs/api-extensions/custom-graph-adapters.md
diff --git a/reference/api-extensions/custom-result-builders.md b/old-docs/api-extensions/custom-result-builders.md similarity index 100% rename from reference/api-extensions/custom-result-builders.md rename to old-docs/api-extensions/custom-result-builders.md
diff --git a/reference/api-reference/README.md b/old-docs/api-reference/README.md similarity index 100% rename from reference/api-reference/README.md rename to old-docs/api-reference/README.md
diff --git a/reference/api-reference/available-decorators.md b/old-docs/api-reference/available-decorators.md similarity index 100% rename from reference/api-reference/available-decorators.md rename to old-docs/api-reference/available-decorators.md
diff --git a/reference/api-reference/available-drivers.md b/old-docs/api-reference/available-drivers.md similarity index 100% rename from reference/api-reference/available-drivers.md rename to old-docs/api-reference/available-drivers.md
diff --git a/reference/api-reference/available-graph-adapters.md b/old-docs/api-reference/available-graph-adapters.md similarity index 100% rename from reference/api-reference/available-graph-adapters.md rename to old-docs/api-reference/available-graph-adapters.md
diff --git a/reference/api-reference/available-result-builders.md b/old-docs/api-reference/available-result-builders.md similarity index 100% rename from reference/api-reference/available-result-builders.md rename to old-docs/api-reference/available-result-builders.md
diff --git a/best-practices/README.md b/old-docs/best-practices/README.md similarity index 100% rename from best-practices/README.md rename to old-docs/best-practices/README.md
diff --git a/best-practices/code-organization.md b/old-docs/best-practices/code-organization.md similarity index 100% rename from best-practices/code-organization.md rename to old-docs/best-practices/code-organization.md
diff --git a/best-practices/common-indices.md b/old-docs/best-practices/common-indices.md similarity index 100% rename from best-practices/common-indices.md rename to old-docs/best-practices/common-indices.md
diff --git a/best-practices/function-modifiers.md b/old-docs/best-practices/function-modifiers.md similarity index 79% rename from best-practices/function-modifiers.md rename to old-docs/best-practices/function-modifiers.md index 4c7a95b..4287986 100644 --- a/best-practices/function-modifiers.md +++ b/old-docs/best-practices/function-modifiers.md
@@ -6,13 +6,13 @@ Hamilton has a bunch of function modifiers, i.e. python decorators, to modify function behavior. -The behaviors vary based on the function modifier. Please see [available-decorators.md](../reference/api-reference/available-decorators.md "mention") for the current list of supported ones. +The behaviors vary based on the function modifier. Please see [available-decorators.md](../api-reference/available-decorators.md "mention") for the current list of supported ones. ## Why would I use them? These function modifiers are either to: -1. Enable you to make lots of functions that vary by inputs concisely. E.g. to keep your code DRYer.  +1. Enable you to make lots of functions that vary by inputs concisely. E.g. to keep your code DRYer. 2. Provide functionality to make Hamilton more powerful. E.g. so you can break apart multiple outputs. There unfortunately isn't an easy thing to say here, other than, read the list of decorators and their functionality, and then if you see the need arise, use them 😀.
diff --git a/best-practices/function-naming.md b/old-docs/best-practices/function-naming.md similarity index 100% rename from best-practices/function-naming.md rename to old-docs/best-practices/function-naming.md
diff --git a/best-practices/loading-data.md b/old-docs/best-practices/loading-data.md similarity index 100% rename from best-practices/loading-data.md rename to old-docs/best-practices/loading-data.md
diff --git a/best-practices/migrating-to-hamilton.md b/old-docs/best-practices/migrating-to-hamilton.md similarity index 78% rename from best-practices/migrating-to-hamilton.md rename to old-docs/best-practices/migrating-to-hamilton.md index ba0e93a..e421c30 100644 --- a/best-practices/migrating-to-hamilton.md +++ b/old-docs/best-practices/migrating-to-hamilton.md
@@ -9,10 +9,10 @@ Create a way to easily & frequently compare results. 1. Integrate with continuous integration (CI) system if you can. -2. 🔎🐛 Having a means that tests code early & often will helps diagnose bugs in your old code (most likely) or your new implementation (less likely). -3. Specifically, have a system to compare the output of your Hamilton code, to compare to the output of your existing system.  +2. 🔎🐛 Having a means that tests code early & often will helps diagnose bugs in your old code (most likely) or your new implementation (less likely). +3. Specifically, have a system to compare the output of your Hamilton code, to compare to the output of your existing system. -.svg>) +.svg>) ## Integrate into your code base via a "_custom wrapper object_" @@ -20,7 +20,7 @@ Specifically, this custom wrapper object class's purpose is to match your existing API expectations. It will act as the translation layer from your existing API expectations, to what running Hamilton requires, and back. In Hamilton terminology, this is a _Custom Driver Wrapper_, since it wraps around the Hamilton Driver class. - + This is a best practice because:
diff --git a/best-practices/output-immutability.md b/old-docs/best-practices/output-immutability.md similarity index 100% rename from best-practices/output-immutability.md rename to old-docs/best-practices/output-immutability.md
diff --git a/best-practices/using-within-your-etl-system.md b/old-docs/best-practices/using-within-your-etl-system.md similarity index 100% rename from best-practices/using-within-your-etl-system.md rename to old-docs/best-practices/using-within-your-etl-system.md
diff --git a/contributing.md b/old-docs/contributing.md similarity index 100% rename from contributing.md rename to old-docs/contributing.md
diff --git a/unorganized-docs/docs-to-be-organized/README.md b/old-docs/docs-to-be-organized/README.md similarity index 100% rename from unorganized-docs/docs-to-be-organized/README.md rename to old-docs/docs-to-be-organized/README.md
diff --git a/unorganized-docs/docs-to-be-organized/writing-drivers.md b/old-docs/docs-to-be-organized/writing-drivers.md similarity index 72% rename from unorganized-docs/docs-to-be-organized/writing-drivers.md rename to old-docs/docs-to-be-organized/writing-drivers.md index 877dec1..1ba2e28 100644 --- a/unorganized-docs/docs-to-be-organized/writing-drivers.md +++ b/old-docs/docs-to-be-organized/writing-drivers.md
@@ -4,6 +4,6 @@ At the end of the day, Hamilton is not particularly opinionated around _how_ or _when_ you run your dataflow. Only around how you write it. So, writing a driver is up to you. That said, we recommend following some general-purpose guidelines: -1. Include logic with side-effects in your drivers. E.G. writing to dbs, etc... For ease/reuse of pipelines, we recommend executing side-effect code once, after the pipeline completes. If you need more complex materialization, etc... see read about [GraphAdapters](../../reference/api-extensions/custom-graph-adapters.md) -- these allow you to customize execution of nodes. +1. Include logic with side-effects in your drivers. E.G. writing to dbs, etc... For ease/reuse of pipelines, we recommend executing side-effect code once, after the pipeline completes. If you need more complex materialization, etc... see read about [GraphAdapters](../api-extensions/custom-graph-adapters.md) -- these allow you to customize execution of nodes. 2. When migrating, build the API to match your current one! E.G. write to the same tables, have the same function signature, etc... This allows you to migrate easily, only changing one component (the dataflow) at a time 3. Control the loading of modules/config within your driver. E.G. which modules are selected, etc... This should function as a shield from the user of your driver (likely you) from the complexity of the underlying DAG.
diff --git a/extensions.md b/old-docs/extensions.md similarity index 89% rename from extensions.md rename to old-docs/extensions.md index 10ee02b..2e60282 100644 --- a/extensions.md +++ b/old-docs/extensions.md Binary files differ
diff --git a/hamilton-community.md b/old-docs/hamilton-community.md similarity index 100% rename from hamilton-community.md rename to old-docs/hamilton-community.md
diff --git a/less-than-15-minutes-to-mastery/README.md b/old-docs/less-than-15-minutes-to-mastery/README.md similarity index 100% rename from less-than-15-minutes-to-mastery/README.md rename to old-docs/less-than-15-minutes-to-mastery/README.md
diff --git a/less-than-15-minutes-to-mastery/digging-in-further.md b/old-docs/less-than-15-minutes-to-mastery/digging-in-further.md similarity index 100% rename from less-than-15-minutes-to-mastery/digging-in-further.md rename to old-docs/less-than-15-minutes-to-mastery/digging-in-further.md
diff --git a/less-than-15-minutes-to-mastery/installing.md b/old-docs/less-than-15-minutes-to-mastery/installing.md similarity index 100% rename from less-than-15-minutes-to-mastery/installing.md rename to old-docs/less-than-15-minutes-to-mastery/installing.md
diff --git a/less-than-15-minutes-to-mastery/running-your-dataflow.md b/old-docs/less-than-15-minutes-to-mastery/running-your-dataflow.md similarity index 100% rename from less-than-15-minutes-to-mastery/running-your-dataflow.md rename to old-docs/less-than-15-minutes-to-mastery/running-your-dataflow.md
diff --git a/less-than-15-minutes-to-mastery/writing-your-first-dataflow.md b/old-docs/less-than-15-minutes-to-mastery/writing-your-first-dataflow.md similarity index 100% rename from less-than-15-minutes-to-mastery/writing-your-first-dataflow.md rename to old-docs/less-than-15-minutes-to-mastery/writing-your-first-dataflow.md
diff --git a/license.md b/old-docs/license.md similarity index 100% rename from license.md rename to old-docs/license.md
diff --git a/overview-of-concepts/README.md b/old-docs/overview-of-concepts/README.md similarity index 100% rename from overview-of-concepts/README.md rename to old-docs/overview-of-concepts/README.md
diff --git a/overview-of-concepts/decorators.md b/old-docs/overview-of-concepts/decorators.md similarity index 92% rename from overview-of-concepts/decorators.md rename to old-docs/overview-of-concepts/decorators.md index 30152eb..f0eb9bd 100644 --- a/overview-of-concepts/decorators.md +++ b/old-docs/overview-of-concepts/decorators.md
@@ -4,11 +4,11 @@ # Decorators -While the one to one mapping of node -> function implementation is powerful, it can sometimes lead to repeated code. In order to avoid this, Hamilton employs decorators to promote reuse of business logic. In this section we will go over a few different classes of decorators with some examples. For reference on specific decorators, see [available-decorators.md](../reference/api-reference/available-decorators.md "mention").   +While the one to one mapping of node -> function implementation is powerful, it can sometimes lead to repeated code. In order to avoid this, Hamilton employs decorators to promote reuse of business logic. In this section we will go over a few different classes of decorators with some examples. For reference on specific decorators, see [available-decorators.md](../api-reference/available-decorators.md "mention"). In Hamilton, decorators allow the author of Hamilton dataflows to modify functions as well as their role in the DAG. Decorators do one (or multiple) of the following: -**Determine whether a function should exist as a transform in the DAG.**  +**Determine whether a function should exist as a transform in the DAG.** `@config.when` is the primary example of this. It allows the user to determine nodes that are only present in certain DAG configurations. For instance, say that your company has different marketing spend source in different geographic regions. You might write code that looks like this: @@ -50,7 +50,7 @@ pass ``` -Note that we have no implementation for the functions! They're automagically replaced. There are other decorators that similar things.  +Note that we have no implementation for the functions! They're automagically replaced. There are other decorators that similar things. **Split the function into multiple nodes in the DAG** @@ -71,6 +71,6 @@ While there are decorators that don't quite fit into the above, these should give you a sense of how and why decorators are used. For more information on available decorators, see: -{% content-ref url="../reference/api-reference/available-decorators.md" %} -[available-decorators.md](../reference/api-reference/available-decorators.md) +{% content-ref url="../api-reference/available-decorators.md" %} +[available-decorators.md](../api-reference/available-decorators.md) {% endcontent-ref %}
diff --git a/overview-of-concepts/lexicon.md b/old-docs/overview-of-concepts/lexicon.md similarity index 100% rename from overview-of-concepts/lexicon.md rename to old-docs/overview-of-concepts/lexicon.md
diff --git a/overview-of-concepts/the-hamilton-driver.md b/old-docs/overview-of-concepts/the-hamilton-driver.md similarity index 88% rename from overview-of-concepts/the-hamilton-driver.md rename to old-docs/overview-of-concepts/the-hamilton-driver.md index e4e51f2..bac2f81 100644 --- a/overview-of-concepts/the-hamilton-driver.md +++ b/old-docs/overview-of-concepts/the-hamilton-driver.md
@@ -11,7 +11,7 @@ 3. Assembling the results. 4. Enabling you to visualize the DAG. -It is provided as an easy way for the user to specify the data she wants without dealing with the complexities of DAGs, function graphs, or nodes.  +It is provided as an easy way for the user to specify the data she wants without dealing with the complexities of DAGs, function graphs, or nodes. The basic structure of using the Hamilton Driver is: @@ -45,9 +45,9 @@ Static dataflows are only so useful. In the real world, we need to be able to configure both the shape of the DAG and the inputs to the DAG as part of the Hamilton driver. The default Hamilton driver comes with three input types that you can control. They both take the form `Dict[str, Any]`, i.e. a dictionary of string keys that maps to any object type. -1. **config** The config is a dictionary of strings to values. This is passed into the constructor of the Hamilton driver, as it is required to create the DAG. It _also_ get's passed into the DAG at runtime, so you have access to parameter values. See [decorators](decorators.md) for how the config can be used.  -2. **inputs **_****_ The _runtime inputs_ to the DAG. These have to be mutually disjoint from the config -- overrides don't make sense here, as the DAG has been constructed assuming fixed configs. -3. **overrides** Values to override nodes in a DAG. During execution, nothing upstream of these are computed.  +1. **config** The config is a dictionary of strings to values. This is passed into the constructor of the Hamilton driver, as it is required to create the DAG. It _also_ get's passed into the DAG at runtime, so you have access to parameter values. See [decorators](decorators.md) for how the config can be used. +2. \*\*inputs \*\*_\*\*\*\*_ The _runtime inputs_ to the DAG. These have to be mutually disjoint from the config -- overrides don't make sense here, as the DAG has been constructed assuming fixed configs. +3. **overrides** Values to override nodes in a DAG. During execution, nothing upstream of these are computed. Let's go through some examples that show you how to write a Hamilton function that allows it to be conditionally used depending on configuration. @@ -67,7 +67,7 @@ When the graph is compiled, the implementation of `avg_rolling_spend` varies based off of the configuration value. You would construct the driver with `config={'region' : 'US'}` , to get the desired behavior. -**You want to pass in the region/business line to change the behavior or a transform.** Say you have a big dataframe of marketing spend with columns representing the region, and also want to filter it out for the individual region. You would define the transform function as follows.  +**You want to pass in the region/business line to change the behavior or a transform.** Say you have a big dataframe of marketing spend with columns representing the region, and also want to filter it out for the individual region. You would define the transform function as follows. ```python def avg_rolling_spend(spend_by_country: pd.DataFrame, region: str) -> pd.Series: @@ -118,7 +118,7 @@ pass ``` -So we have a few implementations see [available-result-builders.md](../reference/api-reference/available-result-builders.md "mention") for the list. +So we have a few implementations see [available-result-builders.md](../api-reference/available-result-builders.md "mention") for the list. To use it, it needs to be paired with a GraphAdapter - onto the next section! @@ -135,9 +135,9 @@ # four functions not shown ``` -The default GraphAdapter is the `base`.`SimplePythonDataFrameGraphAdapter` which by default makes Hamilton try to build a pandas.DataFrame when `.execute()` is called.  +The default GraphAdapter is the `base`.`SimplePythonDataFrameGraphAdapter` which by default makes Hamilton try to build a pandas.DataFrame when `.execute()` is called. -If you want to tell Hamilton to return something else, we suggest starting with the `base.SimplePythonGraphAdapter` and writing a simple class & function that implements the `base.ResultMixin` interface and passing that in. See [available-graph-adapters.md](../reference/api-reference/available-graph-adapters.md "mention") and [available-result-builders.md](../reference/api-reference/available-result-builders.md "mention") for options. +If you want to tell Hamilton to return something else, we suggest starting with the `base.SimplePythonGraphAdapter` and writing a simple class & function that implements the `base.ResultMixin` interface and passing that in. See [available-graph-adapters.md](../api-reference/available-graph-adapters.md "mention") and [available-result-builders.md](../api-reference/available-result-builders.md "mention") for options. Otherwise, let's quickly walk through some options on how to execute a Hamilton DAG. @@ -148,18 +148,18 @@ 1. Do nothing -- and you'll get `base`.`SimplePythonDataFrameGraphAdapter` by default. 2. Use `base.SimplePythonGraphAdapter` and pass in a subclass of `base.ResultMixin` (you can create your own), and then pass that to the constructor of the Driver. - e.g. \ + e.g.\ `adapter = base.SimplePythonGraphAdapter(base.DictResult())` `dr = driver.Driver(..., adapter=adapter)` By passing in `base.DictResult()` we are telling Hamilton that the result of `execute()` should be a dictionary with a map of `output` to computed result. -#### Scaling Hamilton: Multi-core & Distributed Execution +#### Scaling Hamilton: Multi-core & Distributed Execution This functionality is currently in an "experimental" state. We think the code is solid, but it hasn't been used in a production environment for long. Thus the API to these GraphAdapters might change. -See the [`experimental`](https://github.com/stitchfix/hamilton/tree/main/hamilton/experimental) `` package for the current implementations. We encourage you to give them a spin and provide us with feedback. See [available-graph-adapters.md](../reference/api-reference/available-graph-adapters.md "mention") for more details. +See the [`experimental`](https://github.com/stitchfix/hamilton/tree/main/hamilton/experimental) \`\` package for the current implementations. We encourage you to give them a spin and provide us with feedback. See [available-graph-adapters.md](../api-reference/available-graph-adapters.md "mention") for more details. ### Calling Execute() @@ -178,13 +178,13 @@ `dr.visualize_execution(['desired_output1', 'desired_output2'], './my_file.dot', render_args)` -In addition to specifying the outputs you desire, you need to provide a path to save the created dot file and image, and then provide some arguments for rendering -- at minimum, pass in an empty dictionary. +In addition to specifying the outputs you desire, you need to provide a path to save the created dot file and image, and then provide some arguments for rendering -- at minimum, pass in an empty dictionary. #### Visualize the entire DAG constructed `dr.display_all_functions('./my_file.dot', render_args)` -You need to provide a path to save the created dot file and image, and then provide some optional arguments for rendering.  +You need to provide a path to save the created dot file and image, and then provide some optional arguments for rendering. ## Should I define my own Driver? @@ -212,6 +212,5 @@ ``` -That way, you can create the right API constructs to invoke Hamilton in your context, and then delegate to the stock Hamilton Driver. By doing so, it will ensure that your code continues to work, \ +That way, you can create the right API constructs to invoke Hamilton in your context, and then delegate to the stock Hamilton Driver. By doing so, it will ensure that your code continues to work,\ since we intend to honor the Hamilton Driver APIs with backwards compatibility as much as possible. -
diff --git a/overview-of-concepts/writing-transform-functions.md b/old-docs/overview-of-concepts/writing-transform-functions.md similarity index 100% rename from overview-of-concepts/writing-transform-functions.md rename to old-docs/overview-of-concepts/writing-transform-functions.md
diff --git a/old-docs/readme.md b/old-docs/readme.md new file mode 100644 index 0000000..2f47988 --- /dev/null +++ b/old-docs/readme.md
@@ -0,0 +1,79 @@ +--- +description: >- + The open source framework Hamilton (https://github.com/DAGWorks-Inc/hamilton), + originally built to manage and run Stitch Fix's data pipelines. +--- + +# Hamilton + +## Getting Started + +If you want to jump in head first, we have a simple tutorial for getting started! To ask questions, please join our [slack community!](https://join.slack.com/t/hamilton-opensource/shared\_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg) + +{% content-ref url="less-than-15-minutes-to-mastery/" %} +[less-than-15-minutes-to-mastery](less-than-15-minutes-to-mastery/) +{% endcontent-ref %} + +## What is Hamilton? + +It's a general purpose micro-framework for creating [dataflows](https://en.wikipedia.org/wiki/Dataflow) from python functions! + +Specifically, Hamilton defines a novel paradigm, that allows you to specify a flow of (delayed) execution, that forms a Directed Acyclic Graph (DAG). It was original built to solve creating wide (1000+) column dataframes at Stitch Fix. Core to the design of Hamilton is a clear mapping of function name to dataflow output. That is, Hamilton forces a certain paradigm with writing functions, and aims for DAG clarity, easy modifications, _with always unit testable and naturally documentable code!_ + +For the backstory on how Hamilton came about, see the original Stitch Fix [blog post!](https://multithreaded.stitchfix.com/blog/2021/10/14/functions-dags-hamilton/). + +Hamilton's method of defining dataflows presents a new paradigm when it comes to creating, um, dataframes (let's use dataframes as an example, otherwise you can create _ANY_ python object). Rather than thinking about manipulating a central dataframe procedurally, and extracting the data you want, as is normal in some data engineering/data science work, you instead think about the column(s) (a.k.a. outputs) you want to create, and what inputs are required. There is no need for you to think about maintaining how to create this dataframe, meaning you do not need to think about any "glue" code; this is all taken care of by the Hamilton framework. Specifically, Hamilton enables you to run your dataflow with the following steps: + +1. Define your dataflow (a.k.a. pipeline), as a set of transforms using Hamilton's paradigm for writing python functions. +2. Specify the parameter values your dataflow requires +3. Specify which outputs you want to compute from your dataflow + +The rest is delegated to the framework, which handles the computation for you. + +Let's illustrate this with some code. If you were asked to write a simple transform (let's use pandas for the sake of argument), you may decide to write something simple like this: + +```python +df['col_c'] = df['col_a'] + df['col_b'] +``` + +To represent this in a way Hamilton can understand, you write: + +```python +def col_c(col_a: pd.Series, col_b: pd.Series) -> pd.Series: + """Creating column c from summing column a and column b.""" + return col_a + col_b +``` + + + +The Hamilton framework takes the above code, forms it into a computational DAG, and executes it for you! + +## Hamilton Open Source Community + +If you have questions, have ideas, or need help with Hamilton, join us on [slack](https://join.slack.com/t/hamilton-opensource/shared\_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg), and we'll try to help! + +## Installing Hamilton + +Installation should be quick and as simple as: + +``` +> pip install sf-hamilton +``` + +For more information please see [installing.md](less-than-15-minutes-to-mastery/installing.md "mention"). + +## License + +Hamilton is released under the [BSD 3-Clause Clear License](https://github.com/DAGWorks-Inc/hamilton/blob/main/LICENSE). + +## Contributing + +We take contributions, large and small. We operate via a Code of Conduct and expect anyone contributing to do the same. + +## User Guide + +Dive a little deeper and start exploring our API reference to get an idea of everything that's possible with the API: + +{% content-ref url="api-reference/" %} +[api-reference](api-reference/) +{% endcontent-ref %}
diff --git a/talks-or-podcasts-or-blogs-or-papers.md b/old-docs/talks-or-podcasts-or-blogs-or-papers.md similarity index 100% rename from talks-or-podcasts-or-blogs-or-papers.md rename to old-docs/talks-or-podcasts-or-blogs-or-papers.md
diff --git a/tutorials/README.md b/old-docs/tutorials/README.md similarity index 100% rename from tutorials/README.md rename to old-docs/tutorials/README.md
diff --git a/tutorials/dask.md b/old-docs/tutorials/dask.md similarity index 100% rename from tutorials/dask.md rename to old-docs/tutorials/dask.md
diff --git a/tutorials/data-quality-checks.md b/old-docs/tutorials/data-quality-checks.md similarity index 100% rename from tutorials/data-quality-checks.md rename to old-docs/tutorials/data-quality-checks.md
diff --git a/tutorials/integrating-decorators.md b/old-docs/tutorials/integrating-decorators.md similarity index 100% rename from tutorials/integrating-decorators.md rename to old-docs/tutorials/integrating-decorators.md
diff --git a/tutorials/model-training.md b/old-docs/tutorials/model-training.md similarity index 100% rename from tutorials/model-training.md rename to old-docs/tutorials/model-training.md
diff --git a/tutorials/ray.md b/old-docs/tutorials/ray.md similarity index 92% rename from tutorials/ray.md rename to old-docs/tutorials/ray.md index d067da1..ef728d5 100644 --- a/tutorials/ray.md +++ b/old-docs/tutorials/ray.md
@@ -6,7 +6,7 @@ For those eager to just see code, please jump right ahead to a full hello world with Ray [here](https://github.com/stitchfix/hamilton/tree/main/examples/ray). -<figure><img src="../.gitbook/assets/Hamilton + ray post image.png" alt=""><figcaption><p>Hamilton + Ray.</p></figcaption></figure> +<figure><img src="../../.gitbook/assets/Hamilton + ray post image.png" alt=""><figcaption><p>Hamilton + Ray.</p></figcaption></figure> For those unfamiliar with [Ray](https://ray.io/), it is an open source framework that can scale python applications that came out of [UC Berkeley](https://rise.cs.berkeley.edu/projects/ray/). It has a growing ecosystem of tooling that helps with lots of machine learning related workflows. For example, it sells itself as enabling you to scale from your laptop to a cluster very easily, without having to change much code. In terms of real world use, we like to use Ray as a very quick way to implement [multiprocessing in python](https://machinelearningmastery.com/multiprocessing-in-python/) without worry about the details! @@ -54,11 +54,11 @@ pip install "sf-hamilton[ray]" ``` -Next, with Hamilton by default, all your logic is written as python functions. So you write your Hamilton functions as you normally would.  +Next, with Hamilton by default, all your logic is written as python functions. So you write your Hamilton functions as you normally would. -At the Hamilton framework level, Hamilton can easily inject `@ray.remote` for every single function in the directed acyclic graph (DAG) your functions define. That is_, you don’t have to change any of your Hamilton code to make use of Ray!_ All you need to do, to make Hamilton run on Ray, is provide a “_GraphAdapter”_ object to the Hamilton “_Driver”_ class you instantiate.  +At the Hamilton framework level, Hamilton can easily inject `@ray.remote` for every single function in the directed acyclic graph (DAG) your functions define. That is\_, you don’t have to change any of your Hamilton code to make use of Ray!\_ All you need to do, to make Hamilton run on Ray, is provide a “_GraphAdapter”_ object to the Hamilton “_Driver”_ class you instantiate. -A GraphAdapter, is just a simple class that has a few functions defined that enable you to augment how your DAG is walked and executed. See [available-graph-adapters.md](../reference/api-reference/available-graph-adapters.md "mention") for more information. +A GraphAdapter, is just a simple class that has a few functions defined that enable you to augment how your DAG is walked and executed. See [available-graph-adapters.md](../api-reference/available-graph-adapters.md "mention") for more information. In terms of code to add/change, here’s what’s required to augment standard Hamilton driver code — see **numbered comments**:
diff --git a/tutorials/spark.md b/old-docs/tutorials/spark.md similarity index 100% rename from tutorials/spark.md rename to old-docs/tutorials/spark.md
diff --git a/tutorials/using-hamilton-in-a-notebook.md b/old-docs/tutorials/using-hamilton-in-a-notebook.md similarity index 100% rename from tutorials/using-hamilton-in-a-notebook.md rename to old-docs/tutorials/using-hamilton-in-a-notebook.md