Apache Hamilton UI

This contains the code for the new Apache Hamilton UI. For an overview of getting started & features see this documentation. For a lengthier post and intro see our blog post.

One operational UI for all your dataflows

The Apache Hamilton UI is a system that provides the following capabilities:

1. Execution tracking with associated metadata
   • Provides a persistent database to store/manage these
   • Provides a server that allows reading/writing/authentication
2. Data/artifact observability: provides telemetry/observability of Apache Hamilton executions + specific function results/code through a web interface
3. Lineage & provenance: allows you to quickly inspect how code and data is connected.
4. Catalog: everything is observed and cataloged, so you can quickly search and find what exists and when it was run.

The UI is meant to monitor/debug Apache Hamilton dataflows both in development and production. The aim is to enable dataflow authors to move faster during all phases of the software development lifecycle.

For an overview of some of these features you can watch this quick video.

Execution Tracking

Data/Artifact Observability

Lineage & Provenance

Catalog

Getting started

You can watch this video walkthrough on getting set up.

Make sure you have docker running:

# clone the repository if you haven't
git clone https://github.com/apache/hamilton
# change into the UI directory
cd hamilton/ui
# run docker
./run.sh

Once docker is running navigate to http://localhost:8242 and create an email and a project; then follow instructions on integrating with Apache Hamilton.

A fuller guide can be found here.

Architecture

The architecture is simple.

The tracking server stores data on postgres, as well as any blobs on s3. This is stored in a docker volume on local mode. The frontend is a simple React application. There are a few authentication/ACL capabilities, but the default is to use local/unauthenticated (open). Please talk to us if you have a need for more custom authentication.

Development

The structure involves a bit of cleverness to ensure the UI can easily be deployed and served from the CLI.

We have a symlink from backend/hamilton_ui to backend/server, allowing us to work with django's structure while simultaneously allowing for import as hamilton_ui. (this should probably be changed at some point but not worth it now).

To deploy, use the admin.py script in the UI directory.

This:

1. Builds the frontend
2. Copies it into the build/ directory
3. Publishes to the sf-hamilton-ui package on pypi

Then you‘ll run it with hamilton ui after installing sf-hamilton[ui]. Note to talk to it you’ll need the hamilton_sdk pacakge which can be installed with pip install sf-hamilton[sdk].

Building docker

Dev mode

For development you'll want to run

cd hamilton/ui
./dev.sh --build # to build it all
./dev.sh # to pull docker images but use local code

You need 9GB assigned to Docker or more to build the frontend

The frontend build requires around 8GB of memory to be assigned to docker to build. If you run into this, bump your docker memory allocation up to 9GB or more.

Prod mode

For production build you'll want to run

cd hamilton/ui
./run.sh # to pull from docker and run
./run.sh --build # to rebuild images for prod

Caveats:

You'll want to clean the backend/dist/ directory to not add unnecessary files to the docker image.

Pushing

How to push to docker hub:

# retag if needed
docker tag local-image:tagname dagworks/ui-backend:VERSION
# push built image
docker push dagworks/ui-backend:VERSION
# retag as latest
docker tag dagworks/ui-backend:VERSION dagworks/ui-backend:latest
# push latest
docker push dagworks/ui-backend:latest

# retag if needed
docker tag local-image:tagname dagworks/ui-frontend:VERSION
# push built image
docker push dagworks/ui-frontend:VERSION
# retag as latest
docker tag dagworks/ui-backend:VERSION dagworks/ui-backend:latest
# push latest
docker push dagworks/ui-backend:latest