dlt

This example shows dlt + Hamilton can help you cover the full ELT cycle using portable Python code. It is a companion to this documentation page.

Content

It includes a pipeline to ingest messages from Slack channels and generate threads summaries.

• slack/ is the code imported from dlt for the Slack Source
• .dlt/ contains the source's version, config, and secrets for the dlt Pipeline
• transform.py contains the Hamilton code to transform data and build the threads table
• run.py contains the code to execution the dlt Pipeline and the Hamilton dataflow.
• notebook.ipynb contains the equivalent of both transform.py and run.py allowing you to explore the code interactively.

Set up

1. Create a virtual environment and activate it

   python -m venv venv && . venv/bin/active
   

2. Install requirements

   pip install -r requirements.txt
   

3. Follow this dlt guide to ingest Slack data. Make sure to invite your Slack app to your channel!

4. Execute the code. Use --help to learn about accepted arguments.

   python run.py --help
   python run.py general dlt
   

References

• dlt to ingest Slack data
• Reconstructing Slack threads docs