dlt

This example shows dlt + Apache Hamilton can help you cover the full ELT cycle using portable Python code. It is a companion to this documentation page.

Content

It includes a pipeline to ingest messages from Slack channels and generate threads summaries.

slack/ is the code imported from dlt for the Slack Source
.dlt/ contains the source's version, config, and secrets for the dlt Pipeline
transform.py contains the Apache Hamilton code to transform data and build the threads table
run.py contains the code to execution the dlt Pipeline and the Apache Hamilton dataflow.
notebook.ipynb contains the equivalent of both transform.py and run.py allowing you to explore the code interactively.

Set up

Create a virtual environment and activate it

python -m venv venv && . venv/bin/active

Install requirements
```
pip install -r requirements.txt
```
Follow this dlt guide to ingest Slack data. Make sure to invite your Slack app to your channel!
Execute the code. Use --help to learn about accepted arguments.
```
python run.py --help
python run.py general dlt
```

References

dlt to ingest Slack data
Reconstructing Slack threads docs