# Parallelism Example

## Overview
This is a very simple example of dynamically parameterizing sections of the DAG in parallel.
This loads data from the kaggle dataset [Airbnb Prices in European Cities](https://www.kaggle.com/datasets/thedevastator/airbnb-prices-in-european-cities),
and does the following:

- Lists out the cities in the dataset See [list_data.py](list_data.py)
- Loads the files and runs a set of very basic summary statistics. See [process_data.py](process_data.py)
- Aggregates the summary into a single dataframe  See [aggregate_data.py](aggregate_data.py)

## Take home

This demonstrates two powerful capabilities:

1. Dynamically generating sets of nodes based on the result of another node
2. Running these nodes in parallel

Note that this does not do anything particularly complex -- the dataset/computation is meant to illustrate how you could use these powers.
These datasets are small and the data processing is quite simple.

## Running

First, download the data and place inside a directory called `data`.
You can download the data from [here](https://www.kaggle.com/datasets/thedevastator/airbnb-prices-in-european-cities). You'll need a kaggle account.

You can run the basic analysis in the terminal with:

```bash
python run.py
```

And you can play around with the data using the `notebook.ipynb` notebook.