commit	d77b9de3ffd5ac41d5055795ad95e47bc0631486	[log] [tgz]
author	Klesh Wong <zhenmian.huang@merico.dev>	Tue Sep 14 10:06:03 2021 +0800
committer	GitHub <noreply@github.com>	Tue Sep 14 10:06:03 2021 +0800
tree	8901075040070e95106b3933b5e2eb194e8c5983
parent	20510f0e95fc7446564786c24964a65499c3be19 [diff]
parent	b591ea56de97abfa6171c13eeed38ecdf932bb9a [diff]

tree: 8901075040070e95106b3933b5e2eb194e8c5983

README.md

Dev Lake

What is Dev Lake?

Dev Lake is the one-stop solution that integrates, analyzes, and visualizes software development data throughout the software development life cycle (SDLC) for engineering teams.

Why choose Dev Lake?

Supports various data sources (Gitlab, Jira) and more are being added all the time
Relevant, customizable data metrics ready to view as visual charts
Easily build and view new charts and dashboards with Grafana
Easy-to-setup via Docker
Extensible plugin system to add your own data collectors
Designed to process enterprise-scale data

Section	Description	Documentation Link
Data Sources	Links to specific plugin usage & details	View Section
User Setup	Steps to run the project as a user	View Section
Developer Setup	How to setup dev environment	View Section
Tests	Commands for running tests	View Section
Grafana	How to visualize the data	View Section
Build a Plugin	Details on how to make your own	Link
Add Plugin Metrics	Guide to adding plugin metrics	Link
Contributing	How to contribute to this repo	Link

Data Sources We Currently Support

Below is a list of data source plugins used to collect & enrich data from specific sources. Each have a README.md file with basic setup, troubleshooting and metrics info.

For more information on building a new data source plugin see Build a Plugin.

Section	Section Info	Docs
Jira	Metrics, Generating API Token, Find Project/Board ID	Link
Gitlab	Metrics, Generating API Token	Link
Jenkins	Metrics, Generating API Token	Link

User setup

NOTE: If you only plan to run the product, this is the only section you should need NOTE: Commands written like this are to be run in your terminal

Required Packages to Install

NOTE: After installing docker, you may need to run the docker application and restart your terminal

Commands to run in your terminal

Create a directory and download files

git clone https://github.com/merico-dev/lake.git devlake
cd devlake
git checkout go-main
cp .env.example .env

Open .env file with your editor, fill the values in Jira, Gitlab and Jenkins sections with your deployments.

For more info on how to configure plugins, please refer to the data source plugins section

Launch docker-compose
```
make compose
```

Create a http request to trigger data collect tasks, please replace your gitlab projectId and jira boardId in the request body. This can take up to 20 minutes for large projects. (gitlab 10k+ commits or jira 5k+ issues)

curl -XPOST 'http://localhost:8080/task' \
-H 'Content-Type: application/json' \
-d '[
    {
        "plugin": "gitlab",
        "options": {
            "projectId": 8967944
        }
    },
    {
        "plugin": "jira",
        "options": {
            "boardId": 8
        }
    },
    {
        "plugin": "jenkins",
        "options": {}
    }
]'

Navigate to grafana dashboard http://localhost:3002 (username: admin, password: admin).

Setup cron job

Commonly, we have requirement to synchorize data periodly. We providered a tool called lake-cli to meet that requirement. Check lake-cli usage at here.

Otherwise, if you just want to use the cron job, please check docker-compose version at here

Developer Setup

Requirements

Docker
Golang
Make
- Mac (Already installed)
- Windows: Download
- Ubuntu: sudo apt-get install build-essential

How to setup dev environment

Navigate to where you would like to install this project and clone the repository
```
git clone https://github.com/merico-dev/lake.git
cd lake
```
Install go packages
```
make install
```
Copy sample config files to new local file
```
cp .env.example .env
```
Start the docker containers
Make sure the docker application is running before this step
```
make compose
```
Run the project
```
make dev
```

You can now post to /task to create a jira task. This will collect data from Jira

curl -XPOST 'localhost:8080/task' \
-H 'Content-Type: application/json' \
-d '[{
    "plugin": "jira",
    "options": {
        "boardId": 8
    }
}]'

Visualize the data in the Grafana Dashboard
From here you can see existing data visualized from collected & enriched data
- Navigate to http://localhost:3002 (username: admin, password: admin)
- You can also create/modify existing/save dashboards to lake
- For more info on working with Grafana in Dev Lake see Grafana Doc

Tests

To run the tests: make test

Grafana

We use Grafana as a visualization tool to build charts for the data stored in our database. Using SQL queries we can add panels to build, save, and edit customized dashboards.

All the details on provisioning, and customizing a dashboard can be found in the Grafana Doc

Contributing

CONTRIBUTING.md

Need help?

Message us on Discord