BeamMonitoring

This folder contains files required to spin-up metrics dashboard for Beam.

Utilized technologies

• Grafana as dashboarding engine.
• PostgreSQL as underlying DB.

Approach utilized is to fetch data from corresponding system: Jenkins/Jira/GithubArchives/etc, put it into PostreSQL and fetch it to show in Grafana.

Local setup

Install docker

• install docker
  • https://docs.docker.com/install/#supported-platforms
• install docker-compose
  • https://docs.docker.com/compose/install/#install-compose

# Remove old docker
sudo apt-get remove docker docker-engine docker.io

# Install docker
sudo apt-get update
sudo apt-get install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg2 \
     software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce

###################################################
# Commands below here required to spin up docker  #
# containers locally. Can be omitted for kuberctl #
# setup.                                          #
###################################################

# Install docker-compose
sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose


# start docker service if it is not running already
sudo service docker start

# Build images required for spinning up docker-compose
# related containers.
docker-compose build

# Spinup docker-compose related containers.
docker-compose up

Kubernetes setup

1. Configure gcloud & kubectl

• https://cloud.google.com/kubernetes-engine/docs/quickstart

2. Configure PosgreSQL a. https://pantheon.corp.google.com/sql/instances?project=apache-beam-testing b. Check on this link to configure connection from kubernetes to postgresql: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine
3. add secrets for grafana a. kubectl create secret generic grafana-admin-pwd --from-literal=grafana_admin_password=<pwd>
4. create persistent volume claims:

kubectl create -f beam-grafana-etcdata-persistentvolumeclaim.yaml
kubectl create -f beam-grafana-libdata-persistentvolumeclaim.yaml
kubectl create -f beam-grafana-logdata-persistentvolumeclaim.yaml

5. Build and publish sync containers

cd sync/jenkins
docker build -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1 .
docker push gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1

6. Create deployment kubectl create -f beamgrafana-deploy.yaml

Kubernetes update

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

1. Build and publish sync containers

cd sync/jenkins
docker build -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1 .
docker push -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1

1. Update image for container kubectl set image deployment/beamgrafana container=<new_image_name>

Useful Kubernetes commands and hints

# Get pods
kubectl get pods

# Get detailed status
kubectl describe pod <pod_name>

# Get logs
kubectl log <PodName> <ContainerName>

# Set kubectl logging level: -v [1..10]
https://github.com/kubernetes/kubernetes/issues/35054

Useful docker commands and hints

• Connect from one container to another
  • curl <containername>:<port>
• Remove all containers/images/volumes

sudo docker rm $(sudo docker ps -a -q)
sudo docker rmi $(sudo docker images -q)
sudo docker volume prune