Kubernetes example

This example demonstrates how to deploy a stateful function application written in Python to Kubernetes.

Prerequisites

• Helm
• Kubernetes cluster
• Kafka
• StateFun distribution
• StateFun Python SDK

Overview

This examples create a stateful function application, that consumes LoginEvents from a logins Kafka topic, and produces seen count per user, into the seen Kafka topic.

The main example components contains:

• main.py - A StateFun python function that implements the main logic
• module.yaml - defines the ingress, egress and the remote function specification.
• resources - a Helm chart, templates to deploy StateFun cluster and the remote python worker to k8s.
• build-example.sh - Builds StateFun Docker images and k8s resources to deploy it.

Setup

Create Kafka Topics:

This example consumes LoginEvents from the logins topic, and produces SeenCount to the seen topic

 ./kafka-topics.sh --create --topic logins --zookeeper <zookeeper address>:2181 --partitions 1 --replication-factor 1
 ./kafka-topics.sh --create --topic seen --zookeeper <zookeeper address>:2181 --partitions 1 --replication-factor 1

update module.yaml

Make sure that your module.yaml ingress/and egress sections point to your Kafka cluster.

ingresses:
      - ingress:
            ...
          spec:
            address: kafka-service:9092
            ...
    egresses:
      - egress:
            ...
          spec:
            address: kafka-service:9092

Build the Docker images and the k8s resource yamls.

This examples creates two different Docker images, one for the Python remote worker (k8s-demo-python-worker) and one for the statefun cluster (k8s-demo-statefun).

• If you have a remote docker registry (i.e. gcr.io/<project-name>) make sure to update resources/values.yaml relevant image: sections.

• Modify resources/values.yaml and set the value of checkpoint.directory to a filesystem / object store. For example

checkpoint:
  dir: gcs://my-project/my-bucket

Assuming the all prerequisites where completed run:

build-example.sh

This should create the Docker images and generate a k8s-demo.yaml file.

Deploy

• kubectl create -f k8s-demo.yaml
• kubectl create -f python-worker-deployment.yaml
• kubectl create -f python-worker-service.yaml

Generate events

Run:

pip3 install kafka-python 
python3 event_generator.py --address <kafka address> --events 1000

This would generate 1,000 login events into the logins topic