Install the helm chart for Flink on Kubernetes. See Flink on Kubernetes for more details.
Make sure local Docker has been assigned at least 8 cores and 8G memory.
Point docker-cli to Docker running in minikube
eval $(minikube -p minikube docker-env)
Start local Docker registry in minikube Docker (the docker command is talking to minikube Docker)
docker run -d -p 5001:5000 --restart=always --name registry registry:2
We tunnel port 5001 to 5000, you can choose any available port.
We need to mount a local path on the host for all containers in minikube to work with.
mkdir /tmp/minikubedata minikube mount /tmp/minikubedata:/data &
We need a custom docker image that extends the flink base image and adds the hudi flink example jar to it. You can build this docker image by running the following command:
docker build -t localhost:5001/hudi/hudi-flink .
This should have pushed the docker image to the docker registry. You can verify this by running
docker images
This should show you the docker image hudi-flink:latest in the list of images.
REPOSITORY TAG IMAGE ID CREATED SIZE localhost:5001/hudi/hudi-flink latest 87b936181d74 32 minutes ago 1.08GB
Create Minio server in Kubernetes.
kubectl apply -f config/k8s/minio-standalone.yaml
This will create a pod in Kubernetes that runs the Minio server. You can verify this by running
kubectl port-forward svc/minio-svc 9090
and then opening the Minio UI in your browser by hitting on http://localhost:9090.
Use the default credentials: minioadmin and minioadmin to log into the console.
From the UI, manually create a bucket named test.
kubectl delete deployment minio-deploy kubectl replace -f config/k8s/minio-standalone.yaml
We can now submit the Flink job to the Flink cluster running in Kubernetes.
kubectl apply -f config/k8s/flink-deployment.yaml
This will create a pod in Kubernetes that runs the Flink job. You can verify this by running
kubectl port-forward svc/basic-example-rest 8081
and then opening the Flink UI in your browser by hitting on http://localhost:8081.
kubectl delete deployment basic-example kubectl replace deployment basic-example