commit | 0ee9654c31386fe85ea75ec2da9fe78099b12389 | [log] [tgz] |
---|---|---|
author | 小陈 <jamiechen.778@gmail.com> | Sun Sep 29 11:55:58 2024 +0800 |
committer | GitHub <noreply@github.com> | Sun Sep 29 11:55:58 2024 +0800 |
tree | 32c8ce6cf6f0f4dd4610261bb825014271978200 | |
parent | ce372e5f3906ca1891e4918b05be14608eae608e [diff] |
Implement automated chaos testing framework with Chaos Mesh and OpenChaos integration (#6) * Add files for chaos-test * Use helm to deploy app * Update the base image * Fix typo * Fix bug * Use helm to perform cleanup * Remove decoding of secrets * Enable chaos test * Fix path error * Fix bug : refinement of clean-up operations * Make scripts executable * Modify scheduling script parameters * Enable to customise openchaos and chaos-mesh parameters * Reduce duplicate code * Fix bug : mounted volume setup failed * Supports continuous interval injection * Optimize fault injection scheduler to reduce execution delay * Reduce delay * Fix bug * Fix : delete configmap * Refinement of ci judgement conditions * Remove cron scheduler * Simplify openchaos configuration * Fix bug : miss ns when get pods * Update README.md * Move related code to a separate directory * Revert changes to related files * Update README.md * Add a test workflow * Test port forward * Revert "Test port forward" This reverts commit 7634c61126732033a71e78722f48d9f35c409a57. * Install Chaos Mesh according to detected CRI * Update README and example workflow * Update chart version * Use kubevela to deploy rocketmq * Install helm in config phase * Optimize job cleanup logic in chaos test * Refactor CI conditions: Ensure at least once delivery and no message loss
This tool uses Helm and KubeVela to deploy applications and execute E2E tests in Kubernetes. KubeVela needs to be installed in Kubernetes before use.
- uses: apache/rocketmq-test-tool@v1 name: Deploy with: action: "deploy" ask-config: "${{ secrets.KUBE_CONFIG }}" test-version: "v1.0" chart-git: "https://github.com/your-helm-chart.git" chart-branch: "main" chart-path: "." job-id: 1 helm-values: | app: image: repository: ${{env.DOCKER_REPO}} tag: v1.0
- uses: apache/rocketmq-test-tool@v1 name: e2e test with: action: "test" ask-config: "${{ secrets.KUBE_CONFIG }}" test-version: "v1.0" test-code-git: "https://github.com/your-e2e-test.git" test-code-branch: "main" test-code-path: ./ test-cmd: "your test command" job-id: 1 - uses: actions/upload-artifact@v3 if: always() name: Upload test log with: name: testlog.txt path: testlog.txt
Quick Start:
Deploy your application:
- uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991 name: Deploy rocketmq with: action: "deploy" ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}" test-version: "v0.1" job-id: 1 helm-chart-repo: "https://chi3316.github.io/my_chart/" helm-chart-version: "0.0.5" helm-chart: "rocketmq"
Chaos test:
- name: Checkout repository uses: actions/checkout@v2 - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991 name: Chaos test with: action: "chaos-test" ask-config: "${{ secrets.KUBE_CONFIG }}" job-id: 1 openchaos-driver: ".github/chaos-configs/driver.yaml" chaos-mesh-fault-file: ".github/chaos-configs/network-delay.yaml" fault-scheduler-interval: "30" openchaos-args: "-t 240" fault-durition: "30" node-lable: "app.kubernetes.io/name=broker" meta-node-lable: "app.kubernetes.io/name=nameserver" - uses: actions/upload-artifact@v4 with: name: chaos-test-report path: chaos-test-report/
Clean work:
- uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991 name: clean with: action: "clean" ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}" job-id: 1
Scheduling Fault Injection You can schedule fault injection using fault-scheduler-interval for intervals (in seconds).Specify this parameter to inject faults at regular intervals.
OpenChaos and Chaos-Mesh Configuration To configure OpenChaos, you‘ll need to specify the nodes and metaNodes. Ensure that you provide labels that can filter out the regular nodes and metadata nodes correctly based on your cluster’s configuration. Example Configuration:
nodes: - ${node_1} - ${node_2} ... - ${node_n} metaNodes: - ${meta_node_1} - ${meta_node_2} ... - ${meta_node_n}
Important: You must fill the configuration with the exact format shown above. Use placeholders like
${node_1}
,${node_2}
, etc., for regular nodes, and${meta_node_1}
,${meta_node_2}
, etc., for metadata nodes. Make sure to use underscores_
in the placeholders.
For chaos-mesh YAML files, use placeholders like ${app}
and ${ns}
for application names and namespaces.For example, in your YAML files:
selector: namespaces: - '${ns}' labelSelectors: "app.kubernetes.io/name": "${app}"
Note: When installing Chaos Mesh, you might need to adjust settings based on the container runtime of your Kubernetes cluster.
Default Parameters: OpenChaos already has some default parameters:
./start-openchaos.sh --driver driver-rocketmq/openchaos-driver.yaml --output-dir ./report $OPENCHAOS_ARGS
Note: Make sure not to duplicate the parameters in OPENCHAOS_ARGS.
- uses: apache/rocketmq-test-tool@v1 name: clean with: action: "clean" ask-config: "${{ secrets.KUBE_CONFIG }}" test-version: "v1.0" job-id: 1
Apache License, Version 2.0 Copyright (C) Apache Software Foundation