Implement automated chaos testing framework with Chaos Mesh and OpenChaos integration (#6)
* Add files for chaos-test
* Use helm to deploy app
* Update the base image
* Fix typo
* Fix bug
* Use helm to perform cleanup
* Remove decoding of secrets
* Enable chaos test
* Fix path error
* Fix bug : refinement of clean-up operations
* Make scripts executable
* Modify scheduling script parameters
* Enable to customise openchaos and chaos-mesh parameters
* Reduce duplicate code
* Fix bug : mounted volume setup failed
* Supports continuous interval injection
* Optimize fault injection scheduler to reduce execution delay
* Reduce delay
* Fix bug
* Fix : delete configmap
* Refinement of ci judgement conditions
* Remove cron scheduler
* Simplify openchaos configuration
* Fix bug : miss ns when get pods
* Update README.md
* Move related code to a separate directory
* Revert changes to related files
* Update README.md
* Add a test workflow
* Test port forward
* Revert "Test port forward"
This reverts commit 7634c61126732033a71e78722f48d9f35c409a57.
* Install Chaos Mesh according to detected CRI
* Update README and example workflow
* Update chart version
* Use kubevela to deploy rocketmq
* Install helm in config phase
* Optimize job cleanup logic in chaos test
* Refactor CI conditions: Ensure at least once delivery and no message loss
diff --git a/.github/chaos-configs/driver.yaml b/.github/chaos-configs/driver.yaml
new file mode 100644
index 0000000..f74c579
--- /dev/null
+++ b/.github/chaos-configs/driver.yaml
@@ -0,0 +1,31 @@
+name: RocketMQ
+driverClass: io.openchaos.driver.rocketmq.RocketMQDriver
+
+endToEndLatencyCheck: true
+
+# RocketMQ cluster configuration
+
+# Nodes for broker
+nodes:
+ - ${node_1}
+ - ${node_2}
+
+# Nodes for nameserver
+metaNodes:
+ - ${meta_node_1}
+
+# RocketMQ configuration
+rocketmqVersion: 5.2.0
+installDir: /home/rocketmq/rocketmq-5.2.0 # you could set existent location for RocketMQ
+nameServerPort: 9876
+
+# RocketMQ client configuration
+clusterName: DefaultCluster # same value as the config in broker.properties
+vipChannelEnabled: false
+
+# RocketMQ broker configuration
+brokerClusterName: DefaultCluster
+brokerName: rocketmq-broker-0
+storePathRootDir: /home/rocketmq/store
+storePathCommitLog: /home/rocketmq/store/commitlog
+enableDLegerCommitLog: false
diff --git a/.github/chaos-configs/network-delay.yaml b/.github/chaos-configs/network-delay.yaml
new file mode 100644
index 0000000..470a6e1
--- /dev/null
+++ b/.github/chaos-configs/network-delay.yaml
@@ -0,0 +1,17 @@
+apiVersion: chaos-mesh.org/v1alpha1
+kind: NetworkChaos
+metadata:
+ name: network-delay
+spec:
+ action: delay
+ mode: fixed
+ value: "2"
+ selector:
+ namespaces:
+ - ${ns}
+ labelSelectors:
+ "app.kubernetes.io/name": "broker"
+ "app.kubernetes.io/instance": ${app}
+ delay:
+ latency: "100ms"
+ duration: "30s"
diff --git a/.github/workflows/test-chaos-test.yaml b/.github/workflows/test-chaos-test.yaml
new file mode 100644
index 0000000..d538791
--- /dev/null
+++ b/.github/workflows/test-chaos-test.yaml
@@ -0,0 +1,64 @@
+name: Test chaos-test
+on:
+ push:
+ branches:
+ - master
+ workflow_dispatch:
+jobs:
+ deploy:
+ runs-on: ubuntu-latest
+ timeout-minutes: 10
+ strategy:
+ matrix:
+ version: [v2.2]
+ steps:
+ - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991
+ name: Deploy rocketmq
+ with:
+ action: "deploy"
+ ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}"
+ test-version: "v0.1"
+ job-id: ${{ strategy.job-index }}
+ helm-chart-repo: "https://chi3316.github.io/my_chart/"
+ helm-chart-version: "0.0.5"
+ helm-chart: "rocketmq"
+
+ chaos-test:
+ runs-on: ubuntu-latest
+ needs: deploy
+ timeout-minutes: 10
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v2
+ - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991
+ name: Chaos test
+ with:
+ action: "chaos-test"
+ ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}"
+ job-id: ${{ strategy.job-index }}
+ openchaos-driver: ".github/chaos-configs/driver.yaml"
+ chaos-mesh-fault-file: ".github/chaos-configs/network-delay.yaml"
+ fault-scheduler-interval: "30"
+ openchaos-args: "-t 240"
+ fault-durition: "30"
+ node-lable: "app.kubernetes.io/name=broker"
+ meta-node-lable: "app.kubernetes.io/name=nameserver"
+ - name: Upload test report
+ uses: actions/upload-artifact@v4
+ with:
+ name: chaos-test-report
+ path: chaos-test-report/
+ continue-on-error: true
+ clean:
+ if: always()
+ name: Clean
+ needs: [deploy, chaos-test]
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+ steps:
+ - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991
+ name: clean
+ with:
+ action: "clean"
+ ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}"
+ job-id: ${{ strategy.job-index }}
diff --git a/README.md b/README.md
index bdc4a18..e1c837b 100644
--- a/README.md
+++ b/README.md
@@ -44,6 +44,98 @@
name: testlog.txt
path: testlog.txt
```
+## Perform chaos test in Kubernetes
+**Quick Start:**
+1. On the main branch of your repository, copy the example workflow (the entire ./github/ directory) into your own repository.
+2. Configure a GitHub repository secret:
+ - Go to Settings -> Secrets and variables -> Actions.
+ - Click Repository secrets to create a new secret named - ACK_CONFIG_VIRGINA.
+ - Set the value to your K8s cluster access token.
+3. Finally, you can manually trigger the workflow called 'Test chaos-test' in the Actions tab to see it in action.
+
+**Deploy your application:**
+```yaml
+ - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991
+ name: Deploy rocketmq
+ with:
+ action: "deploy"
+ ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}"
+ test-version: "v0.1"
+ job-id: 1
+ helm-chart-repo: "https://chi3316.github.io/my_chart/"
+ helm-chart-version: "0.0.5"
+ helm-chart: "rocketmq"
+```
+**Chaos test:**
+```yaml
+ - name: Checkout repository
+ uses: actions/checkout@v2
+ - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991
+ name: Chaos test
+ with:
+ action: "chaos-test"
+ ask-config: "${{ secrets.KUBE_CONFIG }}"
+ job-id: 1
+ openchaos-driver: ".github/chaos-configs/driver.yaml"
+ chaos-mesh-fault-file: ".github/chaos-configs/network-delay.yaml"
+ fault-scheduler-interval: "30"
+ openchaos-args: "-t 240"
+ fault-durition: "30"
+ node-lable: "app.kubernetes.io/name=broker"
+ meta-node-lable: "app.kubernetes.io/name=nameserver"
+ - uses: actions/upload-artifact@v4
+ with:
+ name: chaos-test-report
+ path: chaos-test-report/
+```
+**Clean work:**
+```yaml
+ - uses: chi3316/rocketmq-test-tool/chaos-test-runner@1cb6d547b8ae65993a3ed0f03ac6c62ba42cf991
+ name: clean
+ with:
+ action: "clean"
+ ask-config: "${{ secrets.ACK_CONFIG_VIRGINA }}"
+ job-id: 1
+```
+
+**Scheduling Fault Injection**
+You can schedule fault injection using fault-scheduler-interval for intervals (in seconds).Specify this parameter to inject faults at regular intervals.
+
+**OpenChaos and Chaos-Mesh Configuration**
+To configure OpenChaos, you'll need to specify the nodes and metaNodes. Ensure that you provide labels that can filter out the regular nodes and metadata nodes correctly based on your cluster's configuration.
+Example Configuration:
+```yaml
+nodes:
+ - ${node_1}
+ - ${node_2}
+ ...
+ - ${node_n}
+
+metaNodes:
+ - ${meta_node_1}
+ - ${meta_node_2}
+ ...
+ - ${meta_node_n}
+```
+> **Important**: You must fill the configuration with the exact format shown above. Use placeholders like `${node_1}`, `${node_2}`, etc., for regular nodes, and `${meta_node_1}`, `${meta_node_2}`, etc., for metadata nodes. Make sure to use underscores `_` in the placeholders.
+
+For chaos-mesh YAML files, use placeholders like `${app}` and `${ns}` for application names and namespaces.For example, in your YAML files:
+```yaml
+selector:
+ namespaces:
+ - '${ns}'
+ labelSelectors:
+ "app.kubernetes.io/name": "${app}"
+```
+> **Note**: When installing Chaos Mesh, you might need to adjust settings based on the container runtime of your Kubernetes cluster.
+
+**Default Parameters:**
+OpenChaos already has some default parameters:
+```shell
+./start-openchaos.sh --driver driver-rocketmq/openchaos-driver.yaml --output-dir ./report $OPENCHAOS_ARGS
+```
+> **Note**: Make sure not to duplicate the parameters in OPENCHAOS_ARGS.
+
## Clean your app in Kubernetes
```yaml
- uses: apache/rocketmq-test-tool@v1
diff --git a/chaos-test-runner/Dockerfile b/chaos-test-runner/Dockerfile
new file mode 100644
index 0000000..6aa609a
--- /dev/null
+++ b/chaos-test-runner/Dockerfile
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Container image that runs your code
+FROM registry.cn-guangzhou.aliyuncs.com/cc-aliyun/test-runner:v0.0.2
+
+RUN mkdir /chaos-test
+COPY ./chaos-test/ /chaos-test
+
+# Copies your code file from your action repository to the filesystem path `/` of the container
+COPY entry.sh /entry.sh
+
+# Code file to execute when the docker container starts up (`entry.sh`)
+ENTRYPOINT ["/entry.sh"]
\ No newline at end of file
diff --git a/chaos-test-runner/action.yaml b/chaos-test-runner/action.yaml
new file mode 100644
index 0000000..7079887
--- /dev/null
+++ b/chaos-test-runner/action.yaml
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# action.yml
+name: 'Cloud Native CI Action'
+description: ''
+inputs:
+ action: # id of input
+ description: 'action'
+ required: true
+ default: ''
+ test-version: # id of input
+ description: 'test version'
+ required: true
+ default: ''
+ ask-config: # id of input
+ description: 'ask config'
+ required: true
+ default: ''
+ docker-repo-username: # id of input
+ description: 'docker repo username'
+ required: false
+ default: ''
+ docker-repo-password: # id of input
+ description: 'docker repo password'
+ required: false
+ default: ''
+ chart-git: # id of input
+ description: 'chart git'
+ required: false
+ default: ''
+ chart-branch: # id of input
+ description: 'chart branch'
+ required: false
+ default: ''
+ chart-path: # id of input
+ description: 'chart path'
+ required: false
+ default: './'
+ test-code-git: # id of input
+ description: 'test code git'
+ required: false
+ default: ''
+ test-code-branch: # id of input
+ description: 'test code branch'
+ required: false
+ default: ''
+ test-code-path: # id of input
+ description: 'test code path'
+ required: false
+ default: ''
+ test-cmd: # id of input
+ description: 'test cmd'
+ required: false
+ default: 'mvn -B test'
+ job-id: # id of input
+ description: 'job id'
+ required: true
+ default: ''
+ helm-values: # id of input
+ description: 'helm values'
+ required: true
+ default: ''
+ openchaos-driver: # id of input
+ description: 'the driver of openchaos'
+ required: true
+ default: ''
+ chaos-mesh-fault-file: # id of input
+ description: 'the defination file of fault'
+ required: true
+ openchaos-args:
+ description: 'the startup arguments for openchaos'
+ required: false
+ default: ''
+ fault-durition:
+ description: 'the duration of a single fault'
+ required: true
+ fault-scheduler-interval:
+ description: 'time interval for injecting faults'
+ required: ture
+ default: '30'
+ helm-chart-repo:
+ description: 'helm chart repo' # TODO : just for test , should use kubevela to deploy application
+ required: true
+ default: ''
+ helm-chart-version:
+ description: 'helm chart version'
+ required: true
+ default: ''
+ helm-chart:
+ description: 'helm chart'
+ required: true
+ default: ''
+ node-lable:
+ description: 'the cluster node lable'
+ required: true
+ default: ''
+ meta-node-lable:
+ description: 'the meta node lable'
+ required: true
+ default: ''
+ runtime:
+ description: 'the container runtime of K8s'
+ required: false
+ default: ''
+ socket-path:
+ description: 'the socket path of container runtime'
+ required: false
+ default: ''
+runs:
+ using: 'docker'
+ image: 'Dockerfile'
+ args:
+ - ${{ inputs.action }}
+ - ${{ inputs.test-version }}
+ - ${{ inputs.ask-config }}
+ - ${{ inputs.docker-repo-username }}
+ - ${{ inputs.docker-repo-password }}
+ - ${{ inputs.chart-git }}
+ - ${{ inputs.chart-branch }}
+ - ${{ inputs.chart-path }}
+ - ${{ inputs.test-code-git }}
+ - ${{ inputs.test-code-branch }}
+ - ${{ inputs.test-code-path }}
+ - ${{ inputs.test-cmd }}
+ - ${{ inputs.job-id }}
+ - ${{ inputs.helm-values }}
+ - ${{ inputs.openchaos-driver }}
+ - ${{ inputs.chaos-mesh-fault-file }}
+ - ${{ inputs.openchaos-args }}
+ - ${{ inputs.fault-durition }}
+ - ${{ inputs.fault-scheduler-interval }}
+ - ${{ inputs.helm-chart-repo }}
+ - ${{ inputs.helm-chart-version }}
+ - ${{ inputs.helm-chart }}
+ - ${{ inputs.node-lable }}
+ - ${{ inputs.meta-node-lable }}
+ - ${{ inputs.runtime }}
+ - ${{ inputs.socket-path }}
\ No newline at end of file
diff --git a/chaos-test-runner/chaos-test/chaos-controller-template.yaml b/chaos-test-runner/chaos-test/chaos-controller-template.yaml
new file mode 100644
index 0000000..279fe67
--- /dev/null
+++ b/chaos-test-runner/chaos-test/chaos-controller-template.yaml
@@ -0,0 +1,52 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: openchaos-controller
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: openchaos-controller
+ template:
+ metadata:
+ labels:
+ app: openchaos-controller
+ spec:
+ initContainers:
+ - name: init-create-file
+ image: registry.cn-guangzhou.aliyuncs.com/cc-aliyun/busybox
+ command:
+ ["sh", "-c", "touch /chaos-framework/report/chaos-mesh-fault"]
+ volumeMounts:
+ - name: report-volume
+ mountPath: /chaos-framework/report
+ containers:
+ - name: openchaos-controller
+ image: registry.cn-guangzhou.aliyuncs.com/cc-aliyun/openchaos:v1.5
+ imagePullPolicy: IfNotPresent
+ env:
+ - name: CHAOS_MESH_LOG_FILE
+ value: /chaos-framework/report/chaos-mesh-fault
+ command: ["/bin/sh"]
+ args: ["-c", "tail -f /dev/null"]
+ ports:
+ - containerPort: 8080
+ volumeMounts:
+ - name: rocketmq-volume
+ mountPath: /chaos-framework/driver-rocketmq
+ - name: report-volume
+ mountPath: /chaos-framework/report
+ - name: sidecar-container
+ image: registry.cn-guangzhou.aliyuncs.com/cc-aliyun/busybox
+ imagePullPolicy: IfNotPresent
+ command: ["/bin/sh"]
+ args: ["-c", "tail -f /dev/null"]
+ volumeMounts:
+ - name: report-volume
+ mountPath: /chaos-framework/report
+ volumes:
+ - name: rocketmq-volume
+ configMap:
+ name: ${configmap_name}
+ - name: report-volume
+ emptyDir: {}
diff --git a/chaos-test-runner/chaos-test/inject-fault.sh b/chaos-test-runner/chaos-test/inject-fault.sh
new file mode 100755
index 0000000..568c3d4
--- /dev/null
+++ b/chaos-test-runner/chaos-test/inject-fault.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+CHAOSMESH_YAML_FILE=$1
+LOG_FILE=$2
+DURITION=$3
+POD_NAME=$4
+NS=$5
+
+export KUBECTL_PATH=/usr/local/bin/kubectl
+
+if [ ! -f "$CHAOSMESH_YAML_FILE" ]; then
+ echo "Chaos Mesh YAML file not found: $CHAOSMESH_YAML_FILE"
+ exit 1
+fi
+
+current_millis() {
+ echo $(( $(date +%s%N) / 1000000 ))
+}
+
+log_fault_event() {
+ event_type=$1
+ fault_type=$2
+ timestamp=$(current_millis)
+ $KUBECTL_PATH exec -i $POD_NAME -n ${NS} -c sidecar-container -- /bin/sh -c "echo -e 'fault\t$fault_type\t$event_type\t$timestamp' >> $LOG_FILE"
+}
+
+inject_fault() {
+ if $KUBECTL_PATH apply -f $CHAOSMESH_YAML_FILE; then
+ log_fault_event "start" "chaos-mesh-fault"
+ else
+ log_fault_event "error" "chaos-mesh-fault"
+ fi
+}
+
+clear_fault() {
+ if $KUBECTL_PATH delete -f $CHAOSMESH_YAML_FILE; then
+ log_fault_event "end" "chaos-mesh-fault"
+ else
+ log_fault_event "error_clear" "chaos-mesh-fault"
+ fi
+}
+
+inject_fault
+
+# Wait for a period of time equal to the duration of a single fault
+sleep $DURITION
+
+clear_fault
diff --git a/chaos-test-runner/chaos-test/interval-scheduler.sh b/chaos-test-runner/chaos-test/interval-scheduler.sh
new file mode 100755
index 0000000..05069bb
--- /dev/null
+++ b/chaos-test-runner/chaos-test/interval-scheduler.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+TOTAL_TIME=$1
+INTERVAL=$2
+DURATION=$3
+SCRIPT_PATH="$4"
+shift 4
+SCRIPT_ARGS="$@"
+
+export KUBECONFIG=$(printenv KUBECONFIG)
+calculate_initial_wait_time() {
+ local total=$1
+ local interval=$2
+ local duration=$3
+ local current_time=$total
+
+ while [ $current_time -gt 0 ]; do
+ if [ $current_time -ge $duration ]; then
+ current_time=$((current_time - duration))
+ else
+ echo $current_time
+ return
+ fi
+
+ if [ $current_time -gt $interval ]; then
+ current_time=$((current_time - interval))
+ else
+ echo $current_time
+ return
+ fi
+ done
+
+}
+
+initial_wait_time=$(calculate_initial_wait_time $TOTAL_TIME $INTERVAL $DURATION)
+sleep $initial_wait_time
+
+fault_inject_time=0
+fault_total_time=$((TOTAL_TIME - initial_wait_time))
+while [ $fault_inject_time -lt $fault_total_time ]; do
+
+ # Inject fault
+ sh $SCRIPT_PATH $SCRIPT_ARGS > /dev/null 2>&1 &
+
+ # Wait for the fault injection operation to complete
+ sleep $DURATION
+
+ # Wait for the next fault injection
+ sleep $INTERVAL
+
+ fault_inject_time=$((fault_inject_time + INTERVAL + DURATION))
+done
\ No newline at end of file
diff --git a/chaos-test-runner/chaos-test/startup.sh b/chaos-test-runner/chaos-test/startup.sh
new file mode 100755
index 0000000..01c7668
--- /dev/null
+++ b/chaos-test-runner/chaos-test/startup.sh
@@ -0,0 +1,77 @@
+#!/bin/sh
+CHAOSMESH_YAML_FILE=$1
+DURITION=$2
+POD_NAME=$3
+NS=$4
+REPORT_DIR=$5
+OPENCHAOS_ARGS=$6
+INTERVAL=$7
+LOG_FILE='/chaos-framework/report/chaos-mesh-fault'
+
+cleanup() {
+ echo "Test done."
+ echo "Performing cleanup..."
+ kubectl cp --retries=10 -n ${NS} --container=openchaos-controller ${POD_NAME}:/chaos-framework/report "$REPORT_DIR"
+ sleep 5
+ ls $REPORT_DIR
+ kubectl delete deployment openchaos-controller -n ${NS}
+ configmap=$(kubectl get pods -n ${NS} ${POD_NAME} -o jsonpath='{.spec.volumes[*].configMap.name}')
+ kubectl delete cm ${configmap} -n ${NS}
+ kubectl delete pod ${POD_NAME} -n ${NS}
+ echo "Cleanup completed..."
+ check_report
+}
+
+check_report() {
+ resulet_file=$(ls ${REPORT_DIR}/*RocketMQ-chaos-queue-result-file 2>/dev/null | head -n 1)
+ # Check report number
+ file_count=$(ls -1q $REPORT_DIR/* | wc -l)
+ if [ "$file_count" -lt 6 ]; then
+ echo "Test failed: Insufficient report files."
+ exit 1
+ fi
+
+ # Check for missing messages
+ lostMessageCount=$(grep "lostMessageCount:" "$resulet_file" | awk '{print $2}')
+ atLeastOnce=$(grep "atLeastOnce:" "$resulet_file" | awk '{print $2}')
+
+ if [ "$lostMessageCount" -eq 0 ] && [ "$atLeastOnce" = "true" ]; then
+ echo "Test Passed: All conditions met."
+ exit 0
+ else
+ echo "Test failed: Conditions not met."
+ if [ "$lostMessageCount" -ne 0 ]; then
+ echo "Error: lostMessageCount is not 0."
+ fi
+ if [ "$atLeastOnce" != "true" ]; then
+ echo "Error: atLeastOnce is not true."
+ fi
+ exit 1
+ fi
+
+}
+
+trap cleanup EXIT
+
+# Test
+echo "Running chaos test..."
+
+# Extract the value of the -t parameter from OPENCHAOS_ARGS
+total_time=$(echo "$OPENCHAOS_ARGS" | sed -n 's/.*-t \([0-9]*\).*/\1/p')
+if [ -z "$total_time" ]; then
+ echo "Error: -t parameter not found in OPENCHAOS_ARGS."
+ exit 1
+fi
+
+# Start openchaos
+kubectl exec -i ${POD_NAME} -n ${NS} -c openchaos-controller -- /bin/sh -c "./start-openchaos.sh --driver driver-rocketmq/openchaos-driver.yaml --output-dir ./report $OPENCHAOS_ARGS" > "$REPORT_DIR/output.log" 2>&1 &
+
+if [ -n "$INTERVAL" ] && [ "$INTERVAL" != "" ] && [ -n "$DURITION" ]; then
+# Use interval scheduler
+ ./interval-scheduler.sh $total_time $INTERVAL $DURITION /chaos-test/inject-fault.sh "$CHAOSMESH_YAML_FILE" "$LOG_FILE" "$DURITION" "$POD_NAME" "$NS"
+else
+ echo "Error: INTERVAL and DURITION must be provided."
+ exit 1
+fi
+
+wait
\ No newline at end of file
diff --git a/chaos-test-runner/entry.sh b/chaos-test-runner/entry.sh
new file mode 100755
index 0000000..81edd4c
--- /dev/null
+++ b/chaos-test-runner/entry.sh
@@ -0,0 +1,342 @@
+#!/bin/sh -l
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+ACTION=$1
+VERSION=$2
+ASK_CONFIG=$3
+DOCKER_REPO_USERNAME=$4
+DOCKER_REPO_PASSWORD=$5
+CHART_GIT=$6
+CHART_BRANCH=$7
+CHART_PATH=$8
+TEST_CODE_GIT=${9}
+TEST_CODE_BRANCH=${10}
+TEST_CODE_PATH=${11}
+TEST_CMD_BASE=${12}
+JOB_INDEX=${13}
+HELM_VALUES=${14}
+OPENCHAOS_DRIVER=${15}
+CHAOSMESH_YAML_FILE=${16}
+OPENCHAOS_ARGS=${17}
+FAULT_DURITION=${18}
+FAULT_SCHEDULER_INTERVAL=${19}
+HELM_CHART_REPO=${20}
+HELM_CHART_VERSION=${21}
+CHART=${22}
+NODE_LABLE=${23}
+META_NODE_LABLE=${24}
+RUNTIME_PARAM=${25}
+SOCKET_PATH_PARAM=${26}
+
+export VERSION
+export CHART_GIT
+export CHART_BRANCH
+export CHART_PATH
+export REPO_NAME=`echo ${GITHUB_REPOSITORY#*/} | sed -e "s/\//-/g" | cut -c1-36 | tr '[A-Z]' '[a-z]'`
+export WORKFLOW_NAME=${GITHUB_WORKFLOW}
+export RUN_ID=${GITHUB_RUN_ID}
+export TEST_CODE_GIT
+export TEST_CODE_BRANCH
+export TEST_CODE_PATH
+export YAML_VALUES=`echo "${HELM_VALUES}" | sed -s 's/^/ /g'`
+
+echo "Start test version: ${GITHUB_REPOSITORY}@${VERSION}"
+
+echo "************************************"
+echo "* Set config... *"
+echo "************************************"
+mkdir -p ${HOME}/.kube
+# kube_config=$(echo "${ASK_CONFIG}" | base64 -d)
+# TODO : use kubevela
+kube_config=$(echo "${ASK_CONFIG}")
+echo "${kube_config}" > ${HOME}/.kube/config
+export KUBECONFIG="${HOME}/.kube/config"
+
+# Install helm
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
+chmod 700 get_helm.sh
+./get_helm.sh
+helm version
+
+VELA_APP_TEMPLATE='
+apiVersion: core.oam.dev/v1beta1
+kind: Application
+metadata:
+ name: ${VELA_APP_NAME}
+ description: ${REPO_NAME}-${WORKFLOW_NAME}-${RUN_ID}@${VERSION}
+spec:
+ components:
+ - name: ${REPO_NAME}
+ type: helm
+ properties:
+ chart: ${CHART_PATH}
+ git:
+ branch: ${CHART_BRANCH}
+ repoType: git
+ retries: 3
+ secretRef: \047\047
+ url: ${CHART_GIT}
+ values:
+${YAML_VALUES}'
+
+echo -e "${VELA_APP_TEMPLATE}" > ./velaapp.yaml
+sed -i '1d' ./velaapp.yaml
+
+env_uuid=${REPO_NAME}-${GITHUB_RUN_ID}-${JOB_INDEX}
+chaos_mesh_ns="chaos-mesh-${GITHUB_RUN_ID}-${JOB_INDEX}"
+
+
+check_pods_status() {
+ local namespace=$1
+
+ pods_status=$(kubectl get pods -n ${namespace} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{range .status.conditions[?(@.type=="Ready")]}{.status}{"\n"}{end}{end}')
+ all_ready=true
+
+ echo "$pods_status" > /tmp/pods_status.txt
+
+ while read -r pod; do
+ pod_name=$(echo "$pod" | awk '{print $1}')
+ pod_phase=$(echo "$pod" | awk '{print $2}')
+ pod_ready=$(echo "$pod" | awk '{print $3}')
+
+ if [ "$pod_phase" != "Running" ] || [ "$pod_ready" != "True" ]; then
+ echo "Pod $pod_name is not ready (Phase: $pod_phase, Ready: $pod_ready)"
+ all_ready=false
+ fi
+ done < /tmp/pods_status.txt
+
+ if [ "$all_ready" = "true" ]; then
+ return 0
+ else
+ return 1
+ fi
+}
+
+wait_for_pods_ready() {
+ local namespace=$1
+ local timeout=$2
+ local count=0
+
+ while true; do
+ if check_pods_status ${namespace}; then
+ echo "All Pods are ready"
+ kubectl get pods -n "${namespace}"
+ break
+ fi
+
+ if [ $count -gt $timeout ]; then
+ echo "Deployment timeout..."
+ exit 1
+ fi
+
+ echo "Waiting for Pods to be ready..."
+ sleep 5
+ count=$((count + 1))
+ done
+}
+
+if [ ${ACTION} == "deploy" ]; then
+ echo "************************************"
+ echo "* Create env and deploy... *"
+ echo "************************************"
+
+ echo ${VERSION}: ${env_uuid} deploy start
+
+ vela env init ${env_uuid} --namespace ${env_uuid}
+
+ export VELA_APP_NAME=${env_uuid}
+ envsubst < ./velaapp.yaml > velaapp-${REPO_NAME}.yaml
+ cat velaapp-${REPO_NAME}.yaml
+
+ vela env set ${env_uuid}
+ vela up -f "velaapp-${REPO_NAME}.yaml"
+
+ app=${env_uuid}
+ status=`vela status ${app} -n ${app}`
+ echo $status
+ res=`echo $status | grep "Create helm release successfully"`
+ let count=0
+ while [ -z "$res" ]
+ do
+ if [ $count -gt 240 ]; then
+ echo "env ${app} deploy timeout..."
+ exit 1
+ fi
+ echo "waiting for env ${app} ready..."
+ sleep 5
+ status=`vela status ${app} -n ${app}`
+ stopped=`echo $status | grep "not found"`
+ if [ ! -z "$stopped" ]; then
+ echo "env ${app} deploy stopped..."
+ exit 1
+ fi
+ res=`echo $status | grep "Create helm release successfully"`
+ let count=${count}+1
+ done
+fi
+
+if [ ${ACTION} == "chaos-test" ]; then
+ echo "************************************"
+ echo "* Chaos test... *"
+ echo "************************************"
+
+ # Deploy chaos-mesh
+ helm repo add chaos-mesh https://charts.chaos-mesh.org
+ kubectl create ns "${chaos_mesh_ns}"
+ container_runtime=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.containerRuntimeVersion}')
+ # Default to /var/run/docker.sock
+ runtime="docker"
+ socket_path="/var/run/docker.sock"
+
+ # Set the value according to the runtime
+ if echo "$container_runtime" | grep -q "docker"; then
+ runtime="docker"
+ socket_path="/var/run/docker.sock"
+ elif echo "$container_runtime" | grep -q "containerd"; then
+ runtime="containerd"
+
+ kubelet_version=$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}')
+ if echo "$kubelet_version" | grep -q "k3s"; then
+ socket_path="/run/k3s/containerd/containerd.sock"
+ elif echo "$kubelet_version" | grep -q "microk8s"; then
+ socket_path="/var/snap/microk8s/common/run/containerd.sock"
+ else
+ socket_path="/run/containerd/containerd.sock"
+ fi
+
+ elif echo "$container_runtime" | grep -q "crio"; then
+ runtime="crio"
+ socket_path="/var/run/crio/crio.sock"
+ else
+ if [ -n "$RUNTIME_PARAM" ] && [ "$RUNTIME_PARAM" != "" ] && [ -n "$SOCKET_PATH_PARAM" ] && [ "$SOCKET_PATH_PARAM" != "" ]; then
+ runtime=$RUNTIME_PARAM
+ socket_path=$SOCKET_PATH_PARAM
+ else
+ echo "Error : Unable to detect cri,please manually specify runtime and socket path."
+ exit 1
+ fi
+ fi
+
+ helm install ${chaos_mesh_ns} chaos-mesh/chaos-mesh --namespace=${chaos_mesh_ns} --set chaosDaemon.runtime=$runtime --set chaosDaemon.socketPath=$socket_path --version 2.6.3
+ sleep 10
+
+ # Check chaos-mesh pod status
+ wait_for_pods_ready ${chaos_mesh_ns} 240
+
+ # Deploy a pod for test ：openchaos-controller
+ # ConfigMap
+ openchaos_driver_file=$(cat "$OPENCHAOS_DRIVER")
+ echo -e "$openchaos_driver_file" > ./openchaos-driver-template.yaml
+
+ # Replace the placeholders in the configuration file with the ip of the worker node and the metaNode node
+ node_ips=$(kubectl get pods -n ${env_uuid} -l "$NODE_LABLE" -o jsonpath='{.items[*].status.podIP}')
+ set -- $node_ips
+
+ i=1
+ for ip in "$@"; do
+ export node_$i=$ip
+ i=$((i + 1))
+ done
+
+ meta_node_ips=$(kubectl get pods -n ${env_uuid} -l "$META_NODE_LABLE" -o jsonpath='{.items[*].status.podIP}')
+ set -- $meta_node_ips
+
+ i=1
+ for ip in "$@"; do
+ export meta_node_$i=$ip
+ i=$((i + 1))
+ done
+ envsubst < ./openchaos-driver-template.yaml > ./openchaos-driver.yaml
+ # openchaos
+ app=${env_uuid}
+ kubectl create configmap ${app}-configmap --from-file=openchaos-driver.yaml --namespace=${env_uuid} -o yaml --dry-run=client > ${app}-configmap.yaml
+ cat ./${app}-configmap.yaml
+ kubectl apply -f ./${app}-configmap.yaml -n ${env_uuid}
+
+ configmap_name="${app}-configmap"
+ export configmap_name
+ envsubst < /chaos-test/chaos-controller-template.yaml > ./chaos-controller.yaml
+
+ kubectl apply -f ./chaos-controller.yaml -n ${env_uuid}
+ sleep 10
+ test_pod_name=$(kubectl get pods -n ${env_uuid} -l app=openchaos-controller -o jsonpath='{.items[0].metadata.name}')
+
+ wait_for_pods_ready ${env_uuid} 240
+
+ chaosmesh_yaml_template=$(cat "$CHAOSMESH_YAML_FILE")
+ echo -e "${chaosmesh_yaml_template}" > ./chaos-mesh-fault.yaml
+
+ ns=${env_uuid}
+ export app
+ export ns
+ envsubst < ./chaos-mesh-fault.yaml > ./network-chaos.yaml
+ fault_file="$(pwd)/network-chaos.yaml"
+ # Check fault file
+ cat $fault_file
+ # Execute the startup script
+ mkdir -p chaos-test-report
+ REPORT_DIR="$(pwd)/chaos-test-report"
+ touch $REPORT_DIR/output.log
+
+ cd /chaos-test
+ sh ./startup.sh "$fault_file" "$FAULT_DURITION" "$test_pod_name" "$ns" "$REPORT_DIR" "$OPENCHAOS_ARGS" "$FAULT_SCHEDULER_INTERVAL"
+ exit_code=$?
+ cd -
+ exit ${exit_code}
+fi
+
+
+if [ ${ACTION} == "clean" ]; then
+ echo "************************************"
+ echo "* Delete app and env... *"
+ echo "************************************"
+
+ env=${env_uuid}
+ app=${env_uuid}
+
+ helm uninstall ${chaos_mesh_ns} -n ${chaos_mesh_ns}
+ vela delete ${env} -n ${env} -y
+ all_pod_name=`kubectl get pods --no-headers -o custom-columns=":metadata.name" -n ${env}`
+ for pod in $all_pod_name;
+ do
+ kubectl delete pod ${pod} -n ${env}
+ done
+
+ sleep 30
+
+ kubectl proxy &
+ PID=$!
+ sleep 3
+
+ DELETE_ENV=${env}
+
+ vela env delete ${DELETE_ENV} -y
+ sleep 3
+ # Delete namespaces if they exist
+ for ns in ${chaos_mesh_ns} ${DELETE_ENV}; do
+ if kubectl get namespace ${ns}; then
+ kubectl delete namespace ${ns} --wait=false
+ # Remove finalizers
+ kubectl get ns ${ns} -o json | jq '.spec.finalizers=[]' > ns-without-finalizers.json
+ cat ns-without-finalizers.json
+ curl -X PUT http://localhost:8001/api/v1/namespaces/${ns}/finalize -H "Content-Type: application/json" --data-binary @ns-without-finalizers.json
+ fi
+ done
+
+ kill $PID
+fi
\ No newline at end of file
