The configuration file is used to integrate all the step configuration content. You can see the sample configuration files for different environments in the examples directory.
There is a quick view about the configuration file, and using the yaml
format.
setup: # set up the environment cleanup: # clean up the environment trigger: # generate traffic verify: # test cases
Support two kinds of the environment to set up the system.
setup: env: kind file: path/to/kind.yaml # Specified kinD manifest file path timeout: 20m # timeout duration init-system-environment: path/to/env # Import environment file steps: # customize steps for prepare the environment - name: customize setups # step name # one of command line or kinD manifest file command: command lines # use command line to setup path: /path/to/manifest.yaml # the manifest file path wait: # how to verify the manifest is set up finish - namespace: # The pod namespace resource: # The pod resource name label-selector: # The resource label selector for: # The wait condition kind: import-images: # import docker images to KinD - image:version # support using env to expand image, such as `${env_key}` or `$env_key` expose-ports: # Expose resource for host access - namespace: # The resource namespace resource: # The resource name, such as `pod/foo` or `service/foo` port: # Want to expose port from resource
The KinD
environment follow these steps:
KinD
cluster according to the config file, expose KUBECONFIG
to environment for help execute kubectl
in the steps.kind.import-images
if needed.--manifests
) or/and run the custom init command (--commands
) by steps.If you want to import docker image from private registries, there are several ways to do this:
imagePullSecrets
to pull images, please take reference from document.kind.import-images
to load images from host.kind: import-images: - skywalking/oap:${OAP_HASH} # support using environment to expand the image name
If you want to access the resource from host, should follow these steps:
setup: kind: expose-ports: - namespace: default # Need to expose resource namespace resource: pod/foo # Resource description, such as `pod/foo` or `service/foo` port: 8080 # Resource port want to expose, support `<resource_port>`, `<bind_to_host_port>:<resource_port>`
trigger: # trigger with specified mapped port, the resource name replace all `/` or `-` as `_` # host format: <resource_name>_host # port format: <resource_name>_<container_port> url: http://${pod_foo_host}:${pod_foo_8080}/
The console output of each pod could be found in ${workDir}/logs/${namespace}/${podName}.log
.
setup: env: compose file: path/to/compose.yaml # Specified docker-compose file path timeout: 20m # Timeout duration init-system-environment: path/to/env # Import environment file steps: # Customize steps for prepare the environment - name: customize setups # Step name command: command lines # Use command line to setup
The docker-compose
environment follow these steps:
init-system-environment
file for help build service and execute steps. Each line of the file content is an environment variable, and the key value is separate by “=”.docker-compose
services.If you want to get the service host and port mapping, should follow these steps:
docker-compose
service ports
config.oap: image: xx.xx:1.0.0 ports: # define the port - 8080
trigger: # trigger with specified mappinged port url: http://${oap_host}:${oap_8080}/
The console output of each service could be found in ${workDir}/logs/{serviceName}/std.log
.
After the Setup
step is finished, use the Trigger
step to generate traffic.
trigger: action: http # The action of the trigger. support HTTP invoke. interval: 3s # Trigger the action every 3 seconds. times: 5 # The retry count before the request success. url: http://apache.skywalking.com/ # Http trigger url link. method: GET # Http trigger method. headers: "Content-Type": "application/json" "Authorization": "Basic whatever" body: '{"k1":"v1", "k2":"v2"}'
The Trigger executed successfully at least once, after success, the next stage could be continued. Otherwise, there is an error and exit.
After the Trigger
step is finished, running test cases.
verify: retry: # verify with retry strategy count: 10 # max retry count interval: 10s # the interval between two attempts, e.g. 10s, 1m. fail-fast: true # when a case fails, whether to stop verifying other cases. This property defaults to true. cases: # verify test cases - actual: path/to/actual.yaml # verify by actual file path expected: path/to/expected.yaml # excepted content file path - query: echo 'foo' # verify by command execute output expected: path/to/expected.yaml # excepted content file path - includes: # including cases - path/to/cases.yaml # cases file path
The test cases are executed in the order of declaration from top to bottom. When the execution of a case fails and the retry strategy is exceeded, it will stop verifying other cases if fail-fast
is true
. Otherwise, the process will continue to verify other cases.
The retry strategy could retry automatically on the test case failure, and restart by the failed test case.
Support two kind source to verify, one case only supports one kind source type:
yaml
format file.yaml
format.After clarifying the content that needs to be verified, you need to write content to verify the real content and ensure that the data is correct.
You need to use the form of Go Template to write the verification file, and the data content to be rendered comes from the real data. By verifying whether the rendered data is consistent with the real data, it is verified whether the content is consistent. You could see many test cases in this directory.
We use go-cmp to show the parts where excepted do not match the actual data. -
prefix represents the expected data content, +
prefix represents the actual data content.
We have done a lot of extension functions for verification functions on the original Go Template.
Extension functions are used to help users quickly locate the problem content and write test cases that are easier to use.
Verify that the number fits the range.
Function | Description | Grammar | Verify success | Verify failure |
---|---|---|---|---|
gt | Verify the first param is greater than second param | {{gt param1 param2}} | param1 | <wanted gt $param2, but was $param1> |
ge | Verify the first param is greater than or equals second param | {{ge param1 param2}} | param1 | <wanted gt $param2, but was $param1> |
lt | Verify the first param is less than second param | {{lt param1 param2}} | param1 | <wanted gt $param2, but was $param1> |
le | Verify the first param is less than or equals second param | {{le param1 param2}} | param1 | <wanted gt $param2, but was $param1> |
regexp | Verify the first param matches the second regular expression | {{regexp param1 param2}} | param1 | <“$param1” does not match the pattern $param2"> |
notEmpty | Verify The param is not empty | {{notEmpty param}} | param | <"" is empty, wanted is not empty> |
hasPrefix | Verify The string param has the same prefix. | {{hasPrefix param1 param2}} | true | false |
hasSuffix | Verify The string param has the same suffix. | {{hasSuffix param1 param2}} | true | false |
Verify the data in the condition list, Currently, it is only supported when all the conditions in the list are executed, it is considered as successful.
Here is an example, It's means the list values must have value is greater than 0, also have value greater than 1, Otherwise verify is failure.
{{- contains .list }} - key: {{ gt .value 0 }} - key: {{ gt .value 1 }} {{- end }}
In order to make the program easier for users to read and use, some code conversions are provided.
Function | Description | Grammar | Result |
---|---|---|---|
b64enc | Base64 encode | {{ b64enc “Foo” }} | Zm9v |
sha256enc | Sha256 encode | {{ sha256enc “Foo” }} | 1cbec737f863e4922cee63cc2ebbfaafcd1cff8b790d8cfd2e6a5d550b648afa |
sha512enc | Sha512 encode | {{ sha512enc “Foo” }} | 4abcd2639957cb23e33f63d70659b602a5923fafcfd2768ef79b0badea637e5c837161aa101a557a1d4deacbd912189e2bb11bf3c0c0c70ef7797217da7e8207 |
You could include multiple cases into one single E2E verify, It's helpful for reusing the same verify cases.
Here is the reused verify cases, and using includes
configuration item to include this into E2E config.
cases: - actual: path/to/actual.yaml # verify by actual file path expected: path/to/expected.yaml # excepted content file path - query: echo 'foo' # verify by command execute output expected: path/to/expected.yaml # excepted content file path
After the E2E finished, how to clean up the environment.
cleanup: on: always # Clean up strategy
If the on
option under cleanup
is not set, it will be automatically set to always
if there is environment variable CI=true
, which is present on many popular CI services, such as GitHub Actions, CircleCI, etc., otherwise it will be set to success
, so the testing environment can be preserved when tests failed in your local machine.
All available strategies:
always
: No matter the execution result is success or failure, cleanup will be performed.success
: Only when the execution succeeds.failure
: Only when the execution failed.never
: Never clean up the environment.