tests/integration/README.md - dubbo-go-pixiu - Git at Google

 # Istio Integration Tests

 This folder contains Istio integration tests that use the test framework checked in at
 [istio.io/istio/pkg/test/framework](https://github.com/istio/istio/tree/master/pkg/test/framework).

 ## Table of Contents

 1. [Overview](#overview)
 1. [Writing Tests](#writing-tests)
     1. [Adding a Test Suite](#adding-a-test-suite)
     1. [Sub-Tests](#sub-tests)
     1. [Parallel Tests](#parallel-tests)
     1. [Using Components](#using-components)
     1. [Writing Components](#writing-components)
 1. [Running Tests](#running-tests)
     1. [Test Parallelism and Kubernetes](#test-parellelism-and-kubernetes)
     1. [Test Selection](#test-selection)
     1. [Running Tests on CI](#running-tests-on-ci)
         1. [Step 1: Add a Test Script](#step-1-add-a-test-script)
         1. [Step 2: Add a Prow Job](#step-2-add-a-prow-job)
         1. [Step 3: Update TestGrid](#step-3-update-testgrid)
 1. [Environments](#environments)
 1. [Diagnosing Failures](#diagnosing-failures)
     1. [Working Directory](#working-directory)
     1. [Enabling CI Mode](#enabling-ci-mode)
     1. [Preserving State (No Cleanup)](#preserving-state-no-cleanup)
     1. [Additional Logging](#additional-logging)
     1. [Running Tests Under Debugger](#running-tests-under-debugger-goland)
 1. [Reference](#reference)
     1. [Helm Values Overrides](#helm-values-overrides)
     1. [Commandline Flags](#command-line-flags)
 1. [Notes](#notes)
     1. [Running on a Mac](#running-on-a-mac)

 ## Overview

 The goal of the framework is to make it as easy as possible to author and run tests. In its simplest
 case, just typing ```go test ./...``` should be sufficient to run tests.

 This guide walks through the basics of writing tests with the Istio test framework. For best
 practices, see [Writing Good Integration Tests](https://github.com/istio/istio/wiki/Writing-Good-Integration-Tests).

 ## Writing Tests

 The test framework is designed to work with standard go tooling and allows developers
 to write environment-agnostics tests in a high-level fashion.

 ### Adding a Test Suite

 All tests that use the framework, must run as part of a *suite*. Only a single suite can be defined per package, since
 it is bootstrapped by a Go `TestMain`, which has the same restriction.

 To begin, create a new folder for your suite under
 [tests/integration](https://github.com/istio/istio/tree/master/tests/integration).

 ```console
 $ cd ${ISTIO}/tests/integration
 $ mkdir mysuite
 ```

 Within that package, create a `TestMain` to bootstrap the test suite:

 ```go
 func TestMain(m *testing.M) {
     framework.
         NewSuite("mysuite", m).
         Run()
 }
 ```

 Next, define your tests in the same package:

 ```go
 func TestMyLogic(t *testing.T) {
     framework.
         NewTest(t).
         Run(func(ctx framework.TestContext) {
             // Create a component
             p := pilot.NewOrFail(ctx, ctx, cfg)

             // Use the component.
             // Apply Kubernetes Config
             ctx.ApplyConfigOrFail(ctx, nil, mycfg)

             // Do more stuff here.
         }
 }
 ```

 The `framework.TestContext` is a wrapper around the underlying `testing.T` and implements the same interface. Test code
 should generally not interact with the `testing.T` directly.

 In the `TestMain`, you can also restrict the test to particular environment, apply labels, or do test-wide setup, such as
  deploying Istio.

 ```go
 func TestMain(m *testing.M) {
     framework.
         NewSuite("mysuite", m).
         // Deploy Istio on the cluster
         Setup(istio.Setup(nil, nil)).
         // Run your own custom setup
         Setup(mySetup).
         Run()
 }

 func mySetup(ctx resource.Context) error {
     // Your own setup code
     return nil
 }
 ```

 ### Sub-Tests

 Go allows you to run sub-tests with `t.Run()`. Similarly, this framework supports nesting tests with `ctx.NewSubTest()`:

 ```go
 func TestMyLogic(t *testing.T) {
     framework.
         NewTest(t).
         Run(func(ctx framework.TestContext) {

             // Create a component
             g := galley.NewOrFail(ctx, ctx, cfg)

             configs := []struct{
                 name: string
                 yaml: string
             } {
                 // Some array of YAML
             }

             for _, cfg := range configs {
                 ctx.NewSubTest(cfg.name).
                     Run(func(ctx framework.TestContext) {
                         ctx.ApplyConfigOrFail(ctx, nil, mycfg)
                         // Do more stuff here.
                     })
             }
         })
 }
 ```

 Under the hood, calling `subtest.Run()` delegates to `t.Run()` in order to create a child `testing.T`.

 ### Parallel Tests

 Many tests can take a while to start up for a variety of reasons, such as waiting for pods to start or waiting
 for a particular piece of configuration to propagate throughout the system. Where possible, it may be desirable
 to run these sorts of tests in parallel:

 ```go
 func TestMyLogic(t *testing.T) {
     framework.
         NewTest(t).
         RunParallel(func(ctx framework.TestContext) {
             // ...
         }
 }
 ```

 Under the hood, this relies on Go's `t.Parallel()` and will, therefore, have the same behavior.

 A parallel test will run in parallel with siblings that share the same parent test. The parent test function
 will exit before the parallel children are executed. It should be noted that if the parent test is prevented
 from exiting (e.g. parent test is waiting for something to occur within the child test), the test will
 deadlock.

 Consider the following example:

 ```go
 func TestMyLogic(t *testing.T) {
     framework.NewTest(t).
         Run(func(ctx framework.TestContext) {
             ctx.NewSubTest("T1").
                 Run(func(ctx framework.TestContext) {
                     ctx.NewSubTest("T1a").
                         RunParallel(func(ctx framework.TestContext) {
                             // Run in parallel with T1b
                         })
                     ctx.NewSubTest("T1b").
                         RunParallel(func(ctx framework.TestContext) {
                             // Run in parallel with T1a
                         })
                     // Exits before T1a and T1b are run.
                 })

             ctx.NewSubTest("T2").
                 Run(func(ctx framework.TestContext) {
                     ctx.NewSubTest("T2a").
                         RunParallel(func(ctx framework.TestContext) {
                             // Run in parallel with T2b
                         })
                     ctx.NewSubTest("T2b").
                         RunParallel(func(ctx framework.TestContext) {
                             // Run in parallel with T2a
                         })
                     // Exits before T2a and T2b are run.
                 })
         })
 }
 ```

 In the example above, non-parallel parents T1 and T2 contain parallel children T1a, T1b, T2a, T2b.

 Since both T1 and T2 are non-parallel, they are run synchronously: T1 followed by T2. After T1 exits,
 T1a and T1b are run asynchronously with each other. After T1a and T1b complete, T2 is then run in the
 same way: T2 exits, then T2a and T2b are run asynchronously to completion.

 ### Using Components

 The framework itself is just a platform for running tests and tracking resources. Without these `resources`, there
 isn't much added value. Enter: components.

 Components are utilities that provide abstractions for Istio resources. They are maintained in the
 [components package](https://github.com/istio/istio/tree/master/pkg/test/framework/components), which defines
 various Istio components such as galley, pilot, and namespaces.

 Each component defines their own API which simplifies their use from test code, abstracting away the
 environment-specific details. This means that the test code can (and should, where possible) be written in an
 environment-agnostic manner, so that they can be run against any Istio implementation.

 For example, the following code creates and then interacts with a Galley and Pilot component:

 ```go
 func TestMyLogic(t *testing.T) {
     framework.
         NewTest(t).
         Run(func(ctx framework.TestContext) {
             // Create the components.
             g := galley.NewOrFail(ctx, ctx, galley.Config{})
             p := pilot.NewOrFail(ctx, ctx, pilot.Config {})

             // Apply configuration via Galley.
             ctx.ApplyConfigOrFail(ctx, nil, mycfg)

             // Wait until Pilot has received the configuration update.
             p.StartDiscoveryOrFail(t, discoveryRequest)
             p.WatchDiscoveryOrFail(t, timeout,
                 func(response *xdsapi.DiscoveryResponse) (b bool, e error) {
                     // Validate that the discovery response has the configuration applied.
                 })
             // Do more stuff...
         }
 }
 ```

 When a component is created, the framework tracks its lifecycle. When the test exits, any components that were
 created during the test are automatically closed.

 ### Writing Components

 To add a new component, you'll first need to create a top-level folder for your component under the
 [components folder](https://github.com/istio/istio/tree/master/pkg/test/framework/components).

 ```console
 $ cd ${ISTIO}/pkg/test/framework/components
 $ mkdir mycomponent
 ```

 You'll then need to define your component's API.

 ```go
 package mycomponent

 type Instance interface {
     resource.Resource

     DoStuff() error
     DoStuffOrFail(t test.Failer)
 }
 ```

 | NOTE: A common pattern is to provide two versions of many methods: one that returns an error as well as an `OrFail` version that fails the test upon encountering an error. This provides options to the calling test and helps to simplify the calling logic. |
 | --- |

 Next you need to implement your component for one or more environments. If possible, create both a native and Kubernetes version.

 ```go
 package mycomponent

 type nativeComponent struct {
     id resource.ID
     // ...
 }

 func newNative(ctx resource.Context) (Instance, error) {
     if config.Galley == nil {
         return nil, errors.New("galley must be provided")
     }

     instance := &nativeComponent{}
     instance.id = ctx.TrackResource(instance)

     //...
     return instance, nil
 }

 func (c *nativeComponent) ID() resource.ID {
     return c.id
 }
 ```

 Each implementation of the component must implement `resource.Resource`, which just exposes a unique identifier for your
 component instances used for resource tracking by the framework. To get the ID, the component must call `ctx.TrackResource`
 during construction.

 Finally, you'll need to provide an environment-agnostic constructor for your component:

 ```go
 package mycomponent

 func New(ctx resource.Context) (i Instance, err error){
     err = resource.UnsupportedEnvironment(ctx.Environment())
     ctx.Environment().Case(environment.Native, func() {
         i, err = newNative(ctx)
     })
     ctx.Environment().Case(environment.Kube, func() {
         i, err = newKube(ctx)
     })
     return
 }

 func NewOrFail(t test.Failer, ctx resource.Context) Instance {
     i, err := New(ctx)
     if err != nil {
         t.Fatal(err)
     }
     return i
 }
 ```

 Now that everything is in place, you can begin using your component:

 ```go
 func TestMyLogic(t *testing.T) {
     framework.
         NewTest(t).
         Run(func(ctx framework.TestContext) {
             // Create the components.
             g := myComponent.NewOrFail(ctx, ctx)

             // Do more stuff...
         }
 }
 ```

 ## Running Tests

 The test framework builds on top of the Go testing infrastructure, and is therefore compatible with
 the standard `go test` command-line.  For example, to run the tests under the `/tests/integration/mycomponent`
 using the default (native) environment, you can simply type:

 ```console
 $ go test -tags=integ ./tests/integration/mycomponent/...
 ```

 Note that samples below invoking variations of ```go test ./...``` are intended to be run from the ```tests/integration``` directory.

 Tests are tagged with the `integ` build target to avoid accidental invocation. If this is not set, no tests will be run.

 ### Test Parellelism and Kubernetes

 By default, Go will run tests within the same package (i.e. suite) synchronously. However, tests in other packages
 may be run concurrently.

 When running in the Kubernetes environment this can be problematic for suites that deploy Istio. The Istio deployment,
 as it stands is a singleton per cluster. If multiple suites attempt to deploy/configure Istio,
 they can corrupt each other and/or simply fail.  To avoid this issue, you have a couple of options:

 1. Run one suite per command (e.g. `go test ./tests/integration/mysuite/...`)
 1. Disable parallelism with `-p 1` (e.g. `go test -p 1 ./...`). A major disadvantage to doing this is that it will also disable
 parallelism within the suite, even when explicitly specified via [RunParallel](#parallel-tests).

 ### Test Selection

 When no flags are specified, the test framework will run all applicable tests. It is possible to filter in/out specific
 tests using 2 mechanisms:

 1. The standard ```-run <regexp>``` flag, as exposed by Go's own test framework.
 1. ```--istio.test.select <filter-expr>``` flag to select/skip framework-aware tests that use labels.

 For example, if a test, or test suite uses labels in this fashion:

 ```go
 func TestMain(m *testing.M) {
     framework.
         NewSuite("galley_conversion", m).
         // Test is tagged with "Presubmit" label
         Label(label.CustomSetup).
         Run()
 ```

 Then you can explicitly select execution of such tests using label based selection. For example, the following expression
 will select only the tests that have the ```label.CustomSetup``` label.

 ```console
 $ go test ./... --istio.test.select +customsetup
 ```

 Similarly, you can exclude tests that use ```label.CustomSetup``` label by:

 ```console
 $ go test ./... --istio.test.select -customsetup
 ```

 You can "and" the predicates by separating with commas:

 ```console
 $ go test ./... --istio.test.select +customsetup,-postsubmit
 ```

 This will select tests that have ```label.CustomSetup``` only. It will **not** select tests that have both ```label.CustomSetup```
 and ```label.Postsubmit```.

 ### Running Tests on CI

 Istio's CI/CD system is composed of 2 parts:

 Tool | Description |
 ---|---
 [Prow](https://github.com/kubernetes/test-infra/tree/master/prow) | Kubernetes-based CI/CD system developed by the Kubernetes community and is deployed in Google Kubernetes Engine (GKE).
 [TestGrid](https://k8s-testgrid.appspot.com/istio-release) | A Kubernetes dashboard used for visualizing the status of the Prow jobs.

 Test suites are defined for each toplevel directory (such as `pilot` and `telemetry`), so any tests added to these directories will automatically be run in CI.

 If you need to add a new test suite, it can be added to the [job configuration](https://github.com/istio/test-infra/blob/master/prow/config/jobs/istio.yaml).

 ## Environments

 The test binaries run in a Kubernetes cluster, but the test logic runs in the test binary.

 ```console
 $ go test ./... -p 1
 ```

 | WARNING: ```-p 1``` is required when running directly in the ```tests/integration/``` folder. |
 | --- |

 You will need to provide a K8s cluster to run the tests against.
 (See [here](https://github.com/istio/istio/blob/master/tests/integration/GKE.md)
 for info about how to set up a suitable GKE cluster.)
 You can specify the kube config file that should be used to use for connecting to the cluster, through
 command-line:

 ```console
 $ go test ./... -p 1 --istio.test.kube.config ~/.kube/config
 ```

 If not specified, `~/.kube/config` will be used by default.

 **Be aware that any existing content will be altered and/or removed from the cluster**.

 Note that the HUB and TAG environment variables **must** be set when running tests in the Kubernetes environment.

 ## Diagnosing Failures

 ### Working Directory

 The test framework will generate additional diagnostic output in its work directory. Typically, this is
 created under the host operating system's temporary folder (which can be overridden using
 the `--istio.test.work_dir` flag). The name of the work dir will be based on the test id that is supplied in
 a tests TestMain method. These files typically contain some of the logging & diagnostic output that components
 spew out as part of test execution

 ```console
 $ go test galley/... --istio.test.work_dir /foo
   ...

 $ ls /foo
   galley-test-4ef25d910d2746f9b38/

 $ ls /foo/galley-test-4ef25d910d2746f9b38/
   dubbo-system-1537332205890088657.yaml
   ...
 ```

 ### Enabling CI Mode

 When executing in the CI systems, the makefiles use the ```--istio.test.ci``` flag. This flag causes a few changes in
 behavior. Specifically, more verbose logging output will be displayed, some of the timeout values will be more relaxed, and
 additional diagnostic data will be dumped into the working directory at the end of the test execution.

 The flag is not enabled by default to provide a better U/X when running tests locally (i.e. additional logging can clutter
 test output and error dumping can take quite a while). However, if you see a behavior difference between local and CI runs,
 you can enable the flag to make the tests work in a similar fashion.

 ### Preserving State (No Cleanup)

 By default, the test framework will cleanup all deployed artifacts after the test run, especially on the Kubernetes
 environment. You can specify the ```--istio.test.nocleanup``` flag to stop the framework from cleaning up the state
 for investigation.

 ### Additional Logging

 The framework accepts standard istio logging flags. You can use these flags to enable additional logging for both the
 framework, as well as some of the components that are used in-line in the native environment:

 ```console
 $ go test ./... --log_output_level=tf:debug
 ```

 The above example will enable debugging logging for the test framework (```tf```) and the MCP protocol stack (```mcp```).

 ### Running Tests Under Debugger (GoLand)

 The tests authored in the new test framework can be debugged directly under GoLand using the debugger. If you want to
 pass command-line flags to the test while running under the debugger, you can use the
 [Run/Debug configurations dialog](https://i.stack.imgur.com/C6y0L.png) to specify these flags as program arguments.

 ## Reference

 ### Command-Line Flags

 The test framework supports the following command-line flags:

 ```plain
   -istio.test.work_dir string
         Local working directory for creating logs/temp files. If left empty, os.TempDir() is used.

  -istio.test.ci
         Enable CI Mode. Additional logging and state dumping will be enabled.

   -istio.test.nocleanup
         Do not cleanup resources after test completion

   -istio.test.select string
         Comma separatated list of labels for selecting tests to run (e.g. 'foo,+bar-baz').

   -istio.test.hub string
         Container registry hub to use (default HUB environment variable)

   -istio.test.tag string
         Common Container tag to use when deploying container images (default TAG environment variable)

   -istio.test.pullpolicy string
         Common image pull policy to use when deploying container images

   -istio.test.kube.config string
         A comma-separated list of paths to kube config files for cluster environments. (default ~/.kube/config)

   -istio.test.kube.deploy
         Deploy Istio into the target Kubernetes environment. (default true)

   -istio.test.kube.deployEastWestGW
         Deploy Istio east west gateway into the target Kubernetes environment. (default true)

   -istio.test.kube.systemNamespace string
         The namespace where the Istio components reside in a typical deployment. (default "dubbo-system")

   -istio.test.kube.helm.values string
         Manual overrides for Helm values file. Only valid when deploying Istio.

   -istio.test.kube.helm.iopFile string
         IstioOperator spec file. This can be an absolute path or relative to the repository root. Defaults to "tests/integration/iop-integration-test-defaults.yaml".

   -istio.test.kube.loadbalancer bool
         Used to obtain the right IP address for ingress gateway. This should be false for any environment that doesn't support a LoadBalancer type.

   -istio.test.revision string
         Overwrite the default namespace label (istio-enabled=true) with revision lable (istio.io/rev=XXX). (default is no overwrite)

   -istio.test.skipVM bool
         Skip all the VM related parts in all the tests. (default is "false")
 ```

 ## Notes

 ### Running on a Mac

 * Currently some _native_ tests fail when being run on a Mac with an error like:

 ```plain
 unable to locate an Envoy binary
 ```

 This is documented in this [PR](https://github.com/istio/istio/issues/13677). Once the Envoy binary is available for the Mac,
 these tests will hopefully succeed.

 * If one uses Docker for Mac for the kubernetes environment be sure to specify the `-istio.test.kube.loadbalancer=false` parameter. The solves an error like:

 ```plain
 service ingress is not available yet
 ```
	# Istio Integration Tests

	This folder contains Istio integration tests that use the test framework checked in at
	[istio.io/istio/pkg/test/framework](https://github.com/istio/istio/tree/master/pkg/test/framework).

	## Table of Contents

	1. [Overview](#overview)
	1. [Writing Tests](#writing-tests)
	1. [Adding a Test Suite](#adding-a-test-suite)
	1. [Sub-Tests](#sub-tests)
	1. [Parallel Tests](#parallel-tests)
	1. [Using Components](#using-components)
	1. [Writing Components](#writing-components)
	1. [Running Tests](#running-tests)
	1. [Test Parallelism and Kubernetes](#test-parellelism-and-kubernetes)
	1. [Test Selection](#test-selection)
	1. [Running Tests on CI](#running-tests-on-ci)
	1. [Step 1: Add a Test Script](#step-1-add-a-test-script)
	1. [Step 2: Add a Prow Job](#step-2-add-a-prow-job)
	1. [Step 3: Update TestGrid](#step-3-update-testgrid)
	1. [Environments](#environments)
	1. [Diagnosing Failures](#diagnosing-failures)
	1. [Working Directory](#working-directory)
	1. [Enabling CI Mode](#enabling-ci-mode)
	1. [Preserving State (No Cleanup)](#preserving-state-no-cleanup)
	1. [Additional Logging](#additional-logging)
	1. [Running Tests Under Debugger](#running-tests-under-debugger-goland)
	1. [Reference](#reference)
	1. [Helm Values Overrides](#helm-values-overrides)
	1. [Commandline Flags](#command-line-flags)
	1. [Notes](#notes)
	1. [Running on a Mac](#running-on-a-mac)

	## Overview

	The goal of the framework is to make it as easy as possible to author and run tests. In its simplest
	case, just typing ```go test ./...``` should be sufficient to run tests.

	This guide walks through the basics of writing tests with the Istio test framework. For best
	practices, see [Writing Good Integration Tests](https://github.com/istio/istio/wiki/Writing-Good-Integration-Tests).

	## Writing Tests

	The test framework is designed to work with standard go tooling and allows developers
	to write environment-agnostics tests in a high-level fashion.

	### Adding a Test Suite

	All tests that use the framework, must run as part of a suite. Only a single suite can be defined per package, since
	it is bootstrapped by a Go `TestMain`, which has the same restriction.

	To begin, create a new folder for your suite under
	[tests/integration](https://github.com/istio/istio/tree/master/tests/integration).

	```console
	$ cd ${ISTIO}/tests/integration
	$ mkdir mysuite
	```

	Within that package, create a `TestMain` to bootstrap the test suite:

	```go
	func TestMain(m *testing.M) {
	framework.
	NewSuite("mysuite", m).
	Run()
	}
	```

	Next, define your tests in the same package:

	```go
	func TestMyLogic(t *testing.T) {
	framework.
	NewTest(t).
	Run(func(ctx framework.TestContext) {
	// Create a component
	p := pilot.NewOrFail(ctx, ctx, cfg)

	// Use the component.
	// Apply Kubernetes Config
	ctx.ApplyConfigOrFail(ctx, nil, mycfg)

	// Do more stuff here.
	}
	}
	```

	The `framework.TestContext` is a wrapper around the underlying `testing.T` and implements the same interface. Test code
	should generally not interact with the `testing.T` directly.

	In the `TestMain`, you can also restrict the test to particular environment, apply labels, or do test-wide setup, such as
	deploying Istio.

	```go
	func TestMain(m *testing.M) {
	framework.
	NewSuite("mysuite", m).
	// Deploy Istio on the cluster
	Setup(istio.Setup(nil, nil)).
	// Run your own custom setup
	Setup(mySetup).
	Run()
	}

	func mySetup(ctx resource.Context) error {
	// Your own setup code
	return nil
	}
	```

	### Sub-Tests

	Go allows you to run sub-tests with `t.Run()`. Similarly, this framework supports nesting tests with `ctx.NewSubTest()`:

	```go
	func TestMyLogic(t *testing.T) {
	framework.
	NewTest(t).
	Run(func(ctx framework.TestContext) {

	// Create a component
	g := galley.NewOrFail(ctx, ctx, cfg)

	configs := []struct{
	name: string
	yaml: string
	} {
	// Some array of YAML
	}

	for _, cfg := range configs {
	ctx.NewSubTest(cfg.name).
	Run(func(ctx framework.TestContext) {
	ctx.ApplyConfigOrFail(ctx, nil, mycfg)
	// Do more stuff here.
	})
	}
	})
	}
	```

	Under the hood, calling `subtest.Run()` delegates to `t.Run()` in order to create a child `testing.T`.

	### Parallel Tests

	Many tests can take a while to start up for a variety of reasons, such as waiting for pods to start or waiting
	for a particular piece of configuration to propagate throughout the system. Where possible, it may be desirable
	to run these sorts of tests in parallel:

	```go
	func TestMyLogic(t *testing.T) {
	framework.
	NewTest(t).
	RunParallel(func(ctx framework.TestContext) {
	// ...
	}
	}
	```

	Under the hood, this relies on Go's `t.Parallel()` and will, therefore, have the same behavior.

	A parallel test will run in parallel with siblings that share the same parent test. The parent test function
	will exit before the parallel children are executed. It should be noted that if the parent test is prevented
	from exiting (e.g. parent test is waiting for something to occur within the child test), the test will
	deadlock.

	Consider the following example:

	```go
	func TestMyLogic(t *testing.T) {
	framework.NewTest(t).
	Run(func(ctx framework.TestContext) {
	ctx.NewSubTest("T1").
	Run(func(ctx framework.TestContext) {
	ctx.NewSubTest("T1a").
	RunParallel(func(ctx framework.TestContext) {
	// Run in parallel with T1b
	})
	ctx.NewSubTest("T1b").
	RunParallel(func(ctx framework.TestContext) {
	// Run in parallel with T1a
	})
	// Exits before T1a and T1b are run.
	})

	ctx.NewSubTest("T2").
	Run(func(ctx framework.TestContext) {
	ctx.NewSubTest("T2a").
	RunParallel(func(ctx framework.TestContext) {
	// Run in parallel with T2b
	})
	ctx.NewSubTest("T2b").
	RunParallel(func(ctx framework.TestContext) {
	// Run in parallel with T2a
	})
	// Exits before T2a and T2b are run.
	})
	})
	}
	```

	In the example above, non-parallel parents T1 and T2 contain parallel children T1a, T1b, T2a, T2b.

	Since both T1 and T2 are non-parallel, they are run synchronously: T1 followed by T2. After T1 exits,
	T1a and T1b are run asynchronously with each other. After T1a and T1b complete, T2 is then run in the
	same way: T2 exits, then T2a and T2b are run asynchronously to completion.

	### Using Components

	The framework itself is just a platform for running tests and tracking resources. Without these `resources`, there
	isn't much added value. Enter: components.

	Components are utilities that provide abstractions for Istio resources. They are maintained in the
	[components package](https://github.com/istio/istio/tree/master/pkg/test/framework/components), which defines
	various Istio components such as galley, pilot, and namespaces.

	Each component defines their own API which simplifies their use from test code, abstracting away the
	environment-specific details. This means that the test code can (and should, where possible) be written in an
	environment-agnostic manner, so that they can be run against any Istio implementation.

	For example, the following code creates and then interacts with a Galley and Pilot component:

	```go
	func TestMyLogic(t *testing.T) {
	framework.
	NewTest(t).
	Run(func(ctx framework.TestContext) {
	// Create the components.
	g := galley.NewOrFail(ctx, ctx, galley.Config{})
	p := pilot.NewOrFail(ctx, ctx, pilot.Config {})

	// Apply configuration via Galley.
	ctx.ApplyConfigOrFail(ctx, nil, mycfg)

	// Wait until Pilot has received the configuration update.
	p.StartDiscoveryOrFail(t, discoveryRequest)
	p.WatchDiscoveryOrFail(t, timeout,
	func(response *xdsapi.DiscoveryResponse) (b bool, e error) {
	// Validate that the discovery response has the configuration applied.
	})
	// Do more stuff...
	}
	}
	```

	When a component is created, the framework tracks its lifecycle. When the test exits, any components that were
	created during the test are automatically closed.

	### Writing Components

	To add a new component, you'll first need to create a top-level folder for your component under the
	[components folder](https://github.com/istio/istio/tree/master/pkg/test/framework/components).

	```console
	$ cd ${ISTIO}/pkg/test/framework/components
	$ mkdir mycomponent
	```

	You'll then need to define your component's API.

	```go
	package mycomponent

	type Instance interface {
	resource.Resource

	DoStuff() error
	DoStuffOrFail(t test.Failer)
	}
	```

	\| NOTE: A common pattern is to provide two versions of many methods: one that returns an error as well as an `OrFail` version that fails the test upon encountering an error. This provides options to the calling test and helps to simplify the calling logic. \|
	\| --- \|

	Next you need to implement your component for one or more environments. If possible, create both a native and Kubernetes version.

	```go
	package mycomponent

	type nativeComponent struct {
	id resource.ID
	// ...
	}

	func newNative(ctx resource.Context) (Instance, error) {
	if config.Galley == nil {
	return nil, errors.New("galley must be provided")
	}

	instance := &nativeComponent{}
	instance.id = ctx.TrackResource(instance)

	//...
	return instance, nil
	}

	func (c *nativeComponent) ID() resource.ID {
	return c.id
	}
	```

	Each implementation of the component must implement `resource.Resource`, which just exposes a unique identifier for your
	component instances used for resource tracking by the framework. To get the ID, the component must call `ctx.TrackResource`
	during construction.

	Finally, you'll need to provide an environment-agnostic constructor for your component:

	```go
	package mycomponent

	func New(ctx resource.Context) (i Instance, err error){
	err = resource.UnsupportedEnvironment(ctx.Environment())
	ctx.Environment().Case(environment.Native, func() {
	i, err = newNative(ctx)
	})
	ctx.Environment().Case(environment.Kube, func() {
	i, err = newKube(ctx)
	})
	return
	}

	func NewOrFail(t test.Failer, ctx resource.Context) Instance {
	i, err := New(ctx)
	if err != nil {
	t.Fatal(err)
	}
	return i
	}
	```

	Now that everything is in place, you can begin using your component:

	```go
	func TestMyLogic(t *testing.T) {
	framework.
	NewTest(t).
	Run(func(ctx framework.TestContext) {
	// Create the components.
	g := myComponent.NewOrFail(ctx, ctx)

	// Do more stuff...
	}
	}
	```

	## Running Tests

	The test framework builds on top of the Go testing infrastructure, and is therefore compatible with
	the standard `go test` command-line. For example, to run the tests under the `/tests/integration/mycomponent`
	using the default (native) environment, you can simply type:

	```console
	$ go test -tags=integ ./tests/integration/mycomponent/...
	```

	Note that samples below invoking variations of ```go test ./...``` are intended to be run from the ```tests/integration``` directory.

	Tests are tagged with the `integ` build target to avoid accidental invocation. If this is not set, no tests will be run.

	### Test Parellelism and Kubernetes

	By default, Go will run tests within the same package (i.e. suite) synchronously. However, tests in other packages
	may be run concurrently.

	When running in the Kubernetes environment this can be problematic for suites that deploy Istio. The Istio deployment,
	as it stands is a singleton per cluster. If multiple suites attempt to deploy/configure Istio,
	they can corrupt each other and/or simply fail. To avoid this issue, you have a couple of options:

	1. Run one suite per command (e.g. `go test ./tests/integration/mysuite/...`)
	1. Disable parallelism with `-p 1` (e.g. `go test -p 1 ./...`). A major disadvantage to doing this is that it will also disable
	parallelism within the suite, even when explicitly specified via [RunParallel](#parallel-tests).

	### Test Selection

	When no flags are specified, the test framework will run all applicable tests. It is possible to filter in/out specific
	tests using 2 mechanisms:

	1. The standard ```-run <regexp>``` flag, as exposed by Go's own test framework.
	1. ```--istio.test.select <filter-expr>``` flag to select/skip framework-aware tests that use labels.

	For example, if a test, or test suite uses labels in this fashion:

	```go
	func TestMain(m *testing.M) {
	framework.
	NewSuite("galley_conversion", m).
	// Test is tagged with "Presubmit" label
	Label(label.CustomSetup).
	Run()
	```

	Then you can explicitly select execution of such tests using label based selection. For example, the following expression
	will select only the tests that have the ```label.CustomSetup``` label.

	```console
	$ go test ./... --istio.test.select +customsetup
	```

	Similarly, you can exclude tests that use ```label.CustomSetup``` label by:

	```console
	$ go test ./... --istio.test.select -customsetup
	```

	You can "and" the predicates by separating with commas:

	```console
	$ go test ./... --istio.test.select +customsetup,-postsubmit
	```

	This will select tests that have ```label.CustomSetup``` only. It will not select tests that have both ```label.CustomSetup```
	and ```label.Postsubmit```.

	### Running Tests on CI

	Istio's CI/CD system is composed of 2 parts:

	Tool \| Description \|
	---\|---
	[Prow](https://github.com/kubernetes/test-infra/tree/master/prow) \| Kubernetes-based CI/CD system developed by the Kubernetes community and is deployed in Google Kubernetes Engine (GKE).
	[TestGrid](https://k8s-testgrid.appspot.com/istio-release) \| A Kubernetes dashboard used for visualizing the status of the Prow jobs.

	Test suites are defined for each toplevel directory (such as `pilot` and `telemetry`), so any tests added to these directories will automatically be run in CI.

	If you need to add a new test suite, it can be added to the [job configuration](https://github.com/istio/test-infra/blob/master/prow/config/jobs/istio.yaml).

	## Environments

	The test binaries run in a Kubernetes cluster, but the test logic runs in the test binary.

	```console
	$ go test ./... -p 1
	```

	\| WARNING: ```-p 1``` is required when running directly in the ```tests/integration/``` folder. \|
	\| --- \|

	You will need to provide a K8s cluster to run the tests against.
	(See [here](https://github.com/istio/istio/blob/master/tests/integration/GKE.md)
	for info about how to set up a suitable GKE cluster.)
	You can specify the kube config file that should be used to use for connecting to the cluster, through
	command-line:

	```console
	$ go test ./... -p 1 --istio.test.kube.config ~/.kube/config
	```

	If not specified, `~/.kube/config` will be used by default.

	Be aware that any existing content will be altered and/or removed from the cluster.

	Note that the HUB and TAG environment variables must be set when running tests in the Kubernetes environment.

	## Diagnosing Failures

	### Working Directory

	The test framework will generate additional diagnostic output in its work directory. Typically, this is
	created under the host operating system's temporary folder (which can be overridden using
	the `--istio.test.work_dir` flag). The name of the work dir will be based on the test id that is supplied in
	a tests TestMain method. These files typically contain some of the logging & diagnostic output that components
	spew out as part of test execution

	```console
	$ go test galley/... --istio.test.work_dir /foo
	...

	$ ls /foo
	galley-test-4ef25d910d2746f9b38/

	$ ls /foo/galley-test-4ef25d910d2746f9b38/
	dubbo-system-1537332205890088657.yaml
	...
	```

	### Enabling CI Mode

	When executing in the CI systems, the makefiles use the ```--istio.test.ci``` flag. This flag causes a few changes in
	behavior. Specifically, more verbose logging output will be displayed, some of the timeout values will be more relaxed, and
	additional diagnostic data will be dumped into the working directory at the end of the test execution.

	The flag is not enabled by default to provide a better U/X when running tests locally (i.e. additional logging can clutter
	test output and error dumping can take quite a while). However, if you see a behavior difference between local and CI runs,
	you can enable the flag to make the tests work in a similar fashion.

	### Preserving State (No Cleanup)

	By default, the test framework will cleanup all deployed artifacts after the test run, especially on the Kubernetes
	environment. You can specify the ```--istio.test.nocleanup``` flag to stop the framework from cleaning up the state
	for investigation.

	### Additional Logging

	The framework accepts standard istio logging flags. You can use these flags to enable additional logging for both the
	framework, as well as some of the components that are used in-line in the native environment:

	```console
	$ go test ./... --log_output_level=tf:debug
	```

	The above example will enable debugging logging for the test framework (```tf```) and the MCP protocol stack (```mcp```).

	### Running Tests Under Debugger (GoLand)

	The tests authored in the new test framework can be debugged directly under GoLand using the debugger. If you want to
	pass command-line flags to the test while running under the debugger, you can use the
	[Run/Debug configurations dialog](https://i.stack.imgur.com/C6y0L.png) to specify these flags as program arguments.

	## Reference

	### Command-Line Flags

	The test framework supports the following command-line flags:

	```plain
	-istio.test.work_dir string
	Local working directory for creating logs/temp files. If left empty, os.TempDir() is used.

	-istio.test.ci
	Enable CI Mode. Additional logging and state dumping will be enabled.

	-istio.test.nocleanup
	Do not cleanup resources after test completion

	-istio.test.select string
	Comma separatated list of labels for selecting tests to run (e.g. 'foo,+bar-baz').

	-istio.test.hub string
	Container registry hub to use (default HUB environment variable)

	-istio.test.tag string
	Common Container tag to use when deploying container images (default TAG environment variable)

	-istio.test.pullpolicy string
	Common image pull policy to use when deploying container images

	-istio.test.kube.config string
	A comma-separated list of paths to kube config files for cluster environments. (default ~/.kube/config)

	-istio.test.kube.deploy
	Deploy Istio into the target Kubernetes environment. (default true)

	-istio.test.kube.deployEastWestGW
	Deploy Istio east west gateway into the target Kubernetes environment. (default true)

	-istio.test.kube.systemNamespace string
	The namespace where the Istio components reside in a typical deployment. (default "dubbo-system")

	-istio.test.kube.helm.values string
	Manual overrides for Helm values file. Only valid when deploying Istio.

	-istio.test.kube.helm.iopFile string
	IstioOperator spec file. This can be an absolute path or relative to the repository root. Defaults to "tests/integration/iop-integration-test-defaults.yaml".

	-istio.test.kube.loadbalancer bool
	Used to obtain the right IP address for ingress gateway. This should be false for any environment that doesn't support a LoadBalancer type.

	-istio.test.revision string
	Overwrite the default namespace label (istio-enabled=true) with revision lable (istio.io/rev=XXX). (default is no overwrite)

	-istio.test.skipVM bool
	Skip all the VM related parts in all the tests. (default is "false")
	```

	## Notes

	### Running on a Mac

	* Currently some _native_ tests fail when being run on a Mac with an error like:

	```plain
	unable to locate an Envoy binary
	```

	This is documented in this [PR](https://github.com/istio/istio/issues/13677). Once the Envoy binary is available for the Mac,
	these tests will hopefully succeed.

	* If one uses Docker for Mac for the kubernetes environment be sure to specify the `-istio.test.kube.loadbalancer=false` parameter. The solves an error like:

	```plain
	service ingress is not available yet
	```