Skip to content

Commit 4052b27

Browse files
technicallytytechnicallyty
committed
adding e2e back
1 parent a9170dc commit 4052b27

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+12834
-0
lines changed

‎e2e/Makefile

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
DOCKER := $(shell which docker)
2+
TEST_CONTAINERS=$(shell docker ps --filter "label=ibc-test" -a -q)
3+
4+
cleanup-ibc-test-containers:
5+
for id in $(TEST_CONTAINERS) ; do \
6+
$(DOCKER) stop $$id ; \
7+
$(DOCKER) rm $$id ; \
8+
done
9+
10+
init:
11+
./scripts/init.sh
12+
13+
e2e-test: init cleanup-ibc-test-containers
14+
./scripts/run-e2e.sh $(test) $(entrypoint)
15+
16+
e2e-suite: init cleanup-ibc-test-containers
17+
RUN_SUITE="true" ./scripts/run-e2e.sh "" $(entrypoint)
18+
19+
.PHONY: cleanup-ibc-test-containers e2e-test init

‎e2e/README.md

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
# Table of Contents
2+
3+
1. [Running Tests](#running-tests)
4+
2. [Adding a new test](#adding-a-new-test)
5+
3. [Test design](#test-design)
6+
- a. [interchaintest](#interchaintest)
7+
- b. [CI configuration](#ci-configuration)
8+
4. [Building images](#building-and-pushing-images)
9+
5. [Compatibility Tests](#compatibility-tests)
10+
- a. [Running Compatibility Tests](#running-compatibility-tests)
11+
- b. [How Compatibility Tests Work](#how-compatibility-tests-work)
12+
6. [Troubleshooting](#troubleshooting)
13+
7. [Importable Workflow](#importable-workflow)
14+
8. [Future Improvements](#future-improvements)
15+
16+
# Running tests
17+
18+
Tests can be run using a Makefile target under the e2e directory. `e2e/Makefile`
19+
20+
The tests can be configured using a configuration file or environment variables.
21+
22+
See the [minimal example](./sample.config.yaml) or [extended example](./sample.config.extended.yaml) to get started. The default location the tests look is `~/.ibc-go-e2e-config.yaml`
23+
But this can be specified directly using the `E2E_CONFIG_PATH` environment variable.
24+
25+
It is possible to maintain multiple configuration files for tests. This can be useful when wanting to run the tests
26+
using different images, relayers etc.
27+
28+
By creating an `./e2e/dev-configs` directory, and placing any number of configurations there. You will be prompted to choose
29+
which configuration to use when running tests.
30+
31+
> Note: this requires fzf to be installed to support the interactive selection of configuration files.
32+
33+
There are several environment variables that alter the behaviour of the make target which will override any
34+
options specified in your config file. These are primarily used for CI and are not required for local development.
35+
36+
See the extended sample config to understand all the available fields and their purposes.
37+
38+
> Note: when running tests locally, **no images are pushed** to the `ghcr.io/cosmos/ibc-go-simd` registry.
39+
> The images which are used only exist locally only.
40+
41+
These environment variables allow us to run tests with arbitrary versions (from branches or releases) of simd and the go / hermes relayer.
42+
43+
Every time changes are pushed to a branch or to `main`, a new `simd` image is built and
44+
pushed [here](https://github.com/orgs/cosmos/packages?repo_name=ibc-go).
45+
46+
On PRs, E2E tests will only run once the PR is marked as ready for review. This is to prevent unnecessary test runs on PRs that are still in progress.
47+
48+
> If you need the E2E tests to run, you can either run them locally, or you can mark the PR as R4R and then convert it back to a draft PR.
49+
50+
## Adding a new test
51+
52+
All tests should go under the [e2e](https://github.com/cosmos/ibc-go/tree/main/e2e) directory. When adding a new test, either add a new test function
53+
to an existing test suite ***in the same file***, or create a new test suite in a new file and add test functions there.
54+
New test files should follow the convention of `module_name_test.go`.
55+
56+
After creating a new test file, be sure to add a build constraint that ensures this file will **not** be included in the package to be built when
57+
running tests locally via `make test`. For an example of this, see any of the existing test files.
58+
59+
New test suites should be composed of `testsuite.E2ETestSuite`. This type has lots of useful helper functionality that will
60+
be quite common in most tests.
61+
62+
Override the default `SetupSuite` function with the number of chains required for the suite. Example:
63+
64+
```go
65+
// SetupSuite sets up chains for the current test suite
66+
func (s *ConnectionTestSuite) SetupSuite() {
67+
s.SetupChains(context.TODO(), 2, nil) // This suite requires at most two chains.
68+
}
69+
```
70+
71+
> Note: see [here](#how-tests-are-run) for details about these requirements.
72+
73+
### Example of running a single test
74+
75+
> NOTE: environment variables can be set to override one or more config file variables, but the config file can still
76+
> be used to set defaults.
77+
78+
```sh
79+
80+
make e2e-test entrypoint=TestInterchainAccountsTestSuite test=TestMsgSubmitTx_SuccessfulTransfer
81+
```
82+
83+
If `jq` is installed, you only need to specify the `test`.
84+
85+
If `fzf` is also installed, you only need to run `make e2e-test` and you will be prompted with interactive test
86+
selection.
87+
88+
```sh
89+
make e2e-test test=TestMsgSubmitTx_SuccessfulTransfer
90+
```
91+
92+
> Note: sometimes it can be useful to make changes to [interchaintest](https://github.com/strangelove-ventures/interchaintest)
93+
> when running tests locally. In order to do this, add the following line to
94+
> e2e/go.mod
95+
96+
`replace github.com/strangelove-ventures/interchaintest => ../../interchaintest`
97+
98+
Or point it to any local checkout you have.
99+
100+
### Example of running a full testsuite
101+
102+
> NOTE: not all tests may support full parallel runs due to possible chain wide modifications such as params / gov
103+
> proposals / chain restarts. See [When to Use t.Parallel()](#when-to-use-tparallel) for more information.
104+
105+
```sh
106+
make e2e-suite entrypoint=TestTransferTestSuite
107+
```
108+
109+
Similar to running a single test, if `jq` and `fzf` are installed you can run `make e2e-suite` and be prompted
110+
to interactively select a test suite to run.
111+
112+
### Running tests outside the context of the Makefile
113+
114+
In order to run tests outside the context of the Makefile (e.g. from an IDE)
115+
116+
The default location for a config file will be `~/.ibc-go-e2e-config.yaml` but this can be overridden by setting the
117+
`E2E_CONFIG_PATH` environment variable.
118+
119+
This should be set to the path of a valid config file you want to use, setting this env will depend on the IDE being used.
120+
121+
## Test design
122+
123+
### interchaintest
124+
125+
These E2E tests use the [interchaintest framework](https://github.com/strangelove-ventures/interchaintest). This framework creates chains and relayers in containers and allows for arbitrary commands to be executed in the chain containers,
126+
as well as allowing us to broadcast arbitrary messages which are signed on behalf of a user created in the test.
127+
128+
### Test Suites
129+
130+
In order for tests to be run in parallel, we create the chains in `SetupSuite`, and each test is in charge of
131+
creating clients/connections/channels for itself.
132+
133+
This is explicitly not being done in `SetupTest` to enable maximum control and flexibility over the channel creation
134+
params. e.g. some tests may not want a channel created initially, and may want more flexibility over the channel creation itself.
135+
136+
### When to use t.Parallel()
137+
138+
tests should **not** be run in parallel when:
139+
140+
- the test is modifying chain wide state such as modifying params via a gov proposal.
141+
- the test needs to perform a chain restart.
142+
- the test must make assertions which may not be deterministic due to other tests. (e.g. the TotalEscrowForDenom may be modified between tests)
143+
144+
### CI Configuration
145+
146+
There are two main github actions for standard e2e tests.
147+
148+
[e2e.yaml](https://github.com/cosmos/ibc-go/blob/main/.github/workflows/e2e.yaml) which runs when collaborators create branches.
149+
150+
In `e2e.yaml`, the `simd` image is built and pushed to [a registry](https://github.com/orgs/cosmos/packages?repo_name=ibc-go) and every test
151+
that is run uses the image that was built.
152+
153+
In `e2e-fork.yaml`, images are not pushed to this registry, but instead remain local to the host runner.
154+
155+
## How Tests Are Run
156+
157+
The tests use the `matrix` feature of Github Actions. The matrix is
158+
dynamically generated using [this tool](https://github.com/cosmos/ibc-go/blob/main/cmd/build_test_matrix/main.go).
159+
160+
> Note: there is currently a limitation that all tests belonging to a test suite must be in the same file.
161+
> In order to support test functions spread in different files, we would either need to manually maintain a matrix
162+
> or update the script to account for this. The script assumes there is a single test suite per test file to avoid an
163+
> overly complex generation process.
164+
165+
Which looks under the `e2e` directory, and creates a task for each test suite function.
166+
167+
This tool can be run locally to see which tests will be run in CI.
168+
169+
```sh
170+
go run cmd/build_test_matrix/main.go | jq
171+
```
172+
173+
This string is used to generate a test matrix in the Github Action that runs the E2E tests.
174+
175+
All tests will be run on different hosts when running `make e2e-test` but `make e2e-suite` will run multiple tests
176+
in parallel on a shared host.
177+
178+
In a CI environment variables are passed to the test runner to configure test parameters, while locally using
179+
environment variables is supported, but it is often more convenient to use configuration files.
180+
181+
## Building and pushing images
182+
183+
If we ever need to manually build and push an image, we can do so with the [Build Simd Image](../.github/workflows/build-simd-image-from-tag.yml) GitHub workflow.
184+
185+
This can be triggered manually from the UI by navigating to
186+
187+
`Actions` -> `Build Simd Image` -> `Run Workflow`
188+
189+
And providing the git tag.
190+
191+
> There are similar workflows for other simapps in the repo.
192+
193+
## Compatibility Tests
194+
195+
### Running Compatibility Tests
196+
197+
To trigger the compatibility tests for a release branch, you can trigger these manually from the Github UI.
198+
199+
This will build an image from the tip of the release branch and run all tests specified in the corresponding
200+
E2E test annotations.
201+
202+
Navigate to `Actions` -> `Compatibility E2E` -> `Run Workflow` -> `release/v8.0.x`
203+
204+
> Note: this will spawn a large number of runners, and should only be used when there is a release candidate and
205+
> and so should not be run regularly. We can rely on the regular E2Es on PRs for the most part.
206+
207+
### How Compatibility Tests Work
208+
209+
The compatibility tests are executed in [this workflow](../.github/workflows/e2e-compatibility.yaml). This workflow
210+
will build an image for a specified release candidate based on the release branch as an input. And run the corresponding
211+
jobs which are maintained under the `.github/compatibility-test-matrices` directory.
212+
213+
> At the moment these are manually maintained, but in the future we may be able to generate these matrices dynamically. See the [future improvements](#future-improvements) section for more details.
214+
215+
See [this example](https://github.com/cosmos/ibc-go/actions/runs/11645461969) to what the output of a compatibility test run looks like.
216+
217+
## Troubleshooting
218+
219+
- On Mac, after running a lot of tests, it can happen that containers start failing. To fix this, you can try clearing existing containers and restarting the docker daemon.
220+
221+
This generally manifests itself as relayer or simd containers timing out during setup stages of the test. This doesn't happen in CI.
222+
223+
```bash
224+
# delete all images
225+
docker system prune -af
226+
```
227+
228+
This issue doesn't seem to occur on other operating systems.
229+
230+
### Accessing Logs
231+
232+
- When a test fails in GitHub. The logs of the test will be uploaded (viewable in the summary page of the workflow). Note: There
233+
may be some discrepancy in the logs collected and the output of interchaintest. The containers may run for a some
234+
time after the logs are collected, resulting in the displayed logs to differ slightly.
235+
236+
### Prerequisites
237+
238+
- In order to run this workflow, a docker container is required with tags for the versions you want to test.
239+
240+
- If you are running an upgrade, Have an upgrade handler in the chain binary which is being upgraded to.
241+
242+
> It's worth noting that all github repositories come with a built-in docker registry that makes it convenient to build and push images to.

‎e2e/ci-e2e-config.yaml

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# This file contains configuration for running e2e tests.
2+
# Many of these fields can be overridden with environment variables.
3+
# All fields that support this have the corresponding environment variable name in a comment beside the field.
4+
5+
# | Environment Variable | Description | Default Value |
6+
# |----------------------|-------------------------------------------|-------------------------------|
7+
# | CHAIN_IMAGE | The image that will be used for the chain | ghcr.io/cosmos/ibc-go-simd |
8+
# | CHAIN_A_TAG | The tag used for chain A | N/A (must be set) |
9+
# | CHAIN_B_TAG | The tag used for chain B | N/A (must be set) |
10+
# | CHAIN_C_TAG | The tag used for chain C | N/A (optional; fallback to A) |
11+
# | CHAIN_D_TAG | The tag used for chain D | N/A (optional; fallback to A) |
12+
# | CHAIN_BINARY | The binary used in the container | simd |
13+
# | RELAYER_TAG | The tag used for the relayer | 1.13.1 |
14+
# | RELAYER_ID | The type of relayer to use (rly/hermes) | hermes |
15+
16+
# see sample.config.yaml for a bare minimum configuration example.
17+
# set env E2E_CONFIG_PATH to point to this file to use it.
18+
---
19+
chains:
20+
# the entry at index 0 corresponds to CHAIN_A
21+
- chainId: chainA-1
22+
numValidators: 4
23+
numFullNodes: 1
24+
image: ghcr.io/cosmos/ibc-go-simd # override with CHAIN_IMAGE
25+
binary: simd # override with CHAIN_BINARY
26+
27+
# the entry at index 1 corresponds to CHAIN_B
28+
- chainId: chainB-1
29+
numValidators: 4
30+
numFullNodes: 1
31+
image: ghcr.io/cosmos/ibc-go-simd # override with CHAIN_IMAGE
32+
binary: simd # override with CHAIN_BINARY
33+
34+
# the entry at index 2 corresponds to CHAIN_C
35+
- chainId: chainC-1
36+
numValidators: 4
37+
numFullNodes: 1
38+
image: ghcr.io/cosmos/ibc-go-simd # override with CHAIN_IMAGE
39+
binary: simd # override with CHAIN_BINARY
40+
41+
# the entry at index 3 corresponds to CHAIN_D
42+
- chainId: chainD-1
43+
numValidators: 4
44+
numFullNodes: 1
45+
image: ghcr.io/cosmos/ibc-go-simd # override with CHAIN_IMAGE
46+
binary: simd # override with CHAIN_BINARY
47+
48+
# activeRelayer must match the id of a relayer specified in the relayers list below.
49+
activeRelayer: hermes # override with RELAYER_ID
50+
51+
relayers:
52+
- id: hermes
53+
image: ghcr.io/informalsystems/hermes
54+
tag: "1.13.1"
55+
- id: rly
56+
image: ghcr.io/cosmos/relayer
57+
tag: "latest"
58+
59+
cometbft:
60+
logLevel: info
61+
62+
debug:
63+
# setting this value to true will force log collection even if the test passes.
64+
dumpLogs: false
65+
# settings this value to true will keep the containers running after the test finishes.
66+
keepContainers: true
67+
68+
upgradePlanName: ""
69+
upgrades:
70+
- planName: "v7"
71+
tag: "v7.0.0"
72+
- planName: "v7.1"
73+
tag: "v7.1.0"
74+
- planName: "v8"
75+
tag: "v8.0.0"
76+
- planName: "v8.1"
77+
tag: "v8.1.0"
78+
- planName: "v10"
79+
tag: "main" # TODO: Update with correct tag as soon as we have a release
80+
- planName: "ibcwasm-v8"
81+
tag: "v8.0.0-e2e-upgrade"

0 commit comments

Comments
 (0)