Skip to content

Commit 08188b6

Browse files
optimisticbenoptimisticben
committed
Infra repo, add op-replica
1 parent ccd9881 commit 08188b6

File tree

164 files changed

+4020
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

164 files changed

+4020
-0
lines changed

‎infra/op-replica/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# op-replica infra
2+
3+
Deployment examples and resources for running an Optimism replica.
4+
5+
[./envs](./envs) contains working example environmenatal files to configure a replica for different networks.
6+
7+
[./scripts](./scripts/) contains helper scripts to prepare or verify an environment.
8+
9+
[./docker-compose](./docker-compose/) provides working docker-compose example deployments
10+
11+
[./kustomize](./kustomize/) has kubernetes base overlays and network examples

‎infra/op-replica/docker-compose/README.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Running a Network Node
2+
3+
This project lets you set up a local replica of the Optimistic Ethereum chain (either the main one or the Kovan testnet). [New
4+
transactions are submitted either to the sequencer outside of Ethereum or to the Canonical Transaction Chain on
5+
L1](https://research.paradigm.xyz/optimism#data-availability-batches). To submit transactions via a replica, set
6+
`SEQUENCER_CLIENT_HTTP` to a sequencer URL.
7+
8+
## Architecture
9+
10+
You need two components to replicate Optimistic Ethereum:
11+
12+
- `data-transport-layer`, which retrieves and indexes blocks from L1. To access L1 you need an Ethereum Layer 1 provider, such as
13+
  [Infura](https://infura.io/).
14+
15+
- `l2geth`, which provides an Ethereum node where you applications can connect and run API calls.
16+
17+
## Resource requirements
18+
19+
The `data-transport-layer` should run with 1 CPU and 256Mb of memory.
20+
21+
The `l2geth` process should run with 1 or 2 CPUs and between 4 and 8Gb of memory.
22+
23+
With this configuration a synchronization from block 0 to current height is expect to take about 8 hours.
24+
25+
## Software Packages
26+
27+
These packages are required to run the replica:
28+
29+
1. [Docker](https://www.docker.com/)
30+
1. [Docker compose](https://docs.docker.com/compose/install/)
31+
32+
## Configuration
33+
34+
To configure the project, clone this repository and copy either `default-kovan.env` or `default-mainnet.env` file to `.env`.
35+
36+
Review the `SHARED_ENV_PATH` configurations, no changes are required, but depending on the use case, you may need copy these examples to a new directory and make your changes.
37+
38+
!! Update DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT to a valid endpoint !!
39+
40+
### Settings
41+
42+
Change any other settings required for your environment
43+
44+
| Variable                 | Purpose                                                  | Default
45+
| ------------------------ | -------------------------------------------------------- | -----------
46+
| COMPOSE_FILE | The yml files to use with docker-compose | replica.yml:replica-shared.yml
47+
| ETH_NETWORK              | Ethereum Layer1 and Layer2 network (mainnet,kovan)       | kovan (change to `mainnet` for the production network)
48+
| DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT | An endpoint for the L1 network, either kovan or mainnet.
49+
| DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT | Optimistic endpoint, such as https://kovan.optimism.io or https://mainnet.optimism.io
50+
| REPLICA_HEALTHCHECK__ETH_NETWORK_RPC_PROVIDER | The L2 endpoint to check the replica against | (typically the same as the DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT)
51+
| SEQUENCER_CLIENT_HTTP | The L2 sequencer to forward tx to | (typically the same as the DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT)
52+
| SHARED_ENV_PATH          | Path to a directory containing env files                 | [a directory under ./kustomize/replica/envs](https://github.com/optimisticben/op-replica/tree/main/kustomize/replica/envs)
53+
| GCMODE | Whether to run l2geth as an `archive` or `full` node | archive
54+
| L2GETH_IMAGE_TAG         | L2geth version                                           | 0.5.8 (see below)
55+
| DTL_IMAGE_TAG            | Data transport layer version                             | latest (see below)
56+
| HC_IMAGE_TAG | Health check version | latest (see below)
57+
| L2GETH_HTTP_PORT         | Port number for the l2geth RPC endpoint                  | 9991
58+
| L2GETH_WS_PORT         | Port number for the l2geth WebSockets endpoint          | 9992
59+
| DTL_PORT | Port number for the DTL endpoint, for troubleshooting | 7878
60+
| GETH_INIT_SCRIPT | The script name to run when initializing l2geth | A file under kustomize/replica/bases/configmaps/
61+
62+
### Docker Image Versions
63+
64+
We recommend using the latest versions of both docker images. Find them as GitHub tags
65+
[here](https://github.com/ethereum-optimism/optimism/tags) and as published Docker images linked in the badges:
66+
67+
| Package                                                                                                                         | Docker                                                                                                                                                                                                              |
68+
| ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
69+
| [`@eth-optimism/l2geth`](https://github.com/ethereum-optimism/optimism/tree/master/l2geth)                                      | [![Docker Image Version (latest by date)](https://img.shields.io/docker/v/ethereumoptimism/l2geth)](https://hub.docker.com/r/ethereumoptimism/l2geth/tags?page=1&ordering=last_updated)                             |
70+
| [`@eth-optimism/data-transport-layer`](https://github.com/ethereum-optimism/optimism/tree/master/packages/data-transport-layer) | [![Docker Image Version (latest by date)](https://img.shields.io/docker/v/ethereumoptimism/data-transport-layer)](https://hub.docker.com/r/ethereumoptimism/data-transport-layer/tags?page=1&ordering=last_updated) |
71+
| [`@eth-optimism/replica-healthcheck`](https://github.com/ethereum-optimism/optimism/tree/master/packages/replica-healthcheck) | [![Docker Image Version (latest by date)](https://img.shields.io/docker/v/ethereumoptimism/replica-healthcheck)](https://hub.docker.com/r/ethereumoptimism/replica-healthcheck/tags?page=1&ordering=last_updated) |
72+
73+
74+
## Usage
75+
76+
77+
| Action | Command |
78+
| - | - |
79+
| Start the replica (after which you can access it at `http://localhost:L2GETH_HTTP_PORT` | `docker-compose up -d` |
80+
| Get the logs for `l2geth` | `docker-compose logs -f l2geth-replica` |
81+
| Get the logs for `data-transport-layer` | `docker-compose logs -f data-transport-layer` |
82+
| Stop the replica | `docker-compose down` |
83+
84+
85+
## Sync Check
86+
 
87+
There is a sync check container. It fails at startup because at that point the replica is not running yet. It exposes metrics on port 3000, which you could pick up with a Prometheus. You can view its status with this command:
88+
89+
```sh
90+
docker-compose logs -f replica-healthcheck
91+
```
92+
93+
## Registration
94+
95+
[Register here](https://groups.google.com/a/optimism.io/g/optimism-announce) to get announcements, such as notifications of when you're supposed to update your replica.

‎infra/op-replica/docker-compose/contrib/traefik-haproxy/README.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Overview
2+
3+
These files allow you to expose the l2geth RPC/WS ports to the Internet, TLS-encrypted,
4+
via a traefik reverse proxy and Let's Encrypt Certs.
5+
6+
Instructions for the CloudFlare and AWS options are kept up-to-date at https://eth-docker.net/docs/Usage/ReverseProxy
7+
8+
Please open issues for this contribution in the fork at https://github.com/CryptoManufaktur-io/op-replica
9+
10+
## Usage
11+
12+
The `.env` in the main project directory needs to contain traefik-specific variables. Make a backup copy with
13+
`cp .env .env.bak`, then bring in the `default.env` from this directory: `cp contrib/traefik-haproxy/default.env .env`.
14+
Adjust replica variables to match what they were and add either `contrib/traefik-haproxy/traefik-cf.yml` or
15+
`contrib/traefik-haproxy/traefik-aws.yml` to `COMPOSE_FILE`.
16+
17+
Then edit traefik-specific variables for CloudFlare or AWS as per above-linked instructions.
18+
19+
Alternatively, if you have traefik running in its own stack, you can add `contrib/traefik-haproxy/ext-network.yml`
20+
and adjust it for the network traefik runs in.
21+
22+
The haproxy files are examples taken from a docker swarm mode installation. They should work
23+
with minor modifications in k8s via kompose or Portainer.
24+
25+
`optimism-haproxy.cfg` is the configuration file for haproxy, adjust the host and domain names you'll
26+
use in there
27+
28+
`haproxy-docker-sample.yml` is an example docker-compose style deployment in docker swarm.
29+
30+
For example, you may have two replicas called `optimism-a.example.com` and `optimism-b.example.com`.
31+
Both are configured in their `traefik.env` to respond to `optimism-lb.example.com`. Haproxy has
32+
a local alias `optimism-lb.example.com` and will forward traffic to a and b servers, which know to respond
33+
to the `optimism-lb` hostname.
34+
35+
`check-ecsync-optimism.sh` is an external check script to take a server out of rotation when it is not
36+
in sync. It relies on the sequencer-health metrics being available via traefik, and assumes that
37+
servers have `-a`, `-b`, `-c` etc suffixes. The maximum slot distance and name of the healthcheck host
38+
without suffix are configured in the script.

‎infra/op-replica/docker-compose/contrib/traefik-haproxy/check-ecsync-optimism.sh

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/sh
2+
MAX_LAG=2
3+
# Assumes that servers are labeled -a, -b, -c etc and contain their domain name, and that corresponding
4+
# sequencer health hosts exist
5+
HEALTH_HOST=optimismhealth
6+
7+
__domain=$(echo -n $HAPROXY_SERVER_NAME | cut -d '.' -f2-)
8+
__host=$(echo -n $HAPROXY_SERVER_NAME | cut -d '.' -f1)
9+
__dashcount=$(echo -n $__host | grep -o "-" | wc -w)
10+
__suffix=$(echo -n $__host | cut -d '-' -f$(($__dashcount+1)))
11+
12+
__sequencerheight=$(curl -s -m2 -N -L "$HEALTH_HOST-$__suffix.$__domain/metrics" | grep ^replica_health_sequencer_height | cut -d' ' -f2)
13+
__replicaheight=$(curl -s -m2 -N -L "$HEALTH_HOST-$__suffix.$__domain/metrics" | grep ^replica_health_height | cut -d' ' -f2)
14+
if [ -z "$__sequencerheight" -o -z "$__replicaheight" ]; then
15+
exit 1
16+
fi
17+
__distance=$(expr $__sequencerheight - $__replicaheight)
18+
if [ $__distance -le $MAX_LAG ]; then
19+
exit 0
20+
else
21+
exit 1
22+
fi
23+

‎infra/op-replica/docker-compose/contrib/traefik-haproxy/default-kovan.env

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
COMPOSE_FILE=replica.yml:replica-shared.yml:replica-toml.yml
2+
ETH_NETWORK=kovan
3+
DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT=WONT_WORK_UNLESS_YOU_PROVIDE_A_VALID_ETHEREUM_L1_ENDPOINT
4+
DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT=https://kovan.optimism.io
5+
REPLICA_HEALTHCHECK__ETH_NETWORK_RPC_PROVIDER=https://kovan.optimism.io
6+
SEQUENCER_CLIENT_HTTP=https://kovan.optimism.io
7+
SHARED_ENV_PATH=../envs/kovan
8+
GCMODE=archive
9+
L2GETH_IMAGE_TAG=0.5.12
10+
DTL_IMAGE_TAG=
11+
HC_IMAGE_TAG=
12+
L2GETH_HTTP_PORT=9991
13+
L2GETH_WS_PORT=9992
14+
DTL_PORT=7878
15+
GETH_INIT_SCRIPT=check-for-chaindata-berlin.sh
16+
17+
RESTART=unless-stopped
18+
19+
# Secure web proxy - advanced use, please see instructions at https://eth-docker.net/docs/Usage/ReverseProxy
20+
DOMAIN=example.com
21+
ACME_EMAIL=[email protected]
22+
CF_EMAIL=[email protected]
23+
CF_API_TOKEN=SECRETTOKEN
24+
AWS_PROFILE=myprofile
25+
AWS_HOSTED_ZONE_ID=myzoneid
26+
L2GETH_HOST=optimism
27+
L2GETH_LB=optimism-lb
28+
L2GETH_WS_HOST=optimismws
29+
L2GETH_WS_LB=optimismws-lb
30+
L2GETH_HEALTH_HOST=optimismhealth
31+
DDNS_SUBDOMAIN=
32+
DDNS_PROXY=false
33+
34+
TRAEFIK_WEB_PORT=443
35+
TRAEFIK_WEB_HTTP_PORT=80

‎infra/op-replica/docker-compose/contrib/traefik-haproxy/default-mainnet.env

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
COMPOSE_FILE=replica.yml:replica-shared.yml:replica-toml.yml
2+
ETH_NETWORK=mainnet
3+
DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT=WONT_WORK_UNLESS_YOU_PROVIDE_A_VALID_ETHEREUM_L1_ENDPOINT
4+
DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT=https://mainnet.optimism.io
5+
REPLICA_HEALTHCHECK__ETH_NETWORK_RPC_PROVIDER=https://mainnet.optimism.io
6+
SEQUENCER_CLIENT_HTTP=https://mainnet.optimism.io
7+
SHARED_ENV_PATH=../envs/mainnet
8+
GCMODE=archive
9+
L2GETH_IMAGE_TAG=0.5.12
10+
DTL_IMAGE_TAG=
11+
HC_IMAGE_TAG=
12+
L2GETH_HTTP_PORT=9991
13+
L2GETH_WS_PORT=9992
14+
DTL_PORT=7878
15+
GETH_INIT_SCRIPT=check-for-chaindata-berlin.sh
16+
17+
RESTART=unless-stopped
18+
19+
# Secure web proxy - advanced use, please see instructions at https://eth-docker.net/docs/Usage/ReverseProxy
20+
DOMAIN=example.com
21+
ACME_EMAIL=[email protected]
22+
CF_EMAIL=[email protected]
23+
CF_API_TOKEN=SECRETTOKEN
24+
AWS_PROFILE=myprofile
25+
AWS_HOSTED_ZONE_ID=myzoneid
26+
L2GETH_HOST=optimism
27+
L2GETH_LB=optimism-lb
28+
L2GETH_WS_HOST=optimismws
29+
L2GETH_WS_LB=optimismws-lb
30+
L2GETH_HEALTH_HOST=optimismhealth
31+
DDNS_SUBDOMAIN=
32+
DDNS_PROXY=false
33+
34+
TRAEFIK_WEB_PORT=443
35+
TRAEFIK_WEB_HTTP_PORT=80

‎infra/op-replica/docker-compose/contrib/traefik-haproxy/ext-network.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
version: "3.4"
2+
3+
networks:
4+
default:
5+
external:
6+
name: traefik_default
7+
services:
8+
l2geth-replica:
9+
labels:
10+
- traefik.enable=true
11+
- traefik.http.routers.l2geth.service=l2geth
12+
- traefik.http.routers.l2geth.entrypoints=websecure
13+
- traefik.http.routers.l2geth.rule=Host(`${L2GETH_HOST}.${DOMAIN}`)
14+
- traefik.http.routers.l2geth.tls.certresolver=letsencrypt
15+
- traefik.http.routers.l2gethlb.service=l2geth
16+
- traefik.http.routers.l2gethlb.entrypoints=websecure
17+
- traefik.http.routers.l2gethlb.rule=Host(`${L2GETH_LB}.${DOMAIN}`)
18+
- traefik.http.routers.l2gethlb.tls.certresolver=letsencrypt
19+
- traefik.http.services.l2geth.loadbalancer.server.port=8545
20+
- traefik.http.routers.l2gethws.service=l2gethws
21+
- traefik.http.routers.l2gethws.entrypoints=websecure
22+
- traefik.http.routers.l2gethws.rule=Host(`${L2GETH_WS_HOST}.${DOMAIN}`)
23+
- traefik.http.routers.l2gethws.tls.certresolver=letsencrypt
24+
- traefik.http.routers.l2gethwslb.service=l2gethws
25+
- traefik.http.routers.l2gethwslb.entrypoints=websecure
26+
- traefik.http.routers.l2gethwslb.rule=Host(`${L2GETH_WS_LB}.${DOMAIN}`)
27+
- traefik.http.routers.l2gethwslb.tls.certresolver=letsencrypt
28+
- traefik.http.services.l2gethws.loadbalancer.server.port=8546
29+
replica-healthcheck:
30+
labels:
31+
- traefik.enable=true
32+
- traefik.http.routers.l2gethhealth.service=l2gethhealth
33+
- traefik.http.routers.l2gethhealth.entrypoints=websecure
34+
- traefik.http.routers.l2gethhealth.rule=Host(`${L2GETH_HEALTH_HOST}.${DOMAIN}`)
35+
- traefik.http.routers.l2gethhealth.tls.certresolver=letsencrypt
36+
- traefik.http.services.l2gethhealth.loadbalancer.server.port=3000

‎infra/op-replica/docker-compose/contrib/traefik-haproxy/haproxy-docker-sample.yml

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
version: "3.4"
2+
x-logging: &logging
3+
logging:
4+
driver: json-file
5+
options:
6+
max-size: 20m
7+
max-file: "3"
8+
9+
services:
10+
optimism-haproxy:
11+
image: haproxy:latest
12+
user: root
13+
entrypoint: ["/bin/sh", "-c"]
14+
command:
15+
- |
16+
apt-get update
17+
apt-get install --no-install-recommends -y curl jq bc ca-certificates
18+
exec haproxy -f /usr/local/etc/haproxy/haproxy.cfg
19+
networks:
20+
default:
21+
aliases:
22+
- optimismws-lb.example.com
23+
- optimism-lb.example.com
24+
configs:
25+
- source: optimism-haproxy.cfg
26+
target: /usr/local/etc/haproxy/haproxy.cfg
27+
- source: check-ecsync-optimism.sh
28+
target: /var/lib/haproxy/check-ecsync.sh
29+
mode: 0555
30+
deploy:
31+
mode: replicated
32+
replicas: 2
33+
placement:
34+
constraints: ["node.role == worker"]
35+
<<: *logging
36+
37+
configs:
38+
optimism-haproxy.cfg:
39+
external: true
40+
check-ecsync-optimism.sh:
41+
external: true

0 commit comments

Comments
 (0)