Skip to content

Commit aed424f

Browse files
aarltaarlt
committed
Initial docker image creation workflows.
1 parent 6a1ad7e commit aed424f

18 files changed

+201
-18
lines changed

‎.circleci/config.yml

Lines changed: 2 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -33,26 +33,11 @@ defaults:
3333
3434
- run_build: &run_build
3535
name: Build
36-
command: |
37-
set -ex
38-
if [ "$CIRCLE_BRANCH" = release -o -n "$CIRCLE_TAG" -o -n "$FORCE_RELEASE" ]; then echo -n > prerelease.txt; else date -u +"nightly.%Y.%-m.%-d" > prerelease.txt; fi
39-
echo -n "$CIRCLE_SHA1" > commit_hash.txt
40-
mkdir -p build
41-
cd build
42-
[ -n "$COVERAGE" -a "$CIRCLE_BRANCH" != release -a -z "$CIRCLE_TAG" ] && CMAKE_OPTIONS="$CMAKE_OPTIONS -DCOVERAGE=ON"
43-
cmake .. -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE:-Release} $CMAKE_OPTIONS -G "Unix Makefiles"
44-
make -j4
36+
command: scripts/ci/build.sh
4537

4638
- run_build_ossfuzz: &run_build_ossfuzz
4739
name: Build_ossfuzz
48-
command: |
49-
mkdir -p build
50-
cd build
51-
protoc --proto_path=../test/tools/ossfuzz yulProto.proto --cpp_out=../test/tools/ossfuzz
52-
protoc --proto_path=../test/tools/ossfuzz abiV2Proto.proto --cpp_out=../test/tools/ossfuzz
53-
protoc --proto_path=../test/tools/ossfuzz solProto.proto --cpp_out=../test/tools/ossfuzz
54-
cmake .. -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE:-Release} $CMAKE_OPTIONS
55-
make ossfuzz ossfuzz_proto ossfuzz_abiv2 -j4
40+
command: scripts/ci/build_ossfuzz.sh
5641

5742
- run_proofs: &run_proofs
5843
name: Correctness proofs for optimization rules

‎.github/workflows/buildpack-deps.yml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: buildpack-deps
2+
3+
on:
4+
pull_request:
5+
branches: [ develop ]
6+
paths:
7+
- 'scripts/docker/buildpack-deps/Dockerfile.emscripten'
8+
- 'scripts/docker/buildpack-deps/Dockerfile.ubuntu1604.clang.ossfuzz'
9+
- 'scripts/docker/buildpack-deps/Dockerfile.ubuntu1804'
10+
- 'scripts/docker/buildpack-deps/Dockerfile.ubuntu2004.clang'
11+
- 'scripts/docker/buildpack-deps/Dockerfile.ubuntu2004'
12+
13+
jobs:
14+
buildpack-deps:
15+
env:
16+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
17+
DOCKER_REPOSITORY: docker.pkg.github.com/${{ github.repository }}
18+
IMAGE_NAME: buildpack-deps
19+
20+
runs-on: ubuntu-latest
21+
22+
strategy:
23+
fail-fast: false
24+
matrix:
25+
image_variant: [emscripten, ubuntu1604.clang.ossfuzz, ubuntu1804, ubuntu2004.clang, ubuntu2004]
26+
27+
steps:
28+
- uses: actions/checkout@v2
29+
with:
30+
fetch-depth: 0
31+
32+
- name: Upgrade ${{ env.IMAGE_NAME }}-${{ matrix.image_variant }}
33+
run: |
34+
echo "${GITHUB_TOKEN}" | docker login docker.pkg.github.com -u "${GITHUB_ACTOR}" --password-stdin
35+
scripts/ci/docker_upgrade.sh ${{ env.IMAGE_NAME }} ${{ matrix.image_variant }} ${{ env.DOCKER_REPOSITORY }}
36+
docker logout docker.pkg.github.com
37+
38+
- name: comment PR
39+
if: "env.DOCKER_IMAGE"
40+
uses: aarlt/[email protected]
41+
with:
42+
msg: "`${{ env.DOCKER_IMAGE }} ${{ env.DOCKER_REPO_DIGEST }}`."

‎.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ prerelease.txt
3232

3333
# Build directory
3434
build/
35-
build*/
35+
/build*/
3636
emscripten_build/
3737
docs/_build
3838
__pycache__

‎scripts/ci/build.sh

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/usr/bin/env bash
2+
set -ex
3+
4+
ROOTDIR="$(dirname "$0")/../.."
5+
cd "${ROOTDIR}"
6+
7+
# shellcheck disable=SC2166
8+
if [ "$CIRCLE_BRANCH" = release -o -n "$CIRCLE_TAG" -o -n "$FORCE_RELEASE" ]; then echo -n >prerelease.txt; else date -u +"nightly.%Y.%-m.%-d" >prerelease.txt; fi
9+
if [ -n "$CIRCLE_SHA1" ]
10+
then
11+
echo -n "$CIRCLE_SHA1" >commit_hash.txt
12+
fi
13+
14+
mkdir -p build
15+
cd build
16+
17+
# shellcheck disable=SC2166
18+
[ -n "$COVERAGE" -a "$CIRCLE_BRANCH" != release -a -z "$CIRCLE_TAG" ] && CMAKE_OPTIONS="$CMAKE_OPTIONS -DCOVERAGE=ON"
19+
20+
# shellcheck disable=SC2086
21+
cmake .. -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE:-Release}" $CMAKE_OPTIONS -G "Unix Makefiles"
22+
23+
make -j 4

‎scripts/ci/build_ossfuzz.sh

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
ROOTDIR="$(dirname "$0")/../.."
5+
BUILDDIR="${ROOTDIR}/build"
6+
7+
mkdir -p "${BUILDDIR}"
8+
cd "${BUILDDIR}"
9+
10+
protoc --proto_path=../test/tools/ossfuzz yulProto.proto --cpp_out=../test/tools/ossfuzz
11+
protoc --proto_path=../test/tools/ossfuzz abiV2Proto.proto --cpp_out=../test/tools/ossfuzz
12+
protoc --proto_path=../test/tools/ossfuzz solProto.proto --cpp_out=../test/tools/ossfuzz
13+
cmake .. -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE:-Release}" -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/libfuzzer.cmake
14+
15+
make ossfuzz ossfuzz_proto ossfuzz_abiv2 -j 4

‎scripts/ci/buildpack-deps_test_emscripten.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../scripts/travis-emscripten/build_emscripten.sh

‎scripts/ci/buildpack-deps_test_ubuntu1604.clang.ossfuzz.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
build_ossfuzz.sh

‎scripts/ci/buildpack-deps_test_ubuntu1804.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
build.sh

‎scripts/ci/buildpack-deps_test_ubuntu2004.clang.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
build.sh

‎scripts/ci/buildpack-deps_test_ubuntu2004.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
build.sh

‎scripts/ci/docker_upgrade.sh

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
function error() {
5+
echo >&2 "ERROR: ${1} Aborting." && false
6+
}
7+
8+
function warning() {
9+
echo >&2 "WARNING: ${1}"
10+
}
11+
12+
[[ $# == 3 ]] || error "Expected exactly 3 parameters: '${0} <IMAGE_NAME> <IMAGE_VARIANT> <DOCKER_REPOSITORY>'."
13+
14+
IMAGE_NAME="${1}"
15+
IMAGE_VARIANT="${2}"
16+
DOCKER_REPOSITORY="${3}"
17+
DOCKERFILE="scripts/docker/${IMAGE_NAME}/Dockerfile.${IMAGE_VARIANT}"
18+
19+
echo "-- check_dockerfile_was_changed"
20+
21+
# exit, if the dockerfile was not changed.
22+
if git diff --quiet origin/develop HEAD -- "${DOCKERFILE}"; then
23+
echo "${DOCKERFILE} was not changed. Nothing to do."
24+
exit 0
25+
fi
26+
27+
echo "-- check_version"
28+
29+
PREV_VERSION=$(git diff origin/develop HEAD -- "${DOCKERFILE}" | grep -e '^\s*-LABEL\s\+version=".*"\s*$' | awk -F'"' '{ print $2 }')
30+
NEXT_VERSION=$(git diff origin/develop HEAD -- "${DOCKERFILE}" | grep -e '^\s*+LABEL\s\+version=".*"\s*$' | awk -F'"' '{ print $2 }')
31+
32+
[[ $NEXT_VERSION != "" ]] || error "No version label defined in Dockerfile. You may need to add 'LABEL version' in '${DOCKERFILE}'."
33+
34+
[[ $PREV_VERSION != "" ]] || {
35+
warning "no previous version found. Will set \$PREV_VERSION = 0."
36+
PREV_VERSION=0
37+
}
38+
39+
if [[ $((PREV_VERSION + 1)) != $((NEXT_VERSION)) ]]; then
40+
error "Version label in Dockerfile was not incremented. You may need to change 'LABEL version' in '${DOCKERFILE}'."
41+
fi
42+
43+
echo "-- build_docker"
44+
45+
# This is a workaround: we run `docker build` twice to prevent the `layer does not exist` problem.
46+
# See https://github.com/moby/moby/issues/37965.
47+
docker build "scripts/docker/${IMAGE_NAME}" --file "scripts/docker/${IMAGE_NAME}/Dockerfile.${IMAGE_VARIANT}" --tag "${IMAGE_NAME}" ||
48+
docker build "scripts/docker/${IMAGE_NAME}" --file "scripts/docker/${IMAGE_NAME}/Dockerfile.${IMAGE_VARIANT}" --tag "${IMAGE_NAME}"
49+
50+
echo "-- test_docker @ '${PWD}'"
51+
52+
docker run --rm --volume "${PWD}:/root/project" "${IMAGE_NAME}" "/root/project/scripts/ci/${IMAGE_NAME}_test_${IMAGE_VARIANT}.sh"
53+
54+
echo "-- push_docker"
55+
56+
VERSION=$(docker inspect --format='{{.Config.Labels.version}}' "${IMAGE_NAME}")
57+
DOCKER_IMAGE_ID="${DOCKER_REPOSITORY}/${IMAGE_NAME}-${IMAGE_VARIANT}"
58+
59+
docker tag "${IMAGE_NAME}" "${DOCKER_IMAGE_ID}:${VERSION}"
60+
docker push "${DOCKER_IMAGE_ID}:${VERSION}"
61+
62+
REPO_DIGEST=$(docker inspect --format='{{.RepoDigests}}' "${DOCKER_IMAGE_ID}:${VERSION}")
63+
64+
docker tag "${IMAGE_NAME}" "${DOCKER_IMAGE_ID}:latest"
65+
docker push "${DOCKER_IMAGE_ID}:latest"
66+
67+
echo "::set-env name=DOCKER_IMAGE::${DOCKER_IMAGE_ID}:${VERSION}"
68+
echo "::set-env name=DOCKER_REPO_DIGEST::${REPO_DIGEST}"

‎.circleci/docker/Dockerfile.emscripten renamed to ‎scripts/docker/buildpack-deps/Dockerfile.emscripten

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
# make version=1.39.15 build
3030
#
3131
FROM emscripten/emsdk:1.39.15 AS base
32+
LABEL version="0"
3233

3334
ADD emscripten.jam /usr/src
3435
RUN set -ex; \
@@ -63,3 +64,4 @@ RUN set -ex; \
6364
cxxflags="-s DISABLE_EXCEPTION_CATCHING=0 -Wno-unused-local-typedef -Wno-variadic-macros -Wno-c99-extensions -Wno-all" \
6465
--prefix=/emsdk/emscripten/sdk/system install; \
6566
rm -r /usr/src/boost_1_73_0
67+

‎.circleci/docker/Dockerfile.ubuntu1604.clang.ossfuzz renamed to ‎scripts/docker/buildpack-deps/Dockerfile.ubuntu1604.clang.ossfuzz

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
# (c) 2016-2019 solidity contributors.
2323
#------------------------------------------------------------------------------
2424
FROM gcr.io/oss-fuzz-base/base-clang as base
25+
LABEL version="0"
2526

2627
ARG DEBIAN_FRONTEND=noninteractive
2728

@@ -99,3 +100,4 @@ FROM base
99100
COPY --from=libraries /usr/lib /usr/lib
100101
COPY --from=libraries /usr/bin /usr/bin
101102
COPY --from=libraries /usr/include /usr/include
103+

‎.circleci/docker/Dockerfile.ubuntu1804 renamed to ‎scripts/docker/buildpack-deps/Dockerfile.ubuntu1804

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
# (c) 2016-2019 solidity contributors.
2323
#------------------------------------------------------------------------------
2424
FROM buildpack-deps:bionic AS base
25+
LABEL version="0"
2526

2627
ARG DEBIAN_FRONTEND=noninteractive
2728

@@ -91,3 +92,4 @@ FROM base
9192
COPY --from=libraries /usr/lib /usr/lib
9293
COPY --from=libraries /usr/bin /usr/bin
9394
COPY --from=libraries /usr/include /usr/include
95+

‎.circleci/docker/Dockerfile.ubuntu2004 renamed to ‎scripts/docker/buildpack-deps/Dockerfile.ubuntu2004

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
# (c) 2016-2019 solidity contributors.
2323
#------------------------------------------------------------------------------
2424
FROM buildpack-deps:focal AS base
25+
LABEL version="0"
2526

2627
ARG DEBIAN_FRONTEND=noninteractive
2728

@@ -60,3 +61,4 @@ FROM base
6061
COPY --from=libraries /usr/lib /usr/lib
6162
COPY --from=libraries /usr/bin /usr/bin
6263
COPY --from=libraries /usr/include /usr/include
64+

‎.circleci/docker/Dockerfile.ubuntu2004.clang renamed to ‎scripts/docker/buildpack-deps/Dockerfile.ubuntu2004.clang

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
# (c) 2016-2019 solidity contributors.
2323
#------------------------------------------------------------------------------
2424
FROM buildpack-deps:focal AS base
25+
LABEL version="0"
2526

2627
ARG DEBIAN_FRONTEND=noninteractive
2728

@@ -62,3 +63,4 @@ FROM base
6263
COPY --from=libraries /usr/lib /usr/lib
6364
COPY --from=libraries /usr/bin /usr/bin
6465
COPY --from=libraries /usr/include /usr/include
66+

‎scripts/docker/buildpack-deps/README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# buildpack-deps docker images
2+
3+
The `buildpack-deps` docker images are used to compile and test solidity within our CI.
4+
5+
## GitHub Workflow
6+
7+
The creation of the images are triggered by a single workflow, defined in `.github/workflows/buildpack-deps.yml`.
8+
For each resulting `buildpack-deps` docker image a strategy is defined in the workflow file - the image variant.
9+
The workflow gets triggered, if any Dockerfile defined in `scripts/docker/buildpack-deps/Dockerfile.*` were changed
10+
within the PR.
11+
12+
### Versioning
13+
14+
The version of the docker images can be defined within the Dockerfile with `LABEL version`. A new docker image
15+
will only be created and pushed, if the new version is incremented by `1` compared with the version of the Dockerfile
16+
located in `develop`.
17+
18+
### Build, Test & Push
19+
20+
Note that the whole workflow - including all defined strategies (image variants) - will be triggered,
21+
even if only a single Dockerfile was change. The full workflow will only gets executed, if the corresponding
22+
Dockerfile was changed. The execution of workflows of unchanged Dockerfiles will not continue and just return success.
23+
See `scripts/ci/docker_upgrade.sh`.
24+
25+
If the version check was successful, the docker image will be built using the Dockerfile located in
26+
`scripts/docker/buildpack-deps/Dockerfile.*`.
27+
28+
The resulting docker image will be tested by executing
29+
the corresponding `scripts/ci/buildpack-deps_test_*` scripts. These scripts are normally symlinked to `scripts/ci/build.sh`,
30+
except for the `buildpack-deps-ubuntu1604.clang.ossfuzz` docker image, that is symlinked to `scripts/ci/build_ossfuzz.sh`.
31+
These scripts `scripts/ci/build.sh` and `scripts/ci/build_ossfuzz.sh` are also used by CircleCI, see `.circleci/config.yml`.
32+
33+
If the tests passed successfully, the docker image will get tagged by the version defined within the corresponding `Dockerfile`.
34+
Finally, a comment will be added to the PR that contains the full repository, version and repository digest
35+
of the freshly created docker image.

‎.circleci/docker/emscripten.jam renamed to ‎scripts/docker/buildpack-deps/emscripten.jam

File renamed without changes.

0 commit comments

Comments
 (0)