Skip to content

Commit 602d9f8

Browse files
TD5TD5
committed
gh: Allow manual running of the full test suite and erlfuzz
This change introduces two new features: - Comment "/run-fuzzer" on a pull request to trigger a bot which will run erlfuzz and report the results by replying on the comment thread - Add the "full-build-and-check" tag to a pull request to force-override CI to always run tests for all applications The two differ in how they are expressed because I have worked on the basis that: - Running the fuzzer is expensive, so we probably want to try to limit that to when we know we really need it and explicitly invoke it. - Running tests for all apps is sometimes needed because the existing mechanism for selecting apps to test is unsound: If you modify stdlib, apps that depend on it won't be analysed, but stdlib itself will. As such, using a tag means that all subsequent test runs for that pull request will return the full signal. This change also factors out the initial build of OTP in CI into `base-build.yaml` so that it can be reused elsewhere, for example, when running erlfuzz. The mechanism of using comments to execute manual CI jobs could later be extended to add other optional or expensive jobs (e.g. eqWAlizer) so that there is an official means of running them, without requiring that they be part of the usual, automatic CI suite.
1 parent 209f718 commit 602d9f8

File tree

5 files changed

+365
-111
lines changed

5 files changed

+365
-111
lines changed

‎.github/scripts/run-fuzzer.sh‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
#!/bin/bash
2+
3+
FUZZER_DIR=${1}
4+
OTP_DIR=${2}
5+
OUT=${3}
6+
N=${4}
7+
8+
9+
set -exo pipefail
10+
11+
# To update erlfuzz, update this to a later commit hash, branch or tag
12+
ERLFUZZ_VERSION=c9364609b8944c71c8e6184abd8793477772862b
13+
14+
# Install Rust non-interactively
15+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain nightly
16+
PATH=$HOME/.cargo/bin:$PATH
17+
sudo apt-get install parallel
18+
19+
mkdir "../${FUZZER_DIR}"
20+
cd "../${FUZZER_DIR}"
21+
git clone https://github.com/WhatsApp/erlfuzz.git
22+
cd erlfuzz
23+
git checkout ${ERLFUZZ_VERSION}
24+
25+
# Be permissive when building erlfuzz - we don't need it to we warning-free for
26+
# erlfuzz to be useful
27+
RUSTFLAGS=-Awarnings $HOME/.cargo/bin/cargo build --release
28+
29+
mkdir -p out-erl
30+
mkdir -p out-erlc-opts
31+
mkdir -p out-jit
32+
mkdir -p interesting-erl
33+
mkdir -p interesting-erlc-opts
34+
mkdir -p interesting-jit
35+
mkdir -p minimized-erl
36+
mkdir -p minimized-erlc-opts
37+
mkdir -p minimized-jit
38+
39+
echo "Fuzzing erl"
40+
echo "Generating ${N} test cases"
41+
42+
seq ${N} | parallel --line-buffer "./target/release/erlfuzz fuzz-and-reduce -c ./run_erl_once.sh --tmp-directory out-erl --interesting-directory interesting-erl --minimized-directory minimized-erl test{}"
43+
seq ${N} | parallel --line-buffer "./target/release/erlfuzz --deterministic --wrapper printing fuzz-and-reduce -c ./verify_erlc_opts.sh --tmp-directory out-erlc-opts --interesting-directory interesting-erlc-opts --minimized-directory minimized-erlc-opts test{}"
44+
seq ${N} | parallel --line-buffer "./target/release/erlfuzz --deterministic --wrapper printing fuzz-and-reduce -c ./verify_erl_jit.sh --tmp-directory out-jit --interesting-directory interesting-jit --minimized-directory minimized-jit test{}"
45+
46+
echo "Fuzzing complete. Collating results."
47+
48+
mv minimized-erl "${OUT}"/minimized-erl
49+
mv minimized-erlc-opts "${OUT}"/minimized-erlc-opts
50+
mv minimized-jit "${OUT}"/minimized-jit
51+
52+
echo "Results written to: ${OUT}/minimized-erl, "${OUT}"/minimized-erlc-opts and "${OUT}"/minimized-jit"

‎.github/workflows/base-build.yaml‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
##
2+
## This workflow factors out the core erlang/OTP build used to as the basis of
3+
## various later CI jobs
4+
##
5+
name: Base build
6+
7+
on:
8+
workflow_call:
9+
inputs:
10+
base-branch:
11+
required: true
12+
type: string
13+
outputs:
14+
changes:
15+
description: "What apps were changed"
16+
value: ${{ jobs.pack.outputs.changes }}
17+
all:
18+
description: "All apps that exist"
19+
value: ${{ jobs.pack.outputs.all }}
20+
21+
env:
22+
BASE_BRANCH: ${{ inputs.base-branch }}
23+
24+
jobs:
25+
26+
build:
27+
name: Build Erlang/OTP (64-bit) for subsequent stages
28+
runs-on: ubuntu-latest
29+
outputs:
30+
changes: ${{ steps.changes-override.outputs.changes }}
31+
all: ${{ steps.apps.outputs.all }}
32+
steps:
33+
- uses: actions/checkout@v3
34+
- name: Get applications
35+
id: apps
36+
run: |
37+
.github/scripts/path-filters.sh > .github/scripts/path-filters.yaml
38+
ALL_APPS=$(grep '^[a-z_]*:' .github/scripts/path-filters.yaml | sed 's/:.*$//')
39+
ALL_APPS=$(jq -n --arg inarr "${ALL_APPS}" '$inarr | split("\n")' | tr '\n' ' ')
40+
echo "all=${ALL_APPS}" >> $GITHUB_OUTPUT
41+
- uses: dorny/paths-filter@v2
42+
id: changes
43+
with:
44+
filters: .github/scripts/path-filters.yaml
45+
- if: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'full-build-and-check') }}
46+
id: should-enable-full-build-and-check
47+
env:
48+
ALL_APPS: ${{ steps.apps.outputs.all }}
49+
run: |
50+
echo "enable-full-build-and-check=1" >> $GITHUB_ENV
51+
- name: Override changes
52+
id: changes-override
53+
env:
54+
ALL_APPS: ${{ steps.apps.outputs.all }}
55+
CHANGED_APPS: ${{ steps.changes.outputs.changes }}
56+
run: |
57+
if [[ enable-full-build-and-check ]]; then
58+
echo "changes=${ALL_APPS}" >> "$GITHUB_OUTPUT"
59+
else
60+
echo "changes=${CHANGED_APPS}" >> "$GITHUB_OUTPUT"
61+
fi
62+
- name: Create initial pre-release tar
63+
run: .github/scripts/init-pre-release.sh otp_archive.tar.gz
64+
- name: Upload source tar archive
65+
uses: actions/upload-artifact@v3
66+
with:
67+
name: otp_git_archive
68+
path: otp_archive.tar.gz
69+
- name: Docker login
70+
uses: docker/login-action@v2
71+
with:
72+
registry: ghcr.io
73+
username: ${{ github.repository_owner }}
74+
password: ${{ secrets.GITHUB_TOKEN }}
75+
- name: Cache BASE image
76+
uses: actions/cache@v3
77+
with:
78+
path: otp_docker_base.tar
79+
key: ${{ runner.os }}-${{ hashFiles('.github/dockerfiles/Dockerfile.ubuntu-base', '.github/scripts/build-base-image.sh') }}
80+
- name: Build BASE image
81+
run: .github/scripts/build-base-image.sh "${BASE_BRANCH}" 64-bit
82+
- name: Cache pre-built tar archives
83+
id: pre-built-cache
84+
uses: actions/cache@v3
85+
with:
86+
path: |
87+
otp_src.tar.gz
88+
otp_cache.tar.gz
89+
key: prebuilt-${{ github.ref_name }}-${{ github.sha }}
90+
restore-keys: |
91+
prebuilt-${{ github.base_ref }}-${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }}
92+
- uses: dorny/paths-filter@v2
93+
id: cache
94+
with:
95+
filters: |
96+
no-cache:
97+
- '.github/**'
98+
deleted:
99+
- deleted: '**'
100+
bootstrap:
101+
- 'bootstrap/**'
102+
configure:
103+
- '**.ac'
104+
- '**.in'
105+
list-files: shell
106+
- name: Restore from cache
107+
env:
108+
NO_CACHE: ${{ steps.cache.outputs.no-cache }}
109+
BOOTSTRAP: ${{ steps.cache.outputs.bootstrap }}
110+
CONFIGURE: ${{ steps.cache.outputs.configure }}
111+
run: |
112+
.github/scripts/restore-from-prebuilt.sh "`pwd`" \
113+
"`pwd`/.github/otp.tar.gz" \
114+
"`pwd`/otp_archive.tar.gz" \
115+
'${{ github.event_name }}' \
116+
'${{ steps.cache.outputs.deleted_files }}' \
117+
'${{ steps.changes.outputs.changes }}'
118+
- name: Upload restored cache
119+
uses: actions/upload-artifact@v3
120+
if: runner.debug == 1
121+
with:
122+
name: restored-cache
123+
path: .github/otp.tar.gz
124+
- name: Build image
125+
run: |
126+
docker build --tag otp \
127+
--build-arg MAKEFLAGS=-j$(($(nproc) + 2)) \
128+
--file ".github/dockerfiles/Dockerfile.64-bit" \
129+
.github/
130+
- name: Build pre-built tar archives
131+
run: |
132+
docker run -v $PWD:/github --entrypoint "" otp \
133+
scripts/build-otp-tar -o /github/otp_clean_src.tar.gz /github/otp_src.tar.gz -b /buildroot/otp/ /github/otp_src.tar.gz
134+
- name: Build cache
135+
run: |
136+
if [ -f otp_cache.tar.gz ]; then
137+
gunzip otp_cache.tar.gz
138+
else
139+
docker run -v $PWD:/github --entrypoint "" otp \
140+
bash -c 'cp ../otp_cache.tar /github/'
141+
fi
142+
docker run -v $PWD:/github --entrypoint "" otp \
143+
bash -c 'set -x; C_APPS=$(ls -d ./lib/*/c_src); find Makefile ./make ./erts ./bin/`erts/autoconf/config.guess` ./lib/erl_interface ./lib/jinterface ${C_APPS} `echo "${C_APPS}" | sed -e 's:c_src$:priv:'` -type f -newer README.md \! -name "*.beam" \! -path "*/doc/*" | xargs tar --transform "s:^./:otp/:" -uvf /github/otp_cache.tar'
144+
gzip otp_cache.tar
145+
- name: Upload pre-built tar archive
146+
uses: actions/upload-artifact@v3
147+
with:
148+
name: otp_prebuilt
149+
path: |
150+
otp_src.tar.gz
151+
otp_cache.tar.gz

‎.github/workflows/fuzz.yaml‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
name: Run fuzz tester
2+
3+
on: # Allow manual runs via special PR comment '/run-fuzzer'
4+
issue_comment:
5+
types: [created, edited]
6+
# Allow manual runs via GitHub web interface, CLI etc.
7+
workflow_dispatch:
8+
9+
# Allow responding to user by writing a comment on the PR
10+
permissions:
11+
pull-requests: write
12+
contents: read
13+
issues: read
14+
packages: read
15+
16+
env:
17+
BASE_BRANCH: ${{ github.base_ref }}
18+
19+
jobs:
20+
21+
starting-fuzzer:
22+
name: Notify user of fuzzing
23+
if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/run-fuzzer') }}
24+
runs-on: ubuntu-latest
25+
steps:
26+
- uses: actions/[email protected]
27+
with:
28+
github-token: ${{secrets.GITHUB_TOKEN}}
29+
script: |
30+
github.rest.issues.createComment({
31+
issue_number: context.issue.number,
32+
owner: context.repo.owner,
33+
repo: context.repo.repo,
34+
body: "🤖 Running `erlfuzz` as requested by **${{ github.actor }}** ⏱️\n\nYou can follow the progress of the job using the following link:\n👉 ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} 👈"
35+
})
36+
37+
build-otp:
38+
if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/run-fuzzer')}}
39+
uses: ./.github/workflows/base-build.yaml
40+
permissions:
41+
packages: read
42+
with:
43+
base-branch: ${{ github.ref_name }}
44+
45+
fuzz:
46+
name: Fuzz test Erlang/OTP
47+
if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/run-fuzzer')}}
48+
runs-on: ubuntu-latest
49+
needs: build-otp
50+
steps:
51+
- uses: actions/checkout@v3
52+
- name: Cache BASE image
53+
uses: actions/cache@v3
54+
with:
55+
path: otp_docker_base.tar
56+
key: ${{ runner.os }}-${{ hashFiles('.github/dockerfiles/Dockerfile.ubuntu-base', '.github/scripts/build-base-image.sh') }}
57+
- name: Docker login
58+
uses: docker/login-action@v2
59+
with:
60+
registry: docker.pkg.github.com
61+
username: ${{ github.repository_owner }}
62+
password: ${{ secrets.GITHUB_TOKEN }}
63+
- name: Build BASE image
64+
run: .github/scripts/build-base-image.sh "${BASE_BRANCH}" 64-bit
65+
- name: Cache pre-built tar archives
66+
uses: actions/cache@v3
67+
with:
68+
path: |
69+
otp_src.tar.gz
70+
otp_cache.tar.gz
71+
key: prebuilt-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}
72+
restore-keys: |
73+
prebuilt-${{ github.ref_name }}-${{ github.sha }}
74+
- name: Build image
75+
run: |
76+
.github/scripts/restore-from-prebuilt.sh `pwd` .github/otp.tar.gz
77+
rm -f otp_{src,cache}.tar.gz
78+
docker build --tag otp \
79+
--build-arg MAKEFLAGS=-j$(($(nproc) + 2)) \
80+
--file ".github/dockerfiles/Dockerfile.64-bit" \
81+
.github/
82+
- name: Run erlfuzz
83+
id: run-erlfuzz
84+
run: |
85+
docker run -v $PWD:/github --entrypoint "" otp /bin/bash -c "/github/.github/scripts/run-fuzzer.sh fuzzer/ /github/ /github/ 1000"
86+
- name: Collect erlfuzz results
87+
if: always()
88+
run: |
89+
tar czf minimized-erl-results.tar.gz minimized-erl
90+
tar czf minimized-erlc-opts-results.tar.gz minimized-erlc-opts
91+
tar czf minimized-jit-results.tar.gz minimized-jit
92+
- name: Upload erl fuzzing results
93+
uses: actions/upload-artifact@v3
94+
if: always()
95+
with:
96+
name: minimized-erl-results
97+
path: minimized-erl-results.tar.gz
98+
- name: Upload erlc opts fuzzing results
99+
uses: actions/upload-artifact@v3
100+
if: always()
101+
with:
102+
name: minimized-erlc-opts-results
103+
path: minimized-erlc-opts-results.tar.gz
104+
- name: Upload jit fuzzing results
105+
uses: actions/upload-artifact@v3
106+
if: always()
107+
with:
108+
name: minimized-jit-results
109+
path: minimized-jit-results.tar.gz
110+
111+
report:
112+
name: Report results of fuzzing
113+
if: ${{ always() }}
114+
runs-on: ubuntu-latest
115+
needs: fuzz
116+
steps:
117+
- name: Fetch results
118+
uses: actions/download-artifact@v3
119+
- name: Analyse results
120+
run: |
121+
echo "::group::Decompressing results"
122+
tar -xvzf minimized-erl-results/minimized-erl-results.tar.gz
123+
tar -xvzf minimized-erlc-opts-results/minimized-erlc-opts-results.tar.gz
124+
tar -xvzf minimized-jit-results/minimized-jit-results.tar.gz
125+
echo "::endgroup::"
126+
ls -laH .
127+
ls -laH minimized-erl
128+
echo "::group::Counting the number of issues found"
129+
echo "NUM_ERL_ISSUES_FOUND=$(ls -1q minimized-erl/*.erl | wc -l)" >> "${GITHUB_ENV}"
130+
echo "NUM_ERLC_OPTS_ISSUES_FOUND=$(ls -1q minimized-erlc-opts/*.erl | wc -l)" >> "${GITHUB_ENV}"
131+
echo "NUM_JIT_ISSUES_FOUND=$(ls -1q minimized-jit/*.erl | wc -l)" >> "${GITHUB_ENV}"
132+
echo "::endgroup::"
133+
- name: Comment on pull request with results
134+
env:
135+
NUM_ERL_ISSUES_FOUND: ${{ env.NUM_ERL_ISSUES_FOUND }}
136+
NUM_ERLC_OPTS_ISSUES_FOUND: ${{ env.NUM_ERLC_OPTS_ISSUES_FOUND }}
137+
NUM_JIT_ISSUES_FOUND: ${{ env.NUM_JIT_ISSUES_FOUND }}
138+
uses: actions/[email protected]
139+
with:
140+
github-token: ${{secrets.GITHUB_TOKEN}}
141+
script: |
142+
const { NUM_ERL_ISSUES_FOUND, NUM_ERLC_OPTS_ISSUES_FOUND, NUM_JIT_ISSUES_FOUND } = process.env
143+
github.rest.issues.createComment({
144+
issue_number: context.issue.number,
145+
owner: context.repo.owner,
146+
repo: context.repo.repo,
147+
body:
148+
"🤖 A run of `erlfuzz` requested by **${{ github.actor }}** is complete ✔️\n\n" +
149+
"Number of issues with erl found: " + NUM_ERL_ISSUES_FOUND + "\n" +
150+
"Number of issues with erlc opts found: " + NUM_ERLC_OPTS_ISSUES_FOUND + "\n" +
151+
"Number of issues with the jit found: " + NUM_JIT_ISSUES_FOUND + "\n\n" +
152+
"You can see the results of the job using the following link, and looking\n" +
153+
"for the archived files containing the affected source files in the artifacts section at the bottom:\n" +
154+
"👉 ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} 👈"
155+
})
156+

0 commit comments

Comments
 (0)