Skip to content

Commit e10e919

Browse files
oaken-sourceoaken-source
committed
finished README
1 parent fabd77c commit e10e919

File tree

1 file changed

+145
-70
lines changed

1 file changed

+145
-70
lines changed

‎README‎

Lines changed: 145 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -2,28 +2,49 @@
22
parabola-riscv64-bootstrap
33
==========================
44

5+
“Don't cry because it's over, smile because it happened.”
6+
― Dr. Seuss
7+
58
1. Introduction
69
---------------
710

811
This project is an attempt to bootstrap a self-contained parabola
912
GNU/Linux-libre system for the riscv64 architecture. The scripts are created
10-
with the goal to be as generic as possible, to make future porting effort
11-
easier. Feel free to adapt them to target any architecture you want by
12-
modifying the variables in create.sh
13+
with the goal to be as architectur agnostic as possible, to make future porting
14+
efforts easier.
1315

14-
The build process is split into several stages, the rationale of which is
15-
outlined in section 2 below.
16+
The build process is split into four stages, the rationale of which is outlined
17+
in section 2 below.
1618

1719
To initiate a complete build of all stages, run:
1820
$> sudo ./create
1921

20-
To discard fragments from an earlier run and start from a clean slate, run:
21-
$> sudo rm -rf build && sudo ./create
22+
The builds can be configured to keep going if the build of a single package
23+
fails, by creating the file `build/.KEEP_GOING`. Otherwise, the build will stop
24+
once an error is encountered. This is useful for getting as much work done as
25+
possible unattended, but will make debugging harder in the later stages,
26+
because temporary build fragments and filesystem trees will be overwritten by
27+
the next package.
28+
29+
The complete console output of a package build process can be found in the
30+
corresponding .MAKEPKGLOG file in the build directory of that package.
31+
32+
1.1. System Requirements
33+
------------------------
2234

23-
You might have to unmount the directories mounted by the stages as noted in the
24-
sections below.
35+
The scripts require, among probably other things, to be running on a fairly
36+
POSIX-conforming GNU/Linux system, and in particulary need the following tools
37+
to be present and functional:
2538

26-
1.1. A note to the reader
39+
* decently up-to-date GNU build toolchain (gcc, glibc, binutils)
40+
* most of the things in base-devel
41+
* pacman, makepkg
42+
43+
I have tried to make the script smart enough to check for required bits and
44+
pieces where needed, and to report when anything is missing ahead of time, but
45+
some requirements may be missing.
46+
47+
1.2. A note to the reader
2748
-------------------------
2849

2950
The scripts assume to be running on a parabola GNU/Linux-libre system, and may
@@ -35,14 +56,15 @@ pay close attention to any output, and be prepared to fix and modify patches
3556
and scripts.
3657

3758
Also, if you found this project useful, and want to chat about anything, you
38-
can email me at <[email protected]>.
59+
can email me at <[email protected]>, or find me as <oaken-source> in
60+
#parabola and others on irc.freenode.org.
3961

40-
1.2. Current state of the project
62+
1.3. Current state of the project
4163
---------------------------------
4264

43-
The stage3 native base-devel makepkg chroot is finished, and stage4 is primed
44-
to begin with native compilation of a first batch of release packages in the
45-
chroot.
65+
All four stages are complete and this repository is now closed. A pointer where
66+
to find future development efforts for the parabola RISC-V port will be added
67+
here in due time.
4668

4769
2. Build Stages
4870
---------------
@@ -51,20 +73,38 @@ The following subsections outline the reasoning behind the separate bootstrap
5173
stages. More details about *how* things are done may be gathered from reading
5274
the inline comments of the respective scripts.
5375

54-
2.1. Stage 1
55-
------------
76+
From Stage 2 onwards, the scripts use DEPTREEs to determine what packages need
77+
to be built. these files are located in the build/stageX/ directories and can
78+
give an insight into what packages are going to be built next and what
79+
dependencies are still missing.
80+
81+
Since risc-v is a fairly new ISA, some packages were packaged with config.sub
82+
and config.guess files that are too old to recognize the target triplet. This
83+
requires config.sub and config.guess files to be refreshed with newer versions
84+
from upstream. This is done automatically for stage 2 and onwards, if
85+
REGEN_CONFIG_FRAGMENTS is set to yes (the default) in create.sh or the
86+
environment.
87+
88+
Packages with the `any' architecture are reused from upstream arch for this
89+
bootstrap, since they should work for any architecture and do not need to be
90+
re-(or cross-)compiled.
5691

57-
The first stage creates a cross-compile toolchain for the target triplet
58-
defined in $CHOST, consisting of binutils, linux-libre-api-headers, gcc and
59-
glibc. This toolchain has been pre-packaged for riscv64-unknown-linux-gnu on
60-
parabola GNU/Linux-libre and may be installed by running:
92+
Additionally, all checkdepends are ignored, and the check() phase of the builds
93+
is skipped for sanity reasons.
6194

62-
$> pacman -S riscv64-unknown-linux-gnu-gcc
95+
Note that the stages use upstream arch and parabola PKGBUILDs and attempt to
96+
apply custom patches to resolve unbuildable or missing dependencies, and fix
97+
risc-v specific build issues. Look for these patches in src/stageX/patches and
98+
be prepared to fix and adapt them for future porting efforts.
99+
100+
2.1. Stage 1
101+
------------
63102

64-
In any case, the toolchain will be bootstrapped by the stage1.sh script, unless
65-
it is already installed. The scripts will check for $CHOST-ar and $CHOST-gcc in
66-
$PATH to determine whether binutils and gcc are installed, and will then
67-
proceed to look for the following files in $CHOST-gcc's sysroot:
103+
The first stage creates and installs a cross-compile toolchain for the target
104+
triplet defined in $CHOST, consisting of binutils, linux-libre-api-headers, gcc
105+
and glibc. The scripts will check for $CHOST-ar and $CHOST-gcc in $PATH to
106+
determine whether binutils and gcc are installed, and will then proceed to look
107+
for the following files in $CHOST-gcc's sysroot:
68108

69109
$sysroot/lib/libc.so.6 # for $CHOST-glibc
70110
$sysroot/include/linux/kernel.h # for $CHOST-linux-libre-api-headers
@@ -90,58 +130,93 @@ src/stage1/toolchain-pkgbuilds/*.
90130
2.2. Stage 2
91131
------------
92132

93-
Stage 2 uses the toolchain created in Stage 1 to cross-compile the packages of
94-
the base-devel group of packages plus transitive runtime dependencies, in order
95-
to bootstrap a functional cross-makepkg librechroot.
133+
Stage 2 uses the toolchain created in Stage 1 to cross-compile a subset of the
134+
packages of the base-devel group plus transitive runtime dependencies.
96135

97136
The script creates an empty skeleton chroot, into which the cross-compiled
98-
packages are installed, and creates a sane makepkg.conf and a patched
99-
makepkg.sh to work in the prepared chroot root directory. To make the sysroot
100-
of the compiler available to builds in the chroot and vice-versa, the /usr
101-
directory of the chroot is mounted into the sysroot. At the end of Stage 2,
102-
this directory is unmounted automatically.
103-
104-
The transitive dependency tree of the base-devel package group is slightly
105-
modified in a way that resolves any cyclic dependencies, and preserves a
106-
somewhat sane build-order of the packages. Some packages that are too painful
107-
to cross-compile are skipped entirely.
108-
109-
To build the packages, the dependency tree is traversed and packages are
110-
rebuilt using upstream PKGBUILDs and custom patches to cross-compile the
111-
packages for the configured target architecture. Packages with the `any'
112-
architecture are simply reused for this stage, since they should work for any
113-
architecture and do not need to be cross-compiled.
114-
115-
As a final note, since the upstream PKGBUILDS change frequently, and the
116-
patches are unlikely to be maintained once the initial bootstrap is done and
117-
stable, they will probably cease to function in the near future. Exercise
118-
caution.
137+
packages are going to be installed, and creates a sane makepkg.conf and a
138+
patched makepkg.sh to work in the prepared chroot root directory. To make the
139+
sysroot of the compiler available to builds in the chroot and vice-versa, the
140+
/usr directory of the chroot is mounted into the sysroot. At the end of Stage
141+
2, or in case of an error, this directory is unmounted automatically.
142+
143+
To build the packages, the DEPTREE is traversed and packages are cross-compiled
144+
using upstream PKGBUILDs and custom patches, and the compiled packages are
145+
installed into the chroot immediately.
146+
147+
Note that this process is a bit fragile and dependent on arbitrary
148+
particularities of the host system, and thus might fail for subtle reasons,
149+
like missing, or superfluous build-time installed packages on the host.
150+
Exercise caution and common sense.
119151

120152
2.3. Stage 3
121153
------------
122154

123155
Stage 3 uses the cross-compiled makepkg chroot created in Stage 2 to natively
124-
compile the base-devel group of packages again, but without using a
125-
cross-compiler and without the need for mandatory package patches. Many
126-
packages `just work' when compiling them using the upstream PKGBUILDs, but some
127-
still need to be altered, and build time dependencies need to be removed when
128-
they are too painful to compile.
129-
130-
This stage requires more packages to complete, since the host system can not
131-
provide things like git for vcs sources. Everything is taken from the makepkg
132-
chroot.
133-
134-
The process is similar to stage2, a clean librechroot is created using the
135-
packages built in Stage 2, a modified libremakepkg.sh is created to inclued a
136-
update hook for config.sub and config.guess scripts, which are unfortunately
137-
too old for most packages to detect the architecture correctly, a dependency
138-
tree is made, and then traversed.
139-
140-
The patches in this stage are fewer and apply less pressure to the packages,
141-
so, while computationally more expensive, this stage is probably a bit easier
142-
to maintain and adapt.
156+
recompile the base-devel group of packages. This stage requires to build more
157+
packages, since a reduced set of make-time dependencies need to be present in
158+
the makepkg chroot, as well as runtime dependencies. Additionally, running the
159+
cross-compiled native compiler instead of the cross compiler takes longer. As a
160+
result, stage 3 is expected to takes much longer than stage 2.
161+
162+
However, since now the process is isolated from the host systems installed
163+
packages, since everything is built cleanly in a chroot, the process is much
164+
more stable and less prone to hard to diagnose problems with the host system.
165+
166+
The scripts create a clean librechroot from the cross-compiled packages
167+
produced in stage 2. A modified libremakepkg script is created to perform
168+
config fragment regeneration and to skip the check() phase, and then packages
169+
are built in order.
170+
171+
Since no cross-compilation is needed from stage 3 onwards, fewer packages need
172+
patching, and the patches are typically smaller and apply less pressure. As a
173+
consequence, the patches probably need less maintenance in the future, and less
174+
work to adapt to a different architecture.
175+
176+
Note that the cross-compiled packages from stage 3 can be a bit derpy at times,
177+
hence the stage 3 build scripts prioritize building bash and make natively, to
178+
avoid some very weird known issues down the line.
143179

144180
2.4. Stage 4
145181
------------
146182

147-
tbd.
183+
Stage 4 does a final recompile of the packages of the base-devel group, similar
184+
to stage 3, with the difference that more make-time dependencies are enabled,
185+
and the packages of the base group are added to the DEPTREE.
186+
187+
Stage 4 relies entirely on the packages natively compiled in stage 3, and no
188+
cross-compiled packages are present in the build chroot at any time. This
189+
results in reliable builds and reproducible build failures. However, since the
190+
number of packages to be built in stage 4 is close to 750, expect the builds to
191+
take a long time (days / weeks, not including work required to fix broken
192+
patches and builds).
193+
194+
The result of stage 4 is a repository of packages that should allow to
195+
bootstrap and boot a risc-v virtual machine, with the packages required to
196+
build the entirety of the arch / parabola package repository. At this point, I
197+
consider the bootstrap process done.
198+
199+
3. Final Words
200+
==============
201+
202+
I would like to thank the awesome abaumann from the archlinux32 project for
203+
pointers on how to bootstrap a PKGBUILD based system for a new architecture,
204+
and for his work on the bootstrap32 project, which helped a lot in getting this
205+
project started:
206+
207+
https://github.com/archlinux32/bootstrap32
208+
209+
Further, I would like to thank the amazing people in #riscv and #fedora-riscv
210+
on irc.freenode.org, especially rwmjones, davidlt and sorear for all the help
211+
getting packages to build and work on risc-v, and for providing the source
212+
packages of all fedora-riscv packages, including all patches and build recipes,
213+
which have been of immense help:
214+
215+
https://fedorapeople.org/groups/risc-v/SRPMS/
216+
217+
Lastly, it was refreshing to see yor names pop up over and over again on the
218+
risc-v ports, patches and pull requests out there, and it made me realize how
219+
much work it really is to get a port like this off the ground. I could not have
220+
made it this far without your work.
221+
222+
― Andreas Grapentin, 2018-04-18

0 commit comments

Comments
 (0)