mirror of
https://gitlab.com/buildroot.org/buildroot.git
synced 2026-08-01 21:23:51 -09:00
1026abbcf9
Buildroot packaging pretty much assumes that the sources it downloads are publicly available. In general, however, Buildroot is also used to download sources from private repositories. Nowadays, that mostly means from a github or gitlab instance. Although git-over-ssh can be used for that, this poses a problem for CI, because the CI runners integrated with github and gitlab only have access to the repository itself, not to other private repositories. And creating ssh key pairs for CI runners is tricky. Therefore, document how standard tools can be used to make private repositories available both to developers and to CI. There are quite a few alternative approaches possible, but they're more complicated or less generically applicable. Signed-off-by: Arnout Vandecappelle <arnout@rnout.be> [Peter: Fix insteadOf example, capitalize SSH/HTTPS] Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
331 lines
12 KiB
Plaintext
331 lines
12 KiB
Plaintext
// -*- mode:doc; -*-
|
|
// vim: set syntax=asciidoc:
|
|
|
|
=== Tips and tricks
|
|
|
|
[[package-name-variable-relation]]
|
|
==== Package name, config entry name and makefile variable relationship
|
|
|
|
In Buildroot, there is some relationship between:
|
|
|
|
* the _package name_, which is the package directory name (and the
|
|
name of the +*.mk+ file);
|
|
|
|
* the config entry name that is declared in the +Config.in+ file;
|
|
|
|
* the makefile variable prefix.
|
|
|
|
It is mandatory to maintain consistency between these elements,
|
|
using the following rules:
|
|
|
|
* the package directory and the +*.mk+ name are the _package name_
|
|
itself (e.g.: +package/foo-bar_boo/foo-bar_boo.mk+);
|
|
|
|
* the _make_ target name is the _package name_ itself (e.g.:
|
|
+foo-bar_boo+);
|
|
|
|
* the config entry is the upper case _package name_ with `.` and `-`
|
|
characters substituted with `_`, prefixed with +BR2_PACKAGE_+ (e.g.:
|
|
+BR2_PACKAGE_FOO_BAR_BOO+);
|
|
|
|
* the +*.mk+ file variable prefix is the upper case _package name_
|
|
with `.` and `-` characters substituted with `_` (e.g.:
|
|
+FOO_BAR_BOO_VERSION+).
|
|
|
|
[[check-package]]
|
|
==== How to check the coding style
|
|
|
|
Buildroot provides a script in +utils/check-package+ that checks new or
|
|
changed files for coding style. It is not a complete language validator,
|
|
but it catches many common mistakes. It is meant to run in the actual
|
|
files you created or modified, before creating the patch for submission.
|
|
|
|
This script can be used for packages, filesystem makefiles, Config.in
|
|
files, etc. It does not check the files defining the package
|
|
infrastructures and some other files containing similar common code.
|
|
|
|
To use it, run the +check-package+ script, by telling which files you
|
|
created or changed:
|
|
|
|
----
|
|
$ ./utils/check-package package/new-package/*
|
|
----
|
|
|
|
If you have the +utils+ directory in your path you can also run:
|
|
|
|
----
|
|
$ cd package/new-package/
|
|
$ check-package *
|
|
----
|
|
|
|
The tool can also be used for packages in a br2-external:
|
|
|
|
----
|
|
$ check-package -b /path/to/br2-ext-tree/package/my-package/*
|
|
----
|
|
|
|
The +check-package+ script requires you install +shellcheck+ and the
|
|
Python PyPi packages +flake8+ and +python-magic+. The Buildroot code
|
|
base is currently tested against version 0.7.1 of ShellCheck. If you
|
|
use a different version of ShellCheck, you may see additional,
|
|
unfixed, warnings.
|
|
|
|
If you have Docker or Podman you can run +check-package+ without
|
|
installing dependencies:
|
|
|
|
----
|
|
$ ./utils/docker-run ./utils/check-package
|
|
----
|
|
|
|
[[testing-package]]
|
|
==== How to test your package
|
|
|
|
Once you have added your new package, it is important that you test it
|
|
under various conditions: does it build for all architectures? Does it
|
|
build with the different C libraries? Does it need threads, NPTL? And
|
|
so on...
|
|
|
|
Buildroot runs http://autobuild.buildroot.org/[autobuilders] which
|
|
continuously test random configurations. However, these only build the
|
|
`master` branch of the git tree, and your new fancy package is not yet
|
|
there.
|
|
|
|
Buildroot provides a script in +utils/test-pkg+ that uses the same base
|
|
configurations as used by the autobuilders so you can test your package
|
|
in the same conditions.
|
|
|
|
First, create a config snippet that contains all the necessary options
|
|
needed to enable your package, but without any architecture or toolchain
|
|
option. For example, let's create a config snippet that just enables
|
|
+libcurl+, without any TLS backend:
|
|
|
|
----
|
|
$ cat libcurl.config
|
|
BR2_PACKAGE_LIBCURL=y
|
|
----
|
|
|
|
If your package needs more configuration options, you can add them to the
|
|
config snippet. For example, here's how you would test +libcurl+ with
|
|
+openssl+ as a TLS backend and the +curl+ program:
|
|
|
|
----
|
|
$ cat libcurl.config
|
|
BR2_PACKAGE_LIBCURL=y
|
|
BR2_PACKAGE_LIBCURL_CURL=y
|
|
BR2_PACKAGE_OPENSSL=y
|
|
----
|
|
|
|
Then run the +test-pkg+ script, by telling it what config snippet to use
|
|
and what package to test:
|
|
|
|
----
|
|
$ ./utils/test-pkg -c libcurl.config -p libcurl
|
|
----
|
|
|
|
By default, +test-pkg+ will build your package against a subset of the
|
|
toolchains used by the autobuilders, which has been selected by the
|
|
Buildroot developers as being the most useful and representative
|
|
subset. If you want to test all toolchains, pass the +-a+ option. Note
|
|
that in any case, internal toolchains are excluded as they take too
|
|
long to build.
|
|
|
|
The output lists all toolchains that are tested and the corresponding
|
|
result (excerpt, results are fake):
|
|
|
|
----
|
|
$ ./utils/test-pkg -c libcurl.config -p libcurl
|
|
armv5-ctng-linux-gnueabi [ 1/11]: OK
|
|
armv7-ctng-linux-gnueabihf [ 2/11]: OK
|
|
br-aarch64-glibc [ 3/11]: SKIPPED
|
|
br-arcle-hs38 [ 4/11]: SKIPPED
|
|
br-arm-basic [ 5/11]: FAILED
|
|
br-arm-cortex-a9-glibc [ 6/11]: OK
|
|
br-arm-cortex-a9-musl [ 7/11]: FAILED
|
|
br-arm-cortex-m4-full [ 8/11]: OK
|
|
br-arm-full [ 9/11]: OK
|
|
br-arm-full-nothread [10/11]: FAILED
|
|
br-arm-full-static [11/11]: OK
|
|
11 builds, 2 skipped, 2 build failed, 1 legal-info failed
|
|
----
|
|
|
|
The results mean:
|
|
|
|
* `OK`: the build was successful.
|
|
* `SKIPPED`: one or more configuration options listed in the config
|
|
snippet were not present in the final configuration. This is due to
|
|
options having dependencies not satisfied by the toolchain, such as
|
|
for example a package that +depends on BR2_USE_MMU+ with a noMMU
|
|
toolchain. The missing options are reported in +missing.config+ in
|
|
the output build directory (+~/br-test-pkg/TOOLCHAIN_NAME/+ by
|
|
default).
|
|
* `FAILED`: the build failed. Inspect the +logfile+ file in the output
|
|
build directory to see what went wrong:
|
|
** the actual build failed,
|
|
** the legal-info failed,
|
|
** one of the preliminary steps (downloading the config file, applying
|
|
the configuration, running `dirclean` for the package) failed.
|
|
|
|
When there are failures, you can just re-run the script with the same
|
|
options (after you fixed your package); the script will attempt to
|
|
re-build the package specified with +-p+ for all toolchains, without
|
|
the need to re-build all the dependencies of that package.
|
|
|
|
The +test-pkg+ script accepts a few options, for which you can get some
|
|
help by running:
|
|
|
|
----
|
|
$ ./utils/test-pkg -h
|
|
----
|
|
|
|
[[github-download-url]]
|
|
==== How to add a package from GitHub
|
|
|
|
Packages on GitHub often don't have a download area with release tarballs.
|
|
However, it is possible to download tarballs directly from the repository
|
|
on GitHub. As GitHub is known to have changed download mechanisms in the
|
|
past, the 'github' helper function should be used as shown below.
|
|
|
|
----
|
|
# Use a tag or a full commit ID
|
|
FOO_VERSION = 1.0
|
|
FOO_SITE = $(call github,<user>,<package>,v$(FOO_VERSION))
|
|
----
|
|
|
|
.Notes
|
|
- The FOO_VERSION can either be a tag or a commit ID.
|
|
- The tarball name generated by github matches the default one from
|
|
Buildroot (e.g.: +foo-f6fb6654af62045239caed5950bc6c7971965e60.tar.gz+),
|
|
so it is not necessary to specify it in the +.mk+ file.
|
|
- When using a commit ID as version, you should use the full 40 hex characters.
|
|
- When the tag contains a prefix such as +v+ in +v1.0+, then the
|
|
+VERSION+ variable should contain just +1.0+, and the +v+ should be
|
|
added directly in the +SITE+ variable, as illustrated above. This
|
|
ensures that the +VERSION+ variable value can be used to match
|
|
against http://www.release-monitoring.org/[release-monitoring.org]
|
|
results.
|
|
|
|
If the package you wish to add does have a release section on GitHub, the
|
|
maintainer may have uploaded a release tarball, or the release may just point
|
|
to the automatically generated tarball from the git tag. If there is a
|
|
release tarball uploaded by the maintainer, we prefer to use that since it
|
|
may be slightly different (e.g. it contains a configure script so we don't
|
|
need to do AUTORECONF).
|
|
|
|
You can see on the release page if it's an uploaded tarball or a git tag:
|
|
|
|
image::github_hash_mongrel2.png[]
|
|
|
|
- If it looks like the image above then it was uploaded by the
|
|
maintainer and you should use that link (in that example:
|
|
'mongrel2-v1.9.2.tar.bz2') to specify +FOO_SITE+, and not use the
|
|
'github' helper.
|
|
|
|
- On the other hand, if there's is *only* the "Source code" link, then
|
|
it's an automatically generated tarball and you should use the
|
|
'github' helper function.
|
|
|
|
[[gitlab-download-url]]
|
|
==== How to add a package from Gitlab
|
|
|
|
In a similar way to the +github+ macro described in
|
|
xref:github-download-url[], Buildroot also provides the +gitlab+ macro
|
|
to download from Gitlab repositories. It can be used to download
|
|
auto-generated tarballs produced by Gitlab, either for specific tags
|
|
or commits:
|
|
|
|
----
|
|
# Use a tag or a full commit ID
|
|
FOO_VERSION = 1.0
|
|
FOO_SITE = $(call gitlab,<user>,<package>,v$(FOO_VERSION))
|
|
----
|
|
|
|
By default, it will use a +.tar.gz+ tarball, but Gitlab also provides
|
|
+.tar.bz2+ tarballs, so by adding a +<pkg>_SOURCE+ variable, this
|
|
+.tar.bz2+ tarball can be used:
|
|
|
|
----
|
|
# Use a tag or a full commit ID
|
|
FOO_VERSION = 1.0
|
|
FOO_SITE = $(call gitlab,<user>,<package>,v$(FOO_VERSION))
|
|
FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2
|
|
----
|
|
|
|
If there is a specific tarball uploaded by the upstream developers in
|
|
+https://gitlab.com/<project>/releases/+, do not use this macro, but
|
|
rather use directly the link to the tarball.
|
|
|
|
[[accessing-private-repos]]
|
|
==== Accessing a private repository for a package
|
|
|
|
If you want to create a package in a br2-external tree and its source
|
|
is in a private repository (e.g. on gitlab, github, bitbucket, ...),
|
|
you have to write it in a way that it is buildable both by developers
|
|
and in CI. This poses a challenge, because you need to authenticate in
|
|
order to access it.
|
|
|
|
There are several ways you can approach this. The following two are the
|
|
most practical ones.
|
|
|
|
===== Using SSH and +insteadOf+
|
|
|
|
Configure your private packages to use SSH.
|
|
|
|
----
|
|
FOO_SITE = git@githosting.com:/<group>/<package>.git
|
|
----
|
|
|
|
Developers already have an ssh key installed so they can access it
|
|
this way. The only limitation is that if they build in docker, they
|
|
have to make sure the ssh key is accessible from within the container.
|
|
Either mount the SSH directory into the container by passing the
|
|
options +-v ~/.ssh:<homedir>/.ssh+, or load the private key into
|
|
ssh-agent and pass +--mount type=bind,source=$SSH_AUTH_SOCK,target=/ssh-agent
|
|
--env SSH_AUTH_SOCK=/ssh-agent+
|
|
|
|
CI builders typically will not have an SSH key that allows
|
|
access to other repositories. For those, you'll need to generate an
|
|
access token. Then you configure git to replace the SSH access with HTTPS
|
|
access. As a preparation step in CI, run the following command.
|
|
|
|
----
|
|
git config --global url."https://<token>:x-oauth-basic@githosting.com/<group>/".insteadOf "git@githosting.com:/<group>/"
|
|
----
|
|
|
|
The way to use a token for basic authentication differs between different
|
|
git hosting providers, and sometimes between different types of tokens.
|
|
Consult your provider's documentation to find out how to access git over
|
|
HTTPS with a token.
|
|
|
|
===== Use HTTPS and +.netrc+
|
|
|
|
If, for any reason, developers don't have an SSH key already, then it may
|
|
be simpler to use HTTPS authentication. For this, every developer will
|
|
have to generate a token that has (read) access to all relevant repositories.
|
|
Some git hosting providers have a command-line utility that can generate
|
|
such a token, otherwise you'll need to generate it in the web interface. The
|
|
token has a limited lifetime so you'll need to regularly refresh it.
|
|
|
|
To make sure the token is used in the Buildroot build, add it to +~/.netrc+
|
|
|
|
----
|
|
machine githosting.com
|
|
login <username>
|
|
password <token>
|
|
----
|
|
|
|
The +<username>+ and +<password>+ to use are again different for different
|
|
git hosting providers.
|
|
|
|
In CI, generate the +.netrc+ file as a preparation step.
|
|
|
|
Configure your private packages to use HTTPS.
|
|
|
|
----
|
|
FOO_SITE = https://githosting.com/<group>/<package>.git
|
|
----
|
|
|
|
Both wget (https) and git will use +.netrc+ to get login information. This
|
|
approach is potentially somewhat less secure because +.netrc+ cannot be
|
|
password-protected. The advantage is that users and CI use the exact same
|
|
way of providing credentials.
|