Commit Graph

389 Commits

Author SHA1 Message Date
Yann E. MORIN
3aacb38098 utils/brmake: drop use of unbuffer
When it was introduced in afdb545b28 (tools: new tool to filter the
output of make), brmake made use of unbuffer, so as to try and ensure
that no output would get lost between make printing it, and the logger
loop time-stamping it and storing it in the logfile, in case the user
would interrup the build (SIGINT, ^C)

However, unbuffer is missing in our reference build image (we could have
added it, but we missed the occasion to do so every time we updated the
reference build image).

Furthermore, the rationale for using unbuffer, although reasonable, is
not so practical: indeed, when the user hits Crtl-C, this is inherently
asynchronous, and they can't expect everything to be entirely cleanly
terminated, especially buffering. Using unbuffer is thus slightly
superfluous.

The timestamps will be slightly off because of buffering, but a build
generates so much output that this won't be a problem in practice.

Drop use of unbuffer.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-18 20:45:00 +01:00
Romain Naour
ef80b710f2 utils/config: fix shellcheck errors
SC2086 is now reported for missing Double quote around
$FN since shellcheck 0.9.0:

In utils/config line 175:
                if grep -q "# ${BR2_PREFIX}$ARG is not set" $FN ; then
                                                            ^-^ SC2086 (info): Double quote to prevent globbing and word splitting.

In utils/config line 178:
                        if V="$(grep "^${BR2_PREFIX}$ARG=" $FN)"; then
                                                           ^-^ SC2086 (info): Double quote to prevent globbing and word splitting.

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2025-02-09 15:39:46 +01:00
Romain Naour
4f78d362aa utils/checkpackagelib: fix shellcheck tests after update to shellcheck 0.9.0
shellcheck tests needs to be updated after shellcheck 0.9.0 update due
to changes in errors/warning reporting:

  utils/checkpackagelib/test_tool.py::test_Shellcheck[missing shebang-empty.sh--expected0] FAILED [ 98%]
  utils/checkpackagelib/test_tool.py::test_Shellcheck[2 warnings-unused.sh-unused=""-expected3] FAILED [ 99%]
  utils/checkpackagelib/test_tool.py::test_Shellcheck[tab-tab.sh-\t#!/bin/sh-expected4] FAILED [100%]

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2025-02-09 12:53:35 +01:00
Julien Olivain
fa98239074 utils/check-symbols: allow BR2_BROKEN to be unused
Commit [1] introduced the BR2_BROKEN hidden symbol. The intent of
this symbol is to temporarily mark a configuration known to be broken.
This commit was added to fix an undefined usage of the symbol, in a
previous commit [2].

With this somewhat special symbol, it is also perfectly normal to
have no use of it.

Running the command "utils/check-symbols" in a state there is no usage
of the BR2_BROKEN, the script fails with output:

    Config.in:15: BR2_BROKEN defined but not referenced

This commit adds an exception for this symbol, allowing it to be
unused in the whole Buildroot tree.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/8925433382

[1] 2ddc5808cd
[2] 11a8cdd2bb

Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
2025-02-05 11:01:05 +01:00
Brandon Maier
80d26f091f utils/check-package: support finding files from patches
For the `b4` tool to support check-package, check-package must support
reading patch files from stdin.

It would be complicated to make check-package actually run checks on
patch files. So instead we search the patch files to figure out what
files are modified in the repo, then run check-package on the modified
files directly.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
[Arnout: rename variable "files" to "files_to_check"]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-04 18:44:35 +01:00
Yann E. MORIN
717f1fdaeb utils/get-developers: add option to report Cc: lines
It is very common to use the output of get-developers to add cc: lines
in the commit log.

Add an option so that get-developers reports Cc: lines ready to be
pasted in a commit log. That new option behaves similarly to the
existing -e option: it only affects the output when parsing a patch.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:20 +01:00
Yann E. MORIN
3177ecd260 utils/get-developers: sort reported developers
The list of reported developers is not ordered: that may leave the
impression (when receiving a patch) that a Cc is more important than
another, by virtue of being earlier in the list.

Also, the ordering changes on every call.

Report the developers in an alphabetically order, so that there is no
confusion anymore, and so the ordering is reproducible across calls.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:17 +01:00
Yann E. MORIN
d10d22221f utils/get-developers: read patch from stdin when it's not a tty
It is very common that get-developers be used with its stdin a pipe from
git-show:
    git show |./utils-get-developers -

In this case, the '-' is superfluous: we can very easily deduce that the
user wants to read stdin as the patch.

So, if no other action was requested, and stdin is not a tty, use it as
the source of the patch, and thus '-' is then no longer required.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:13 +01:00
Yann E. MORIN
35f381b93e utils/get-developers: use parser.error() rather than canned print+return
parser.error() reports a nice error message, that also displays a short
reminder of the available options.

Adapt the test-suite accordingly: previously, the error string was an
exact string in the stdout list, while it now is a substring in one of
the strings in stderr. The exit code changes, too.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:10 +01:00
Yann E. MORIN
cdcb3f56e8 utils/get-developers: don't offload parse_args()
Offloading parser.parse_args() to a helper function does not bring much,
if at all; it even is restrictive: indeed, we can't use parser.error()
to report errors and thus have to resort to a canned print+return
sequence...

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:06 +01:00
Thomas Petazzoni
f97b67be74 utils/genrandconfig: more igh-ethercat tweaks
utils/genrandconfig already had some logic to disable building
igh-ethercat drivers since they are highly kernel version
specific. Commit
eec3a1108a ("package/igh-ethercat: add
configuration options") recently added a number of additional config
options to build more igh-ethercat drivers, which causes a number of
autobuilder failures. This commit therefore updates genrandconfig to
also not build those new drivers.

Fixes:

  https://autobuild.buildroot.org/results/9489b68967a35aa398e608d663824b2d2be9cae5/ (stmmac driver)
  https://autobuild.buildroot.org/results/ec2c24f16b365a3e7de8964fc8d73cb570c74abe/ (igc driver)
  https://autobuild.buildroot.org/results/65e17f8e03ab14dd9f39386f2176629677b8ba8c/ (genet driver)

Cc: Jannik Becher <becher.jannik@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2024-12-17 00:13:08 +01:00
James Hilliard
0205f4c2b8 package/pkg-python.mk: add poetry setup type
As we have many python packages using the poetry(poetry-core) build
system we should add a setup type for it so that we don't have to
manually specify the host-python-poetry-core dependency.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-11-04 23:19:55 +01:00
Thomas Petazzoni
b07d21ab98 utils/genrandconfig: handle BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y
When BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y,
BR2_LINUX_KERNEL_IMAGE_TARGET_NAME is supposed to be non-empty. But in
the context of genraconfig, we don't know to what value
BR2_LINUX_KERNEL_IMAGE_TARGET_NAME can be set, so let's avoid cases
where BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y. By dropping this option,
kconfig will revert back to the default image format for the selected
architecture.

Fixes:

  http://autobuild.buildroot.net/results/1d104a051c83bb31e98565369a2ec7badfa21eca/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-27 19:48:06 +01:00
Brandon Maier
6ffcdb52e8 utils/check-package: add inline script requirements
The check-package tool requires some PyPi package to be installed before
it can run. This is typically done by manually installing them into the
user's global Python environment or setting up a virtual environment,
then manually installing each dependency.

Python recently defined a format for managing script dependencies as
inline metadata[1]. This can be used with the `uv` tool to run a Python
script and automatically install the minimum required version of Python
and PyPi dependencies.

With this change, it's now possible to run check-package with

  uv run -s ./utils/check-package

Note that, because check-package does not have the '.py' file extension
we must specify the `-s` or `--script` argument. That argument was added
very recently in release 0.4.19[2].

I set the minimum python to 3.9 as that is the oldest version still
supported[3]. I verified 3.9 works by running

  uv run -p 3.9 -s ./utils/check-package `git ls-tree -r --name-only HEAD` --ignore-list=.checkpackageignore

[1] https://packaging.python.org/en/latest/specifications/inline-script-metadata/#script-type
[2] https://github.com/astral-sh/uv/releases/tag/0.4.19
[3] https://devguide.python.org/versions/

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-26 13:37:39 +02:00
James Hilliard
1459c0a6f0 package/pkg-python.mk: add hatch setup type
As we have many python packages using the hatch(hatchling) build
system we should add a setup type for it so that we don't have to
manually specify the host-python-hatchling dependency.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-23 23:00:03 +02:00
Arnout Vandecappelle
8aad67f157 utils/brmake: add option to run in docker
At the moment, it is difficult to combine brmake with docker-run.
`docker-run brmake ...` doesn't work because our docker image doesn't
have unbuffer. In addition, inside the container the timezone is UTC,
but you probably want the timestamps added by brmake to be in local
time. Therefore, it's better to have the call to docker-run nested
inside brmake.

Currently, brmake doesn't have any way to pass parameters, all of "$@"
is passed unchanged to the `make` invocation. Thus, there is no
established way to pass in the option whether or not to use docker. We
choose to use an environment variable to pass in the option. The
convention is that such a buildroot-specific environment variable should
start with BR2_, so we choose BR2_DOCKER.

Run make inside docker-run if the BR2_DOCKER environment variable is
set. Update utils/readme.txt (the only existing documentation of brmake)
with this information.

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-23 22:09:50 +02:00
James Hilliard
74764683f5 utils/scanpypi: add missing setuptools fallback
Some projects use pyproject.toml without setting a build backend, per
pep517 we should simply fallback to using setuptools when no build
backend is specified.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-10 23:17:16 +02:00
James Hilliard
699b3c38fd utils/scanpypi: refactor setuptools handling to not use imp
The imp module is deprecated as of python verison 3.12.

Refactor setuptools handling to remove monkeypatching hack and
instead do pep517 metadata generation and dependency resolution.

This is effectively done by implementing the minimal neccesary
pep517 frontend hooks needed for dependency resolution in scanpypi.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Fiona Klute (WIWA) <fiona.klute@gmx.de>
Reviewed-by: Fiona Klute (WIWA) <fiona.klute@gmx.de>
[Arnout: add license info]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-10-08 22:07:28 +02:00
Peter Korsgaard
f1ceb3687f utils/update-rust: fix 'following' typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:39:10 +02:00
Peter Korsgaard
aa79ae24a8 utils/scanpypi: fix typos
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[yann.morin.1998@free.fr: s/contents/content/]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:38:59 +02:00
Peter Korsgaard
37574ab12a utils/readme.txt: fix 'typos' typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:19:00 +02:00
Peter Korsgaard
ed2cb54252 utils/getdeveloperlib.py: fix retrieve typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:18:56 +02:00
Peter Korsgaard
6bcdbccb7f utils/docker-run: fix symmetry typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:18:52 +02:00
Peter Korsgaard
b3e5bcec65 utils/config: fix don't typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:18:48 +02:00
Yann E. MORIN
f9e6d39e8e utils/check-package: require exactly 1 TAB and 2 SP on help text 1st line
Test that the first line starts exactly with one TAB and exactly two
spaces before the text.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-09-14 22:24:55 +02:00
James Hilliard
ea6bb507b1 utils/genrandconfig: allow overriding KCONFIG_PROBABILITY
Tweaking this variable should allow us to get better coverage of
packages with larger dependency trees.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-08-23 19:40:12 +02:00
Yann E. MORIN
4dbb87bb66 utils/genrandconfig: fix fallout of dropping toolchain CSV
Commit 2f260084d5 (utils/genrandconfig: remove support for toolchain
CSV) kept the --no-toolchains-csv option, but in the rework forgot to
keep it as a bool, while argparse default is to expect a string.

Rather than re-introduce the action="store_true" which implies the
argument is a bool, explicit make it a bool.

Fixes: 2f260084d5

Reported-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-08-20 23:22:59 +02:00
Thomas Petazzoni
ce3dedc26b utils/genrandconfig: improve logging
Right now, genrandconfig just spits out the random messages from the
different make invocations, which isn't terribly useful. Instead,
let's redirect the output of make invocations to oblivion, and add
some more high level logging.

As part of this logging, we're interested to see how many iterations
were needed to find a valid configuration, so changed the loop logic
to count from 0 to 100 instead of from 100 to 0 so that we can easily
show the iteration number.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:08:10 +02:00
Thomas Petazzoni
da5c25c9f9 utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds
In order to test that upstream sites are still working, we need to NOT
fallback to sources.buildroot.net for some builds.

As there is anyway a local cache in the autobuilder instances, we need
to do quite a lot of builds without any BR2_BACKUP_SITE configured to
have a chance to catch issues, which is why a 50% chance is used to
unset BR2_BACKUP_SITE.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:08:05 +02:00
Thomas Petazzoni
3d33d394c2 utils/genrandconfig: rework fine-tuning logic
Before calling randpackageconfig/randconfig, we were pre-generating a
snippet of .config with:

 (1) minimal.config
 (2) BR2_CURL/BR2_WGET settings
 (3) some random selection of init system, debug, runtime debug, etc
 (4) enabling BR2_REPRODUCIBLE=y when diffoscope was found

Now that we only use randconfig, this whole fine-tuning is completely
irrelevant, as it gets overridden by "make randconfig".

(1) and (3) above are useless, as randconfig does all the
randomization that is needed.

However, we want to preserve (2) and (4) above, so we re-implement
those fixups, but *after* randconfig has done its job.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:07:56 +02:00
Thomas Petazzoni
2f260084d5 utils/genrandconfig: remove support for toolchain CSV
Now that the support for generating a fully random configuration has
been well-tested, the whole mechanism based on a toolchain CSV isn't
really useful anymore, so let's drop it to simplify the logic.

Note that the autobuilder code still uses --{,no-}toolchains-csv, so we
can't remove those or the autobuilders would fail. Once all supported
branches no longer use those argumetns, we can drop them from the
autobuilder code, then ask people to update their runners, and we will
finally be able to drop those arguments. Eventually.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: keep --{,no-}toolchains-csv and explain why]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:07:21 +02:00
Thomas Petazzoni
9e33882568 utils/genrandconfig: remove fixups related to untested CT-NG toolchains
We have accumulated a whole bunch of very old fixups to avoid issues
with super old CT-NG toolchains, which we are not testing anymore, so
remove those fixups.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 11:56:30 +02:00
Yann E. MORIN
32b4912a75 utils/checkpackagelib: instruct shellcheck to follow included files
We do have a few shell scripts that have source directives (either with
'source' or with '.'), and they all are currently either globally
ignored in .checkpackageignore, or have shellcheck directives to ignore
the source statement (SC1090).

So, we can safely instruct shellcheck where to lookup for sourced files;
that does not trigger any new error.

This will alow fixing some shellcheck errors in later commits.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
[Arnout: use long option]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 10:09:07 +02:00
Yann E. MORIN
fc2dedcfc6 utils/check-package: add option to run the test-suite
Currently, how to run the check-package test-suite is thoroughly under-
documented. There is one hint in the commit log for commit fc254881e6
(utils/checkpackagelib: add unit tests), and another in commit
242e9d72e7 (utils/docker-run: new script); the former is hard to find,
and the latter is about an unrelated script, so harder yet to find...

Add a new option to check-package, that will make it run its test-suite.
Since pytest is only needed for the test-suite, only import it in that
case.

pytest will be default create a cache of the tests (not sure what it is
about, though), so instruct it not to (esp. since the tree can be
read-only).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
[Arnout: move the code into checkpackagelib.base]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 10:09:06 +02:00
Yann E. MORIN
5836b79762 utils/checkpackagelib: extend hint about unprefixed variables
User may get confused when they see the current hint, and take that as
the proper replacement, while we're only reporting the stem of the
variable name:

    .../foo.mk:16: possible typo: BLA -> *FOO*

There is usually no easy way to actually suggest the proper variable
name, though, so let's make it a little bit more obvious that we meant
the variable was improperly prefixed:

    .../foo.mk:16: possible typo, variable not properly prefixed: BLA -> *FOO_XXX*

And while at it, throw in the URL to the corresponding manual entry.

Adapt the test accordingly.

Reported-by: "Frager, Neal" <neal.frager@amd.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Reviewed-by: Neal Frager <neal.frager@amd.com>
[Arnout: also update new test, scoped -> prefixed]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 08:53:40 +02:00
Yann E. MORIN
b0964df557 utils/checkpackagelib: really check variable name
Currently, when a package defines an unprefixed variable, but its value
contains a properly prefixed expansion (or even just the name of a
variable), there is not error reported (e.g. with the recently fixed
composer issue):

    BASE_SITE = https://getcomposer.org/download/$(COMPOSER_VERSION)/composer.phar

The reason is that he check is done on the whole line, rather than on
the variable that is being set.

We fix that by really looking at the variable we found, instead of
looking in the whole line.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 08:53:36 +02:00
Ricardo Martincoski
ccc12c0f24 utils/check-package: check linux-tools
Each linux tool uses a fragment of a .mk file, named, for instance:
package/linux-tools/linux-tool-cpupower.mk.in
So currently check-package does not check these files.

Add the support in check-package script.
At the same time, factor out a function to derive package prefix from
the filename being checked, so the fix (calling os.path.splitext twice)
can be applied in a single place.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add docstring and explain double splitext to
         get_package_prefix_from_filename]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-12 20:30:40 +02:00
Ricardo Martincoski
cca315f814 utils/check-package: fix check for <pkg>_REDISTRIBUTE
Commit "8e87d76c12 check-package: check *.mk files" added a typo that
prevents check-package from checking <PKG>_REDISTRIBUTE default value.

Fix the typo: _INSTALL_REDISTRIBUTE -> _REDISTRIBUTE

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-12 20:30:36 +02:00
Yann E. MORIN
a316aeb7da utils/genrandconfig: do not check certificates with curl
genrandconfig is used in autobuilders, and some autobuilders are running
on old distributions that are lacking the most recent CAs, causing build
failures because package sources can't be retrieved.

Do for the curl backend what we already did a while back for the wget
backend, with commit 0866a280e4 (utils/genrandconfig: use
--no-check-certificate in wget by default); in curl, the equivalent
would be --insecure, and applies to the ftps transport.

The integrity of the downloads are validated against our bundled hashes
so there is no risk of corruption of the downloaded files. The only
issue would be that an MITM could inspect the transaction, the same way
as for the wget --no-check-certificate in 0866a280e4, but this is not
considered a high-level issue (we're anyway talking FTPS here, that's a
legacy protocol that has other issues).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-12 14:36:10 +02:00
Yann E. MORIN
881a8f1346 utils/genrandconfig: stop passing --passive-ftp to wget
Since we no longer use WGET to retrieve FTP-hosted files, we can drop
the --passive-ftp option from genrandconfig, as it would cause
problems on systems that use wget2.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-12 14:34:57 +02:00
James Hilliard
0fa7343b6b utils/update-rust: use pathlib with relative paths
This avoids the requirement of running update-rust from TOPDIR and
is slightly cleaner.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-10 23:03:42 +02:00
James Hilliard
8436a5141b utils/update-rust: use urllib.request from stdlib instead of requests
We can use urllib.request without adding any addition lines of code
compared with requests which is not part of the stdlib.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-10 22:40:52 +02:00
Vincent Fazio
76cd14167f package/python3: bump python to 3.12.4
Along with the version bump, a number of the patches have been dropped
or reworked.

Compiled shared module builds are now disabled via py_cv_module_* = n/a
when possible (see PY_STDLIB_MOD_SET_NA). Leveraging this method allows
dropping numerous patches.

Note: this method does not work when access to a compiled module is
wrapped by a python-based module such as sqlite, tkinter, etc. In these
cases, configure knobs are still required to disable these modules and
their wrappers.

Both setup.py and distutils are no longer utilized upstream, so patches
have been dropped or adjusted accordingly.

Patch 0007 has been dropped, the Xtensa architecture can avoid compile
errors by using a toolchain with kernel headers newer than 3.17 [0].
That would require a new, non-trivial dependency to be added, but it
would also need to be propagated to all packages that select python3; as
we consider xtensa with old headers to be a corner case, we decide to
not address the issue, and rather add an exclusion in genrandconfig to
avoid the issue in autobuilders.

Patch 0026 has been dropped, the issue is fixed in uClibc v1.0.37+ [1].
Note: uClibc is not generally supported by upstream CPython.

While we're making assumptions about uClibc, ac_cv_func_wcsftime=no is
no longer set as it was fixed in v1.0.0 [2].

Co-developed-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Co-developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[See 36e635d2d5]

Co-developed-by: Roy Kollen Svendsen <roy.kollen.svendsen@akersolutions.com>
[See 687b96db4d]

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=f61bf8e7d19e0a3456a7a9ed97c399e4353698dc
[1]: https://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/commit/libcrypt/crypt.c?id=2f6076cdaada2132b670b5ef25ad80c76a916c5a
[2]: https://lists.uclibc.org/pipermail/uclibc/2014-November/048723.html

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
[yann.morin.1998@free.fr;
  - extend the commit log for the xtensa headers case
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-21 21:10:37 +02:00
Yann E. MORIN
2ab21855bb package/rust: update to 1.79.0
In rust 1.75.0, the follwong targets were dropped [0]:
    mips-unknown-linux-musl
    mips64-unknown-linux-muslabi64
    mips64el-unknown-linux-muslabi64
    mipsel-unknown-linux-musl

[0] 03870154f4

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-21 21:10:33 +02:00
Yann E. MORIN
df899041fb utils/update-rust: add script to automatically update the rust version
When we update the rust version, we need to update a large list of
hashes for rust-bin, and update the hash for rust (source). This is very
tedious and very error-prone.

Add a helper script that generates the hash files automatically, and
also iupdate the _VERSION in the .mk.

We decided not to carry the hint about the verification of the tarballs
against the upstream GPG signature for three reasons:

  - it requires that all the tarballs be downloaded, which can take
    quite some time;

  - the hash files are available for all the needed archives;

  - the hash files are downloaded over https, so if someone managed to
    get a hold of the rust server to provide backdoored archives, they
    can also change the hash files;

  - properly verifying the archives would require a chain of trust
    between the person running the upsate script, and the rust GPG key,
    which is not a given, and verifying sigantures using an unverified
    key is not providing much security, if at all.

Regenerate the hash files with that script.

Note (for the future, maybe): there are manifest files for each release,
https:/static.rust-lang.org/dist/channel-rust-VERSION.toml, that we
could use to generate the RUST_HOSTS and RUST_TARGETS list. However,
adding new hosts or new targets implies updating the corresponding
_ARCH_SUPPORTS and accompanying symbols, so better leave that to a
manual operation, at least for now.

Notes: Many thanks to James for providing an initial script with the
same purpose. Given the feedback from previous reviews, it was simpler
to rewrite it from scratch; it should now be much, much simpler.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: James Hilliard <james.hilliard1@gmail.com>
Tested-by: James Hilliard <james.hilliard1@gmail.com>
2024-06-21 21:10:31 +02:00
Peter Korsgaard
876d24578c Merge branch 'next' 2024-06-13 11:16:07 +02:00
Yann E. MORIN
8b8f5e3366 utils/genrandconfig: fix matching for a few lines
The matching in genrandconfig is idiomatically done by matching whole
lines, i.e. with the terminating \n but a few places are missing that.

Those are only matching against '=y', a boolean symbol, so it is in
practice not causing any issue. Still, for consistency, fix those.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-10 21:54:21 +02:00
James Hilliard
929a491f40 utils/genrandconfig: drop glibc Linaro toolchain version check
It has been well over 10 years since glibc 2.14 was released; the last
Debian version that had an earlier glibc was Wheezy, which Freexian
stopped to maintain as an ELTS in June 2020, 4 years ago, while the
oldest still maintained Ubuntu has glibc 2.21. It is now safe to assume
glibc 2.14 on all major, relevant distributions nowadays.

The distutils module is no longer bundled with python 3.12 so this
eliminates the need to install additional python modules under python
3.12.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[yann.morin.1998@free.fr: add Debian and Ubuntu references]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-03 22:27:37 +02:00
Julien Olivain
f97e6293de utils/docker-run: set podman userns option in environment
Commit 9a629f5 "utils/docker-run: allow running with Podman" added an
option on system providing the podman command. This case is mainly
for Fedora systems.

Fedora repositories have a podman-docker package, that provides the
docker command for compatibility. See [1]. In that case, invoking
"docker" redirects to podman.

When this package is installed on a Fedora system, both the docker and
podman commands are available. Since the docker command is checked
before podman, the --userns option is not passed in that case. This
brings "permission denied" errors.

Other cases are also possible, like a host system providing the real
Docker alongside a podman installation. In such a case, to avoid
unexpected change of behavior of the docker-run script, the original
search order is preserved (search for "docker" first, then "podman").

This commit changes the way the podman user namespace mode is set.
Rather than adding the "--userns=keep-id" command line option only in
the podman case, it is globally set using the PODMAN_USERNS=keep-id
environment variable [2].

Doing so makes sure that the variable will be consumed by the "docker"
compatibility command, and just ignored by the real "docker"
implementation.

[1] https://packages.fedoraproject.org/pkgs/podman/podman-docker/
[2] https://docs.podman.io/en/latest/markdown/podman-run.1.html

Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-01 15:51:11 +02:00
James Hilliard
584ebdea6e utils/genrandconfig: switch to async/await format
This requires python 3.5 or newer but is a bit cleaner than the
previous coroutine method.

This should also fix a python3.12 issue:
[Tue, 28 May 2024 13:09:05] INFO: generate the configuration
Traceback (most recent call last):
  File "/home/autobuild/autobuild/instance-0/buildroot/utils/genrandconfig", line 833, in <module>
    ret = asyncio.run(gen_config(args))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 89, in run
    raise ValueError("a coroutine was expected, got {!r}".format(coro))
ValueError: a coroutine was expected, got <generator object gen_config at 0xffff7bd822c0>
[Tue, 28 May 2024 13:09:06] WARN: failed to generate configuration

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-05-31 21:20:03 +02:00