mirror of
https://gitlab.com/buildroot.org/buildroot.git
synced 2026-08-08 08:30:47 -09:00
c9df1b64b2
We have our own GitLab-CI runner, but with only one we can't run many
jobs in parallel so it takes a very long time before all the tests have
completed. In addition, if that runner goes down, we have nothing at
all.
GitLab offers the following machine types for hosted runners on Linux
x86-64 [1]. The default is the "small" runner. Using a larger runner
increases the "Cost factor" [2].
For opensource projects, the Cost factor is reduced to 0.5 (1 minute per
2 minutes of job time) whatever the runner type.
Runner Tag vCPUs Memory Storage Cost factor (OSS)
saas-linux-small-amd64 (default) 2 8 GB 30 GB 1 (0.5)
saas-linux-medium-amd64 4 16 GB 50 GB 2 (0.5)
saas-linux-large-amd64 (Premium and Ultimate only) 8 32 GB 100 GB 3 (0.5)
saas-linux-xlarge-amd64 (Premium and Ultimate only) 16 64 GB 200 GB 6 (0.5)
saas-linux-2xlarge-amd64 (Premium and Ultimate only) 32 128 GB 200 GB 12 (0.5)
Compute minutes consumed by a job is calculated by:
Job duration / 60 * Cost factor
(Job duration: The time, in seconds, that a job took to run, not
including time spent in the created or pending statuses.)
Thanks to the GitLab OSS program [3], Buildroot benefits from a free
Ultimate subscription and can use GitLab shared runners tagged with
saas-linux-{large, xlarge, 2xlarge}-amd64. In addition, we receive
50,000 free runner minutes per month.
In order to use one of those tags in Buildroot GitLab-CI jobs, we have
to classify all tests by resource requirement, to make sure the job
doesn't fail because it times out or has insufficient memory or disk
space. While we usually shouldn't use the largest runner for
everything, we can use larger runner without cost penalty thanks to
the cost factor reduced to 0.5 for opensource projects. This will
reduce the CI minutes consumed by a CPU intensive job.
First we introduce some new templates used to add the corresponding
runner tag to a runtime test job (reusing the GitLab terminology).
.runner-{small,medium,large,xlarge,2xlarge}
Most of our tests are fast (checkpackage, test_external_bootlin...), so
saas-linux-small-amd64 runner tag is enough. Default to this tag if
nothing else is specified.
Add a comment next to the test class to provide the runner tag.
This runner tag is retrieved when generating the
generated-gitlab-ci.yml file used to create the child pipeline where
the runtime test jobs are executed.
We use the list of runtime tests returned by node2:
"tests.boot.test_edk2.TestEdk2.test_run"
We convert each element of this list to get the path to the test source
file and the name of the test:
"support/testing/tests/boot/test_edk2.py"
TestEdk2
With that, we can grep into the test source file to retrieve the runner
tag placed one line above the test class:
# GitLab-runner: large
class TestEdk2(infra.basetest.BRTest):
Once the runner tag is retrieved, it's used to use the corresponding
runner template to the runtime test job:
tests.boot.test_edk2.TestEdk2.test_run: { extends: [ .runtime_test_base, .runner-large ]}
GitLab runners hosted by the Buildroot project should be able to run
any jobs, so they should be tagged with Gitlab runner tags
(saas-linux-{small,medium,large,xlarge,2xlarge}-amd64).
A specific runner tag "buildroot-runner" can be used to allow running
a job only on such runners.
If a test can't be executed by any shared GitLab-CI runners, we have
to use a runner owned by the Buildroot project. In this case we have
to use a specific template ".runner-buildroot-runner-only" in order to
add the specific runner tag "buildroot-runner" to the job running the
test. There is no such runtime test at the moment.
The proposed classification is based on a previous pipeline analysis
[5]:
- Tests lasting more than 3 hours will use 2xlarge runners.
- Tests lasting more than 2 hours will use xlarge runners.
- Tests lasting more than 1 hours will use large runners.
- Tests building a kernel or a toolchain will use medium runners.
- All other tests will use small runners when possible.
CI minute cost estimate:
tests.package.test_clang.TestClangCompilerRT.test_run lasts 4h25 on the
Buildroot runner. If we this duration for 2xlarge runners, the CI
minute consumed would be:
(15900 / 60) * 0.5 = 133
With 6 jobs using a 2xlarge runners we used ~795 CI minutes.
tests.package.test_kmscube.TestKmsCube.test_run list 2h04 on the
Buildroot runner. If we this duration for xlarge runners, the CI
minute consumed would be:
(7440 / 60) * 0.5 = 62
With 4 jobs using a xlarge runners we used ~248 CI minutes.
tests.package.test_weston.TestWeston.test_run last 1h15 on th
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(4500 / 60) * 0.5 = 37.5
With 28 jobs using a large runners we used ~1050 CI minutes.
tests.package.test_gstreamer1.TestGstreamer1.test_run last 46min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(2760 / 60) * 0.5 = 23
With 31 jobs using a medium runners we used ~713 CI minutes.
tests.package.test_python.TestPython3Py.test_run last 13min on the
Buildroot runner. If we this duration for large runners, the CI
minute consumed would be:
(780 / 60) * 0.5 = 6.5
With 691 jobs using a medium runners we used ~4491 CI minutes.
In total, one pipeline for the runtime tests cost ~7297 CI minutes.
After a first try [6], we are actually using 8000 CI minutes per
pipeline.
We run such pipeline once a week (on Monday), one for each Buildroot
releases every 3 month, one for each stable and LTS release per month,
and one for each release candidate (3).
Worst case (release month):
(4 weeks + 1 release + 1 stable + 1 LTS + 3 release candidate) * 8000 CI
minutes: 80000 CI minutes / 50000.
So we would spend the minutes very quickly in the worst case scenario.
We have to keep one pipeline under 5000 CI minutes.
[1] https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
[2] https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors
https://docs.gitlab.com/ci/pipelines/compute_minutes/#compute-usage-calculation
https://docs.gitlab.com/ci/pipelines/compute_minutes/#cost-factors-of-hosted-runners-for-gitlabcom
[3] https://gitlab.com/buildroot.org/gitlab-oss
[4] https://docs.gitlab.com/ci/runners/hosted_runners/#gitlabcom-hosted-runner-workflow
[5] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2416603721
[6] https://gitlab.com/buildroot.org/buildroot/-/pipelines/2562421098
Signed-off-by: Romain Naour <romain.naour@smile.fr>
[Arnout:
- simplify parsing of test_file and test_name;
- match the entire test_name instead of substring;
- assume "small" by default;
- remove the "small" tags;
- use "gitlab-runner" instead of "Gitlab-runner".
]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
243 lines
9.7 KiB
Python
243 lines
9.7 KiB
Python
import os
|
|
import pexpect
|
|
import infra.basetest
|
|
|
|
|
|
class TestXenBase(infra.basetest.BRTest):
|
|
"""A class to test Xen for multiple architectures."""
|
|
|
|
# We use the same rootfs contents for both the host and the guest.
|
|
base_config = \
|
|
"""
|
|
BR2_TOOLCHAIN_EXTERNAL=y
|
|
BR2_ROOTFS_POST_BUILD_SCRIPT="support/testing/tests/package/test_xen/common/post-build.sh"
|
|
BR2_LINUX_KERNEL=y
|
|
BR2_LINUX_KERNEL_CUSTOM_VERSION=y
|
|
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.18.4"
|
|
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
|
|
BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y
|
|
BR2_PACKAGE_BRIDGE_UTILS=y
|
|
BR2_PACKAGE_XEN=y
|
|
BR2_PACKAGE_XEN_HYPERVISOR=y
|
|
BR2_PACKAGE_XEN_TOOLS=y
|
|
BR2_TARGET_ROOTFS_EXT2=y
|
|
BR2_TARGET_ROOTFS_EXT2_4=y
|
|
BR2_TARGET_ROOTFS_EXT2_SIZE="128M"
|
|
# BR2_TARGET_ROOTFS_TAR is not set
|
|
BR2_TARGET_UBOOT=y
|
|
BR2_TARGET_UBOOT_BUILD_SYSTEM_KCONFIG=y
|
|
BR2_TARGET_UBOOT_CUSTOM_VERSION=y
|
|
BR2_TARGET_UBOOT_CUSTOM_VERSION_VALUE="2026.01"
|
|
BR2_TARGET_UBOOT_NEEDS_OPENSSL=y
|
|
BR2_TARGET_UBOOT_NEEDS_GNUTLS=y
|
|
BR2_PACKAGE_HOST_DOSFSTOOLS=y
|
|
BR2_PACKAGE_HOST_GENIMAGE=y
|
|
BR2_PACKAGE_HOST_MTOOLS=y
|
|
"""
|
|
|
|
def get_dom_uuid(self) -> str:
|
|
out, rc = self.emulator.run("cat /sys/hypervisor/uuid")
|
|
self.assertEqual(rc, 0, "Failed to get domain UUID")
|
|
return out[0]
|
|
|
|
def assertNumVM(self, x: int) -> None:
|
|
out, rc = self.emulator.run("xl vm-list")
|
|
self.assertEqual(rc, 0, "Failed to get VM list")
|
|
num_vm = len(out) - 1
|
|
self.assertEqual(num_vm, x, f"Expected {x} VM(s) but found {num_vm}")
|
|
|
|
def run_xen_test(self, arch: str, options: list[str]) -> None:
|
|
"""This functions tests Xen for multiple architectures.
|
|
The arch and options parameters are passed to the emulator.
|
|
|
|
Here is the network setup we use in the test:
|
|
|
|
: dom0 : dom1 :
|
|
: : :
|
|
: br0 : :
|
|
: 10.0.2.42 : :
|
|
gw : | : :
|
|
10.0.2.2 -:- eth0 --+-- vif1.0 -:-- eth0 :
|
|
: : 10.0.2.43 :
|
|
|
|
The VMs use static IP addresses.
|
|
We create a bridge in dom0, which allows dom0 to reach the gateway.
|
|
vif1.0 is added to the bridge automatically when dom1 is created.
|
|
|
|
There is a stability issue for Armv7 with Qemu < 9.1, which makes the
|
|
dom1 -> gw connection unreliable. As a workaround, we interact only with
|
|
dom0 from dom1 (hence the fixed IP addresses).
|
|
|
|
Here is the disk setup we use in the test:
|
|
|
|
Image/part. Dom0 Dom1 Description
|
|
----------- ---------- ------- -----------
|
|
disk.img vda
|
|
+-- boot vda1, /mnt Contains Xen and kernel
|
|
+-- root vda2, / Dom0 rootfs
|
|
`- guest vda3 xvda, / Dom1 rootfs
|
|
"""
|
|
|
|
# Boot the emulator.
|
|
# The system should automatically boot Xen and a Dom0.
|
|
self.emulator.boot(arch=arch, options=options)
|
|
self.emulator.login()
|
|
|
|
# Verify that we are indeed running under Xen.
|
|
self.assertRunOk("xl info")
|
|
|
|
# Check that we are dom0.
|
|
uuid = self.get_dom_uuid()
|
|
dom0_uuid = "00000000-0000-0000-0000-000000000000"
|
|
self.assertEqual(uuid, dom0_uuid, f"Unexpected dom UUID {uuid}")
|
|
|
|
# Check that we have one VM running.
|
|
self.assertNumVM(1)
|
|
|
|
# Create a network bridge.
|
|
self.assertRunOk("brctl addbr br0")
|
|
self.assertRunOk("brctl addif br0 eth0")
|
|
self.assertRunOk("brctl show")
|
|
|
|
# Bring up the network in the dom0.
|
|
self.assertRunOk("ifconfig eth0 up")
|
|
self.assertRunOk("ifconfig br0 10.0.2.42")
|
|
self.assertRunOk("ifconfig -a")
|
|
|
|
# Verify that we can ping the gateway.
|
|
self.assertRunOk("ping -c 3 -A 10.0.2.2")
|
|
|
|
# Create dom1 with console attached and login.
|
|
self.emulator.qemu.sendline("xl create -c /etc/xen/dom1.cfg")
|
|
self.emulator.login()
|
|
|
|
# Check that we are not talking to dom0 anymore.
|
|
uuid = self.get_dom_uuid()
|
|
self.assertNotEqual(uuid, dom0_uuid, "Unexpected dom0 UUID")
|
|
|
|
# Verify dom1 block device.
|
|
# (This is a bit redundant, as if the test runs this far we already know
|
|
# that dom1 found its rootfs.)
|
|
self.assertRunOk("ls -laF /sys/class/block/xvda")
|
|
|
|
# Bring up the network in the dom1.
|
|
self.assertRunOk("ifconfig eth0 10.0.2.43")
|
|
self.assertRunOk("ifconfig -a")
|
|
|
|
# Verify that we can ping the dom0.
|
|
self.assertRunOk("ping -c 3 -A 10.0.2.42")
|
|
|
|
# Detach from dom1's console with CTRL-].
|
|
# dom1 is still running in the background after that.
|
|
self.emulator.qemu.send(chr(0x1d))
|
|
mult = self.emulator.timeout_multiplier
|
|
index = self.emulator.qemu.expect(["#", pexpect.TIMEOUT],
|
|
timeout=2 * mult)
|
|
self.assertEqual(index, 0, "Timeout exiting guest")
|
|
|
|
# Check that we are talking to dom0 again.
|
|
uuid = self.get_dom_uuid()
|
|
self.assertEqual(uuid, dom0_uuid, f"Unexpected dom UUID {uuid}")
|
|
|
|
# Check that we have two VMs running.
|
|
self.assertNumVM(2)
|
|
|
|
# Print the block setup for debugging.
|
|
self.assertRunOk("xl block-list dom1")
|
|
|
|
# Print the bridge setup for debugging.
|
|
self.assertRunOk("brctl show")
|
|
|
|
|
|
# gitlab-runner: large
|
|
class TestXenAarch64(TestXenBase):
|
|
# Test Xen on 64b Arm.
|
|
# Boot flow: Qemu Devicetree -> U-Boot -> Xen UEFI -> Linux
|
|
# We need to boot Xen in UEFI to read xen.cfg.
|
|
# We use U-Boot as our UEFI firmware.
|
|
# We have a custom kernel config to reduce build time.
|
|
# Our genimage.cfg is inspired from qemu_aarch64_ebbr_defconfig as we boot
|
|
# Xen with UEFI.
|
|
config = TestXenBase.base_config + \
|
|
"""
|
|
BR2_aarch64=y
|
|
BR2_ROOTFS_OVERLAY="support/testing/tests/package/test_xen/common/overlay \
|
|
support/testing/tests/package/test_xen/aarch64/overlay"
|
|
BR2_ROOTFS_POST_IMAGE_SCRIPT="support/testing/tests/package/test_xen/aarch64/post-image.sh support/scripts/genimage.sh"
|
|
BR2_ROOTFS_POST_SCRIPT_ARGS="-c support/testing/tests/package/test_xen/aarch64/genimage.cfg"
|
|
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="support/testing/tests/package/test_xen/aarch64/linux.config"
|
|
BR2_TARGET_UBOOT_BOARD_DEFCONFIG="qemu_arm64"
|
|
"""
|
|
|
|
def test_run(self):
|
|
uboot_bin = os.path.join(self.builddir, "images", "u-boot.bin")
|
|
disk_img = os.path.join(self.builddir, "images", "disk.img")
|
|
|
|
# We need to run Qemu with virtualization to run Xen.
|
|
qemu_opts = [
|
|
"-bios", uboot_bin,
|
|
"-cpu", "cortex-a53",
|
|
"-device", "virtio-blk-device,drive=hd0",
|
|
"-device", "virtio-net-device,netdev=eth0",
|
|
"-drive", f"file={disk_img},if=none,format=raw,id=hd0",
|
|
"-m", "1G",
|
|
"-netdev", "user,id=eth0,restrict=yes",
|
|
"-machine", "virt,gic-version=3,virtualization=on,acpi=off",
|
|
"-smp", "2"
|
|
]
|
|
|
|
# Run Xen test.
|
|
self.run_xen_test(arch="aarch64", options=qemu_opts)
|
|
|
|
|
|
# gitlab-runner: large
|
|
class TestXenArmv7(TestXenBase):
|
|
# Test Xen on 32b Arm v7.
|
|
# Boot flow: Qemu Devicetree -> U-Boot -> Xen -> Linux
|
|
# Xen does not boot with UEFI on 32-bit Arm v7.
|
|
# We use U-Boot and a script to load the Dom0 images and amend the
|
|
# Devicetree for Xen dynamically.
|
|
# We have a custom kernel config to reduce build time.
|
|
# We need to build host-qemu to workaround an issue with Xen guest block
|
|
# devices on 32b Arm.
|
|
# This was fixed by commit 7175a562f157 ("hw/intc/arm_gic: Fix deactivation
|
|
# of SPI lines") and the fix is present in Qemu versions >= 9.1.0.
|
|
# When the CI docker image will have a recent-enough Qemu version with the
|
|
# fix, we can remove the host-qemu package from the test configuration.
|
|
config = TestXenBase.base_config + \
|
|
"""
|
|
BR2_arm=y
|
|
BR2_cortex_a15=y
|
|
BR2_ROOTFS_OVERLAY="support/testing/tests/package/test_xen/common/overlay \
|
|
support/testing/tests/package/test_xen/arm/overlay"
|
|
BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
|
|
BR2_ROOTFS_POST_SCRIPT_ARGS="-c support/testing/tests/package/test_xen/arm/genimage.cfg"
|
|
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="support/testing/tests/package/test_xen/arm/linux.config"
|
|
BR2_TARGET_UBOOT_BOARD_DEFCONFIG="qemu_arm"
|
|
BR2_PACKAGE_HOST_UBOOT_TOOLS=y
|
|
BR2_PACKAGE_HOST_UBOOT_TOOLS_BOOT_SCRIPT=y
|
|
BR2_PACKAGE_HOST_UBOOT_TOOLS_BOOT_SCRIPT_SOURCE="support/testing/tests/package/test_xen/arm/boot.cmd"
|
|
BR2_PACKAGE_HOST_QEMU=y
|
|
BR2_PACKAGE_HOST_QEMU_SYSTEM_MODE=y
|
|
"""
|
|
|
|
def test_run(self):
|
|
uboot_bin = os.path.join(self.builddir, "images", "u-boot.bin")
|
|
disk_img = os.path.join(self.builddir, "images", "disk.img")
|
|
|
|
# We need to run Qemu with virtualization to run Xen.
|
|
qemu_opts = [
|
|
"-bios", uboot_bin,
|
|
"-cpu", "cortex-a15",
|
|
"-device", "virtio-blk-device,drive=hd0",
|
|
"-device", "virtio-net-device,netdev=eth0",
|
|
"-drive", f"file={disk_img},if=none,format=raw,id=hd0",
|
|
"-m", "1G",
|
|
"-machine", "virt,virtualization=on,acpi=off",
|
|
"-netdev", "user,id=eth0,restrict=yes",
|
|
"-smp", "2"
|
|
]
|
|
|
|
# Run Xen test.
|
|
self.run_xen_test(arch="armv7", options=qemu_opts)
|