CI: aports version check: modernize with new rules (MR 3615)

Update the script once more with the goal that we no longer need to
comment on pkgrel/pkgver bumps. Parse not only the version of the old
and new package, but also the checksums and remote sources.

Replace the previous check of simply having $pkgver-r$pkgrel higher
than the upstream APKBUILD with more detailed rules:

  APKBUILD has remote sources?      ------>       Did sha512sums change?
   |                                  No            |         |
   | Yes                                            | No      | Yes
   v                                                |         v
  Did the file name of a remote                     |     * Increase pkgver
  source change?                    ----.           |     * Set pkgrel=0
   |                                    |           |
   | Yes                                | No        |
   v                                    v           v
  Is pkgver=9999?         ----->   * Don't change pkgver
  (Forked from Alpine)     Yes     * Bump pkgrel by 1
   |
   | No
   v
  * Set the pkgver to the version
    of the source tarball
  * Set pkgrel=0

I'll also put this in the wiki at:
https://postmarketos.org/howto-bump-pkgrel-pkgver

Besides the new rules, print a detailed error message below each package
where the test failed that explains exactly what was expected and what
the reasoning behind that is. Also add a hint that users can run this
locally with 'pmbootstrap ci' now.

Use logging instead of print, so we get colored ERROR: and NOTE:
statements from pmbootstrap code, this should make it easier to read.

.ci/apkbuild-lint.sh

@@ -0,0 +1,22 @@
+#!/bin/sh -e
+# Description: run apkbuild-lint on modified APKBUILDs
+# Options: native
+# Use 'native' because it requires git commit history.
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
+	sh ./install_pmbootstrap.sh
+	exec su "${TESTUSER:-pmos}" -c "sh -e $0"
+fi
+
+# Wrap pmbootstrap to use this repository for --aports
+pmaports="$(cd $(dirname $0)/..; pwd -P)"
+_pmbootstrap="$(command -v pmbootstrap)"
+pmbootstrap() {
+	"$_pmbootstrap" --aports="$pmaports" "$@"
+}
+
+.ci/lib/apkbuild_linting.py
+

.ci/build-aarch64.sh

@@ -0,0 +1 @@
+lib/build_changed_aports.sh

.ci/build-armhf.sh

@@ -0,0 +1 @@
+lib/build_changed_aports.sh

.ci/build-armv7.sh

@@ -0,0 +1 @@
+lib/build_changed_aports.sh

.ci/build-riscv64.sh

@@ -0,0 +1 @@
+lib/build_changed_aports.sh

.ci/build-x86.sh

@@ -0,0 +1 @@
+lib/build_changed_aports.sh

.ci/build-x86_64.sh

@@ -0,0 +1 @@
+lib/build_changed_aports.sh

.ci/build.sh

@@ -1,8 +0,0 @@
-#!/bin/sh -e
-# Convenience wrapper for short arch-specific build jobs in .gitlab-ci.yml
-
-export PYTHONUNBUFFERED=1
-JOB_ARCH="${CI_JOB_NAME#build-}"
-
-set -x
-su pmos -c ".ci/build_changed_aports.py $JOB_ARCH"

.ci/check-dtb-install-location.sh

@@ -1,7 +0,0 @@
-#!/bin/sh -e
-
-if grep -r 'INSTALL_DTBS_PATH="$pkgdir"/usr/share/dtb'; then
-	echo 'Please do not install dtbs to /usr/share/dtb!'
-	echo 'Unless you have a good reason not to, please put them in /boot/dtbs'
-	exit 1
-fi

.ci/check_changed_aports_versions.py

@@ -1,156 +0,0 @@
-#!/usr/bin/env python3
-# Copyright 2021 Oliver Smith
-# SPDX-License-Identifier: GPL-3.0-or-later
-
-import glob
-import tempfile
-import sys
-import subprocess
-
-# Same dir
-import common
-
-# pmbootstrap
-import testcases.add_pmbootstrap_to_import_path  # noqa
-import pmb.parse
-import pmb.parse.version
-import pmb.helpers.logging
-
-
-def get_package_version(args, package, revision, check=True):
-    # Redirect stderr to /dev/null, so git doesn't complain about files not
-    # existing in upstream branch for new packages
-    stderr = None
-    if not check:
-        stderr = subprocess.DEVNULL
-
-    # Run something like "git show upstream/master:main/hello-world/APKBUILD"
-    pmaports_dir = common.get_pmaports_dir()
-    pattern = pmaports_dir + "/**/" + package + "/APKBUILD"
-    path = glob.glob(pattern, recursive=True)[0][len(pmaports_dir + "/"):]
-    apkbuild_content = common.run_git(["show", revision + ":" + path], check,
-                                      stderr)
-    if not apkbuild_content:
-        return None
-
-    # Save APKBUILD to a temporary path and parse it from there. (Not the best
-    # way to do things, but good enough for this CI script.)
-    with tempfile.TemporaryDirectory() as tempdir:
-        with open(tempdir + "/APKBUILD", "w", encoding="utf-8") as handle:
-            handle.write(apkbuild_content)
-        parsed = pmb.parse.apkbuild(tempdir + "/APKBUILD", False, False)
-
-    return parsed["pkgver"] + "-r" + parsed["pkgrel"]
-
-
-def version_compare_operator(result):
-    """ :param result: return value from pmb.parse.version.compare() """
-    if result == -1:
-        return "<"
-    elif result == 0:
-        return "=="
-    elif result == 1:
-        return ">"
-
-    raise RuntimeError("Unexpected version_compare_operator input: " + result)
-
-
-def exit_with_error_message():
-    branch = common.get_upstream_branch()
-    print()
-    print("ERROR: Modified package(s) don't have an increased version or a")
-    print("new package has a nonzero pkgrel!")
-    print()
-    print("This can happen if you added a new package with a nonzero")
-    print("pkgrel, or if you did not change the pkgver/pkgrel")
-    print("variables in the APKBUILDs. Or you did change them, but the")
-    print(f"packages have been updated in the official '{branch}' branch, and")
-    print("now your versions are not higher anymore.")
-    print()
-    print("Your options:")
-    print("a) If you made changes to the packages, and did not increase the")
-    print("   pkgrel/pkgver: increase them now, and force push your branch.")
-    print("   => https://postmarketos.org/howto-bump-pkgrel-pkgver")
-    print("b) If you had already increased the package versions, rebase on")
-    print(f"   '{branch}', increase the versions again and then force push:")
-    print("   => https://postmarketos.org/rebase")
-    print("c) If you made a change, that does not require rebuilding the")
-    print("   packages, such as only changing the arch=... line: you can")
-    print("   disable this check by adding '[ci:skip-vercheck]' to the")
-    print("   latest commit message, then force push.")
-    print()
-    print("Thank you and sorry for the inconvenience.")
-    exit(1)
-
-
-def check_versions(args, packages):
-    error = False
-
-    # Get relevant commits: compare HEAD against upstream branch or HEAD~1
-    # (the latter if this CI check is running on upstream branch). Note that
-    # for the common.get_changed_files() code, we don't check against
-    # upstream branch HEAD, but against the latest common ancestor. This is not
-    # desired here, since we already know what packages changed, and really
-    # want to check if the version was increased towards *current* upstream
-    # branch HEAD.
-    commit = f"upstream/{common.get_upstream_branch()}"
-    if common.run_git(["rev-parse", "HEAD"]) == common.run_git(["rev-parse",
-                                                                commit]):
-        print(f"NOTE: {commit} is on same commit as HEAD, comparing"
-              " HEAD against HEAD~1.")
-        commit = "HEAD~1"
-
-    for package in packages:
-        # Get versions, skip new packages
-        head = get_package_version(args, package, "HEAD")
-        upstream = get_package_version(args, package, commit, False)
-        if not upstream:
-            if head.rpartition('r')[2] != "0":
-                print(f"- {package}: {head} (HEAD) (new package) [ERROR]")
-                error = True
-            else:
-                print(f"- {package}: {head} (HEAD) (new package)")
-            continue
-
-        # Compare head and upstream versions
-        result = pmb.parse.version.compare(head, upstream)
-        if result != 1:
-            error = True
-
-        # Print result line ("- hello-world: 1-r2 (HEAD) > 1-r1 (HEAD~1)")
-        formatstr = "- {}: {} (HEAD) {} {} ({})"
-        if result != 1:
-            formatstr += " [ERROR]"
-        operator = version_compare_operator(result)
-        print(formatstr.format(package, head, operator, upstream, commit))
-
-    if error:
-        exit_with_error_message()
-
-
-if __name__ == "__main__":
-    # Get and print modified packages
-    common.add_upstream_git_remote()
-    packages = common.get_changed_packages()
-    print(f"Changed packages: {packages}")
-
-    # Verify modified package count
-    common.get_changed_packages_sanity_check(len(packages))
-    if len(packages) == 0:
-        print("no aports changed in this branch")
-        exit(0)
-
-    # Potentially skip this check
-    if common.commit_message_has_string("[ci:skip-vercheck]"):
-        print("WARNING: not checking for changed package versions"
-              " ([ci:skip-vercheck])!")
-        exit(0)
-
-    # Initialize args (so we can use pmbootstrap's APKBUILD parsing)
-    sys.argv = ["pmbootstrap.py", "chroot"]
-    args = pmb.parse.arguments()
-    pmb.helpers.logging.init(args)
-
-    # Verify package versions
-    print("checking changed package versions...")
-    check_versions(args, packages)

.ci/commits.sh

@@ -0,0 +1,20 @@
+#!/bin/sh -e
+# Copyright 2022 Oliver Smith
+# SPDX-License-Identifier: GPL-3.0-or-later
+# Description: check pkgver/pkgrel bumps, amount of changed pkgs etc
+# Options: native
+# Use 'native' because it requires git commit history.
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	# In .gitlab-ci.yml currently .ci/pytest.sh runs before this and
+	# already downloads and runs install_pmbootstrap.sh.
+	if ! [ -e install_pmbootstrap.sh ]; then
+		wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
+		sh ./install_pmbootstrap.sh
+	fi
+	exec su "${TESTUSER:-pmos}" -c "sh -e $0"
+fi
+
+.ci/lib/check_changed_aports_versions.py

.ci/ec.sh

@@ -0,0 +1,13 @@
+#!/bin/sh -e
+# Description: editorconfig-checker: lint for trailing whitespaces etc.
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	apk -q add \
+		editorconfig-checker
+	exec su "${TESTUSER:-build}" -c "sh -e $0"
+fi
+
+set -x
+ec

.ci/flake8.sh

@@ -0,0 +1,14 @@
+#!/bin/sh -e
+# Description: lint CI related python scripts/tests
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	apk -q add py3-flake8
+	exec su "${TESTUSER:-build}" -c "sh -e $0"
+fi
+
+set -x
+
+# shellcheck disable=SC2046
+flake8 --ignore E501,F401,E722,W504,W605 $(find .ci -name '*.py')

.ci/grep.sh

@@ -0,0 +1,24 @@
+#!/bin/sh -e
+# Description: check various bad patterns with grep
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	apk -q add grep
+	exec su "${TESTUSER:-build}" -c "sh -e $0"
+fi
+
+# Find CHANGEMEs in APKBUILDs
+if grep -qr '(CHANGEME!)' *; then
+	echo "ERROR: Please replace '(CHANGEME!)' in the following files:"
+	grep --color=always -r '(CHANGEME!)' *
+	exit 1
+fi
+
+# DTBs installed to /usr/share/db
+# FIXME: doesn't pass currently, fix it and enable it afterwards
+# if grep -r 'INSTALL_DTBS_PATH="$pkgdir"/usr/share/dtb'; then
+# 	echo 'Please do not install dtbs to /usr/share/dtb!'
+# 	echo 'Unless you have a good reason not to, please put them in /boot/dtbs'
+# 	exit 1
+# fi

.ci/kconfig.sh

@@ -0,0 +1,22 @@
+#!/bin/sh -e
+# Description: check all kernel configs with 'pmbootstrap kconfig check'
+# Options: native
+# Use 'native' because it requires running pmbootstrap.
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
+	sh ./install_pmbootstrap.sh
+	exec su "${TESTUSER:-pmos}" -c "sh -e $0"
+fi
+
+# Wrap pmbootstrap to use this repository for --aports
+pmaports="$(cd $(dirname $0)/..; pwd -P)"
+_pmbootstrap="$(command -v pmbootstrap)"
+pmbootstrap() {
+	"$_pmbootstrap" --aports="$pmaports" "$@"
+}
+
+set -x
+pmbootstrap kconfig check

.ci/lib/add_pmbootstrap_to_import_path

@@ -0,0 +1 @@
+../testcases/add_pmbootstrap_to_import_path

.ci/lib/build_changed_aports.py

@@ -7,7 +7,7 @@ import sys
 import common
 
 # pmbootstrap
-import testcases.add_pmbootstrap_to_import_path
+import add_pmbootstrap_to_import_path
 import pmb.parse
 import pmb.parse._apkbuild
 import pmb.helpers.pmaports

.ci/lib/build_changed_aports.sh

@@ -0,0 +1,19 @@
+#!/bin/sh -e
+# Description: build modified packages for this architecture
+# Options: native slow
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
+	sh ./install_pmbootstrap.sh
+	exec su "${TESTUSER:-pmos}" -c "sh -e $0"
+fi
+
+export PYTHONUNBUFFERED=1
+
+# Get the architecture from the symlink we are running
+arch="$(echo "$0" | cut -d '-' -f 2 | cut -d '.' -f 1)"
+
+set -x
+.ci/lib/build_changed_aports.py "$arch"

.ci/lib/check_changed_aports_versions.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+# Copyright 2022 Oliver Smith
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+import glob
+import logging
+import os
+import subprocess
+import sys
+import tempfile
+
+# Same dir
+import common
+
+# pmbootstrap
+import add_pmbootstrap_to_import_path  # noqa
+import pmb.parse
+import pmb.parse.version
+import pmb.helpers.logging
+
+
+def get_package_version_sources(args, package, revision, check=True):
+    """
+    Get the version ($pkgver-r$pkgrel), checksums and a list of remote files
+    (not bundled with the APKBUILD) from a specific git revision.
+    :param package: name of the package
+    :param revision: git revision to look at
+    :param check: complain if the file does not exist
+    :returns: {"checksums": see lib.common.parse_apkbuild_checksums(),
+               "pkgrel": 3,
+               "pkgver": "1.0.0",
+               "sources_remote": ["x-1.0.0-r2.tar.bz2", ...],
+               "version": "1.0.0-r2"}
+    """
+    # Redirect stderr to /dev/null, so git doesn't complain about files not
+    # existing in upstream branch for new packages
+    stderr = None
+    if not check:
+        stderr = subprocess.DEVNULL
+
+    # Run something like "git show upstream/master:main/hello-world/APKBUILD"
+    pmaports_dir = common.get_pmaports_dir()
+    pattern = pmaports_dir + "/**/" + package + "/APKBUILD"
+    path = glob.glob(pattern, recursive=True)[0][len(pmaports_dir + "/"):]
+    apkbuild_content = common.run_git(["show", revision + ":" + path], check,
+                                      stderr)
+    if not apkbuild_content:
+        return None
+
+    # Save APKBUILD to a temporary path and parse it from there. (Not the best
+    # way to do things, but good enough for this CI script.)
+    with tempfile.TemporaryDirectory() as tempdir:
+        apkbuild_path = f"{tempdir}/APKBUILD"
+        with open(apkbuild_path, "w", encoding="utf-8") as handle:
+            handle.write(apkbuild_content)
+        parsed = pmb.parse.apkbuild(apkbuild_path, False, False)
+        checksums = common.parse_apkbuild_checksums(args, apkbuild_path)
+
+    # Build sources_remote from files in checksums that are not bundled.
+    # To make the logic not overly complicated, only glob in the currently
+    # checked out directory. This is inaccurate for the previous commit, but
+    # good enough for how it's used in check_versions() below.
+    sources_remote = []
+    for filename, checksum in checksums.items():
+        bundled_path_pattern = f"{os.path.dirname(path)}/**/{filename}"
+        if not glob.glob(bundled_path_pattern, recursive=True):
+            sources_remote += filename
+
+    return {"checksums": checksums,
+            "pkgrel": int(parsed['pkgrel']),
+            "pkgver": parsed['pkgver'],
+            "sources_remote": sources_remote,
+            "version": f"{parsed['pkgver']}-r{parsed['pkgrel']}"}
+
+
+def version_compare_operator(result):
+    """ :param result: return value from pmb.parse.version.compare() """
+    if result == -1:
+        return "<"
+    elif result == 0:
+        return "=="
+    elif result == 1:
+        return ">"
+
+    raise RuntimeError("Unexpected version_compare_operator input: " + result)
+
+
+def exit_with_error_message():
+    branch = common.get_upstream_branch()
+    logging.error("")
+    logging.error("NOTE: use 'pmbootstrap ci commits' to run this check"
+                  " on your PC.")
+    logging.error("")
+    logging.error("Some more hints:")
+    logging.error(f"* You may need to rebase on {branch}.")
+    logging.error("* Detailed rules for bumping pkgrel/pkgver:")
+    logging.error("  https://postmarketos.org/howto-bump-pkgrel-pkgver")
+    logging.error("* If the error doesn't make sense (e.g. you want to")
+    logging.error("  intentionally downgrade a package), add [ci:skip-build]")
+    logging.error("  with reasoning to the last commit message.")
+    logging.error("")
+    logging.error("Thank you and sorry for the inconvenience.")
+    exit(1)
+
+
+def check_versions(args, packages):
+    error = False
+
+    # Get relevant commits: compare HEAD against upstream branch or HEAD~1
+    # (the latter if this CI check is running on upstream branch). Note that
+    # for the common.get_changed_files() code, we don't check against
+    # upstream branch HEAD, but against the latest common ancestor. This is not
+    # desired here, since we already know what packages changed, and really
+    # want to check if the version was increased towards *current* upstream
+    # branch HEAD.
+    commit = f"upstream/{common.get_upstream_branch()}"
+    if common.run_git(["rev-parse", "HEAD"]) == common.run_git(["rev-parse",
+                                                                commit]):
+        logging.info(f"NOTE: {commit} is on same commit as HEAD, comparing"
+                     " HEAD against HEAD~1.")
+        commit = "HEAD~1"
+
+    for package in packages:
+        logging.info("")
+        # Get versions and sources, skip new packages
+        head = get_package_version_sources(args, package, "HEAD")
+        head_v = head["version"]
+        upstream = get_package_version_sources(args, package, commit, False)
+        if not upstream:
+            if head["pkgrel"] != 0:
+                logging.error(f"- {package}: {head_v} (HEAD) (new package)")
+                logging.error("  ERROR: new package should have pkgrel=0")
+                error = True
+            else:
+                logging.info(f"- {package}: {head_v} (HEAD) (new package)")
+            continue
+        upstream_v = upstream["version"]
+
+        # Compare head and upstream versions based on rules from:
+        # https://postmarketos.org/howto-bump-pkgrel-pkgver
+        error_str = ""
+        if head["sources_remote"]:
+            if head["sources_remote"] == upstream["sources_remote"]:
+                pkgver = upstream["pkgver"]
+                pkgrel = upstream["pkgrel"] + 1
+                if head["pkgver"] != pkgver or head["pkgrel"] != pkgrel:
+                    error_str = f"expected pkgver={pkgver}, pkgrel={pkgrel}" \
+                                " (pkgver unchanged, pkgrel + 1), because" \
+                                " this package has unchanged remote sources"
+            else:  # remote sources changed
+                if upstream["pkgver"] == "9999":
+                    pkgver = "9999"
+                    pkgrel = upstream["pkgrel"] + 1
+                    if head["pkgver"] != pkgver or head["pkgrel"] != pkgrel:
+                        error_str = f"expected pkgver={pkgver}," \
+                                    f" pkgrel={pkgrel} (pkgver unchanged," \
+                                    " pkgrel + 1), because this package has" \
+                                    " pkgver=9999"
+                elif head["pkgver"] == upstream["pkgver"] or head["pkgrel"] != 0:
+                    error_str = "expected pkgver to change and pkgrel=0," \
+                                " because this package has changed remote" \
+                                " sources"
+        else:  # no remote sources
+            if head["checksums"] == upstream["checksums"]:
+                pkgver = upstream["pkgver"]
+                pkgrel = upstream["pkgrel"] + 1
+                if head["pkgver"] != pkgver or head["pkgrel"] != pkgrel:
+                    error_str = f"expected pkgver={pkgver}, pkgrel={pkgrel} " \
+                                " (pkgver unchanged, pkgrel + 1), because" \
+                                " this package has no remote sources and the" \
+                                " bundled sources didn't change"
+            else:  # checksums changed
+                if head["pkgver"] == upstream["pkgver"] or head["pkgrel"] != 0:
+                    error_str = "expected pkgver to change and pkgrel=0," \
+                                " because this package has no remote sources" \
+                                " but the bundled files changed"
+
+        result = pmb.parse.version.compare(head_v, upstream_v)
+        if result != 1:
+            error_str = "new version is not higher than the previous version"
+
+        # Print result line ("- hello-world: 1-r2 (HEAD) > 1-r1 (HEAD~1)")
+        formatstr = "- {}: {} (HEAD) {} {} ({})"
+        operator = version_compare_operator(result)
+        logging.error(formatstr.format(package, head_v, operator, upstream_v,
+                                       commit))
+
+        # Detailed error message should help the user to resolve it and explain
+        # why it failed.
+        if error_str:
+            logging.error(f"  ERROR: {error_str}")
+            error = True
+
+    if error:
+        exit_with_error_message()
+
+
+if __name__ == "__main__":
+    # Get and print modified packages
+    common.add_upstream_git_remote()
+    packages = common.get_changed_packages()
+    logging.info(f"Changed packages: {packages}")
+
+    # Verify modified package count
+    common.get_changed_packages_sanity_check(len(packages))
+    if len(packages) == 0:
+        logging.info("no aports changed in this branch")
+        exit(0)
+
+    # Potentially skip this check
+    if common.commit_message_has_string("[ci:skip-vercheck]"):
+        logging.warning("WARNING: not checking for changed package versions"
+                        " ([ci:skip-vercheck])!")
+        exit(0)
+
+    # Initialize args (so we can use pmbootstrap's APKBUILD parsing)
+    sys.argv = ["pmbootstrap.py", "chroot"]
+    args = pmb.parse.arguments()
+    pmb.helpers.logging.init(args)
+
+    # Verify package versions
+    logging.info("Checking changed package versions...")
+    check_versions(args, packages)

.ci/lib/check_devices_in_wiki.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# Copyright 2021 Oliver Smith
+# Copyright 2022 Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
 
 import argparse
@@ -13,7 +13,7 @@ def get_devices():
     """:returns: list of all devices"""
     ret = []
     pmaports = (os.path.realpath(os.path.join(os.path.dirname(__file__) +
-                "/..")))
+                "/../..")))
     for path in glob.glob(pmaports + "/device/*/device-*/"):
         device = os.path.dirname(path).split("device-", 1)[1]
 
@@ -110,7 +110,7 @@ def main():
         print("")
         print("Now it's time to add some documentation:")
         print("1) Create a device specific wiki page as described here:")
-        print("   <https://wiki.postmarketos.org/wiki/Help:Device_Page>")
+        print("   https://wiki.postmarketos.org/wiki/Help:Device_Page")
         print("2) Set 'booting = yes' in the infobox of your device page.")
         print("3) Run these tests again with an empty commit in your MR:")
         print("   $ git commit --allow-empty -m 'run tests again'")

.ci/lib/common.py

@@ -2,7 +2,8 @@
 # Copyright 2021 Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
 
-# Various functions used in CI scripts
+# Various functions used in CI scripts and tests. This file is in
+# .ci/lib/common.py and symlinked to .ci/testcases/common.py.
 
 import configparser
 import os
@@ -17,7 +18,7 @@ def get_pmaports_dir():
     global cache
     if "pmaports_dir" in cache:
         return cache["pmaports_dir"]
-    ret = os.path.realpath(os.path.join(os.path.dirname(__file__) + "/.."))
+    ret = os.path.realpath(os.path.join(os.path.dirname(__file__) + "/../.."))
     cache["pmaports_dir"] = ret
     return ret
 
@@ -190,3 +191,51 @@ def get_changed_packages():
         ret.add(os.path.basename(dirname))
 
     return ret
+
+
+def parse_apkbuild_checksums(args, apkbuild_path):
+    """
+    Read the APKBUILD file and parse source files from the checksums at the
+    bottom. This has always the same format, even if $source is built with hard
+    to parse shell code (like in postmarketos-base).
+
+    :param apkbuild_path: full path to the APKBUILD
+    :returns: dict of parsed "source" filenames and checksums, e.g.:
+              {"first.patch": "b4dc4f3…",
+               "second.patch": "b4dc4f3…"}
+    """
+    start = 'sha512sums="'
+    in_block = False
+    ret = {}
+
+    with open(apkbuild_path, encoding="utf-8") as handle:
+        for line in handle.readlines():
+            # Find start
+            if not in_block:
+                if line.startswith(start):
+                    in_block = True
+                else:
+                    continue
+
+            # sha512sums may have lines without checksums:
+            # https://gitlab.alpinelinux.org/alpine/abuild/-/merge_requests/73
+            if "  " not in line:
+                continue
+
+            try:
+                checksum, filename = line.rstrip().split("  ", 2)
+            except ValueError:
+                raise ValueError("Failed to parse checksums. Try to delete the"
+                                 " checksums and generate them again with"
+                                 f" 'pmbootstrap checksum': {apkbuild_path}")
+
+            # Cut off 'sha512sums="' if the first checksum is in that line
+            if checksum.startswith(start):
+                checksum = checksum[len(start):]
+
+            # Find end
+            if filename.endswith('"'):
+                filename = filename[:-1]
+
+            ret[filename] = checksum
+    return ret

.ci/lib/gitlab_prepare_ci.sh

@@ -0,0 +1,17 @@
+#!/bin/sh -e
+# Copyright 2022 Oliver Smith
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+if ! mount | grep -q /proc/sys/fs/binfmt_misc; then
+	echo "Mounting binfmt_misc"
+	mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
+fi
+
+if id "build" > /dev/null 2>&1; then
+	echo "User 'build' exists already"
+else
+	echo "Creating build user"
+	adduser -D build
+	chown -R build:build .
+	echo 'build ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
+fi

.ci/pytest.sh

@@ -1,10 +1,21 @@
 #!/bin/sh -e
-# Copyright 2021 Oliver Smith
+# Copyright 2022 Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
+# Description: lint with various python tests
+# Options: native
+# Use 'native' because it requires pmbootstrap.
+# https://postmarktos.org/pmb-ci
 
-# Require pmbootstrap
-if ! command -v pmbootstrap > /dev/null; then
-	echo "ERROR: pmbootstrap needs to be installed."
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
+	sh ./install_pmbootstrap.sh pytest
+	exec su "${TESTUSER:-pmos}" -c "sh -e $0"
+fi
+
+# Require pytest to be installed on the host system
+if [ -z "$(command -v pytest)" ]; then
+	echo "ERROR: pytest command not found, make sure it is in your PATH."
 	exit 1
 fi
 

.ci/shellcheck.sh

@@ -1,16 +1,15 @@
 #!/bin/sh -e
-# Copyright 2021 Oliver Smith
+# Copyright 2022 Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
+# Description: lint all shell scripts
+# https://postmarktos.org/pmb-ci
 
-set -e
-DIR="$(cd "$(dirname "$0")" && pwd -P)"
-cd "$DIR/.."
+DIR="$(cd "$(dirname "$0")/.." && pwd -P)"
 
-# Find CHANGEMEs in APKBUILDs
-if grep -qr '(CHANGEME!)' *; then
-	echo "ERROR: Please replace '(CHANGEME!)' in the following files:"
-	grep --color=always -r '(CHANGEME!)' *
-	exit 1
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	apk -q add shellcheck
+	exec su "${TESTUSER:-build}" -c "sh -e $0"
 fi
 
 # Shell: shellcheck
@@ -44,8 +43,9 @@ sh_files="
 
 	$(find . -path '.ci/*.sh')
 "
+
 for file in $sh_files; do
 	echo "Test with shellcheck: $file"
-	cd "$DIR/../$(dirname "$file")"
+	cd "$DIR/$(dirname "$file")"
 	shellcheck -e SC1008 -x "$(basename "$file")"
 done

.ci/testcases/common.py

@@ -0,0 +1 @@
+../lib/common.py

.ci/testcases/test_source.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+# Copyright 2022 Oliver Smith
+# SPDX-License-Identifier: GPL-3.0-or-later
+# Various checks on source= in the APKBUILDs
+
+import glob
+import logging
+import os
+import pytest
+import sys
+
+# Same dir
+import common
+
+# pmbootstrap
+import add_pmbootstrap_to_import_path
+import pmb.parse
+import pmb.parse._apkbuild
+import pmb.parse.apkindex
+import pmb.helpers.repo
+
+
+def test_aports_unreferenced_files(args):
+    """
+    Raise an error if an unreferenced file is found
+    """
+    for apkbuild_path in glob.iglob(args.aports + "/**/APKBUILD", recursive=True):
+        # pmbootstrap parser has some issues with complicated APKBUILDs, skip those.
+        if apkbuild_path.startswith(args.aports + "/cross/"):
+            continue
+
+        apkbuild = pmb.parse.apkbuild(apkbuild_path)
+        sources_chk = common.parse_apkbuild_checksums(args, apkbuild_path)
+
+        # Collect files from subpackages
+        subpackage_installs = []
+        subpackage_triggers = []
+        if apkbuild["subpackages"]:
+            for subpackage in apkbuild["subpackages"].values():
+                if not subpackage:
+                    continue
+                subpackage_installs += subpackage.get("install", [])
+                subpackage_triggers += subpackage.get("triggers", [])
+
+        # Collect trigger files
+        trigger_sources = []
+        for trigger in apkbuild["triggers"] + subpackage_triggers:
+            trigger_sources.append(trigger.split("=")[0])
+
+        dirname = os.path.dirname(apkbuild_path)
+        for file in glob.iglob(dirname + "/**", recursive=True):
+            rel_file_path = os.path.relpath(file, dirname)
+            # Skip APKBUILDs and directories
+            if rel_file_path == "APKBUILD" or os.path.isdir(file):
+                continue
+
+            if os.path.basename(rel_file_path) not in sources_chk \
+                    and rel_file_path not in apkbuild["install"] \
+                    and rel_file_path not in subpackage_installs \
+                    and rel_file_path not in trigger_sources:
+                raise RuntimeError(f"{apkbuild_path}: found unreferenced file: {rel_file_path}")
+
+
+def test_distfiles_conflict(args):
+    """
+    Make sure that each filename mentioned in any source= of any APKBUILD
+    always has the same checksum. This is important because apk caches
+    downloaded source files in a flat distfiles directory. So if two APKBUILDs
+    both download a file with the same filename but different checksum, and the
+    user builds both after each other, abuild will fail on the second build
+    with a checksum error.
+    """
+    source_all = {}
+    for apkbuild_path in glob.iglob(f"{args.aports}/**/APKBUILD", recursive=True):
+        source = common.parse_apkbuild_checksums(args, apkbuild_path)
+        dir_path = os.path.dirname(apkbuild_path)
+        apkbuild_rel = os.path.relpath(apkbuild_path, args.aports)
+        for filename, checksum in source.items():
+            # Files bundled with the APKBUILD don't get copied to the distfiles
+            # cache, so not relevant for this check. Use glob.glob here and not
+            # iglob, because we don't want an iterator.
+            if glob.glob(f"{dir_path}/**/{filename}", recursive=True):
+                continue
+
+            # First time seeing this file
+            if filename not in source_all:
+                source_all[filename] = {"checksum": checksum,
+                                        "apkbuild_rel": apkbuild_rel}
+                continue
+
+            # Saw this file already with same checksum
+            if checksum == source_all[filename]["checksum"]:
+                continue
+
+            # Saw this file already with different checksum
+            logging.error("")
+            logging.error(f"ERROR: the source file '{filename}' has different"
+                          " checksums in the following files:")
+            logging.error(f"- {source_all[filename]['apkbuild_rel']}:")
+            logging.error(f"  {source_all[filename]['checksum']}")
+            logging.error(f"- {apkbuild_rel}:")
+            logging.error(f"  {checksum}")
+            logging.error("")
+            logging.error("Fix this by setting a different target filename in"
+                          " the package you modified:")
+            logging.error("https://wiki.alpinelinux.org/wiki/APKBUILD_Reference#source")
+            logging.error("")
+            raise RuntimeError(f"Conflict with source file '{filename}'")

.ci/testcases/test_unreferenced_files.py

@@ -1,99 +0,0 @@
-#!/usr/bin/env python3
-# Copyright 2021 Oliver Smith
-# SPDX-License-Identifier: GPL-3.0-or-later
-
-import glob
-import pytest
-import sys
-import os
-
-import add_pmbootstrap_to_import_path
-import pmb.parse
-import pmb.parse._apkbuild
-import pmb.parse.apkindex
-import pmb.helpers.repo
-
-
-def parse_source_from_checksums(args, apkbuild_path):
-    """
-    Read the APKBUILD file and parse source files from the checksums at the
-    bottom. This has always the same format, even if $source is built with hard
-    to parse shell code (like in postmarketos-base).
-
-    :param apkbuild_path: full path to the APKBUILD
-    :returns: list of parsed "source" files, e.g.:
-              ["first.patch", "second.patch"]
-    """
-    start = 'sha512sums="'
-    in_block = False
-    ret = []
-
-    with open(apkbuild_path, encoding="utf-8") as handle:
-        for line in handle.readlines():
-            # Find start
-            if not in_block:
-                if line.startswith(start):
-                    in_block = True
-                else:
-                    continue
-
-            # sha512sums may have lines without checksums:
-            # https://gitlab.alpinelinux.org/alpine/abuild/-/merge_requests/73
-            if "  " not in line:
-                continue
-
-            try:
-                _, filename = line.rstrip().split("  ", 2)
-            except ValueError:
-                raise ValueError("Failed to parse checksums. Try to delete the"
-                                 " checksums and generate them again with"
-                                 f" 'pmbootstrap checksum': {apkbuild_path}")
-
-            # Find end
-            if filename.endswith('"'):
-                ret += [filename[:-1]]
-                break
-
-            ret += [filename]
-    return ret
-
-
-def test_aports_unreferenced_files(args):
-    """
-    Raise an error if an unreferenced file is found
-    """
-    for apkbuild_path in glob.iglob(args.aports + "/**/APKBUILD", recursive=True):
-        # pmbootstrap parser has some issues with complicated APKBUILDs, skip those.
-        if apkbuild_path.startswith(args.aports + "/cross/"):
-            continue
-
-        apkbuild = pmb.parse.apkbuild(apkbuild_path)
-        sources_chk = parse_source_from_checksums(args, apkbuild_path)
-
-        # Collect files from subpackages
-        subpackage_installs = []
-        subpackage_triggers = []
-        if apkbuild["subpackages"]:
-            for subpackage in apkbuild["subpackages"].values():
-                if not subpackage:
-                    continue
-                subpackage_installs += subpackage.get("install", [])
-                subpackage_triggers += subpackage.get("triggers", [])
-
-        # Collect trigger files
-        trigger_sources = []
-        for trigger in apkbuild["triggers"] + subpackage_triggers:
-            trigger_sources.append(trigger.split("=")[0])
-
-        dirname = os.path.dirname(apkbuild_path)
-        for file in glob.iglob(dirname + "/**", recursive=True):
-            rel_file_path = os.path.relpath(file, dirname)
-            # Skip APKBUILDs and directories
-            if rel_file_path == "APKBUILD" or os.path.isdir(file):
-                continue
-
-            if os.path.basename(rel_file_path) not in sources_chk \
-                    and rel_file_path not in apkbuild["install"] \
-                    and rel_file_path not in subpackage_installs \
-                    and rel_file_path not in trigger_sources:
-                raise RuntimeError(f"{apkbuild_path}: found unreferenced file: {rel_file_path}")

.ci/wiki.sh

@@ -0,0 +1,15 @@
+#!/bin/sh -e
+# Description: verify devices are documented in the wiki
+# Options: native
+# Run natively so we don't need to set up a chroot, python3 is the only dep.
+# https://postmarktos.org/pmb-ci
+
+if [ "$(id -u)" = 0 ]; then
+	set -x
+	apk -q add \
+		python3
+	exec su "${TESTUSER:-build}" -c "sh -e $0"
+fi
+
+set -x
+.ci/lib/check_devices_in_wiki.py --booting

.gitignore

@@ -20,7 +20,6 @@ dist/
 downloads/
 eggs/
 .eggs/
-lib/
 lib64/
 parts/
 sdist/

.gitlab-ci.yml

@@ -3,7 +3,7 @@
 # global settings
 image: alpine:latest
 after_script:
-  - .ci/move_logs.sh $CI_PROJECT_DIR
+  - .ci/lib/move_logs.sh $CI_PROJECT_DIR
 stages:
   - lint
   - build
@@ -31,13 +31,12 @@ wiki:
       - merge_requests
     changes:
       - device/*/device-*/*
-  before_script:
-    - apk -q add python3
   script:
-    - .ci/check_devices_in_wiki.py --booting
+    - .ci/lib/gitlab_prepare_ci.sh
+    - .ci/wiki.sh
 
 # testcases linting
-testcases-linting:
+flake8:
   stage: lint
   <<: *only-default
   only:
@@ -49,38 +48,35 @@ testcases-linting:
   before_script:
     - apk -q add py-flake8
   script:
-    - flake8 --ignore E501,F401,E722,W504,W605
+    - .ci/lib/gitlab_prepare_ci.sh
+    - .ci/flake8.sh
 
-# shellcheck
-shellcheck:
+# shellcheck and various grep checks
+shellcheck-grep:
   stage: lint
   <<: *only-default
   image: alpine:edge
-  before_script:
-    - apk -q add shellcheck
   script:
+    - .ci/lib/gitlab_prepare_ci.sh
     - .ci/shellcheck.sh
+    - .ci/grep.sh
 
 editor-config:
   stage: lint
   <<: *only-default
   image: alpine:edge
-  before_script:
-    - apk -q add editorconfig-checker
   script:
-    - ec
+    - .ci/lib/gitlab_prepare_ci.sh
+    - .ci/ec.sh
 
 # aports checks (generic)
-aports-static:
+pytest-commits:
   stage: lint
   <<: *only-default
-  before_script:
-    - wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
-    - sh ./install_pmbootstrap.sh pytest
   script:
-    - su pmos -c ".ci/run_testcases.sh"
-    - su pmos -c ".ci/check_changed_aports_versions.py"
-    - su pmos -c ".ci/check-dtb-install-location.sh"
+    - .ci/lib/gitlab_prepare_ci.sh
+    - .ci/pytest.sh
+    - .ci/commits.sh
   artifacts:
     when: on_failure
     paths:
@@ -89,26 +85,12 @@ aports-static:
       - pmbootstrap.cfg
     expire_in: 1 week
 
-distfile-check:
-  stage: lint
-  <<: *only-default
-  before_script:
-    - apk -q add make findutils
-    - wget "https://gitlab.com/postmarketOS/ci-common/-/archive/master/ci-common-master.tar.bz2"
-    - tar -xf ci-common-master.tar.bz2
-    - mv ci-common-master/distfile-check/*.sh ci-common-master/distfile-check/Makefile .
-    - rm -r ci-common-master ci-common-master.tar.bz2
-  script:
-    - make -j999
-
 # APKBUILD linting
 aport-lint:
   stage: lint
-  before_script:
-    - wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
-    - sh ./install_pmbootstrap.sh
   script:
-    - su pmos -c ".ci/apkbuild-linting.py"
+    - .ci/lib/gitlab_prepare_ci.sh
+    - .ci/apkbuild-lint.sh
   only:
     - merge_requests
   allow_failure: true
@@ -123,11 +105,9 @@ kernel-kconfig:
     changes:
       - device/*/linux-*/config-*
       - main/linux-*/config-*
-  before_script:
-    - wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
-    - sh ./install_pmbootstrap.sh pytest
   script:
-    - su pmos -c "pmbootstrap kconfig check"
+    - .ci/lib/gitlab_prepare_ci.sh
+    - .ci/kconfig.sh
 
 # MR settings
 # (Checks for "Allow commits from members who can merge to the target branch")
@@ -146,10 +126,8 @@ mr-settings:
   stage: build
   <<: *only-default
   before_script:
-    - wget "https://gitlab.com/postmarketOS/ci-common/-/raw/master/install_pmbootstrap.sh"
-    - sh ./install_pmbootstrap.sh
-  script:
-    - .ci/build.sh
+    - .ci/lib/gitlab_prepare_ci.sh
+  after_script:
     - cp -r /home/pmos/.local/var/pmbootstrap/packages/ packages/ || true
   artifacts:
     expire_in: 1 week
@@ -159,18 +137,30 @@ mr-settings:
 
 build-x86_64:
   extends: .build
+  script:
+    - .ci/build-x86_64.sh
 
 build-x86:
   extends: .build
+  script:
+    - .ci/build-x86.sh
 
 build-aarch64:
   extends: .build
+  script:
+    - .ci/build-aarch64.sh
 
 build-armv7:
   extends: .build
+  script:
+    - .ci/build-armv7.sh
 
 build-armhf:
   extends: .build
+  script:
+    - .ci/build-armhf.sh
 
 build-riscv64:
   extends: .build
+  script:
+    - .ci/build-riscv64.sh