This commit makes them use spaces instead, so they can be more easily amended by Makefiles using them. Also define them in a more uniform way, and use the newly introduced PREREQ_RUN variable to fill them, which in turn can also be appended to before that.
At present, the PREREQ-variable is effectively only used to detect if prerequiste packages haven't run "make all" before make is run in a given package. Also, it's only useful in $(TOPDIR). This commit splits the variable up into PREREQ_BUILD and PREREQ_RUN, and makes the variables available in every Makefile of a package by placing them in defs.mk instead of topdir.mk.
This also fixes a problem that PREREQ was cached before being filled, hence empty. Which effectively wasn't much of a problem, because it was basically unused, but still.
make/Makefile is responsible to generate $(TOPDIR)/cache-projects.mk. The variables are taken from .cache-project.mk, with some variables intentionally omitted, but their ifndef / endif blocks remain in place. Not harmful but ugly. Make sed range-delete the left-over blocks entirely.
Aside from PYTHONPATH, ldlibpath.mk runs jw-pkg.py for determining other paths, too, which is often unneeded and can impact performance. Split the PYTHONPATH detection into a dedicated py-path.mk, and include it from ldlibpath.mk, so it can be used instead where needed.
Add support for PY_INIT_SUBMODULES to py-mod.mk. If it is defined in a Makefile including py-mod.mk, the listed submodules will be added to __init__.py and thus included in the list of things that can be imported from a module.
This commit also adds support for --submodules to python-tools.sh for that to happen.
Letting python-tools.sh rewrite symbols is more robust than rewriting an entire __init__.py with PY_INIT_FILTER in the including Makefile. The latter can break in non-obvious ways if python-tools.sh changes __init__.py's format.
Make python-tools.sh support --symbol-filter to remedy that. The option takes an sed script which should expect a string of two non-whitespace tokens: The module from which the symbol is imported, and the name of the symbol in that module. It's output will then be used as the symbol to be exported from __init__.py.
Also, support the PY_SYMBOL_FILTER variable in py-mod.mk. If it's defined, it is used for --symbol-filter.
Make use of the newly introduced --prefix option to the pythonpath command, and generate what's subseqently used to fill in mypy_path in pyproject.toml.
By decoupling it from PYTHONPATH, this commit makes the creation of mypy_path less involved and easier to understand. It also obviates the need replace the relatively heavy ldlibpath.mk by the relatively lightweight projects.mk, thereby enhancing performance.
pyproject.toml is currently copied unchanged from conf/topdir to the toplevel directory. Set up machinery in py-topdir.mk to render it from a template in conf/templates instead, replacing {mypypath} in the process.
Replace variable PY_SRC_ROOT by PY_CHECK_ROOTS. The name PY_SRC_ROOT was a bad choice, given that it isn't immediately obvious that it a) can contain multiple root locations to be checked, and that it b) specifically concerns static type checking.
As of this commit, it's possible to limit the type checking scope with PY_CHECK_ROOTS as in
PY_CHECK_ROOTS="src/python/jw/pkg/CmdBase.py src/python/jw/lib" \
make check
Running pyright in a minimal docker container gives this error:
$ pyright
/usr/bin/npm-default: No such file or directory
Traceback (most recent call last):
File "/usr/bin/pyright-3.13", line 6, in <module>
sys.exit(entrypoint())
~~~~~~~~~~^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 31, in entrypoint
sys.exit(main(sys.argv[1:]))
~~~~^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 18, in main
return run(*args, **kwargs).returncode
~~~^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 22, in run
pkg_dir = install_pyright(args, quiet=None)
File "/usr/lib/python3.13/site-packages/pyright/_utils.py", line 69, in install_pyright
node.run(
~~~~~~~~^
'npm',
^^^^^^
...<5 lines>...
stderr=subprocess.PIPE if silent else sys.stderr,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/lib/python3.13/site-packages/pyright/node.py", line 144, in run
subprocess.run(node_args, **kwargs),
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/npm', 'install', \
'pyright@1.1.409']' returned non-zero exit status 255.
This means that on openSUSE, python3-pyright tries to pull in packages from the NPM registry. This increases the CI supply chain attack surface inacceptably, so remove pyright from the release prerequisites. That should be enough to remove it from the prerequisites of target check as well and allow it to succeed.
The pyright check machinery itself remains useful, so keep it in place for developers who install python3-pyright manually.
By the time projects-dir.mk is used during onboarding, it's already cloned, and so is jw-pkg in all its glory. So better use a ssh-wrapper.sh directly under jw-pkg's version control instead of plainly generating one with echo some-script-logic > ssh-wrapper.sh.
This has the main benefit of allowing a more elaborate script. The one added by this commit removes "-l user" from remotes which have a standard-user@gitserver form, typically because they differentiate users via their SSH pubkeys only, and which would deny access if both -l user and standard-user@ were specified.
ssh-wrapper.sh still needs to be a target which is updated by a recipe, because the version found in jw-pkg can't be trusted to be executable during bootstrapping, because "make all" has not run, yet.
The git-get-pub does not have the same effect as the other git-get-% targets, and this commit makes it.
The other git-get-% targets run pgit.sh, which rebases the current branch onto the fetched branch, and git-get-pub doesn't. Since devops merges contributor forges fast-forward without a merge-commit, the pub remote's master needs to be the last to be rebased on, because otherwise it will not allow to force-push the result.
As soon as multiple forges with protected master branches contribute, fast-forward merging of the master branch will need to be abolished anyway, and the release machinery will need an overhaul.
pkg.requires.os.release = python3-pyright breaks CI on Kali Linux. It is present in the janware repos, but using those would cross a line: jw-pkg must be buildable from the base repositories alone, so don't make pyright mandatory for Debian, because that pulls it in for Kali, too.
Ironically, the Debian repo provides it. Which makes it obvious that we will need another entry in the os cascade for Debian proper to allow pulling in such packages on Debian.
Target all should create all necessary files in topdir. Currently they're only needed for static file checks, but they might well be prerequistes for the build to succeed in the future, so make target all depend on topdir.
Also, place target all before the block of includes, so that the execution order is defined in topdir.mk rather than the included snippets.
This file can be included from toplevel Makefiles of projects which are part of a jw-pkg project tree but don't want to provide necessary targets themselves. The targets in this makefile simply do nothing. Feel free to add functionality to whatever target seems useful to you.
make topdir doesn't reliably regenerate pyrightconfig.json because of .SECONDEXPANSION. Adding a second dollar symbol $$(TD_GENERATE_FILES) solves the problem.
Currently, only mypy is run from py-check-syntax, but the targets are meant as entry point for further linter jobs in the future.
There's also
py-format-assignments:
which is meant to add spaces around "=" in multi-line assignment blocks, but, pending future experiments, I thing that target is going to be removed again. Left in for now.
Add py-ns-dir.mk. It's first intended use is inside the $(TOPDIR)/src/python/jw directories of each package. These are to be treated specially, because they contribute to the same namespace: jw.
This is true when installed, which means that no __init__.py should be packaged from that directory, see the guide linked below.
And it's also true when not installed, in which case there has to be a path-extending __init__.py in the directory. Normally, it should work without __init__.py in that case, as well, and it does for running Python code with PYTHONPATH pointing to the respective package roots. However, pyright doesn't seem to pull in code scattered over multiple locations without an old-style namespace path-extender, so we're likely going to add (but not install) an __init__.py there.
Not sure if it's going to be auto-generated py py-ns-dir.mk or committed to version control, we'll see with more testing.
Here's where I have my info from, aside from experiments.
If $(wildcard py.typed) is found in a python module directory, install it. py.typed should be used by every repo that declares properly typed code, for jw-pkg that would be at the jw.pkg node, i.e installed to <site-dir>/jw/jw-pkg/py.typed.
Add generic machinery to dynamically create files in $(TOPDIR). The need arises because version controlled configuration files for linters are going to be introduced.
For that, this commit introduces a variable $(TD_GENERATE_FILES), which target all depends on, and which topdir.clean removes.
It defaults to another variable also introduced by this commit, $(TD_COPY_FILES), which in turn defaults to $(TOPDIR)/conf/topdir.
This commit also adds support for JW_PKG_TOPDIR_COPY_PATH. It supports a PATH-style syntax, which allows pointing to multiple directories to be checked for source files. If they exist, they will be appended to the files found in $(TOPDIR)/conf/topdir after copying. Defining arbitray files to copy is not supported before security implications during CI runs are better understood.
Having the copy prerequisites work comes at the cost of having to add .SECONDEXPANSION. Since it's limited to the toplevel Makefile, I suppose that's acceptable.
The __init__.py files as gnerated by python-tools.sh contain multiple issues, fix them:
- Make the machinery fail if the same type name is imported from
different modules
- Support relative imports from .Module import Module instead of
having to use the entire module path as import source
- Import types explicitly re-exported with "as":
from .Module import Module as Module
Otherwise ruff will regard the type as "imported but not used"
- Add "# ruff: noqa: E501" near the top. The import lines can get
long and are beyond manual control (except for renaming the
modules themselves, that is). This can cause ruff to fail, so get
it to accept long lines in __init__.py. The style violation
doesn't make much of a difference in generated code, anyway,
because nobody reads that. Plus what's happening in the code
isn't rocket science, so good style wouldn't help much with
understanding, either.
This promptly digs up two symbol name conflicts lib.pm.dpkg and lib.pm.rpm. Fix them along with this commit to keep it from breaking the build.
Remove JW_PKG_EXTRA_SSH_OPTS before git pull in the context of get-pub / git-get-pub because it contains -l username, which collides with Forge-style git@<forge> remote URIs.
Add get-pub for top directory / projects directory level. This is a janware specific target needed for CI. It integrates the current master branch from the new Forgejo-based Git repos.
That said, this will likely need to go along with other release machinery. Packaging is good, but releasing over a bunch of directories is an intricate process, as-is only usable by janware itself, and doesn't need to be part of jw-pkg.
If a project is not initialized via make pkg-init-%, it doesn't contain a VERSION file. When CI tries to build and package such a project, it auto-creates a VERSION file, but a broken one: The revision isn't properly seperated by a dash but by a dot, which makes CI give up while parsing it.
Add a new target git-show-ahead-of-master, which does what git-show-pushable did up to now: List all repos where the currently checked-out branch is ahead of origin/master.
git-show-pushable now shows repos that have their checked-out branch ahead of their respective upstream branches. Which is truer to what the target's name suggests.
Rename command "distro" to "pkg" together with "info", its last remaining subcommand. "distro" is often used in the sense of "Linux distribution", which would be too narrow for the targets jw-pkg could theoretically support.
With the exception of the "info" subcommand, nearly all of distro's subcommands deal with package managing, so push them into their own command category.
pgit.sh get is not invoked with --create-remote-user-repos by default. For the janware Git servers, this makes CI fail over new repos: Repos from the maintainers are pulled, but before pkg-rebuild-reinstall, repos that the devops user has no remote repo for are purged again and are subseqently missing during build.
Passing --create-remote-user-repos to all pgit.sh get invocations should fix the problem, so do that.
CmdCanonicalizeRemotes / canonicalize-remotes and the respective target in topdir.mk remove the /srv/git portion from all remotes' URLs pointing to git.janware.com.
