The __flip_dep_graph(graph) call sits inside the while loop and performs redundant graph flipping on every iteration. Hoist it outside to compute once and reuse the result.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Remove App.get_section() which parses raw file sections by scanning for section headers and accumulating lines. This method is no longer needed since ProjectConf now handles all config file parsing.
Replace the @functools.lru_cache(maxsize=None) decorator with @functools.cache throughout App. functools.cache is a shorthand for functools.lru_cache(maxsize=None) introduced in Python 3.9 and is more concise and readable with identical behaviour.
The global --topdir-format make:XXX option to jw-pkg is half-baked at best, and __find_dir() ignores it entirely. Make __find_dir() return some Makefile-syntax-formatted output if the option is present. Not used anywhere, currently, and, hence, badly tested, but still better than the situation before.
The global --topdir-format option governs how a project's root directory is represented in paths output by various queries. "absolute" means as absolute path, "unaltered" means verbatim as specified via --topdir, make:xyz means replaced by the string $(xyz), for later expansion in a makefile variable.
This commit adds another variant: "relative" yields the shortest possible output format of the output path in question relative to --topdir, with "shortest possible" in this context meaning canonicalized and leading "./" stripped.
App.get_projects_refs() is a versatile tool, but what it does isn't obvious. Use the simpler method .get_value() instead for get_libname(), and return None if a project doesn't provide a linkable library.
- Rename variable dep and deps to val and vals, respectively,
because that's more what they are values of key-value pairs. In
some cases that can represent dependencies, in some case other
things.
- Make a scope case distinction a little clearer by mentioning all
possible cases in a match / case block
Allow find_dir() to return None in case it couldn't find a directory, that's a legal outcome. Add a boolean parameter "throw" to support throwing an exception if the existence needs to be asserted.
It would probably be nicer for the type checkers to split this up into a throwing and non-throwing function. Postponed.
App.is_excluded_from_build() uses the wrong function entirely to query the [build.exclude] section of project.conf (App.get_project_refs() instead of App.get_value()). This has obviously never worked. It rose to prominence because commit 6db73873 introduced App.__proj_dir(), which now raises an Exception if passed garbage, which in turn surfaces as
Exception: No project path found for module "debian"
Use the correct function for that: App.get_value().
The previous commits have put rules for linting and formatting via ruff, yapf, mypy and pyright into place. They are checked with the make check target, and this commit adds the fixes for the target to succeed.
It does some refactoring where type checking dug up dirty bits, and also adds lots of churn in the Python code. To a good deal, that's owed to mere formatting changes. It would have been better to seperate those from syntax and refactoring fixes into multiple commits, so that the interesting changes don't drown in the formatting nose. However, that would have been a lot of additional work only to be thrown away by later commits, hence this commit has a big diff in one piece. The size of the diff is regrettable but hopefully a one-off: What it buys is automatic format checking for CI and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
- ruff check .
- mypy .
- pyright
- Format check:
- yapf --diff --recursive .
The refactoring includes:
- Turn the Result class into a more elaborate object, capable of
doing more heavy lifting around stderr and stdout decoding,
summarizing outcome, and matching error strings.
Aside from fixing broken type checks, this also removes lots of
boilerplate calling code which is currently used for handling
possible call outcome scenarios. Trying to access an inexistent,
decoded string should raise a meaningful exception by itself now,
which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
- Add the AbstractCmd class above Cmd. This is necessary because
the checker rightfully complains it can't instantiate a Cmd
instance where constructor arguments were needed. They never
were, but the type used at the instantiating code's location in
jw.pkg.App so claims.
- Lots of sub- and sub-subcommands are derived from the base
class of the invoking command. That provides some properties
shared across the ancestor hierarchy of a command, but is
semantically unsound. Fix that by introducing jw.pkg.BaseCmd
class as a place to provide basic helpers shared across all
commands used in a jw.pkg.App's context, and derive all command
classes from that afresh. The parent command is still reachable
via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
The line was needed by Python 2 which is not supported anylonger.
For Python 3, the default encoding is UTF-8, anyway.
- Allow to run "make py-format" without having it produce any
changes. It's basically "yapf --in-place --recursive ." with some
code style settings, see conf/topdir/pyproject.toml. The settings
may be debatable. I've had custom tweaks in place on that target,
too, but then again, IDEs would have more hassle to integrate
that.
- Introduce a 88 character line length limit
- One import per line, reshuffle them semantically, see
[tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
if TYPE_CHECKING
- Spaces around assignments accounts for much churn. Having having
no spaces in inline parameter list assignments and default
parameter values would arguably be more compact where it's
useful. On the other hand, I have not found a code formatter
which allows spaces around assignments in parameter lists broken
into one per line and that's often better than a wall of text.
- Add two spaces before # export, as this seems to be mandated by
PEP-8
--uri is unnecessarily generic in that it could mean the URI of anything. --target makes it clearer that operations are to be exectuted on that target.
Add global --pkg-filter argument, defaulting to JW_DEFAULT_PKG_FILTER. If it's specified, instantiate a PackageFilterString from it, and initialize App's Distro instance with it.
Commit a19679fec reverted the first attempt to make AsyncSSH reuse one connection during an instance lifetime. That failed because a lot of distribution-specific properties were filled in a new event loop thread started by AsyncRunner, and AsyncSSH didn't like that.
This commit is the first part of the solution: Move those properties from the App class to the Distro class, and load the Distro class in an async loader. As soon as it's instantiated, it can provide all its properties without cluttering the code with async keywords.
App.distro_info() accepts and returns str instances, interpret anything passed as fmt parameter which is not a str as iterable, and return lists of expanded strings in that case.
For now, move the definiions of Result, Input and InputMode from ExecContext into lib.base. Having to import them from the ExecContect module is too heavy-handed for those simple types.
Python's platform.system() outputs 'Linux', and to use it is tempting. Sadly, that's wrong, because it reflects the host's idea of the target system, not the execution context's, so replace it with straight 'linux' if the distro is known, or, failing that, the output of uname -s.
Even with --interactive=[true|auto], there's no point in trying to read /etc/os-release interactively, so don't do that. Most notably, this commit keeps the property method from spilling /etc/os-release's content over the terminal.
ExecContext's .sudo() omits many of run()'s parameters, and this commit adds them. To avoid redundancy around repeating and massaging the long parameter list of both functions and their return values, it also adds some deeper changes:
- Make run(), _run(), sudo() and _sudo() always return instances of
Result. Before it was allowed to return a triplet of stdout,
stderr, and exit status.
- Have ExecContext stay out of the business of decoding the result
entirely. Result provides a convenience method .decode()
operating on stdout and stderr and leaves the decision to the
caller.
This entails miniscule adaptations in calling code, namely in
App.os_release, util.get_profile_env() and CmdListRepos._run().
- Wrap the _run() and _sudo() callbacks in a context manager object
of type CallContext to avoid code duplication.
- Consistently name the first argument to run(), _run(), sudo() and
_sudo() "cmd", not "args". The latter suggests that the caller is
omitting the executable, which is not the case.
Don't open and parse /etc/os-release with Python built-in functions. Spawn "cat /etc/os-release" as a subprocess and capture the output for parsing instead. The obvious advantage is that this also works with a remote shell.
DistroBase's option --id is now redundant to the new global option --distro-id in the App class, so remove --id. The only added value DistroBase then brings to the table is its .distro property, which can be provided by App just fine at this point, given that App has all it needs to construct a Distro object, so add .distro to App and remove the entire DistroBase class.
For convenience, also make App.distro available as a newly added cmds.Cmd.distro property. This also obviates the need for the distro-related properties in the .distro.Cmd class, remove all that.
Add the --verbose global option, which is made available as the App.verbose property.
Some functions still take a verbose parameter, but the type of these parameters is converted from bool to bool|None. The idea is that, if they are None, their verbosity falls back to the global default.
The code below lib.distro, as left behind by the previous commit, is geared towards being directly used as a command-line API. This commit introduces the abstract base class Distro, a proxy for distribution-specific interactions. The proxy abstracts distro specifics into an API with proper method prototypes, not argparse.Namespace contents, and can thus be more easily driven by arbitrary code.
The Distro class is initialized with a member variable of type ExecContext, another new class introduced by this commit. It is designed to abstract the communication channel to the distribution instance. Currently only one specialization exists, Local, which interacts with the distribution and root file system it is running in, but is planned to be subclassed to support interaction via SSH, serial, chroot, or chains thereof.
get-os.sh returned "suse" for SuSE-like distros, and that seems more appropriate since SLES is not OpenSUSE but should share and ID with other SuSE variants.
Add more fields to the OS cascade returned by App.os_cascade, based on the ID field in /etc/os-release. This includes some new ones, prefixed by pkg-, revealing which package format is used.
Major - but not yet sufficient - code beautification starting from jw.pkg.App.
- Make more methods private
- Rename methods to be more self-explanatory
- Same for method arguments, notably clean up some inconsistent
uses of "module" vs "project"
- Add more type hints
ResultCache is a home-grown result cache. The @lru_cache decorator, now available in Python 3, accomplishes the same thing, so try to ditch ResultCache for it.
Sadly, this doesn't entirely work as of now, because it uses hash() to hash the arguments, which won't work for the two list-type arguments to add_modules_from_project_txt() (buf and visited).
