janware/jw-pkg
1
Code Pull requests Activity Actions

jw.pkg: Fix "make check" static code check fallout

Browse source 
The previous commits have put rules for linting and formatting via ruff, yapf, mypy and pyright into place. They are checked with the make check target, and this commit adds the fixes for the target to succeed.
It does some refactoring where type checking dug up dirty bits, and also adds lots of churn in the Python code. To a good deal, that's owed to mere formatting changes. It would have been better to seperate those from syntax and refactoring fixes into multiple commits, so that the interesting changes don't drown in the formatting nose. However, that would have been a lot of additional work only to be thrown away by later commits, hence this commit has a big diff in one piece. The size of the diff is regrettable but hopefully a one-off: What it buys is automatic format checking for CI and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
      - ruff check .
      - mypy .
      - pyright
- Format check:
     - yapf --diff --recursive .
The refactoring includes:
  - Turn the Result class into a more elaborate object, capable of
    doing more heavy lifting around stderr and stdout decoding,
    summarizing outcome, and matching error strings.
    Aside from fixing broken type checks, this also removes lots of
    boilerplate calling code which is currently used for handling
    possible call outcome scenarios. Trying to access an inexistent,
    decoded string should raise a meaningful exception by itself now,
    which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
    - Add the AbstractCmd class above Cmd. This is necessary because
      the checker rightfully complains it can't instantiate a Cmd
      instance where constructor arguments were needed. They never
      were, but the type used at the instantiating code's location in
      jw.pkg.App so claims.
    - Lots of sub- and sub-subcommands are derived from the base
      class of the invoking command. That provides some properties
      shared across the ancestor hierarchy of a command, but is
      semantically unsound. Fix that by introducing jw.pkg.BaseCmd
      class as a place to provide basic helpers shared across all
      commands used in a jw.pkg.App's context, and derive all command
      classes from that afresh. The parent command is still reachable
      via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
    The line was needed by Python 2 which is not supported anylonger.
    For Python 3, the default encoding is UTF-8, anyway.
  - Allow to run "make py-format" without having it produce any
    changes. It's basically "yapf --in-place --recursive ." with some
    code style settings, see conf/topdir/pyproject.toml. The settings
    may be debatable. I've had custom tweaks in place on that target,
    too, but then again, IDEs would have more hassle to integrate
    that.
- Introduce a 88 character line length limit
  - One import per line, reshuffle them semantically, see
    [tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
      if TYPE_CHECKING
  - Spaces around assignments accounts for much churn. Having having
    no spaces in inline parameter list assignments and default
    parameter values would arguably be more compact where it's
    useful. On the other hand, I have not found a code formatter
    which allows spaces around assignments in parameter lists broken
    into one per line and that's often better than a wall of text.
  - Add two spaces before # export, as this seems to be mandated by
    PEP-8
- Use single quotes by default
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit is contained in:
Jan Lindemann 2026-05-27 07:16:05 +02:00
commit 6db73873e7
Signed by: Jan Lindemann
GPG key ID: 3750640C9E25DD61
97 changed files with 3229 additions and 1893 deletions
Download patch file Download diff file Expand all files Collapse all files

208
src/python/jw/pkg/cmds/projects/CmdBuild.py
View file

@ -1,65 +1,120 @@
# -*- coding: utf-8 -*-
import datetime
import os
import re
import os, re, sys, subprocess, datetime, time
from argparse import Namespace, ArgumentParser
from argparse import ArgumentParser, Namespace
from functools import lru_cache
from ...lib.util import get_profile_env
from ...lib.log import *
from ..Cmd import Cmd
from ..CmdProjects import CmdProjects
from ...App import Scope
from ...lib.log import DEBUG, ERR, NOTICE, log
from ...lib.util import get_profile_env, pretty_cmd
from .Cmd import Cmd, Parent
class CmdBuild(Cmd): # export
class CmdBuild(Cmd): # export
def __init__(self, parent: CmdProjects) -> None:
super().__init__(parent, 'build', help='janware software project build tool')
def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'build', help = 'janware software project build tool')
def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser)
parser.add_argument('--exclude', default='', help='Space seperated ist of modules to be excluded from build')
parser.add_argument('-n', '--dry-run', action='store_true',
default=False, help='Don\'t build anything, just print what would be done.')
parser.add_argument('-O', '--build-order', action='store_true',
default=False, help='Don\'t build anything, just print the build order.')
parser.add_argument('-I', '--ignore-deps', action='store_true',
default=False, help='Don\'t build dependencies, i.e. build only modules specified on the command line')
parser.add_argument('--env-reinit', action='store_true',
default=False, help='Source /etc/profile before each build step. Discard environment unless --env-keep is specified')
parser.add_argument('--env-keep', default='none', help='Comma seperated list of environment variables to keep, '
+ '"all" or "none", only meaningful if --env-reinit is specified')
parser.add_argument('target', default='all', help='Build target')
parser.add_argument('modules', nargs='+', default='', help='Modules to be built')
parser.add_argument(
'--exclude',
default = '',
help = 'Space seperated ist of modules to be excluded from build',
)
parser.add_argument(
'-n',
'--dry-run',
action = 'store_true',
default = False,
help = "Don't build anything, just print what would be done.",
)
parser.add_argument(
'-O',
'--build-order',
action = 'store_true',
default = False,
help = "Don't build anything, just print the build order.",
)
parser.add_argument(
'-I',
'--ignore-deps',
action = 'store_true',
default = False,
help = (
"Don't build dependencies, i.e. build only modules specified "
'on the command line'
),
)
parser.add_argument(
'--env-reinit',
action = 'store_true',
default = False,
help = (
'Source /etc/profile before each build step. Discard environment '
'unless --env-keep is specified'
),
)
parser.add_argument(
'--env-keep',
default = 'none',
help = (
'Comma seperated list of environment variables to keep, '
'"all" or "none", only meaningful if --env-reinit is specified'
),
)
parser.add_argument(
'target',
default = 'all',
help = 'Build target',
)
parser.add_argument(
'modules',
nargs = '+',
default = '',
help = 'Modules to be built',
)
async def _run(self, args: Namespace) -> None:
@lru_cache(maxsize=None)
def read_deps(cur, prereq_type):
@lru_cache(maxsize = None)
def read_deps(cur, prereq_type: str) -> list[str]:
# dep cache doesn't make a difference at all
if prereq_type in dep_cache:
if cur in dep_cache[prereq_type]:
return dep_cache[prereq_type][cur]
else:
dep_cache[prereq_type]: dict[str, str] = {}
dep_cache[prereq_type] = {}
ret = self.app.get_project_refs([ cur ], ['pkg.requires.jw'],
prereq_type, scope = Scope.Subtree, add_self=False, names_only=True)
ret = self.app.get_project_refs(
[cur],
['pkg.requires.jw'],
prereq_type,
scope = Scope.Subtree,
add_self = False,
names_only = True,
)
log(DEBUG, 'prerequisites = ' + ' '.join(ret))
if cur in ret:
ret.remove(cur)
log(DEBUG, 'inserting', prereq_type, "prerequisites of", cur, ":", ' '.join(ret))
log(
DEBUG,
(f'Inserting {prereq_type}, prerequisites of {cur}: {" ".join(ret)}'),
)
dep_cache[prereq_type][cur] = ret
return ret
def add_dep_tree(cur, prereq_types, tree, all_deps):
log(DEBUG, "adding prerequisites " + ' '.join(prereq_types) + " of module " + cur)
log(DEBUG, 'Adding deps "{" ".join(prereq_types)}" of module {cur)')
if cur in all_deps:
log(DEBUG, 'already handled module ' + cur)
log(DEBUG, 'Already handled module "{cur}"')
return 0
deps = set()
all_deps.add(cur)
for t in prereq_types:
log(DEBUG, "checking prereqisites of type " + t)
log(DEBUG, 'Checking deps of type "{t}"')
deps.update(read_deps(cur, t))
for d in deps:
add_dep_tree(d, prereq_types, tree, all_deps)
@ -70,17 +125,20 @@ class CmdBuild(Cmd): # export
all_deps = set()
dep_tree = {}
for m in modules:
log(DEBUG, "--- adding dependency tree of module " + m)
log(DEBUG, '--- Adding dependency tree of module "{m}"')
add_dep_tree(m, prereq_types, dep_tree, all_deps)
while len(all_deps):
# Find any leaf
for d in all_deps:
if not len(dep_tree[d]): # Dependency d doesn't have dependencies itself
break # found
else: # no Leaf found
# Dependency d doesn't have dependencies itself
if not len(dep_tree[d]):
break # found
else: # no Leaf found
print(all_deps)
raise Exception("fatal: the dependencies between these modules are unresolvable")
order.append(d) # do it
raise Exception(
'Fatal: the dependencies between these modules are unresolvable'
)
order.append(d) # do it
# bookkeep it
all_deps.remove(d)
for k in dep_tree.keys():
@ -88,56 +146,68 @@ class CmdBuild(Cmd): # export
dep_tree[k].remove(d)
return 1
async def run_make(module, target, cur_project, num_projects):
async def run_make(module, target, cur_project, num_projects) -> None:
patt = self.app.is_excluded_from_build(module)
if patt is not None:
title = f'---- {module}'
log(NOTICE, f',{title} >')
log(NOTICE, f'| Configured to skip build on platform >{patt}<')
log(NOTICE, f'`{title} <')
return
make_cmd = [ "make", target ]
wd = self.app.find_dir(module, pretty=False)
title = '---- [%d/%d]: Running "%s" in %s -' % (cur_project, num_projects, ' '.join(make_cmd), wd)
make_cmd = ['make', target]
wd = self.app.find_dir(module, pretty = False)
title = '---- [%d/%d]: Running "%s" in %s -' % (
cur_project,
num_projects,
' '.join(make_cmd),
wd,
)
mod_env = None
if args.env_reinit:
keep: bool|list[str] = False
keep: bool | list[str] = False
if args.env_keep is not None:
match args.env_keep:
case 'all':
keep=True
keep = True
case 'none':
keep=False
keep = False
case _:
keep = args.env_keep.split(',')
mod_env = await get_profile_env(keep=keep)
mod_env = await get_profile_env(keep = keep)
try:
await self.app.exec_context.run(
make_cmd,
wd=wd,
throw=True,
verbose=True,
mod_env=mod_env,
title=title
wd = wd,
throw = True,
verbose = True,
mod_env = mod_env,
title = title,
)
except Exception as e:
log(ERR, f'Failed to make target "{target}" in module "{module}" below base {self.app.projs_root}: {str(e)}')
log(
ERR,
(
f'Failed to make target "{target}" in module "{module}" '
f'below base {self.app.projs_root}: {str(e)}'
),
)
raise
async def run_make_on_modules(modules, order, target):
cur_project = 0
num_projects = len(order)
if target in ["clean", "distclean"]:
if target in ['clean', 'distclean']:
for m in reversed(order):
cur_project += 1
await run_make(m, target, cur_project, num_projects)
if m in modules:
modules.remove(m)
if not len(modules):
log(NOTICE, "All modules cleaned")
log(NOTICE, 'All modules cleaned')
return
else:
for m in order:
@ -146,7 +216,7 @@ class CmdBuild(Cmd): # export
async def run(args):
log(DEBUG, "----------------------------------------- running ", ' '.join(sys.argv))
log(DEBUG, f'-------------------------------------- running {pretty_cmd()}')
modules = args.modules
exclude = args.exclude.split()
@ -154,19 +224,19 @@ class CmdBuild(Cmd): # export
env_exclude = os.getenv('BUILD_EXCLUDE', '')
if len(env_exclude):
log(NOTICE, "Exluding modules from environment: " + env_exclude)
exclude += " " + env_exclude
log(NOTICE, 'Exluding modules from environment: ' + env_exclude)
exclude += ' ' + env_exclude
# -- build
order = []
glob_prereq_types = [ "build" ]
if re.match("pkg-.*", target) is not None:
glob_prereq_types = [ "build", "run", "release", "devel" ]
glob_prereq_types = ['build']
if re.match('pkg-.*', target) is not None:
glob_prereq_types = ['build', 'run', 'release', 'devel']
if target != 'order' and not args.build_order:
log(NOTICE, "Using prerequisite types " + ' '.join(glob_prereq_types))
log(NOTICE, "Calculating order for modules ... ")
log(NOTICE, 'Using prerequisite types ' + ' '.join(glob_prereq_types))
log(NOTICE, 'Calculating order for modules ... ')
calculate_order(order, modules, glob_prereq_types)
if args.ignore_deps:
@ -177,18 +247,24 @@ class CmdBuild(Cmd): # export
exit(0)
cur_project = 0
log(NOTICE, "Building target %s in %d projects:" % (target, len(order)))
log(NOTICE, 'Building target %s in %d projects:' % (target, len(order)))
for m in order:
cur_project += 1
log(NOTICE, " %3d %s" % (cur_project, m))
log(NOTICE, ' %3d %s' % (cur_project, m))
if args.dry_run:
exit(0)
await run_make_on_modules(modules, order, target)
log(NOTICE, 'Build done at %s' % (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")))
log(
NOTICE,
(
'Build done at %s' %
(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
),
)
dep_cache: dict[dict[str, str]] = {}
dep_cache: dict[str, dict[str, list[str]]] = {}
await run(args)