pycontribs / mk Goto Github PK

mk ease contributing to any open source repository by exposing most common actions you can run. Inspired by make, tox and other cool tools!

Home Page: https://mk.readthedocs.io/

License: MIT License

Python 99.95% Shell 0.05%

tox git cli productivity pep-621 mk

mk's Introduction

Documentation of mk tool

mk is a CLI tool that aims to ease contribution and maintenance for projects by hiding repository implementation details from the casual contributor. With it, you can contribute without having to know all the build and testing tools that the project team already uses, which often have strange requirements.

If you ever asked yourself one of the below questions, probably you would want to try mk and if it can help you

How do I run tests locally?
Which are the test suites I can run?
Is my change ready to be reviewed?
How can I propose a change for review?

Run mk inside any cloned repository to display which options you have. No configuration file is needed as the tool will look for common tools used by the repository and expose their commands.

mk is inspired by tools such make, waf, taskfile, tox, nox, npm, yarn and pre-commit, but it does not aim to replace them. Instead, it aims to provide a unified interface for calling them that is friendly even for those that never used these tools.

Installation

We recommend using pipx to install mk to avoid potential dependency conflicts. You can use pip3 install --user mk as well.

pipx install mk

How it works

mk inspects the current core repository and detects build tools used by the project, like pre-commit, tox, npm and exposes their commands to the user in a predictable way.

For example, you should be able to lint any code repository running only mk lint, regardless of author preference for picking one way to execute them or another.

Be assured that mk does not make use of AI to guess what needs to run. As most projects use relatively similar patterns, it is easy to identify the one to execute.

At this moment, if two tools expose the same command name, the tool will add a number to its name. In the future, we may decide to either chain them under a single name or allow some tools to shadow others and avoid duplicates.

What are the main benefits

One of the benefits of mk is that it should reduce the amount of how-to-contribute documentation the author needs to write.

A considerable amount of maintainer effort can go into producing documentation that makes it easier for someone to contribute.

Some projects are less affected than others. That is usually related to how well the potential contributors know the practices used by the project. Still, if your project has a wide range of uses, you will quickly discover that newbie contributors may hit a knowledge wall. Such a barrier will likely prevent most of them from becoming active contributors. The remaining ones will flood the project with questions, distracting other maintainers from doing more advanced tasks.

Unless you want to deter contributions, you should plan to make it as easy as possible for people to contribute. That is one area where mk aims to help.

Aliases

Similar to git aliases, mk allows typing as little as possible by automatically aliasing commands. For example, you can run mk lint just by typing mk l as long there is no other command starting with the same letter. Aliases are available for one, two and three letters prefixes.

Using mk to propose changes to projects

Instead of writing a long list of tasks to follow, we can use a tool that tells him what to do next. For example, mk has a built-in command named up(load) that aims to ease preparing a local change from being proposed to the project.

This command detects if it should use GitHub workflow or Gerrit and will run the appropriate commands for opening or updating a CR/PR. Users will be allowed to upload a change only after passing the minimal set of local tests, preventing noisy mistakes or clog CI/CD pipelines.

In addition to linting, it will also check that the repository is not in dirty status or that the testing did not leave untracked files.

Planned features

A persistent state of each command run - This means that it will know if a specific command was run and if it failed or not. The state would be linked to the repository state, so modifying a tracked file would reset the state to be unknown. (#20)
Configuration file where additional actions can be added. (#21)
Dependencies between commands. While some tools support dependencies, many do not. You should be able to declare that a specific command will run only after another one already passed. (#22)
Ability to generate CI/CD pipelines so the user would spend less time writing non-portable configurations. (#23)

mk's People

Contributors

Stargazers

Watchers

Forkers

ssbarnea bristea pombredanne isabella232 aesopius ansibuddy ssbarnea-bot gotmax23 hmajid2301 fox-forks joroque

mk's Issues

Allow mk to be installed using homebrew

To ease installation for MacOS users that do not master pip/pipx secrets, it would be useful to allow installation using:

brew install mk

While I already made a formula that allows that, it should be noted that homebrew maintainers have a some minimal popularity requirements for adding new formulas.

If you want to help, be sure that you do star the repo and fork it, both of them are counted towards those limits. Once we rich these limits I will propose it again.

The current homebrew popularity requirements are: 30 forks, 30 watchers and 75 stars

Related: Homebrew/homebrew-core#74903

If you want to make this happen sooner, please be sure that you roke, watch and star the repository, so we can fit the requirements for adding a homebrew recipe.

2.5.0 doesn't seem to see Taskfile anymore.

I'm still troubleshooting, but mk 2.5.0 isn't parsing or picking up taskfile.yml (or it's various filenames) anymore.

I have a Taskfile in my home directory, which I guess can't be picked up by mk anymore

❯ mk detect
CRITICAL Current version of mk works only within git repos.
mk detect...

which is fine, but it's not picking up taskfiles in my repo either.

❯ ls ./*askfile* ls ~/*askfile*
./taskfile.yml  /Users/bdmorin/Taskfile.yml

❯ task --list-all
task: Available tasks for this project:
* check-gum:
* default:
* kustomize-*:
* quick-commit:
* trigger-build-branch:

❯ task -t ~/Taskfile.yml
task: Available tasks for this project:
* authorize-kubectl:          Authorize kubectl to access GKE cluster
* default:                    blargh
* verify-gcloud-config:       Verify current gcloud configuration
* whoami:                     Display your current gcloud information.

❯ mk detect
mk detect...

 mk --help

 Usage: mk [OPTIONS] COMMAND [ARGS]...

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --version                                                                                                                                                                                                       │
│ --verbose             -v      INTEGER  Increase verbosity. [default: 0]                                                                                                                                         │
│ --install-completion                   Install completion for the current shell.                                                                                                                                │
│ --show-completion                      Show completion for the current shell, to copy it or customize the installation.                                                                                         │
│ --help                -h               Show this message and exit.                                                                                                                                              │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ commands   List all commands available.                                                                                                                                                                         │
│ detect     Display detected information about current project.                                                                                                                                                  │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ git ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ up         Upload current change by creating or updating a CR/PR. git                                                                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────

Hopefully I'll figure out what's going on and post my findings.

Command aliases (shortcuts)

User should be able to tun mk l and call mk lint just like git does as long there are no other commands that share the same prefix. If there are duplicates and error message should mention which options are available.

Unable to publish mk on test.pypi.org

Upstream request made at pypi/support#995

Drop dependency on typer

As there are no signs of typer improving its maintenance status, removing it now looks like the right approach, especially as it is not compatible with newer click versions.

Related: fastapi/typer#300

Configuration file

Implement configuration loading with an minimal feature that allows us to define custom actions:

actions:
  foo:
    commands: ... # shell code

While custom commands can be implemented using the Tool class like other external tools, the purpose of this feature is to allow us to add other configurable options.

recognize scripts under tools/

Executable scripts found under tools/ folder should also be exposed as commands, allowing user to easily extend possible test commands by only adding new scripts.

Rebase before running up

I think that rebasing code before pushing would be desirable in order to keep your PR in sync with target branch.

Add ability to generate CI/CD pipeline configs

Add CMake support and some thoughts

I just wanted to say that mk is really cool, but there are a couple things I wanted to mention:

There should be CMake support because many projects use it and it's tricky to use for beginners, but there are a few patterns that you can follow, such as a build directory and CMakeLists.txt being present
mk shouldn't try to execute .so files, as it provided the option to do so, but only the glibc .so system libraries can actually work this way.
mk should inform you if up won't work before you actually use it, as it never mentioned that I need to be in a non-default branch to use up when running mk

Overall, I think mk has some serious potential. :)

Describe purpose of the project

Documenting what mk aims to do and not to do should happen even before adding the first lines of code.

v1 MVP

Support an initial set of builders: tox, npm (or yarn) for start, and "raw" one where each command is defined manually in our own config file.
Builders should avoid reading config files themselves and use APIs, so we would not endup having problems when tools decide to change how they store their configuration.
Each builder should be able to report available commands
List commands available: mk ls shoud list provide an output similar to tox -va
Implicit command should be run, so if someone types mk lint (being translated to mk run lint) and there is a tox environment called lint, it will effectively run tox -e lint.

v2

If two builders expose the same command, the tool should be able to chain them
mk should be able to remember result of particular task, and reset it if tracked files changed since the run. A side effect of this is that a list command should be able to also expose which tasks were already tested or not.
User should be able to define dependencies between tasks. This means that a task will not be run until all its deps report success

v3

Add ability to generate CI/CD pipelines based on current config, GHA for start.

How to best describe what mk does in a single sentence?

As mk is clearly not yet-another-build-tool and it does not replace any other tool, it is little bit hard to describe it.

What it does

detects project own build tools
allows user to easily run exposed test/build commands
minimize the keystrokes needed for running these commands
reports if a required build tool is missing
it is very easy to extend
it has zero configuration (at least for now) as its configuration is basically loaded from detected tools

Any single-sentence suggestion to describe what mk is would be more than welcomed, we need such description. Waiting for suggestions...

recognize python projects using only setup.py

Some python projects do not use tox or nox but they still have a setup.cfg but we should still be able to recognize on them a minimal set of commands:

install
uninstall
package
test -- pytest in their case

Example of such projects:

Determine terminology to use

When it comes to set of actions and group of actions that are executed by most build tools and CI/CD systems there is a great variance in terminology, one that often causes confusions for users. I want to pick good names while mk is still in research phase, so we do not repeat others mistakes.

Jenkins has jobs > steps
GHA has workflows > jobs > steps
Tox has environments > commands
NPM/yarn have scripts
Makefile has targets

It should also be noted that we use Click/Typer which means that we already have a meaning for command, so a good reason to avoid using the term to define something else.

As of today the Runner class does discover actions exposed by tools mentioned above. These are translated into cli commands but we can rename them at any time.

Add ability to declare dependencies using requires/provides blocks

Once we have a configuration file, user should be able to declare dependencies like:

# Example config that will prevent `build` from running unless `lint` already passed.
actions:
    build:
       requires: lint

Please keep in mind that the commands themselves can be exposed by different tools and that this configuration only add extra metadata to those. As you can see lint may not be defined inside the config, but this example should be valid as is.

Obviously that no lint action is found to be exposed by any known tool, an error similar to Unknown action lint was required by build. should be displayed.

That feature is very useful for cases where multiple tools are used which do not glue together well (tox and npm for example).

While I am inclined to used requires/provides pair, I do also consider before/after as alternatives. If you have any preference feel free to share it before we implement this feature.

recognize ansible playbooks as commands

Mk should be able to expose any files found under playbooks/*.yml as commands.

autocomplete can sometimes get stuck (dependency caused)

$ mk --show-completion zsh
WARNING  Repo is dirty on chore/gha                                                                                                        
^CTraceback (most recent call last):
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/c/os/mk/.tox/lower/lib/python3.12/site-packages/subprocess_tee/__init__.py", line 102, in _stream_subprocess
    await asyncio.wait(set(tasks))
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/asyncio/tasks.py", line 464, in wait
    return await _wait(fs, timeout, return_when, loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/asyncio/tasks.py", line 550, in _wait
    await waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ssbarnea/c/os/mk/.tox/lower/bin/mk", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/ssbarnea/c/os/mk/src/mk/__main__.py", line 122, in cli
    for action in ctx.runner.actions:
                  ^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/c/os/mk/src/mk/runner.py", line 67, in actions
    _actions.extend(c.actions())
                    ^^^^^^^^^^^
  File "/Users/ssbarnea/c/os/mk/src/mk/tools/tox.py", line 31, in actions
    result = run_or_fail(
             ^^^^^^^^^^^^
  File "/Users/ssbarnea/c/os/mk/src/mk/exec.py", line 59, in run_or_fail
    return run(*args, check=True, cwd=cwd, tee=tee, env_overrides=env_overrides)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/c/os/mk/src/mk/exec.py", line 36, in run
    result = subprocess_tee.run(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/c/os/mk/.tox/lower/lib/python3.12/site-packages/subprocess_tee/__init__.py", line 142, in run
    result = asyncio.run(_stream_subprocess(cmd, **kwargs))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/asyncio/runners.py", line 123, in run
    raise KeyboardInterrupt()
KeyboardInterrupt

It does not reproduce with on all platforms/dependencies combinations so far..

Apparently is caused by tox command:"tox", "-qq", "--colored", "no", "--hashseed", "1", "--showconfig" which seems to do paging when is not expected to do.

implement CI/CD pipelines

idea: should mk up be idempotent for gerrit?

At this moment the command git review returns error if the change was already uploaded because that is what git server replies with:

remote:
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done
To ssh://review.rdoproject.org:29418/rdo-jobs
 ! [remote rejected] HEAD -> refs/for/master%topic=bump_octavia (no new changes)
error: failed to push some refs to 'ssh://review.rdoproject.org:29418/rdo-jobs'
ERROR    Received exit code 1 from: git review

I am considering adding some code to mk up so it would look for no new changes string and ignore the error returned by git, making the command "idempotent", so you can use it multiple times and not get an error if everything is in desired state.

Still, I would like to hear from others before I do this. Maybe some love to see the error.

Persist command results

When running commands the result of the command should be persistent between runs, avoiding the need to run twice if it passed
State must depend on current state of the repository, modifying a file must reset the state

Add support for taskfile

https://taskfile.dev/ should be supported

Adding support for nox

As nox is a relatively popular build tool (1k githb stars) with one notable project using it (pip and pipx), mk should support it.

One current issue with nox is that it does not have a parsable output format for its list of targets:

$ nox --list
Automation using nox.

Sessions defined in /Users/ssbarnea/c/os/pip/noxfile.py:

- test-3.7
- test-3.8
- test-3.9
- test-3.10
- test-3.11
- test-pypy3
- docs
- docs-live
* lint
- vendoring
- coverage
- prepare-release
- build-release
- upload-release

sessions marked with * are selected, sessions marked with - are skipped.

I do not even know if nox allows descriptions for these as I did not see any on pip. I think it would be prefered if nox itself would be able to add a JSON output format.

Depends-On: wntrblm/nox#648

tools need to be able to shadow others

While the tool detection can easily identify all tools used inside a repository we need a way to avoid running same tasks twice.

Example one

pre-commit can also be called as standalone, from inside tox or even from a Makefile. We do not want to endup running it three times.

Implementing heuristics on this can help but I am afraid they will never be able to be perfect, so it may be better to just ignore duplicated ones via config items.

Run subprocesses in interactive mode

I observed that current implementation does not run an interactive shell, and that might break some commands that require user interaction, yarn run deps on vscode-ansible project.