If there are too many lints curses style updates work poorly

If you scroll up to view the lints out of the viewport you start getting further and further behind in terminal history

Lintrunner Init threw incorrect Torch version error.

As a developer for Pytorch, I have the trunk version of Pytorch installed.
When I ran lintrunner init after a lintrunner run prompted me to rerun init, the following error happened.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.10.1 requires torch==1.9.1, but you have torch 1.13.0a0+git43c7ca3 which is incompatible.

Full logs: P508494855

Upstream pytorch adaptors to lintrunner

I find lintrunner extremely useful in pytorch and hope it use in other projects as well. It would be awesome if some of the adaptors in pytorch are upstremed to lintrunner itself so that they can be used directly for popular linters.

MacOS and Windows distribution missing for v0.11

https://pypi.org/project/lintrunner/0.11.0/#files

And the version was not tagged: https://github.com/suo/lintrunner/tags

Feature request: check for doc formatting errors, too

Docs are difficult to build locally (our documentation for how to do so is often out of date) and formatting errors in CI appear without line numbers (in at least some cases). Some recent example errors/warnings:

/opt/conda/lib/python3.7/site-packages/torch/__init__.py:docstring of torch.set_float32_matmul_precision:22: ERROR: Unexpected indentation.
/opt/conda/lib/python3.7/site-packages/torch/__init__.py:docstring of torch.set_float32_matmul_precision:23: WARNING: Block quote ends without a blank line; unexpected unindent.

If lintrunner could detect doc formatting issues it would making adding/updating docs much easier.

Replace `+` in the persistent data path too

I omitted the case when ppl are on a different time zone (+1 for example).

We should replace + with _ in

lintrunner/src/persistent_data.rs

Line 50 in 88da12f

self.timestamp.clone().replace(":", "-") + "_" + &args

too

lintrunner should only consider files

filter out directories from being passed to the linter adapters

lintrunner doesn't work on deleted files

This is not an urgent issue, but I deleted a file as part of my changes and lintrunner -a fails.

rage:

lintrunner rage report:
timestamp: 2022-08-31T13:23:00.243-07:00
args: 'lintrunner' '-a' '-v'
exit code: 1
err msg: Some("Failed to find file while gathering files to lint: /private/home/howardhuang/pytorch-projects/pytorch/torch/csrc/distributed/c10d/OpsImpl.hpp")

========= BEGIN LOGS =========
[2022-08-31T20:23:00Z DEBUG lintrunner] Version: 0.9.2
[2022-08-31T20:23:00Z DEBUG lintrunner] Passed args: Args { inner: ["lintrunner", "-a", "-v"] }
[2022-08-31T20:23:00Z DEBUG lintrunner] Computed args: Args { verbose: 1, config: ".lintrunner.toml", apply_patches: true, paths_cmd: None, paths_from: None, revision: None, merge_base_with: None, skip: None, take: None, output: Default, cmd: None, paths: [], force_color: false, data_path: None, tee_json: None, all_files: false }
[2022-08-31T20:23:00Z DEBUG lintrunner] Current rev: d956dae3dd5d03bc845b62e9972fa1fc81c33f3f
[2022-08-31T20:23:00Z DEBUG lintrunner::lint_config] Found linters: {"MYPY", "NEWLINE", "NOQA", "CIRCLECI", "NATIVEFUNCTIONS", "CALL_ONCE", "PYBIND11_INCLUDE", "FLAKE8", "SPACES", "INCLUDE", "CLANGTIDY", "TABS", "PYPIDEP", "EXEC", "PYBIND11_SPECIALIZATION", "CUBINCLUDE", "WORKFLOWSYNC", "UFMT", "SHELLCHECK", "ACTIONLINT", "RAWCUDA", "MYPYSTRICT", "CLANGFORMAT", "TYPEIGNORE", "ONCE_FLAG", "CMAKE", "TESTOWNERS"}
[2022-08-31T20:23:00Z DEBUG lintrunner::persistent_data] Checking data file '/private/home/howardhuang/.local/share/lintrunner/bead93266017a343c0460d4b7f1e341fd1377ace6b639541fd53a8ced23af967/.lintrunner.toml' to see if config has changed
[2022-08-31T20:23:00Z DEBUG lintrunner] Running linters: ["FLAKE8", "CLANGFORMAT", "MYPY", "MYPYSTRICT", "CLANGTIDY", "TYPEIGNORE", "NOQA", "CIRCLECI", "NATIVEFUNCTIONS", "NEWLINE", "SPACES", "TABS", "INCLUDE", "PYBIND11_INCLUDE", "PYBIND11_SPECIALIZATION", "PYPIDEP", "EXEC", "CUBINCLUDE", "RAWCUDA", "CMAKE", "SHELLCHECK", "ACTIONLINT", "TESTOWNERS", "CALL_ONCE", "ONCE_FLAG", "WORKFLOWSYNC", "UFMT"]
[2022-08-31T20:23:00Z TRACE lintrunner::log_utils] Linting commit diff files: {"torch/csrc/distributed/c10d/OpsImpl.cpp", "torch/csrc/distributed/c10d/OpsImpl.hpp", "test/distributed/test_c10d_gloo.py", "build_variables.bzl", "test/forward_backward_compatibility/check_forward_backward_compatibility.py", "test/distributed/test_c10d_common.py", "test/distributed/test_c10d_nccl.py"}
[2022-08-31T20:23:00Z TRACE lintrunner::log_utils] Linting working tree diff files: {"torch/csrc/distributed/c10d/Ops.cpp", "torch/csrc/distributed/c10d/OpsImpl.cpp", "test/distributed/test_c10d_gloo.py"}
[2022-08-31T20:23:00Z DEBUG lintrunner::persistent_data] Writing run info to /private/home/howardhuang/.local/share/lintrunner/bead93266017a343c0460d4b7f1e341fd1377ace6b639541fd53a8ced23af967/runs/2022-08-31T13:23:00.243-07:00_7f8c328e06d0cb89332e8065321ffcbd903d13f421ea1a9a7c8004f12c742224

Compact view on results

Currently the source context is printed as part of the lint results. When there are many results, e.g. the change is big or when a new linter is introduced, it helps to have an option to display a compact view of the lint results, including

path:line for each lint message, so one can command click in an editor (vscode) and get straight to that line.
Exclude source so the messages are easier to navigate.

Single linter can cause -a conflicts with itself

For example pytorch's grep linter with a replacement, if the replacement applies multiple times in a file lintrunner will complain that multiple linters edited the same file. But it was one linter. Also if you keep running lintrunner -a over and over again eventually it works.

lintrunner -a results in stale lines for linters

Lintrunner uses line numbers to print lines, but the printed lines are extracted after autoformatting is done, but the lines are from before the autoformatting is done, so they're out of sync.

lintrunner panics on Windows

Running lintrunner or lintrunner init fails with "The filename, directory name, or volume label syntax is incorrect. (os error 123)"

cargo test on Windows:

Error: Failed to find file while gathering files to lint: \\?\C:\Users\justinchu\AppData\Local\Temp\.tmpG29xS4/test_3.txt

Caused by:
    The filename, directory name, or volume label syntax is incorrect. (os error 123)
test git::tests::doesnt_detect_unchanged ... FAILED
Error: Failed to find file while gathering files to lint: \\?\C:\Users\justinchu\AppData\Local\Temp\.tmpziOhRn/new.txt

`lintrunner init` fails in WSL

lintrunner rage report:
timestamp: 2023-01-25T07:33:24.637-08:00
args: 'lintrunner' 'init' '-v'
exit code: 1
err msg: Some("No such file or directory (os error 2)")

========= BEGIN LOGS =========
[2023-01-25T15:33:24Z DEBUG lintrunner] Version: 0.10.5
[2023-01-25T15:33:24Z DEBUG lintrunner] Passed args: Args { inner: ["lintrunner", "init", "-v"] }
[2023-01-25T15:33:24Z DEBUG lintrunner] Computed args: Args { verbose: 1, config: ".lintrunner.toml", apply_patches: false, paths_cmd: None, paths_from: None, revision: None, merge_base_with: None, skip: None, take: None, output: Default, cmd: Some(Init { dry_run: false }), paths: [], force_color: false, data_path: None, tee_json: None, all_files: false }
[2023-01-25T15:33:24Z DEBUG lintrunner] Current rev: 791ab90a320dd9a5948c951146f28aa808a1e840
[2023-01-25T15:33:24Z DEBUG lintrunner::lint_config] Found linters: {"FLAKE8", "PYLINT", "CLANGFORMAT", "NAMESPACE", "EDITORCONFIG-CHECKER", "MYPY", "BLACK-ISORT"}
[2023-01-25T15:33:24Z DEBUG lintrunner] Initializing linters: ["FLAKE8", "MYPY", "BLACK-ISORT", "NAMESPACE", "CLANGFORMAT", "PYLINT", "EDITORCONFIG-CHECKER"]
[2023-01-25T15:33:24Z INFO lintrunner::linter] Initializing linter: 'FLAKE8'
[2023-01-25T15:33:24Z DEBUG lintrunner::linter] Running: python '-m' 'lintrunner_adapters' 'run' 'pip_init' '--dry-run=0' 'flake8==5.0.4' 'flake8-bugbear==22.10.27' 'flake8-pyi==22.10.0' 'dlint==0.13.0'
[2023-01-25T15:33:24Z DEBUG lintrunner::persistent_data] Writing run info to /home/justinchu/.local/share/lintrunner/a68d229237ed30daadd2150156b0198386f18ad5bfed583651b4dbb6600b093a/runs/2023-01-25T07-33-24.637-08-00_e7da52f428fc352d2ec8d0dbcb839d986877fd833296a61efc912bf272b77352

lintrunner check doesn't fail if command is obviously wrong

with

[[linter]]
code = 'BARF'
include_patterns = [
    'tools/codegen/**/*.py'
]
command = [
    'python3',
    'missing_file.py',
    '--',
    '@{{PATHSFILE}}'
]

when I run

(base) ezyang-mbp:pytorch ezyang$ lintrunner --take BARF
ok No lint issues.

which is obviously wrong since the command totally failed

Black commands fail with debug build of python

Using a pydebug build of python 3.8, any command for black fails with

  Advice (BLACK) command-failed
    COMMAND (exit code -11)
    /home/alban/local/pytorch/3.8_debug_source_env/bin/python3 -mblack
    --stdin-filename /home/alban/local/pytorch/3.8_debug_source/torchgen/
    model.py -
    
    STDERR
    Objects/typeobject.c:5247: PyType_Ready: Assertion "type->tp_call !=
    ((void *)0)" failed
    Enable tracemalloc to get the memory block allocation traceback
    
    object address  : 0x562b83994360
    object refcount : 1
    object type     : 0x562b81bfe8e0
    object type name: type
    object repr     :
    
    STDOUT
    (empty)

Any idea where this might come from or where we should report it?

Issue 1

For this PR (pytorch/pytorch#83735) the linter failed to run but still printed out "Successfully applied all patches." Do you know how I can fix the flake8 issue?

Command:
lintrunner -a

Output:

...
>>> General linter failure:

  Error (FLAKE8) command-failed
    COMMAND (exit code 1)
    /private/home/howardhuang/.conda/envs/pytorch/bin/python3 -mflake8 --exit-
    zero /private/home/howardhuang/pytorch-projects/pytorch/test/distributed/
    test_c10d_common.py
    
    STDERR
    Traceback (most recent call last):
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/plugins/manager.py", line 157, in load_plugin
        self._load()
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/plugins/manager.py", line 134, in _load
        self._plugin = self.entry_point.load()
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/
    importlib/metadata.py", line 77, in load
        module = import_module(match.group('module'))
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/
    importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in
    _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 783, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in
    _call_with_frames_removed
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/pyi.py", line 12, in <module>
        from pyflakes.checker import PY2, ClassDefinition
    ImportError: cannot import name 'PY2' from 'pyflakes.checker' (/private/
    home/howardhuang/.local/lib/python3.8/site-packages/pyflakes/checker.py)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/
    runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/
    runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/__main__.py", line 4, in <module>
        cli.main()
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/main/cli.py", line 22, in main
        app.run(argv)
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/main/application.py", line 360, in run
        self._run(argv)
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/main/application.py", line 347, in _run
        self.initialize(argv)
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/main/application.py", line 328, in initialize
        self.find_plugins(config_finder)
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/main/application.py", line 159, in find_plugins
        self.check_plugins.load_plugins()
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/plugins/manager.py", line 415, in load_plugins
        plugins = list(self.manager.map(load_plugin))
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/plugins/manager.py", line 302, in map
        yield func(self.plugins[name], *args, **kwargs)
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/plugins/manager.py", line 413, in load_plugin
        return plugin.load_plugin()
      File "/private/home/howardhuang/.conda/envs/pytorch/lib/python3.8/site-
    packages/flake8/plugins/manager.py", line 164, in load_plugin
        raise failed_to_load
    flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "Y0"
    due to cannot import name 'PY2' from 'pyflakes.checker' (/private/home/
    howardhuang/.local/lib/python3.8/site-packages/pyflakes/checker.py).
    
    STDOUT
    (empty)
Successfully applied all patches.

Issue 2

Command

lintrunner rage

output:

error:        No such file or directory (os error 2)

lintrunner diff view removes trailing whitespace

Example: https://github.com/pytorch/pytorch/runs/6817042373?check_suite_focus=true

suo / lintrunner Goto Github PK

lintrunner's People

Contributors

Stargazers

Watchers

Forkers

lintrunner's Issues

Issue 1

Issue 2

Recommend Projects

Recommend Topics

Recommend Org