Git Product home page Git Product logo

action-sqlfluff's Introduction

action-sqlfluff

Test reviewdog depup release GitHub release (latest SemVer) action-bumpr supported

This is a github action to lint and fix SQL with sqlfluff. The action has the two modes corresponding to sqlfluff lint and sqlfluff fix. One is to automatically leaves comments about SQL violation using reviewdog The other is to automatically suggests code formatting on github pull request with reviewdog too.

Lint mode

The lint mode leaves comments on github pull requests. Comments are pointed out by sqlfluff. github-pr-review demo (lint)

Fix mode

The fix mode suggests code formatting based on sqlfluff fix. github-pr-review demo (fix)

Example

name: sqlfluff with reviewdog
on:
  pull_request:
jobs:
  test-check:
    name: runner / sqlfluff (github-check)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: yu-iskw/action-sqlfluff@v4
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "3.0.7"
          sqlfluff_command: "fix" # Or "lint"
          config: "${{ github.workspace }}/.sqlfluff"
          paths: "${{ github.workspace }}/models"
      - name: "Show outputs (Optional)"
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

Input

inputs:
  github_token:
    description: "GITHUB_TOKEN"
    required: true
    default: "${{ github.token }}"
  github_base_ref:
    description: "Pull request target branch name"
    required: false
    default: "${{ github.base_ref }}"
  working-directory:
    description: "working directory"
    required: false
    default: "${{ github.workspace }}"
  ### Flags for reviewdog ###
  level:
    description: "Report level for reviewdog [info,warning,error]"
    required: false
    default: "error"
  reporter:
    description: "Reporter of reviewdog command [github-check,github-pr-review]."
    required: false
    default: "github-check"
  filter_mode:
    description: |
      Filtering mode for the reviewdog command [added,diff_context,file,nofilter].
      Default is file.
    required: false
    default: "file"
  fail_on_error:
    description: |
      Exit code for reviewdog when errors are found [true,false]
      Default is `false`.
    required: false
    default: "false"
  reviewdog_version:
    description: "reviewdog version"
    required: false
    default: "v0.17.4"
  ### Flags for sqlfluff ###
  sqlfluff_version:
    description: |
      sqlfluff version. Use the latest version if not set.
    required: false
    default: "3.0.7"
  sqlfluff_command:
    description: "The sub command of sqlfluff. One of lint and fix"
    required: false
    default: "lint"
  paths:
    description: |
      PATH is the path to a sql file or directory to lint.
      This can be either a file ('path/to/file.sql'), a path ('directory/of/sql/files'), a single ('-') character to indicate reading from *stdin* or a dot/blank ('.'/' ') which will be interpreted like passing the current working directory as a path argument.
    required: true
  file_pattern:
    description: "The regular expression to filter checked files"
    required: false
    default: '\.sql$'
  encoding:
    description: "Specifiy encoding to use when reading and writing files. Defaults to autodetect."
    required: false
    default: ""
  config:
    description: |
      Include additional config file.
      By default the config is generated from the standard configuration files described in the documentation.
      This argument allows you to specify an additional configuration file that overrides the standard configuration files.
      N.B. cfg format is required.
    required: false
    default: ""
  exclude-rules:
    description: |
      Exclude specific rules.
      For example specifying –exclude-rules L001 will remove rule L001 (Unnecessary trailing whitespace) from the set of considered rules.
      This could either be the allowlist, or the general set if there is no specific allowlist.
      Multiple rules can be specified with commas e.g. –exclude-rules L001,L002 will exclude violations of rule L001 and rule L002.
    required: false
    default: ""
  rules:
    description: |
      Narrow the search to only specific rules.
      For example specifying –rules L001 will only search for rule L001 (Unnecessary trailing whitespace).
      Multiple rules can be specified with commas e.g. –rules L001,L002 will specify only looking for violations of rule L001 and rule L002.
    required: false
    default: ""
  templater:
    description: "The templater to use"
    required: false
    default: ""
  disable-noqa:
    description: "Set this flag to ignore inline noqa comments."
    required: false
    default: ""
  dialect:
    description: "The dialect of SQL to lint"
    required: false
    default: ""
  processes:
    description: "The number of parallel processes to run."
    required: false
    default: "2"
  extra_requirements_txt:
    description: |
      A path to your custom `requirements.txt` to install extra modules for your dbt adapters.
      Please make sure not to contain `sqlfluff` and its dependent packages, because the action can be broken by the conflicts.
    required: false
    default: ""

Outputs

The outputs are available only when the sqlfluff_command input is lint.

outputs:
  sqlfluff-results:
    description: "The JSON object string of sqlfluff results"
    value: ${{ steps.sqlfluff-with-reviewdog-in-composite.outputs.sqlfluff-results }}
  sqlfluff-exit-code:
    description: "The exit code of sqlfluff"
    value: ${{ steps.sqlfluff-with-reviewdog-in-composite.outputs.sqlfluff-exit-code }}
  sqlfluff-results-rdjson:
    description: "The JSON object string of sqlfluff results"
    value: ${{ steps.sqlfluff-with-reviewdog-in-composite.outputs.sqlfluff-results-rdjson }}
  reviewdog-return-code:
    description: "The exit code of reviewdog"
    value: ${{ steps.sqlfluff-with-reviewdog-in-composite.outputs.reviewdog-return-code }}

Development

Release

You can bump version on merging Pull Requests with specific labels (bump:major,bump:minor,bump:patch). Pushing tag manually by yourself also work.

This action updates major/minor release tags on a tag push. e.g. Update v1 and v1.2 tag when released v1.2.3. ref: https://help.github.com/en/articles/about-actions#versioning-your-action

Lint - reviewdog integration

This reviewdog action template itself is integrated with reviewdog to run lints which is useful for Docker container based actions.

reviewdog integration

Supported linters:

Dependencies Update Automation

This repository uses reviewdog/action-depup to update reviewdog version.

reviewdog depup demo

Contributors

yu-iskw
Yu Ishikawa
vgw-chriskruger
Chris Kruger
onesuper
Dreamsome
snyk-bot
Snyk Bot
kieronellis
Null

action-sqlfluff's People

Contributors

github-actions[bot] avatar kieronellis avatar onesuper avatar renovate[bot] avatar snyk-bot avatar vgw-chriskruger avatar yu-iskw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

action-sqlfluff's Issues

Not working with dbt Snowflake adapter

When I run this action with the following setup:

 name: Lint files
        id: lint-files
        uses: yu-iskw/action-sqlfluff@v3
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_command: "lint" # Or "lint"
          config: "${{ github.workspace }}/.sqlfluff"
          paths: '${{ github.workspace }}/models'
          templater: dbt

I get the following error:

 Running sqlfluff 🐶 ...
  19:42:09  Error importing adapter: No module named 'dbt.adapters.snowflake'
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 183, in _credentials_from_profile
      cls = load_plugin(typename)
    File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 215, in load_plugin
      return FACTORY.load_plugin(name)
    File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 70, in load_plugin
      raise RuntimeException(f'Could not find adapter type {name}!')
  dbt.exceptions.RuntimeException: Runtime Error
    Could not find adapter type snowflake!

I suspect this is because this action seems to install dbt-core and not the adapter specific dbt.

What is the best way to use inside docker-compose project?

I have dbt inside docker compose with all dependencies.

Currently, I'm running check that compilation successful in PR's like this

name: dbt
on: pull_request
jobs:
  compile:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Start docker compose
        run: docker compose up -d
      - name: Compile sources
        run: docker compose exec service dbt compile
      - name: Stop compose
        if: always()
        run: |
          docker-compose down
          docker network prune -f

I'm trying to add sqlfluff linter, and it requires all database dependencies to be set manually.
However I can run command inside compose

docker compose exec service sqlfluff lint --format json --dialect postgres --templater dbt ...

Is it possible to utilize this? Or should I build workflow from scratch?

Alternate working directory configuration not respected (dbt_project.yml)

Is it possible to use this action template on dbt when the dbt_project.yml file lives somewhere other than the root of the repository? If so, how to configure it? The below describes what I have attempted.

In a git action template, I am setting the working-directory for yu-iskw/action-sqlfluff@v3 as follows:

working-directory: ${{ github.workspace }}/dbt_transform_code

In the SQL Fluff configuration file, I've also experimented with setting the project_dir to '/home/runner/work/<repo-name>/<repo-name>/dbt_transform_code'

In either case, I see the following error from the git action run:

    File "/usr/local/bin/sqlfluff", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
      return __callback(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/sqlfluff/cli/commands.py", line 830, in fix
      result = lnt.lint_paths(
    File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/linter/linter.py", line 1164, in lint_paths
      for i, linted_file in enumerate(runner.run(expanded_paths, fix), start=1):
    File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/linter/runner.py", line 142, in run
      for lint_result in self._map(
    File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 870, in next
      raise value
  dbt.exceptions.DbtProjectError: Runtime Error
    no dbt_project.yml found at expected path /github/workspace/dbt_project.yml

Which appears to be telling me the process is looking for the dbt_project.yml file to exist at the root of the repository.

(conclusion=failure)

I'm trying to get this working and I think i'm close.
For some reason at the moment "Running reviewdog 🐶 ..." process breaks with an error:
2024/03/25 23:30:19 [sqlfluff-fix] reported: https://github.com/<my_repo>/runs/23080766658 (conclusion=failure)
Not sure what that means or where to begin looking for a fix. Any suggestions appreciated.

SQLFluff linting job failing due to missing CompilationException from dbt.exceptions

The SQLFluff linting was working as expected with the following versions: SQLFluff version 1.4.5 with DBT Version 1.2 in DBT Cloud. But recently it started failing with the following error:

Traceback (most recent call last):
    File "/usr/local/bin/sqlfluff", line 5, in <module>
      from sqlfluff.cli.commands import cli
    File "/usr/local/lib/python3.10/site-packages/sqlfluff/cli/commands.py", line 518, in <module>
      def lint(
    File "/usr/local/lib/python3.10/site-packages/sqlfluff/cli/commands.py", line 218, in core_options
      get_plugin_manager().hook.get_templaters()
    File "/usr/local/lib/python3.10/site-packages/sqlfluff/core/plugin/host.py", line 12, in get_plugin_manager
      pm.load_setuptools_entrypoints(project_name)
    File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
      plugin = ep.load()
    File "/usr/local/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
      module = import_module(match.group('module'))
    File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "/usr/local/lib/python3.10/site-packages/sqlfluff_templater_dbt/__init__.py", line 3, in <module>
      from sqlfluff_templater_dbt.templater import DbtTemplater
    File "/usr/local/lib/python3.10/site-packages/sqlfluff_templater_dbt/templater.py", line 17, in <module>
      from dbt.exceptions import (
  ImportError: cannot import name 'CompilationException' from 'dbt.exceptions' (/usr/local/lib/python3.10/site-packages/dbt/exceptions.py)

"Show output - optional" step shows empty results

code I'm using:

name: sqlfluff with reviewdog
on:
pull_request:
jobs:
test-check:
name: Lint-and-Parse-SQL
runs-on: ubuntu-latest

steps:
  - uses: actions/checkout@v2
  - uses: yu-iskw/[email protected]
    id: lint-sql
    #id: sqlfluff-with-reviewdog-in-composite
    with:
      github_token: ${{ secrets.github_token }}
      reporter: github-pr-review
      sqlfluff_version: "2.0.7"
      sqlfluff_command: "lint" # Or "lint"
      config: "${{ github.workspace }}/.sqlfluff"
      paths: '${{ github.workspace }}/sql'
      fail_on_error: true
  - name: 'Show outputs (Optional)'
    shell: bash
    run: |
      echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
      echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'
      echo '${{ steps.lint-sql.outputs.sqlfluff-exit-code }}' | jq -r '.'
      echo '${{ steps.lint-sql.outputs.reviewdog-return-code }}' | jq -r '.'

image

The check results as a success even when violations are found. I believe, Brewdog is supposed to fail the check with fail_on_error: true parameter set but I think there is a brewdog bug waiting for a fix to address this issue.

So for now, I'm trying to obtain the output value, specifically for sqlfluff-exit-code and reviewdog-return-code to allow to use these to dictate whether the check should be a success or a fail but I can't seem to get the output values to show!

Any help would be appreciated, thanks!

Wrong JSON output with tee and dbt templater

When running with the dbt templater the process might add more output lines that aren't actual JSON format and then it will fail trying to run jq here

e.g:

�[0m14:02:51  Unable to do partial parsing because config vars, config profile, or config target have changed
�[0m14:02:52  Unable to do partial parsing because config vars, config profile, or config target have changed
[{"filepath":  ...

I think one possible solution is to replace tee with --write-output from sqlfluff itself that was added since version 0.10.1, this way the resulting file only contains the final report.

I could try to make a PR to fix this, what do you think?

sqlfluff v2.0

maybe this action is not compatible sqlfluff v2.0 (I test with 2.0.3)
is there any update plan?

action-sqlfluff GH action outputs raw format instead of inline code suggestion

@yu-iskw The issue reported in #56 is still present when the diff produced by sqlfluff fix is bigger than a few lines (including a screenshot below). Do we have a fix/workaround for it?

image

The relevant config from the workflow yaml file

    steps:
      - uses: actions/checkout@v2
      - uses: yu-iskw/action-sqlfluff@v3
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "1.4.5"
          sqlfluff_command: "fix"
          config: "${{ github.workspace }}/.sqlfluff"
          paths: "${{ github.workspace }}/models"
          extra_requirements_txt: "${{ github.workspace }}/requirements.txt"
      - name: "Show outputs (Optional)"
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

Fail action when there are violations

The action always shows a green check-mark even if there are violations after linting
Screen Shot 2022-06-08 at 9 12 00 AM

Is there any of the inputs that need to be changed to make it show a red x when there are any violations after linting?

my yml file has this

name: Code QA
on:
  pull_request:
jobs:
  test-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: yu-iskw/action-sqlfluff@v3
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "0.13.0"
          sqlfluff_command: "lint" # Or "fix"
          fail_on_error: 'true'
          config: "${{ github.workspace }}/.github/linters/.sqlfluff"
          paths: '${{ github.workspace }}/dbt/models'
      - name: 'Show outputs (Optional)'
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

I was hoping fail_on_error would do the trick but it didn't

Parse error when running on latest version

Don't understand the error I am facing

Action.yml

name: Lint Check

on:
  pull_request:
    branches:
      - master
env:
  DBT_PROFILES_DIR: ${{ github.workspace }}/.dbt/
  TRINO_DEV_URL: trino-dev-coordinator-int.de.razorpay.com
  DBT_USER: github-actions-dbt
jobs:
  Lint_Check:
    name: Lint_Check
    runs-on: [self-hosted]

    steps:
      - name: Check out
        uses: actions/checkout@v2
      - uses: yu-iskw/action-sqlfluff@v3
        name: sql-lint-check
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          config: "${{ github.workspace }}/.sqlfluff"
          paths: "${{ github.workspace }}/models"
          extra_requirements_txt: "${{ github.workspace }}/requirements.txt"

sqlfluff config

[sqlfluff]
templater = dbt
dialect = ansi
sql_file_exts = .sql
exclude_rules = L027

[sqlfluff:rules]
max_line_length = 150

[tool.sqlfluff.templater.jinja]
apply_dbt_builtins = True

[sqlfluff:templater:dbt]
project_dir = .
profiles_dir = ./.dbt
profile = trino
target = trino-dev

ERROR screenshots

Screenshot 2022-06-09 at 6 56 01 PM

SqlFluff action causes Deprecation Warning on GitHub

This loc: https://github.com/yu-iskw/action-sqlfluff/blob/main/entrypoint.sh#L24

Throws:

Running sqlfluff 🐶 ...
  ==== finding fixable violations ====
    [1 templating/parsing errors found]
  ==== no fixable linting violations found ====
  All Finished!
  Warning: The `set-output` command is deprecated and will be disabled soon.
  Please upgrade to using Environment Files. For more information see:
  https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Thank you.

Can't get changed files.

My repository is like this.

.
├── README.md
├── other_directory
│   └── other_files
├── dbt
│   ├──.sqlfluff
│   ├──packages.yml
|   └── models

And I want to lint to dbt/models/. So my yaml for workflow write like this.

name: sqlfluff with reviewdog
on:
  pull_request:
jobs:
  test-check:
    name: runner / sqlfluff (github-check)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: auth
        uses: google-github-actions/setup-gcloud@v0
        with:
          project_id: unipos-dev-diu
          service_account_key: ${{ secrets.gcp_key }}
          export_default_credentials: true
      - uses: yu-iskw/[email protected]
        id: lint-sql
        env:
          GCLOUD_PROJECT: my-project
        with:
          working-directory: ./dbt
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "0.13.0"
          sqlfluff_command: "lint"
          config: "./.sqlfluff"
          paths: './models'
          extra_requirements_txt: './extra_requirements.txt'
      - name: 'Show outputs (Optional)'
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

I think setting woriking-directory is required, because of running dbt deps. But, then here is not working well...
I expect models/marts/hoge.sql, but git diff --name-only --no-color sqlfluff_bot HEAD returns dbt/models/marts/hoge.sql.
There is any solutions ?

SQLFluff not leaving comments on GitHub Pull Requests on lint mode

Wasn't sure if this was an issue or the way I've configured SQLFluff, but it looks like SQLFluff isn't leaving comments on Github for the lint mode. This has been an issue for more than a month, not sure exactly when it happened. It used to run properly beforehand and I haven't changed any of the settings since last time.

Below is a snippet from the SQLFluff run on Github actions [Run yu-iskw/[email protected]] - it looks like it's correctly running and finding formatting issues:

image

EDIT: Ignore the below comment, was misremembering.
However, the show outputs section looks empty, whereas in the past it had outputs:

image

Here's my workflow file:

image

Please let me know if there's any more details you need or if this isn't an issue, happy to move/delete.

SQLFluff failing due to ImportError

sqlfluff has been failing for us since yesterday around 3pm GMT, with the following error:

ImportError: cannot import name 'CompilationException' from 'dbt.exceptions' (/usr/local/lib/python3.9/site-packages/dbt/exceptions.py).

We haven't changed any configs or anything - think the issue started around the same time that dbt went down too.

Rest of the error is below:

Traceback (most recent call last): File "/usr/local/bin/sqlfluff", line 5, in <module> from sqlfluff.cli.commands import cli File "/usr/local/lib/python3.9/site-packages/sqlfluff/__init__.py", line 6, in <module> from sqlfluff.api import lint, fix, parse, list_rules, list_dialects File "/usr/local/lib/python3.9/site-packages/sqlfluff/api/__init__.py", line 4, in <module> from sqlfluff.api.simple import lint, fix, parse, APIParsingError File "/usr/local/lib/python3.9/site-packages/sqlfluff/api/simple.py", line 4, in <module> from sqlfluff.core import ( File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/__init__.py", line 9, in <module> from sqlfluff.core.linter import Linter File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/linter/__init__.py", line 6, in <module> from sqlfluff.core.linter.linter import Linter File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/linter/linter.py", line 23, in <module> from sqlfluff.core.rules import get_ruleset File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/rules/__init__.py", line 12, in <module> for plugin_rules in get_plugin_manager().hook.get_rules(): File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/plugin/host.py", line 12, in get_plugin_manager pm.load_setuptools_entrypoints(project_name) File "/usr/local/lib/python3.9/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints plugin = ep.load() File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load module = import_module(match.group('module')) File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/usr/local/lib/python3.9/site-packages/sqlfluff_templater_dbt/__init__.py", line 3, in <module> from sqlfluff_templater_dbt.templater import DbtTemplater File "/usr/local/lib/python3.9/site-packages/sqlfluff_templater_dbt/templater.py", line 17, in <module> from dbt.exceptions import ( ImportError: cannot import name 'CompilationException' from 'dbt.exceptions' (/usr/local/lib/python3.9/site-packages/dbt/exceptions.py)

Screenshot below:

image

Action doesn't pick up space in paths

Hi,

There is an issue with including spacing in the 'paths' input for the action. I have tried several methods (ubuntu way of escaping space, env variables, repo variables) and nothing has worked out.

image

Thanks!

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/yu-iskw/action-sqlfluff/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.


This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Enhancement: enable --show-lint-violations for sqlfluff fix

Currently: unfixable violations when running sqlfluff fix causes failed check with the error [n unfixable linting violations found]
Screenshot 2023-01-23 at 4 11 17 PM

Enhancement: be able to see what rules failed in the output with the sqlfluff flag --show-lint-violations so that the dev can go into the code and manually fix. In general have an option to pass additional options/flags to the action when running fix or lint

here is sqlfluff feature:
sqlfluff/sqlfluff#3788

Current version of sqlfluff is 1.4.5

I note the current release version of sqlfluff is 1.4.5.

I looked at what appeared to be the last release bump in this repository #42 but it doesn't actually seem to have updated this sqlfluff version so I am not a little confused by what the supported version means in terms of this particular action?

i.e. What is actually the version of sqlfluff installed within this action?

reviewdog outputs raw format instead of code suggestion

When running sqlfluff fix on changes that end up being more than a few lines of sql, instead of a code suggestion on pr review it gets outputted in raw format. Wondering why that's the case or if there is something with my workflow implementation?

small change:
Screenshot 2023-01-12 at 3 12 33 PM

change that is over 20 lines long:
Screenshot 2023-01-12 at 3 21 11 PM

.sqlfluff config:

dialect = bigquery
templater = dbt
rules = core
exclude_rules = L008, L032, L031, L040

[sqlfluff:templater:dbt]
project_dir = .

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

[sqlfluff:rules]
capitalisation_policy = consistent
extended_capitalisation_policy = consistent

[sqlfluff:rules:L016]
max_line_length = 200

workflow:

steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 1

      - uses: yu-iskw/[email protected]
        id: fix-sql
        env: 
          DBT_PROFILES_DIR: '${{ github.workspace }}/dbt'
        with: 
          github_token: ${{ secrets.GITHUB_TOKEN }}
          working-directory: '${{ github.workspace }}/dbt'
          level: 'warning'
          reporter: github-pr-review
          filter_mode: 'added'
          reviewdog_version: '0.14.1'
          sqlfluff_version: '1.4.5'
          sqlfluff_command: 'fix'
          paths:  "${{ github.workspace }}/dbt/models"
          config: "${{ github.workspace }}/dbt/.sqlfluff"
          extra_requirements_txt: "${{ github.workspace }}/dbt/sqlfluff_requirements.txt"
      
      - name: 'Show outputs (Optional)'
        shell: bash
        run: |
          echo '${{ steps.fix-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.fix-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

SQLFluff failing due to ModuleNotFoundError: No module named 'pytz'

Hi all,

sqlfluff has been failing for us since this morning with the following error:

ModuleNotFoundError: No module named 'pytz'

Rest of the error is below:

Traceback (most recent call last):
    File "/usr/local/bin/sqlfluff", line 5, in <module>
      from sqlfluff.cli.commands import cli
    File "/usr/local/lib/python3.9/site-packages/sqlfluff/cli/commands.py", line 518, in <module>
      def lint(
    File "/usr/local/lib/python3.9/site-packages/sqlfluff/cli/commands.py", line 218, in core_options
      get_plugin_manager().hook.get_templaters()
    File "/usr/local/lib/python3.9/site-packages/sqlfluff/core/plugin/host.py", line 12, in get_plugin_manager
      pm.load_setuptools_entrypoints(project_name)
    File "/usr/local/lib/python3.9/site-packages/pluggy/_manager.py", line [287](https://github.com/motorway/motorway-dbt/actions/runs/4292196911/jobs/7478336098#step:4:289), in load_setuptools_entrypoints
      plugin = ep.load()
    File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
      module = import_module(match.group('module'))
    File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "/usr/local/lib/python3.9/site-packages/sqlfluff_templater_dbt/__init__.py", line 3, in <module>
      from sqlfluff_templater_dbt.templater import DbtTemplater
    File "/usr/local/lib/python3.9/site-packages/sqlfluff_templater_dbt/templater.py", line 13, in <module>
      from dbt.config import read_user_config
    File "/usr/local/lib/python3.9/site-packages/dbt/config/__init__.py", line 2, in <module>
      from .profile import Profile, read_user_config  # noqa
    File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 10, in <module>
      from dbt.contracts.connection import Credentials, HasCredentials
    File "/usr/local/lib/python3.9/site-packages/dbt/contracts/connection.py", line 28, in <module>
      from dbt.contracts.util import Replaceable
    File "/usr/local/lib/python3.9/site-packages/dbt/contracts/util.py", line 7, in <module>
      from dbt import deprecations
    File "/usr/local/lib/python3.9/site-packages/dbt/deprecations.py", line 6, in <module>
      import dbt.tracking
    File "/usr/local/lib/python3.9/site-packages/dbt/tracking.py", line 25, in <module>
      import pytz
  ModuleNotFoundError: No module named 'pytz'

Screenshot below:

image

working-directory input not working

When I try to run the action, it is always running in /github/home and this is inaccessible.

I have attempted to move my profiles.yml file to /github/home/.dbt but I do not have permissions to create that directory and it does not exist.

I have attempted to change the working directory as an input but each time it runs the action runs in the /github/home directory looking for .dbt/ which again does not exist and cannot be created due to permissions.

I also checked the ${{ github.workspace }} which is supposed to be the default, but for my step setup dbt this is different to where the action is running. For my custom step it is /home/runner/work/dbt-bq-elt/dbt-bq-elt and where the action tries to run is /github/home.

Here is my configuration for the action. Note I am attempting to create a profiles.yml file in the setup dbt section in order for SQLFluff to correctly connect to our BigQuery warehouse:

name: sqlfluff with reviewdog
on:
  pull_request:
jobs:
  test-check:
    name: runner / sqlfluff (github-check)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: 'setup dbt'
        shell: bash
        run: |
          echo "${{ secrets.gcloud_service_key }}" > "${{ github.workspace }}/dbt-service-account.json"
          sed s/template-project-id/${{ secrets.GOOGLE_PROJECT_ID }}/g profiles.github.yaml > profiles.github1.yaml
          sed -i s,template-json-directory,${{ github.workspace }},g profiles.github1.yaml > profiles.github2.yaml
          mkdir -p "~/.dbt"
          mkdir "${{ github.workspace }}/.dbt"
          mv profiles.github2.yaml "~/.dbt/profiles.yml"
      - uses: yu-iskw/action-sqlfluff@v2
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "0.9.1"
          working-directory: /home/runner
          config: "${{ github.workspace }}/.sqlfluff"
          paths: '${{ github.workspace }}/models'
          fail_on_error: true
      - name: 'Show outputs (Optional)'
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

fatal: --unshallow on a complete repository does not make sense

When using this GitHub Action, I encounter the following error message:

🐶 Get changed files
  fatal: --unshallow on a complete repository does not make sense

I suspect the error here, but do not understand why this occurs.

Maybe this condition could help solving this:

if $(git rev-parse --is-shallow-repository); then
    ... repository is shallow ...
fi

Source: StackOverflow


My workflow file:

    steps:
      - name: Checkout repo
        uses: actions/checkout@v3

      - name: Lint added and modified .sql files
        uses: yu-iskw/action-sqlfluff@v3
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "1.4.5"
          sqlfluff_command: "lint"
          config: "${{ github.workspace }}/.sqlfluff"
          paths: "${{ github.workspace }}/models"
          extra_requirements_txt: "${{ github.workspace }}/.github/config/requirements.txt"

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

dockerfile
Dockerfile
  • python 3.12-slim
github-actions
.github/workflows/contributors-list.yml
  • akhilmhdh/contributors-readme-action v2.3.10
.github/workflows/depup.yml
  • actions/checkout v4
  • reviewdog/action-depup v1
.github/workflows/dockerimage.yml
  • actions/checkout v4
.github/workflows/labels.yml
  • actions/checkout v4
  • lannonbr/issue-label-manager-action 4.0.0
.github/workflows/release.yml
  • actions/checkout v4
  • haya14busa/action-bumpr v1
  • haya14busa/action-update-semver v1
  • haya14busa/action-cond v1
  • shogo82148/actions-create-release v1
  • actions/checkout v4
  • haya14busa/action-bumpr v1
.github/workflows/reviewdog.yml
  • actions/checkout v4
  • haya14busa/action-cond v1
  • reviewdog/action-shellcheck v1
  • actions/checkout v4
  • reviewdog/action-shfmt v1
  • actions/checkout v4
  • reviewdog/action-actionlint v1
  • actions/checkout v4
  • reviewdog/action-misspell v1
  • actions/checkout v4
  • reviewdog/action-alex v1
.github/workflows/test.yml
  • actions/checkout v4
  • actions/checkout v4
pip_requirements
requirements/requirements.txt
  • setuptools >=65.5.1
  • pre-commit >=3.7,<3.8
  • pytz >=2015.7
testdata/test_failed_dbt/extra_requirements.txt
  • dbt-postgres >=1.0.0,<2.0
  • dbt-bigquery >=1.0.0,<2.0
  • dbt-redshift >=1.0.0,<2.0
  • dbt-snowflake >=1.0.0,<2.0
  • dbt-spark >=1.0.0,<2.0
testdata/test_failed_dbt/requirements.txt
  • dbt-bigquery >=1.5,<1.6
  • dbt-core >=1.5,<1.6
  • sqlparse >=0.5.0

  • Check this box to trigger a request for Renovate to run again on this repository

Check only added and changed files

Motivation

It takes a long time to check many SQL files with sqlfluff. So, the action should check only added and changed SQL files to shorten the elapsed time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.