—env VALIDATE_ALL_CODEBASE=false does not work in docker about megalinter HOT 12 CLOSED

Found it :)
And it had already been found in #1515 (comment) , we really need to update the documentation :)

Here is a working job, with 2 updates:

• checkout step
• remove --env GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \: token remains set by checkout step

jobs:
  # Run MegaLinter to detect linting and security issues
  - job: MegaLinter
    pool:
      vmImage: ubuntu-latest
    steps:

      # Checkout repo
      - checkout: self
        fetchDepth: 0
        persistCredentials: true
        displayName: Git Checkout

      # Pull MegaLinter docker image
      - script: docker pull oxsecurity/megalinter:v7
        displayName: Pull MegaLinter

      # Run MegaLinter
      - script: |
          docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
            --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
            -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
            -e VALIDATE_ALL_CODEBASE=false \
            oxsecurity/megalinter:v7
        displayName: Run MegaLinter

      # Upload MegaLinter reports
      - task: PublishPipelineArtifact@1
        condition: succeededOrFailed()
        displayName: Upload MegaLinter reports
        inputs:
          targetPath: "$(System.DefaultWorkingDirectory)/megalinter-reports/"
          artifactName: MegaLinterReport

Result:

@RolfMoleman please can you confirm it works on your side with such updates ?

from megalinter.

@white-james @DanBerns

For visibility

from megalinter.

I see that the formed URL to fetch might not be well written for azure DevOps. But can you make sure that if Azure DevOps has some permission controls to prevent fetching the repo, that the action (inside docker), is allowed to pull/fetch)? Not validating all code base means that we use git to know what files have changed.

from megalinter.

@nvuillam do you have experience with Azure DevOps ? I don't really

from megalinter.

@echoix @nvuillam This used to work in v7.6 and I think v7.7 (I can test and confirm) but has been seemingly broken since v7.8.

The permissions in Azure DevOps remain unchanged so it definitely seems to be a sicker image issue. I'll see what else I can find and update here.

from megalinter.

@cmdmescall you might be interested in this

from megalinter.

I'm currently investigating the issue in an azure repo

from megalinter.

I succeed to reproduce the problem, that's a start ^^

from megalinter.

Did they (azure) change their permissions or some sort?

Is the same bug now reproduced with an older image? (To see if it is the image version or the environment that changes the result)

from megalinter.

It also crashes with 7.7.0

I wonder if it ever worked with VALIDATE_ALL_CODEBASE=false on Azure

MegaLinter now collects the files to analyse (expand for details)
Listing updated files in [/tmp/lint] using git diff.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/megalinter/run.py", line 14, in <module>
    linter.run()
  File "/megalinter/MegaLinter.py", line 203, in run
    self.collect_files()
  File "/megalinter/MegaLinter.py", line 645, in collect_files
    all_files = self.list_files_git_diff()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/megalinter/MegaLinter.py", line 755, in list_files_git_diff
    repo.git.fetch("origin", f"{remote_ref}:{local_ref}")
  File "/usr/local/lib/python3.11/site-packages/GitPython-3.1.40-py3.11.egg/git/cmd.py", line 736, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/GitPython-3.1.40-py3.11.egg/git/cmd.py", line 1316, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/GitPython-3.1.40-py3.11.egg/git/cmd.py", line 1111, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch origin HEAD:refs/remotes/origin/HEAD
  stderr: 'fatal: unable to access 'https://dev.azure.com/nicolasvuillamy/test-megalinter/_git/test-megalinter/': The requested URL returned error: 400'

for the moment I tried to add an extra step to persist git credentials, but it still triggers a git access error :/

      # Checkout repo
      - checkout: self
        fetchDepth: 0
        persistCredentials: true
        displayName: Git Checkout

from megalinter.

Hi @nvuillam i can confirm removal of

remove --env GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \

Does indeed work. We've got some final tweaks to perform at our end but am happy to supply our job template for use in docs as a more complex Azure devops example demonstrating different run types based on pipeline trigger types and publishing of junit test results and sarif reports to code analysislogs if it would be beneficial?

from megalinter.

@RolfMoleman I'm glad it now works for you :)

And that would indeed be nice to enhance the default azure template if it handles more business cases ;-) (i love the sarif one ^^)

from megalinter.