Git Product home page Git Product logo

azure-pipelines-tasks-common-packages's Introduction

Shared Npm packages

These packages are only meant for use by in the box tasks - contributions will only be accepted for those purposes.

Publishing should happen automatically on CI builds, for any issues with this process please reach out to @tkasparek/@martinmrazik and/or the Azure Pipelines Platform team.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

azure-pipelines-tasks-common-packages's People

Contributors

aleksandrlevochkin avatar alexvtor avatar csigs avatar denisnikulin5 avatar denisrumyantsev avatar dmitriibobreshev avatar embetten avatar ivanduplenskikh avatar jessehouwing avatar jiaw37 avatar kirill-ivlev avatar konstantintyukalov avatar lefttwixwand avatar max-zaytsev avatar merlynomsft avatar microsoftopensource avatar microtuld avatar mmrazik avatar onetocny avatar ozanmsft avatar patelchandni avatar sergey-koryshev avatar starkmsu avatar tintse-thxsky-msft avatar tkasparek avatar v-mohithgc avatar viralpandya avatar vmapetr avatar yevheniikholodkov avatar zhenghao104 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

azure-pipelines-tasks-common-packages's Issues

Use Name reference for secure files instead of ID

Current mechanism

Currently the tasks use secure files objects referenced by id.

Proposed mechanism

The tasks should reference every secure file by name and only resolve the id when it is necessary like creating a download ticket.

Justification

The name of the secure file is also a unique identifier. In a project only 1 secure file can exist with any given name.
The mechanism of referencing by id also breaks every task that was storing the id as a ref when someone wants to update the content of the secure file because the update itself removes the secure file and then recreates it with the same name.

Fix

#307

Tasks

Managed Identity Service Connection Not Supported in All Agent Runtime Environments

The following code in azure-arm-common.ts:357-361 is dated and too prescriptive, resulting in a failure to leverage Managed Identity in Azure Container Apps hosting my Azure Pipelines agents:

let apiVersion = "2018-02-01";
webRequest.uri = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=" + apiVersion + "&resource=" + resourceId;
webRequest.headers = {
    "Metadata": true
};

The current code results in the following debug output after two minutes from a pipeline using the AzureWebApp@1 task.

2024-06-16T10:07:46.2198904Z Got service connection details for Azure App Service:'[redacted]'
2024-06-16T10:07:46.2199146Z ##[debug]MSAL - getMSALToken called. force=undefined
2024-06-16T10:07:46.2199319Z ##[debug]agent.proxyurl=undefined
2024-06-16T10:07:46.2199492Z ##[debug]agent.proxybypasslist=undefined
2024-06-16T10:07:46.2199705Z ##[debug][Sun, 16 Jun 2024 10:07:43 GMT] : @azure/[email protected] : Info - acquireTokenByClientCredential called
2024-06-16T10:07:46.2199986Z ##[debug][Sun, 16 Jun 2024 10:07:44 GMT] : [a3cbad3d-90fb-452e-bac8-76e65feb7c77] : @azure/[email protected] : Info - Using appTokenProvider extensibility.
2024-06-16T10:07:46.2200221Z ##[debug]MSAL - ManagedIdentity is used.
2024-06-16T10:07:46.2200389Z ##[debug]CLIENT_RESETSTREAMONRETRY=undefined
2024-06-16T10:07:46.2200610Z ##[debug][GET]http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/
2024-06-16T10:07:46.7059248Z ##[debug]Agent environment resources - Disk: / Available 10771.00 MB out of 20064.00 MB, Memory: Used 1434.00 MB out of 2667.00 MB, CPU: Usage 13.57%
2024-06-16T10:07:51.7054738Z ##[debug]Agent environment resources - Disk: / Available 10774.00 MB out of 20064.00 MB, Memory: Used 1448.00 MB out of 2667.00 MB, CPU: Usage 12.94%
2024-06-16T10:07:56.7060122Z ##[debug]Agent environment resources - Disk: / Available 10774.00 MB out of 20064.00 MB, Memory: Used 1451.00 MB out of 2667.00 MB, CPU: Usage 11.26%
...
2024-06-16T10:09:53.9885966Z ##[debug]Encountered a retriable error:ETIMEDOUT. Message: connect ETIMEDOUT 169.254.169.254:80.

As an example, the following is the constructed URL for obtaining MSI tokens based on the runtime environment of one replica of my agents.

http://localhost:42356/msi/token?api-version=2019-08-01&resource=https://management.azure.com

See an image capture from an example bash console session below:

image

I would suggest a backwards-compatible rewrite of the code to the following to leverage a more recent MSI API:

const useMsi2019 = process.env.IDENTITY_ENDPOINT && process.env.IDENTITY_HEADER;
const apiVersion = useMsi2019 ? "2019-08-01" : "2018-02-01";
const tokenEndpoint = useMsi2019 ? process.env.IDENTITY_ENDPOINT : "http://169.254.169.254/metadata/identity/oauth2/token";
webRequest.uri = `${tokenEndpoint}?api-version=${apiVersion}&resource=${resourceId}`;

webRequest.headers = {
    "Metadata": true
};

if (useMsi2019) {
    webRequest.headers["X-Identity-Header"] = process.env.IDENTITY_HEADER;
}

If this repository is accepting contributions, please tag me if desired and I'll submit a pull request with this information.

Remove 'Tests' from azure-pipelines-tasks-webdeployment-common package

The webdeployment-common package has a Tests directory. This directory should be removed from the published package. The reason for that in my particular case is that the file common-npm-packages/webdeployment-common/Tests/L1ZipUtility/potentially_malicious.zip is detected by the anti-virus/malware scanner.
Is it possible to remove the Test sources from the published package?

InstallAppleCertificate assumes OpenSSL supports algorithm RC2-40-CBC

Apple's development certificates are issued using algorithm RC2-40-CBC, which is supported by LibreSSL 3.3 shipped with MacOS Sonoma (the binary confusingly named "openssl"), but is not supported by OpenSSL 3.3 without the -legacy flag. Because this task uses which to find the openssl binary, if OpenSSL 3 is installed on the system (via Homebrew for example), the task will fail without the pipeline definition explicitly providing the -legacy flag.

The task should check whether the algorithm of the provided certificate is supported by OpenSSL, and add that flag automatically if required.

'Failed to enable code coverage: TypeError: Cannot read properties of undefined (reading 'split')' error when enabling JaCoCo coverage on Ant task without supplying optional input

The issue was previously opened in the azure-pipelines-tasks repo, as issue #19724, but I will be re-opening the issue in this repo (the correct one).

To summarize, when creating an Azure DevOps build pipeline, and working with Ant@V1 task, enabling JaCoCo code coverage is failing, and is generating the following error:

##[warning]Failed to enable code coverage: TypeError: Cannot read properties of undefined (reading 'split')
Here is the log output:

masking file paths with (...)

##[debug]classFilter=undefined
##[debug]classFilesDirectories=build/src/
##[debug]srcDirectories=undefined
##[debug]rm -rf (...)/cobertura.ser
##[debug]rm -rf (...)/CCReport43F6D5EF
##[debug]removing directory
##[debug]rm -rf (...)/CCReportBuildA4D283EG.xml
##[debug]removing file
##[debug]rm -rf (...)/InstrumentedClasses
##[debug]Input parameters: {"buildfile":"(...)/build.xml","classfilesdirectories":"build/src/","summaryfile":"summary.xml","reportdirectory":"(...)/CCReport43F6D5EF","ccreporttask":"CodeCoverage_9064e1d0","reportbuildfile":"(...)/CCReportBuildA4D283EG.xml"}
##[debug]Extracting Azure Pipelines filter: undefined
##[debug]Applying the filter pattern:  op: 
##[debug]Applying the filter pattern:  op: 
##[debug]Reading XML file: (...)/build.xml
##[debug]Converting XML file to JSON
##[warning]Failed to enable code coverage: TypeError: Cannot read properties of undefined (reading 'split')
##[debug]Processed: ##vso[task.issue type=warning;]Failed to enable code coverage: TypeError: Cannot read properties of undefined (reading 'split')

This error was generated with the following YAML configuration for the Ant@V1 task:

codeCoverageToolOptions: 'JaCoCo'
codeCoverageClassFilesDirectories: 'build/src/'

After several days of debugging, our team found the issue was able to be resolved by adding the following optional field to the YAML configuration for the Ant@V1 task:

codeCoverageSourceDirectories: 'components/src/'

This is inconsistent with the Ant@V1 documentation, which states the codeCoverageSourceDirectories input is optional when codeCoverageToolOptions != None. I looked into the JaCoCo code coverage functions and believe I found the root cause of the issue in the enableCodeCoverage() function in the jacoco.ant.cc.enabler class.

In enableCodeCoverage(), the global sourceDirs variable is set to the value of ccProps["sourcedirectories"], which contains the value of the INPUT_SRCDIRECTORIES environment variable passed from the Ant task.

From there, addCodeCoverageData() is called, which then calls getSourceFilter().

    protected getSourceFilter(): string {
        let srcData = "";
        let srcDirs = this.sourceDirs === null ? "" : this.sourceDirs;
        srcDirs.split(",").forEach(dir => {
            if (!util.isNullOrWhitespace(dir)) {
                srcData += `<fileset dir="${dir}"/>`;
                srcData += os.EOL;
            }
        });
        if (util.isNullOrWhitespace(srcData)) {
            srcData = `<fileset dir="."/>`;
            srcData += os.EOL;
        }
        return srcData;
    }

If the codeCoverageSourceDirectories YAML input is not given a value, then INPUT_SRCDIRECTORIES is undefined, meaning the value of global var sourceDirs is set to undefined. Thus, calling split() on an undefined value generates the error and causes JaCoCo code coverage to fail on the pipeline.

I believe the issue is due to a lack of handling undefined values. At the same time, that doesn't seem to affect that classFilter variable when that value is undefined. I have reproduced this on several build pipelines, all to the same effect. This probably just needs a bare minimum change like

    if (srcDirs) {
        srcDirs.split(",").forEach(dir => {
               // continues if srcDirs != undefined
        });
    }
    else {
        // continues if srcDirs == undefined
    }

but not sure if anyone has run into this issue ever...

Npm@1 - Translation missing from debug call breaks build due to new warning

There is a translated debug print statement here in getSystemAccessToken() in locationUtilities.ts, but the translation doesn't exist in resources/en-US/resources.resjson

This generates a warning ##[warning]Can't find loc string for key: Info_GotAndMaskAuth which prevents us from merging changes due to the Build Quality Checks plugin preventing us from increasing the warning counts of our software builds.

Please fix the warning/add the missing translation

Unable to change app service slot setting back to false

Using the yaml task AzureAppServiceSettings@1 here , I am able to make any variable's slotSetting equal to true in my app service's appsettings configuration, but I am unable to change it back to false without manual intervention (which is not possible in my prod environment). The value of the setting always updates but not the slotSetting boolean value.

After some investigation I've noticed that AzureAppServiceSettings@1 calls appServiceUtility.updateAndMonitorAppSettings(...) which is here, and that calls both appService.patchApplicationSettings(...) (line 164) and appService.patchApplicationSettingsSlot(...) (line 192) which is here. The value of the property is always updated by patchApplicationSettings, but not the 'slotSetting' boolean value. I believe the problem is in patchApplicationSettingsSlot. If I set an appsetting with slotSetting = true then the code here will update the value and slotSetting value to true, however if slotSetting = false then it ignores it completely and will not edit the slotSetting value back to false.

This is causing a problem in our deployment because there are some variables that were previously set as slotSetting = true that we now wish to change to slotSetting = false so that the value will move over when we swap slots for our app service in azure from staging to production. However because of this bug the slotSetting is never changed to false even though we explicitly tell it to.

Request to fix error related to KuduStackTraceURL

#217 was opened to try and address this.
I think the issue here is that it was added at the wrong level.

common-npm-packages/azure-arm-rest/Strings/resources.resjson/en-US/resources.resjson

Whereas it should be added at https://github.com/microsoft/azure-pipelines-tasks-common-packages/blob/main/common-npm-packages/azurermdeploycommon/azure-arm-rest/Strings/resources.resjson/en-US/resources.resjson and is still missing from there.

The Task imports it from here.

 catch(error) {
        let stackTraceUrl:string = this._appServiceKuduService.getKuduStackTraceUrl();
        tl.error(tl.loc('PackageDeploymentFailed'));
        tl.error(tl.loc('KuduStackTraceURL', stackTraceUrl));
        throw Error(error);
    }

import { Kudu } from 'azure-pipelines-tasks-azure-arm-rest/azure-arm-app-service-kudu';

Can you please look at this again?

It is happening still. From the last 30 days.

WarningMessage TaskVersion count_
Can't find loc string for key: KuduStackTraceURL 1.231.0 6662
Can't find loc string for key: KuduStackTraceURL 2.231.0 3300

AddLabel does not work with variable values with spaces (non escaped characters)

When I use the Docker@V2 build command with a 'teamproject' name with a space in it, the build command fails.
The following label causes the issue: com.azure.dev.image.system.teamproject
It is added here

The cause of this issue is that the addLabel function doesn't properly escape the variable value here.

Example error when using Docker@V2:
/usr/bin/docker build -f /home/ubuntuserver/myagent/_work/95/s/xxx/Dockerfile --label xxx.image.system.teamfoundationcollectionuri=https://xxx/ --label xxx.image.system.teamproject=My Project --label xxx.image.build.repository.name=omp-companion --label xxx.image.build.sourceversion=c0f460b6923ca2c7a3949614d2ce9933dd8b7700 --label xxx.image.build.repository.uri=
"docker build" requires exactly 1 argument.

Hard Coded API version in API call not compatible with Azure Stack Hub

The function with the issue:

getSitePublishingCredentialPolicies() in azure-pipelines-tasks-common-packages/common-npm-packages/azurermdeploycommon/azure-arm-rest/azure-arm-app-service.ts

The issue:

This function passes api version '2022-03-01' which is not supported by Azure Stack Hub. This is called by Azure Devop's 'Azure App Service deploy' task (https://github.com/microsoft/azure-pipelines-tasks/tree/c61bf51acafa7f1b51b2b7271d766a0e5bdc6003/Tasks/AzureRmWebAppDeploymentV4) which is used on Azure DevOps Server. The upgrade of Azure DevOps Server installed the newer incompatible version which broke all deployments in our organization using that deployment task.

The fix:

Pass the API version of '2020-12-01' or better yet, check what versions are supported and don't use an unsupported version.

Line of code reference:

Bug introduced in this commit:

885a11f#diff-0a2aa407bd15f8009b4661e830703df84cef0cc4cced168290d308c3feaacb7a

Use buildx instead of build for docker Images

We get a warning in our azdo pipeline (Docker@2):

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Are there plans, to support the "new" buildx command?
Code:

Reference: https://docs.docker.com/engine/deprecated/#deprecated-engine-features-1 (Legacy Builder for Linux Images)

[Issue]: Agent install being flagged for CVE-2023-49210

Hi,

My company is using Wiz to scan various virtual machine resources, and we are running some Windows build agents through Azure DevOps.

Agent version - 3.241.0

The agent and work directories for each agent service are being flagged by Wiz due to having older version of openssl
(1.0.2l)

See vulnerability reference here: [https://github.com/advisories/GHSA-75w2-qv55-x7fv] ([
"https://gist.github.com/mcoimbra/b05a55a5760172dccaa0a827647ad63e",
"https://github.com/ossf/malicious-packages/tree/main/malicious/npm",
"https://www.npmjs.com/package/openssl"])

A specific example of one of these flags:

image

image

Is there currently a PR in progress to address this, or otherwise an ETA for resolution? Any input would be appreciated.

Agent version: 3.241.0
Azure DevOps Server type: dev.azure.com
Operation system: Windows 11
Version control system: GitHub

Best Regards,

Marcelo Calado

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.