lukka / cppbuildtasks Goto Github PK

I've been trying out CppBuildTasks for some cross-platform builds (and thanks for a great resource for this!), and have run into a small issue with caching the results of run-vcpkg when vcpkg is not a submodule. I realise that's not the recommended use case, but thought I'd report what I found just in case this is a real bug or requires additional documentation/warning on the non-submodule case.

What seems to happen is that in the non-submodule case, the run-vcpkg task expects a file named vcpkg_remote_url.last to exist in the vcpkg root. The task creates this file fine, but it is not listed as an exception in the generated .artifactsignore file created for a connected Cache@2 task. Whilst the cache tasks restores vcpkg.exe, .git and installed o.k., the lack of the vcpkg_remore_url.last file causes the run-vcpkg task to remove the vcpkg directory and start from scratch (so rebuilding vcpkg.exe and all wanted packages).

I was able to get things working for this case but setting the hidden vcpkgArtifactIgnoreEntries variable to the form before #15 was merged:

- task: run-vcpkg@0 
    displayName: 'Run vcpkg'
    inputs:
      vcpkgGitCommitId: $(vcpkgGitRef)
      vcpkgTriplet: x64-windows
      vcpkgArguments: fmt libgit2 gtest
      vcpkgArtifactIgnoreEntries: "buildtrees\ndownloads\npackages\n"

That's crude, but gets a cache that's restored and reused o.k.

This was only an experiment for a small library project, hence not using vcpkg as a submodule. I can go in that direction if that's going to be the only method supported by CppBuildTasks going forward.

I use vcpkg to create native nuget packages. For some of them I package both (x86 and x64) flavours. From looking at the source, I figured out that the triplet parser only reports the first occurance of --triplet. But I would like to use run-vcpkg in a way like this

libmodbus
--triplets
x86-windows
x64-windows

or like this

libmodbus
--triplet
x86-windows
libmodbus
--triplet
x64-windows

vcpkg command line already supports this:

libmodbus:x86-windows
libmodbus:x64-windows

Currently I tried to use two run-vcpkg steps, but this seems not to be optimal. It also seems to have issues when being combined with the Cache functionality.

Or am I getting something wrong?

It looks like theres a way to pass arguments as build time, but no way to pass arguments at generation time to run-cmake. Am I missing something obvious, or is this something that actually can't be done?

hi,

i have a build configured for 3 agents (osx/win/linux) win an linux are working just fine... but for osx agent I'm getting this:

2021-02-22T15:39:12.6138780Z ##[section]Starting: Running vcpkg
2021-02-22T15:39:12.6150190Z ==============================================================================
2021-02-22T15:39:12.6150550Z Task         : Run vcpkg
2021-02-22T15:39:12.6150830Z Description  : Run vcpkg to build dependencies.
2021-02-22T15:39:12.6151100Z Version      : 0.9.196
2021-02-22T15:39:12.6151320Z Author       : Luca Cappa
2021-02-22T15:39:12.6151700Z Help         : [More information](https://github.com/lukka/CppBuildTasks/blob/master/README.md)
2021-02-22T15:39:12.6152130Z ==============================================================================
2021-02-22T15:39:15.4226760Z Set task output variable 'RUNVCPKG_VCPKG_ROOT_OUT' to value: /Users/runner/work/1/s/tchain/vcpkg
2021-02-22T15:39:15.5240150Z Fetching the commit id at /Users/runner/work/1/s/tchain/vcpkg
2021-02-22T15:39:15.5675600Z [command]/usr/local/bin/git rev-parse HEAD
2021-02-22T15:39:15.5753530Z ec6fe06e8da05a8157dc8581fa96b36b571c1bd5
2021-02-22T15:39:15.5990920Z Checking whether vcpkg's repository is updated to commit id 'ec6fe06e8da05a8157dc8581fa96b36b571c1bd5' ...
2021-02-22T15:39:15.7373200Z [command]/usr/local/bin/git submodule status /Users/runner/work/1/s/tchain/vcpkg
2021-02-22T15:39:15.7442290Z  ec6fe06e8da05a8157dc8581fa96b36b571c1bd5 tchain/vcpkg (2020.04-1442-gec6fe06e8)
2021-02-22T15:39:15.7592940Z 'vcpkg' is detected as a submodule, adding '.git' to the ignored entries in '.artifactignore' file (for excluding it from caching).
2021-02-22T15:39:15.7695560Z File '.artifactsignore' content: ''buildtrees', 'packages', 'downloads', '.git''
2021-02-22T15:39:15.7795220Z Is vcpkg repository updated? Yes
2021-02-22T15:39:15.7853680Z Checking last vcpkg build commit id in file '/Users/runner/work/1/s/tchain/vcpkg/vcpkgLastBuiltCommitId' ...
2021-02-22T15:39:15.8156280Z vcpkg executable is up to date with sources.
2021-02-22T15:39:15.9256510Z [command]/bin/chmod +x /Users/runner/work/1/s/tchain/vcpkg/vcpkg
2021-02-22T15:39:15.9465980Z Running 'vcpkg remove --outdated --recurse' in directory '/Users/runner/work/1/s/tchain/vcpkg' ...
2021-02-22T15:39:16.0110430Z [command]/Users/runner/work/1/s/tchain/vcpkg/vcpkg remove --outdated --recurse
2021-02-22T15:39:16.0210230Z /Users/runner/work/1/s/tchain/vcpkg/vcpkg: /Users/runner/work/1/s/tchain/vcpkg/vcpkg: cannot execute binary file
2021-02-22T15:39:16.0509590Z ##[error]Last command execution failed with error code '126'.
2021-02-22T15:39:16.0624780Z ##[error]Error: Last command execution failed with error code '126'.
2021-02-22T15:39:16.0792350Z ##[error]vcpkg failed with error: 'Error: Last command execution failed with error code '126'.'.
2021-02-22T15:39:16.0930500Z ##[section]Finishing: Running vcpkg

Issue:

Using azure cache beta feature, I am able to cache $(Build.BinariesDirectory)/vcpkg/installed after I use vcpkg to install all the required packages. So in the following run, my pipeline restore $(Build.BinariesDirectory)/vcpkg/installed cache and then run the same vcpkg task again.

Expect: Vcpkg will skip installing the packages as all of them have been restored.
Actual: Vcpkg re-install the packages again.

I realised that before I run vcpkg task, $(Build.BinariesDirectory) will be empty and running vcpkg task is doing a git checkout to vcpkg branch as shown in the flowchat. I suspect that git checkout override the restored $(Build.BinariesDirectory)/vcpkg/installed ? Any thoughts?

Work around:

I have to run vcpkg task before the cache restore with --dry-run argument, then run another vcpkg task to install the packages. This work fine, however both vcpkg run execute bootstrap-vcpkg.bat which took roughly 1.5 minutes each.

Suggestion

I think it will be good if there is a build vcpkg task that is seperated from run vcpkg. Hence the pipeline can be:

git checkout vcpkg -> run bootstrap-vcpkg.bat -> restore cache -> vcpkg install packages

This will give more control with the vcpkg task.

the workflow chart for the run-vcpkg task contains two errors: the initial restore cache, and the final 'save cache', both of them are not existing in this task.

Those are only available on the run-vcpkg GitHub action, for the Azure DevOps run-vcpkg task there is no automatic caching mechanism at all.

Documentation should be splitted in the shared submodule 'libs'.

Reused, unless the task run-cmake is provided with a triplet explicitly in its parameter.

I had a weird error when using the runcmake task where it would fail right away with the following error message. Turns out this was due to CMakeSettings.json being saved as UTF8 with BOM, when changing the encoding to UTF8 (without BOM) using VSCode, the task could proceed.

SyntaxError: Unexpected token  in JSON at position 0

Please refers to this bug reported on the Cache task.

Its common to run ctest-based tests as part of continuous integration builds. My attempt to run the built tests after a build was:

- script: 'ctest -j 1 --interactive-debug-mode 0 --output-on-failure -R "in_memory"'
    workingDirectory: '$(Build.ArtifactStagingDirectory/<config-name'
    displayName: 'Run Ctest'

But it always failed to discover the tests. My assumption here is that I am missing the right parameter to specify the build directory.

In CMakeSettings.json, Visual Studio still uses the old style of specifying the generator architecture "Visual Studio 16 2019 Win64", while CMake does not support this style anymore and instead requests using "Visual Studio 16 2019" and additionally the parameter "-A Win64".

The task runcmake seems to use the exact generator as written in the CMakeSettings.json, but it should detect VS2019 and higher and extract the architecture from the generator field. Otherwise the build on azure breaks with the error message:

Processing configuration: 'x64-Release'. 
  | :\Program Files\CMake\bin\cmake.exe" -G "Visual Studio 16 2019 Win64" -DCMAKE_BUILD_TYPE=RelWithDebInfo  d:\a\1\s |  
  | CMake Error: Could not create named generator Visual Studio 16 2019 Win64 |  
  |   |  
  | Generators |  
  | * Visual Studio 16 2019         = Generates Visual Studio 2019 project files. |  
  |                                   Use -A option to specify architecture.

Hi, is there any easy way to install the CppBuildTasks even on a small local DevOps Server running Azure DevOpsServer 2019 ?
Thanks for your support.

e.g. if this is provided

```"inheritEnvironments": [ "msvc_x64_x64" ]````

then the x64_x64 cmd prompt should be used to establish the environment for MSVC.

Only on Windows and if related to MSVC the task should be using the inheritEnvironments value if provided.

Hi,
Trying to migrate to your actions from my own scripted vcpkg/cmake which work but don't cache.
I am getting this error in the action's output in the set up page, my workflow is based on any of your samples...
##[error]An action could not be found at the URI 'https://api.github.com/repos/lukka/run-vcpkg/tarball/dev/disable_cache'

This is before any of my params are even read...
Current runner version: '2.272.0'
Operating System
Ubuntu
18.04.4
LTS
Virtual Environment
Environment: ubuntu-18.04
Version: 20200806.0
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200806.0/images/linux/Ubuntu1804-README.md
Prepare workflow directory
Prepare all required actions
Download action repository 'actions/checkout@v1'
Download action repository 'lukka/get-cmake@latest'
Download action repository 'lukka/set-shell-env@master'
Download action repository 'lukka/run-vcpkg@dev/disable_cache'
##[error]An action could not be found at the URI 'https://api.github.com/repos/lukka/run-vcpkg/tarball/dev/disable_cache'

I recently stumbled to build some libs (boost, ceres and qt-5-base) on a Microsoft hosted agent in DevOps due to insufficient disk space. It took me a while to figure out it was a disk space issue and then again some to to find out I can workaround by cleaning the buildtrees folder using the vcpkg option --clean-after-build.

I wasn't sure if it works if I just put it as last line into the pkg file, but it seems to do the trick.

boost-math:x64-linux
boost-test:x64-linux
boost-property-tree:x64-linux
ceres:x64-linux
qt5-base:x64-linux
--clean-after-build

I think it is woth to highlight this problem somewhere in the docs (Disk space limit is mentioned here) and to extend the run-vcpkg task with an "clean" option which makes it easy to discover and set.

Hello!

There is a configuration

• azure-devops.cmakesettings.json

{
 "configurations": [
   {
      "name": "x64-windows",
      "configurationType": "Release",
      "inheritEnvironments": [ "msvc_x64" ],
      "buildCommandArgs": "",
      "cmakeCommandArgs": "",
      "ctestCommandArgs": "",
      "generator": "Visual Studio 16 2019 Win64"
   }
 ]
}

• azure-pipelines.yml

  - task: run-cmake@0
    displayName: 'Run CMake with CMakeSettings.json'
    inputs:
      cmakeListsOrSettingsJson: 'CMakeSettingsJson'
      cmakeSettingsJsonPath: 'azure-devops.cmakesettings.json'
      useVcpkgToolchainFile: false
      configurationRegexFilter: '^(x64-windows)$'

CMake configure stage is correct:

Overriding build directory to: 'C:\agent\_work\1\a\x64-windows'
"C:\Program Files\CMake\bin\cmake.exe" -G "Visual Studio 16 2019" -A x64 -DCMAKE_BUILD_TYPE=Release C:\agent\_work\1\s
-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.17763.
-- The CXX compiler identification is MSVC 19.24.28314.0
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Failed
-- Performing Test COMPILER_HAS_DEPRECATED
-- Performing Test COMPILER_HAS_DEPRECATED - Success
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_BUILD_TYPE


-- Build files have been written to: C:/agent/_work/1/a/x64-windows

But build stage is build Debug configuration:

Building with CMake in build directory 'C:\agent\_work\1\a\x64-windows' ...
"C:\Program Files\CMake\bin\cmake.exe" --build . --
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  Checking Build System
  Building Custom Rule C:/agent/_work/1/s/CMakeLists.txt
  main.cpp
     Creating library C:/agent/_work/1/a/x64-windows/Debug/sample-addin.lib and object C:/agent/_work/1/a/x64-windows/Debug/sample-addin.exp
  sample-addin.vcxproj -> C:\agent\_work\1\a\x64-windows\bin\Debug\sample-addin.dll
  Building Custom Rule C:/agent/_work/1/s/test/CMakeLists.txt
  main.cpp
  test-sample-addin.vcxproj -> C:\agent\_work\1\a\x64-windows\bin\Debug\test-sample-addin.exe
  Building Custom Rule C:/agent/_work/1/s/CMakeLists.txt

This is because no configuration is specified in cmake --build
Similar problem

buildWithCMakeArgs: '--config Release' is ignored for CMakeSettingsJson

Сan you fix it please. Ninja generator doesn't always fit, and I'm not very into JS to fix it.
buildWithCMakeArgs

Thanks.

I'd like to use the run-cmake task without the need for actually building anything in vcpkg. I still need to build vcpkg since run-cmake relies on vcpkg env so for now I'm building a package I don't need.

In azure-pipelines.yml I have a run-cmake task script as shown below:

- task: run-cmake@0
  inputs:
    cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
    cmakeListsTxtPath: '$(Build.SourcesDirectory)/CMakeLists.txt'
    useVcpkgToolchainFile: true
    buildDirectory: '$(Build.ArtifactStagingDirectory)'
    ninjaPath: '$(Build.BinariesDirectory)/downloads/'
    cmakeAppendedArgs: '-G Ninja -DCMAKE_BUILD_TYPE=Release'

When I do a build, I am getting error saying CMakeLists.txt not found in "D:/a/1/a" which seems to be a wrong directory. I specified the cmakeListsTxtPath to $(Build.SourcesDirectory)/CMakeLists.txt as shown in the yml script above.

##[section]Starting: runcmake
==============================================================================
Task         : Run CMake
Description  : Run CMake (and optionally vcpkg and Ninja) with a CMakeLists.txt or with CMakeSettings.json created with Visual Studio
Version      : 0.3.11
Author       : Luca Cappa
Help         : [More information](https://github.com/lukka/CppBuildTasks/blob/master/README.md)
==============================================================================
[command]"C:\Program Files\CMake\bin\cmake.exe" -G Ninja -DCMAKE_BUILD_TYPE=Release "-DCMAKE_MAKE_PROGRAM=d:\a\1\b\downloads\" d:\a\1\s"
CMake Warning:
  No source or binary directory provided.  Both will be assumed to be the
  same as the current working directory, but note that this warning will
  become a fatal error in future CMake releases.


CMake Error: The source directory "D:/a/1/a" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
##[error]Error: CMake failed with error: '1'.
##[error]CMake failed with error: 'Error: CMake failed with error: '1'.'.
##[section]Finishing: runcmake

I do not understand why it is looking into a wrong directory although I have already specified a directory. Any idea?

When using this task with a self-hosted agent which does not set clean: True in checkout, vcpkg is not re-bootstrapped. One could either bootstrap anytime the commit pointed to by the submodule changes or the output of vcpkg version can be compared to toolsrc/VERSION.txt for example.

See this CI run.

Currently, the README says:

It is highly recommended to use vcpkg as a Git submodule.

and

when vcpkg is NOT a submodule (not recommended)

I believe it would be useful if FAQ offered an explanation of what are the reasons a submodule is the recommended way to set it up.

• Drop the submodule containing the code shared with run-vcpkg and run-cmake action.

• Use the NPM packages from lukka/run-cmake-vcpkg-action-libs.

• Ensure functional validation on Azure DevOps Pipelines.

Hi @lukka, I have been using CppBuildTasks for awhile and it has been great. Just have a few question here with pipeline caching feature.

Question 1:
Your example is caching '$(Build.BinariesDirectory)/vcpkg', the entire folder of vcpkg. For my case, the whole vcpkg folder is getting up to around 5GB which is big to cache. I wonder if you have an example to cache '$(Build.BinariesDirectory)/vcpkg/installed' folder?

Question 2:
After restored the cache, run vcpkg task will remove the outdated packages and reinstall them. This all go well except the cache task does not recache the updated packages because the cache key isn't invalidated. The only solution I can think of at the moment is to use a new cache key, invalidate the existing cache key and redo all vcpkg task from scratch which is not nice.

Let say I want to install one more package and update two packages, I would wish that the cache task will restore the installed packages, vcpkg remove outdated packages and reinstall the new version, vcpkg install the new added package, and finally recache the whole installed packages in post-job. This will avoid to redo vcpkg task from scratch if there is new package version or just adding one new package.

Any thoughts?

Will this extension work with vcpkg in manifest mode? Does anything need to change with regards to the vcpkgGitRef?

• The hardcoded --recurse flag forces vcpkg into classic mode.
• The required vcpkgGitCommitId seems to clash with manifest mode's baseline versioning.