Git Product home page Git Product logo

dotnet-buildtools-prereqs-docker's Introduction

Dockerfiles for .NET Core Builds

The Dockerfiles in this repository are used for building the .NET Core product. As such there are Dockerfiles for the various supported Linux distributions which setup the necessary prerequisites to build the .NET Core product.

Where are the published images

The images produced from the Dockerfiles are published to the mcr.microsoft.com/dotnet-buildtools/prereqs Docker repository.

How to identify an image

The tag format used by an image is mcr.microsoft.com/dotnet-buildtools/prereqs:<linux-distribution-name>-<version>-<variant>-<architecture>-<date-time>-<dockerfile-commit-sha>

  • <linux-distribution-name> - name of the Linux distribution the image is based on
  • <version> - version of the Linux distribution
  • <variant> - name describing the specialization purpose of the image. Often special dependencies are needed for certain parts of the product. It can be beneficial to separate these dependencies into a separate Dockerfile/image.
  • <architecture> - the docker image architecture (amd64 shall be implied if not specified).
  • <date-time> - UTC timestamp (yyyyMMddhhmmss) of when the image was built
  • <dockerfile-commit-sha> - Git commit SHA of the folder containing the Dockerfile the image was produced from

How to modify or create a new image

There will be a need for modifying existing Dockerfiles or creating new ones. For example, when a new Linux distribution/version needs to be supported, a corresponding Dockerfile will need to be created. The following steps are a guideline for modifying/creating Dockerfiles.

  1. Edit Dockerfiles

    • Add/Update the Dockerfile(s)
    • If new Dockerfile(s) were added:
      • Update the manifest
      • Update the CODEOWNERS with the respective team code owner(s) (not individual users) for the Dockerfile(s) and list @dotnet/dotnet-docker-reviewers as a secondary owner. Team code owners must be assigned to each Dockerfile for maintenance and issue assignment purposes.
  2. Validate the changes locally by running build.ps1. It is strongly suggested to specify the -DockerfilePath option to avoid the overhead of building all the images.

    For example, if editing the Fedora 30 Dockerfile, then run the following command to build just that Dockerfile.

    .\build.ps1 -DockerfilePath "*fedora/30/amd64*"

    It is a good practice to use --dry-run option on the first attempt to verify what commands will get run.

    .\build.ps1 -DockerfilePath "*fedora/30/amd64*" -ImageBuilderCustomArgs "--dry-run"

    Partial paths and wildcards in the -DockerfilePath option are also supported. The following example will build all the Fedora Dockerfiles.

    .\build.ps1 -DockerfilePath "*fedora/*"
  3. Prepare a PR

Additional Info

Source Folder Structure

The folder structure used in src aligns with the tagging convention - <linux-distribution-name>-<version>-<variant>-<architecture>. For example, the Dockerfile used to produce the mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.9-50f0d02-20191210224540 image is stored in the src/alpine/3.9/amd64 folder.

Manifest

The manifest.json contains metadata used by the build infrastructure to produce the Docker images. The metadata describes which Dockerfiles to build, what tags to produce, and where to publish the images. It is critical that the manifest gets updated appropriately when Dockerfiles are added/removed. The manifest at the root of the repo represents the global manifest. It has references to sub-manifests within each of the OS folders (e.g. src/alpine/manifest.json). When adding or modifying entries for Dockerfiles, those changes should be made to the appropriate OS-specific sub-manifest file. Each Dockerfile will have an entry that looks like the following.

{
  "platforms": [
    {
      "dockerfile": "alpine/3.9/amd64",
      "os": "linux",
      "tags": {
        "alpine-3.9-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {}
      }
    }
  ]
},
{
  "platforms": [
    {
      "architecture": "arm",
      "dockerfile": "src/debian/9/arm32v7",
      "os": "linux",
      "tags": {
        "debian-9-arm32v7-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {}
      },
      "variant": "v7"
    }
  ]
},
  • architecture - architecture of the image (default is amd64) [amd64/arm/arm64]
  • dockerfile - relative path to the Dockerfile to build
  • os - (linux/windows) the OS type the Docker image is based on
  • tags - the collection of tags to create for the image
  • variant - architecture variant of the image
  • $(System:TimeStamp) and $(System:DockerfileGitCommitSha) - built in variable references that are evaluated at build time and substituted

Note: The position in manifest determines the sequence in which the image will be built.

Image Dependency

A precondition for building an image is to ensure that the base image specified in the FROM statement of the Dockerfile is available either locally or can be pulled from a Docker registry. Some of the Dockerfiles depend on images produced from other Dockerfiles (e.g. src/ubuntu/22.04/debpkg). In these cases, the FROM reference should not include the <date-time>-<dockerfile-commit-sha> portion of the tags. This is referred to as a stable tag as it does not change from build to build. This pattern is used so that the Dockerfiles do not need constant updating as new versions of the base images are built. To support this scenario, the manifest entry for the base image must be defined to produce the stable tag.

"platforms": [
    {
        "dockerfile": "ubuntu/22.04",
        "os": "linux",
        "tags": {
            "ubuntu-22.04-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
            "ubuntu-22.04": {
                "isLocal": true
            }
        }
    }
]

Hooks

In certain cases, it is necessary to run custom logic before and after the Dockerfiles are built. For example, to build the Dockerfiles that are used for cross-gen builds, the rootfs that gets copied into the Docker image needs to be built on the host OS. To support these scenarios a pre-build or post-build bash or PowerShell script can be placed in a hooks folder next to the Dockerfile. The scripts will get invoked by the build process.

Note that multi-stage docker builds can be used to accomplish the same without build hooks, and are easier to iterate on locally because this takes advantage of docker image caching to avoid re-running steps when nothing has changed (whereas pre-build hooks run every time the dockerfile is built).

Warning: It is generally recommended to avoid the need to use hooks whenever possible.

Image-Builder

The underlying tool used to build the Dockerfiles is called Image-Builder. Its source is located at dotnet/docker-tools


For any questions, please contact [email protected]

dotnet-buildtools-prereqs-docker's People

Contributors

akoeplinger avatar am11 avatar directhex avatar dotnet-docker-bot avatar dotnet-maestro-bot avatar ellahathaway avatar hoyosjs avatar janvorli avatar jkoritzinsky avatar joperezr avatar lbussell avatar liveans avatar manickap avatar mattgal avatar michaelsimons avatar michellemcdaniel avatar mmitche avatar mthalman avatar nikolamilosavljevic avatar omajid avatar pavelsavara avatar radekdoulik avatar ravimeda avatar richlander avatar sbomer avatar steveisok avatar thaystg avatar vargaz avatar wfurt avatar wtgodbe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dotnet-buildtools-prereqs-docker's Issues

Alpine 3.6 images no longer build

15:43:56 Step 5/5 : RUN apk -X https://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache lldb-dev
15:43:57 ---> Running in a844d5b9f3e7
15:43:59 fetch https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
15:44:00 fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
15:44:00 fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
15:44:00 �[91mWARNING: This apk-tools is OLD! Some packages might not function properly.
15:44:00 �[0m�[91mERROR: unsatisfiable constraints:
15:44:00 �[0m so:libLLVM-5.0.so (missing):
15:44:00 required by: lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:00 lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:00 lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:00 lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:00 lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:00 lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:00 lldb-5.0.1-r0[so:libLLVM-5.0.so]
15:44:13 The command '/bin/sh -c apk -X https://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache lldb-dev' returned a non-zero code: 3

Latest images contain files which were intended to be removed to save space

Latest images contain files which were intended to be removed to save space.

[sdmaclea@EASTPOINT-1604 ~]$ sudo docker run --name cross-arm64 -it -d --privileged microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-cross-arm64-e435274-20180317125354 bash
71c4584b2f8563597f54efc71cc51e09a9a42a5992906f4f1a3328a51d252511
[sdmaclea@EASTPOINT-1604 ~]$ sudo docker exec cross-arm64 du -h -t 10M  -d 4 /crossrootfs/arm64/
13M	/crossrootfs/arm64/usr/share/i18n
11M	/crossrootfs/arm64/usr/share/locale
20M	/crossrootfs/arm64/usr/share/perl/5.22.1
20M	/crossrootfs/arm64/usr/share/perl
15M	/crossrootfs/arm64/usr/share/doc
17M	/crossrootfs/arm64/usr/share/man
99M	/crossrootfs/arm64/usr/share14M	/crossrootfs/arm64/usr/include/llvm-3.9/llvm
14M	/crossrootfs/arm64/usr/include/llvm-3.9
40M	/crossrootfs/arm64/usr/include
19M	/crossrootfs/arm64/usr/lib/aarch64-linux-gnu/perl
233M	/crossrootfs/arm64/usr/lib/aarch64-linux-gnu
24M	/crossrootfs/arm64/usr/lib/python3.5
13M	/crossrootfs/arm64/usr/lib/python2.7
53M	/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu
53M	/crossrootfs/arm64/usr/lib/gcc
129M	/crossrootfs/arm64/usr/lib/llvm-3.9/lib
15M	/crossrootfs/arm64/usr/lib/llvm-3.9/bin148M	/crossrootfs/arm64/usr/lib/llvm-3.9
475M	/crossrootfs/arm64/usr/lib
43M	/crossrootfs/arm64/usr/bin
659M	/crossrootfs/arm64/usr
148M	/crossrootfs/arm64/var/cache/apt/archives
203M	/crossrootfs/arm64/var/cache/apt
206M	/crossrootfs/arm64/var/cache
134M	/crossrootfs/arm64/var/lib/apt/lists
134M	/crossrootfs/arm64/var/lib/apt
13M	/crossrootfs/arm64/var/lib/dpkg/info
13M	/crossrootfs/arm64/var/lib/dpkg
147M	/crossrootfs/arm64/var/lib
353M	/crossrootfs/arm64/var
16M	/crossrootfs/arm64/lib/aarch64-linux-gnu
13M	/crossrootfs/arm64/lib/udev
39M	/crossrootfs/arm64/lib
1.1G	/crossrootfs/arm64/

This line is wrong. https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/ubuntu/build-scripts/build-rootfs.sh#L65

rm -rf /rootfs/*/var/cache/apt/archives/* /rootfs/*/var/lib/apt/lists/*

Doesn't work because the wild cards are expanded in the local shell. I failed to notice in my testing because I had been testing rootfs and so my host machine had similar files.

Replace it with

bash -c 'rm -rf /rootfs/*/{var/cache/apt/archives/*,var/lib/apt/lists/*,usr/share/doc,usr/share/man}'

Trial run with slightly modified args to adjust for different naming convention.

[sdmaclea@EASTPOINT-1604 ~]$ sudo docker exec cross-arm64 bash -c 'rm -rf /*rootfs/*/{var/cache/apt/archives/*,var/lib/apt/lists/*,usr/share/doc,usr/share/man}'
[sdmaclea@EASTPOINT-1604 ~]$ sudo docker exec cross-arm64 du -h -t 10M  -d 4 /crossrootfs/arm64/
13M	/crossrootfs/arm64/usr/share/i18n
11M	/crossrootfs/arm64/usr/share/locale
20M	/crossrootfs/arm64/usr/share/perl/5.22.1
20M	/crossrootfs/arm64/usr/share/perl
68M	/crossrootfs/arm64/usr/share
14M	/crossrootfs/arm64/usr/include/llvm-3.9/llvm
14M	/crossrootfs/arm64/usr/include/llvm-3.9
40M	/crossrootfs/arm64/usr/include
19M	/crossrootfs/arm64/usr/lib/aarch64-linux-gnu/perl
233M	/crossrootfs/arm64/usr/lib/aarch64-linux-gnu
24M	/crossrootfs/arm64/usr/lib/python3.5
13M	/crossrootfs/arm64/usr/lib/python2.7
53M	/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu
53M	/crossrootfs/arm64/usr/lib/gcc
129M	/crossrootfs/arm64/usr/lib/llvm-3.9/lib
15M	/crossrootfs/arm64/usr/lib/llvm-3.9/bin
148M	/crossrootfs/arm64/usr/lib/llvm-3.9
475M	/crossrootfs/arm64/usr/lib
43M	/crossrootfs/arm64/usr/bin
627M	/crossrootfs/arm64/usr
55M	/crossrootfs/arm64/var/cache/apt
58M	/crossrootfs/arm64/var/cache
13M	/crossrootfs/arm64/var/lib/dpkg/info
13M	/crossrootfs/arm64/var/lib/dpkg
14M	/crossrootfs/arm64/var/lib
72M	/crossrootfs/arm64/var
16M	/crossrootfs/arm64/lib/aarch64-linux-gnu
13M	/crossrootfs/arm64/lib/udev
39M	/crossrootfs/arm64/lib
753M	/crossrootfs/arm64/

Separate cross-build images into their own images

The current architecture will force the crossbuild image to include each rootfs. For multi-arch builds it is nice to only target one cross image. That being said, it would be nice to have arch specific images for space saving purposes. For example, building coreclr-arm does not require the x86, and arm64 rootfs.

Add build image for RPM packages

We have e.g. ubuntu-14.04-debpkg for Debian packages, we need something similar to build RPM packages. This would let Core-Setup and others use an actively supported build image when producing RPM packages.

I think we just need to create a new centos-7-rpmpkg Dockerfile that extends the CentOS Dockerfile with:

RUN yum clean all \
    && yum install -y \
        gcc \
        make \
        rpm-build \
        ruby-devel \
        sudo \
    && gem install --no-ri --no-rdoc fpm \
    && yum clean all

This isn't blocking me.

Alpine 3.6 arm64 Dockerfile fails to build

-- EXECUTING: docker build -t mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.6-arm64v8-3e800f1-20190501200618 -f src/alpine/3.6/arm64v8/Dockerfile src/alpine/3.6/arm64v8
Sending build context to Docker daemon   2.56kB
...
Step 5/5 : RUN apk -X https://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache         lldb-dev
 ---> Running in 5e59e3e73382
fetch https://dl-cdn.alpinelinux.org/alpine/edge/testing/aarch64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/aarch64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/aarch64/APKINDEX.tar.gz
WARNING: This apk-tools is OLD! Some packages might not function properly.‌
ERROR: unsatisfiable constraints:‌
  so:libLLVM-5.0.so (missing):‌
    required by: lldb-5.0.2-r0[so:libLLVM-5.0.so]
                 lldb-5.0.2-r0[so:libLLVM-5.0.so]
                 lldb-5.0.2-r0[so:libLLVM-5.0.so]
                 lldb-5.0.2-r0[so:libLLVM-5.0.so]
                 lldb-5.0.2-r0[so:libLLVM-5.0.so]
                 lldb-5.0.2-r0[so:libLLVM-5.0.so]
                 lldb-5.0.2-r0[so:libLLVM-5.0.so]
The command '/bin/sh -c apk -X https://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache         lldb-dev' returned a non-zero code: 3

Enable CI

Enable CI to ensure that the Dockerfiles are in a valid (build'able) state when changes are made.

[Linux Arm64] Infrastructure does not support building linux arm64v8 images

In order to build the following images we need the ability to run build cross architecture docker builds.

base image arch requires arm64 emulation or hardware Example dockerfile
arm64v8/ubuntu:16.04 arm64 yes https://gist.github.com/jashook/9eb11fa6228901234baecee195f8b19e
arm64v8/ubuntu:16.04 or ubuntu:16.04 armhf maybe https://gist.github.com/jashook/5b0daff1f00e8bf92e41909e033081ea
arm64v8/alpine:3.6 arm64 yes https://gist.github.com/jashook/2515f81048ada333f664337754af4702
arm64v8/debian:9 arm64 yes https://gist.github.com/jashook/08a3c23e8f71d39e115b9f14694ca67d

We should these docker images on arm64 hardware

Alpine 3.9 builds failing when installing lldb-dev

Step 5/5 : RUN apk -X http://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache         lldb-dev
 ---> Running in fa246b9618ae
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:‌
  so:libLLVM-8.so (missing):‌
    required by: lldb-8.0.0-r0[so:libLLVM-8.so]
The command '/bin/sh -c apk -X http://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache         lldb-dev' returned a non-zero code: 3

Looks to be related to #55

Relevant Dockerfile: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/alpine/3.9/amd64/Dockerfile

Add checked in yaml based VSTS build definitions

Checked in yaml based VSTS build definitions should be added. This will ease the maintenance of the official builds.

A separate RHEL build definition was recently created because of some special requirements RHEL has. With yaml based builds a single definition is possible with special RHEL conditional steps/demands/etc where necessary.

Add RHEL CI tests

#57 added a RHEL 7.5 Dockerfile. It isn't testable with the current infrastructure because the image must be built on a "real" RHEL VM with an active Red Hat subscription.

I'm working on adding RHEL to the dotnet VSTS instance for source-build CI. Once this repo moves to VSTS for public CI, it should be easy to add a RHEL leg.

/cc @MichaelSimons

Alpine CI is failing

docker build -t mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.6-7928975-20190320202248 -t mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.6 -f src/alpine/3.6/amd64/Dockerfile src/alpine/3.6/amd64
...
13:24:10 Step 5/5 : RUN apk -X https://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache         lldb-dev
13:24:11  ---> Running in 2fc2ea9d8bad
13:24:11 fetch https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
13:24:11 fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
13:24:11 fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
13:24:12 �[91mWARNING: This apk-tools is OLD! Some packages might not function properly.
13:24:12 �[0m�[91mERROR: unsatisfiable constraints:
13:24:12 �[0m  so:libLLVM-5.0.so (missing):
13:24:12     required by: lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12                  lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12                  lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12                  lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12                  lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12                  lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12                  lldb-5.0.2-r0[so:libLLVM-5.0.so]
13:24:12 The command '/bin/sh -c apk -X https://dl-cdn.alpinelinux.org/alpine/edge/testing add --no-cache         lldb-dev' returned a non-zero code: 3

Curl CLI issues in ubuntu 18.04 images

I'm trying to build corefx using the prereqs:ubuntu-18.04-* images, however they're all invariably failing with curl errors:

$ docker run -it mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-3e800f1-201908 
14211606 curl
curl: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory

Host images on mcr.microsoft.com

All Microsoft images are to be hosted on MCR. The image repo should be

mcr.microsoft.com/dotnet-buildtools/prereqs

Since these images are engineering infrastructure for the dotnet open source project, they should be hosted in a community repo on Docker Hub versus a verified publisher repo.

Ubuntu 16.04 Cross image unable to build CoreCLR on ARM32

In dotnet/coreclr#23999 I tried to update the docker image that we use to build CoreCLR on Linux ARM to be based on Ubuntu 16.04 instead of 14.04 so we can move off 14.04 before it EOLs. For whatever reason, the build with the 16.04 image fails. I don't know enough about how our Docker images work to efficiently debug this issue myself.
Here's a link to the failed build on Azure DevOps: https://dev.azure.com/dnceng/public/_build/results?buildId=156148&view=logs&jobId=d37b6b50-5a2d-5da1-80ef-d0fdb8e5bff2

From my first glance it seems that the pre-build hook might not be getting run correctly. Does that seem possible?

Fedora 23 Dockerfile fails CI

Fedora 23 CI started failing - indication of a non-repeatable Dockerfile.

15:30:27 Step 3/5 : RUN dnf install -y clang cmake lldb-devel llvm-devel make python which && dnf clean all
15:30:27 ---> Running in 556ae072cb0e
15:30:36 �[91mError: Failed to synchronize cache for repo 'fedora'
15:30:36 �[0mThe command '/bin/sh -c dnf install -y clang cmake lldb-devel llvm-devel make python which && dnf clean all' returned a non-zero code: 1

What caused /usr/bin/sh setlocale: LC_ALL: cannot change locale (en_US.UTF-8) on Linux?

This is a question. I think you might have a better answer. Do you know what caused the error /usr/bin/sh setlocale: LC_ALL: cannot change locale (en_US.UTF-8) on Linux?
https://dev.azure.com/dnceng/public/_build/results?buildId=386114&view=logs

It happens when the docker image does not have the correct locale. However I don't know where in the stack this error happens, and what is the solution since so far I cannot make SDK work on Fedora 30

Ubuntu 14.04 official build flaky: apt-get "Failed to fetch" "Hash Sum mismatch"

https://devdiv.visualstudio.com/DevDiv/_build/index?buildId=1560649

W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages.gz is not what the server reported 1328333 1328339
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-amd64/Packages  Hash Sum mismatch

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.
Error executing command, exiting
The command '/bin/sh -c apt-get update     && apt-get install -y wget     && wget -O - https://deb.nodesource.com/setup_8.x | bash     && apt-get install -y         git         zip         tar         nodejs     && rm -rf /var/lib/apt/lists/*     && npm install -g [email protected] --unsafe-perm=true     && npm cache clean -f' returned a non-zero code: 1

I'm rerunning in case this is just flakiness (it didn't happen in CI). https://devdiv.visualstudio.com/DevDiv/_build/index?buildId=1560916

This solution sounds like what we need, and we already do the rm -rf /var/lib/apt/lists/* in some Dockerfiles: https://unix.stackexchange.com/a/273441

You can delete the cached metadata on your system and try again. To do this, first run apt-get clean followed by rm -rf /var/lib/apt/lists/*. Next, run apt-get update. This will re-download all the metadata. If the repository you are trying to connect to doesn't update itself while you are running apt-get update, you will be fine. Otherwise, you'll have to do this again.

Centos_6 CI is failing

12:50:08 Step 5/7 : RUN wget http://download.icu-project.org/files/icu4c/57.1/icu4c-57_1-RHEL6-x64.tgz && tar -xf icu4c-57_1-RHEL6-x64.tgz -C / && rm icu4c-57_1-RHEL6-x64.tgz
12:50:09 ---> Running in d16807219e28
12:50:09 �[91m--2018-03-01 20:50:09-- http://download.icu-project.org/files/icu4c/57.1/icu4c-57_1-RHEL6-x64.tgz
12:50:09 Resolving download.icu-project.org... �[0m�[91m208.109.101.195
12:50:09 Connecting to download.icu-project.org|208.109.101.195|:80... �[0m�[91mconnected.
12:50:09 HTTP request sent, awaiting response... �[0m�[91m302 Found
12:50:09 Location: https://downloads.sourceforge.net/project/icu/ICU4C/57.1/icu4c-57_1-RHEL6-x64.tgz [following]
12:50:09 --2018-03-01 20:50:09-- https://downloads.sourceforge.net/project/icu/ICU4C/57.1/icu4c-57_1-RHEL6-x64.tgz
12:50:09 �[0m�[91mResolving downloads.sourceforge.net... �[0m�[91m52.85.131.102, 52.85.131.22, 52.85.131.167, ...
12:50:09 Connecting to downloads.sourceforge.net|52.85.131.102|:443... �[0m�[91mconnected.
12:50:09 �[0m�[91mHTTP request sent, awaiting response... �[0m�[91m302 Moved Temporarily
12:50:10 Location: https://downloads.sourceforge.net/#!/project/icu/ICU4C/57.1/icu4c-57_1-RHEL6-x64.tgz [following]
12:50:10 --2018-03-01 20:50:10-- https://downloads.sourceforge.net/
12:50:10 �[0m�[91mReusing existing connection to downloads.sourceforge.net:443.
12:50:10 HTTP request sent, awaiting response... �[0m�[91m200 OK
12:50:10 Length: 652 [text/html]
12:50:10 Saving to: icu4c-57_1-RHEL6-x64.tgz' 12:50:10 12:50:10 0K �[0m�[91m100% 194M=0s 12:50:10 12:50:10 �[0m�[91m2018-03-01 20:50:10 (194 MB/s) - icu4c-57_1-RHEL6-x64.tgz' saved [652/652]
12:50:10
12:50:10 �[0m�[91m
12:50:10 gzip: stdin: not in gzip format
12:50:10 �[0m�[91mtar: Child returned status 1
12:50:10 tar: Error is not recoverable: exiting now
12:50:10 �[0mThe command '/bin/sh -c wget http://download.icu-project.org/files/icu4c/57.1/icu4c-57_1-RHEL6-x64.tgz && tar -xf icu4c-57_1-RHEL6-x64.tgz -C / && rm icu4c-57_1-RHEL6-x64.tgz' returned a non-zero code: 2
12:50:10 System.InvalidOperationException: Failed to execute docker build -t microsoft/dotnet-buildtools-prereqs:centos-6-d770b8b-20180301200846 -f src/centos/6/Dockerfile src/centos/6
12:50:10 at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Func`2 executor, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 69
12:50:10 at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.BuildImages() in /image-builder/src/Commands/BuildCommand.cs:line 61
12:50:10 at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.ExecuteAsync() in /image-builder/src/Commands/BuildCommand.cs:line 28
12:50:10 at Microsoft.DotNet.ImageBuilder.ImageBuilder.Main(String[] args) in /image-builder/src/ImageBuilder.cs:line 53

Can this repo be more "self serve"?

Per this comment #92 (comment)

If the product team is responsible for the special docker image, I hope we can cut down most of the back and forth. The following is the flow I am currently in to updating docker files (to use the latest Linux distros)

  1. Change the docker file inside SDK
  2. Send WIP PR to test the change actually works
  3. Send PR to https://github.com/dotnet/dotnet-buildtools-prereqs-docker
  4. Wait for approval and get a build
  5. Use the image in SDK repos

If there is something wrong I need to start from 3-5 again. The whole process is lengthy. I am wondering if the process could be more self service like cutting the review so there is no wait on a human in the process since in the end as long as it can build, it is enough.

How do I see the latest built and published image for a given Dockerfile?

This is the best way I can think of right now to find the latest version of a specific image I want to use:

  1. Find the dockerfile I'm interested in on GitHub.
  2. Open the full list of tags (https://registry.hub.docker.com/v1/repositories/microsoft/dotnet-buildtools-prereqs/tags).
  3. Search for the initial part of the name (like ubuntu-18.04) using the convention in the readme.
  4. Pick the one with the latest date. Unfortunately the tags are sorted by name, and commit hash is in the name before the build date. Besides making a tool, the most efficient way I can think to do find latest is put the images in a text editor, edit out the commit, sort again, and see which image corresponds to the latest.

Is there any status on something like publishing this info to dotnet/versions or the Arcade BAR?

/cc @MichaelSimons @dleeapho

Ubuntu 16.04 Cross Images unable to build CoreCLR

The docker images for Ubuntu 16.04 for cross building against 14.04 ARM, 16.04 ARM64, and Alpine ARM64 fail to build CoreCLR due to what looks like a misconfigured/incorrectly generated rootfs.

This broke at some point between 2019-05-28 (the newest image CoreCLR is currently using) and 2019-08-14.

Sample build here: https://dev.azure.com/dnceng/public/_build/results?buildId=360236&view=logs&j=7df7e7c1-9be0-5cba-8a92-3fe105a7dcf9

This is blocking us from upgrading our images to the ones built on 2019-09-18 with the changes in #185

rootfs build errors and silently ignored errors

Looking at CI logs, all logs show `"arm" and "arm64" rootfs builds having errors. However, the errors are silently ignored and the build succeeds.

See debug notes in https://github.com/dotnet/coreclr/issues/16900

I looked at the visible console logs starting at build #5. I see the error for all "arm" and "arm64" attempts to build a rootfs.

The errors occur when attempting to run the initial qemu-debootstrap command. Missing package?

@MichaelSimons @janvorli

Linux jobs in PR build succeed w/ issues

Due to the changes in #158, the build jobs succeed "with issues" because the Copy Test Results task fails but is allowed to fail without failing the job. It fails because there's no data to copy since the test that is run is just a stub. This should be fixed so that this task doesn't fail and produces a "clean" build.

Centos images attempting download from inaccessible FTP server

The Centos Dockerfiles have the following RUN command: wget ftp://sourceware.org/pub/binutils/snapshots/binutils-2.29.1.tar.xz. This command is failing because the hostname is inaccessible in the build agents (see dotnet/core-eng#5478). That issue was closed because the recommendation is to not be accessing an insecure host to download files from, especially with no kind of verification. In that context, the subsequent downloads using HTTP are also a concern: wget http://releases.llvm.org/3.9.1/cfe-3.9.1.src.tar.xz. Please work with core-eng to arrive at a secure and workable solution.

Ubuntu 17.10 Dockerfiles no longer build

Ubuntu 17.10 is EOL therefore these Dockerfiles should be cleaned up.

-- EXECUTING: docker build -t mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-17.10-3e800f1-20190508213131 -t mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-17.10 -f src/ubuntu/17.10/Dockerfile src/ubuntu/17.10
Sending build context to Docker daemon  5.632kB

Step 1/4 : FROM ubuntu:17.10
 ---> e211a66937c6
Step 2/4 : RUN apt-get update     && apt-get install -y wget     && echo "deb http://llvm.org/apt/artful/ llvm-toolchain-artful main" | tee /etc/apt/sources.list.d/llvm.list     && echo "deb http://llvm.org/apt/artful/ llvm-toolchain-artful-3.9 main" | tee -a /etc/apt/sources.list.d/llvm.list     && wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | apt-key add -     && apt-get update     && apt-get install -y         clang-3.9         cmake         gdb         liblldb-3.9-dev         lldb-3.9         llvm-3.9         make         python-lldb-3.9         sudo     && rm -rf /var/lib/apt/lists/*
 ---> Running in 453df8481301
Ign:1 http://security.ubuntu.com/ubuntu artful-security InRelease
Err:2 http://security.ubuntu.com/ubuntu artful-security Release
  404  Not Found [IP: 91.189.91.26 80]
Ign:3 http://archive.ubuntu.com/ubuntu artful InRelease
Ign:4 http://archive.ubuntu.com/ubuntu artful-updates InRelease
Ign:5 http://archive.ubuntu.com/ubuntu artful-backports InRelease
Err:6 http://archive.ubuntu.com/ubuntu artful Release
  404  Not Found [IP: 91.189.88.162 80]
Err:7 http://archive.ubuntu.com/ubuntu artful-updates Release
  404  Not Found [IP: 91.189.88.162 80]
Err:8 http://archive.ubuntu.com/ubuntu artful-backports Release
  404  Not Found [IP: 91.189.88.162 80]
Reading package lists...
E: The repository &#x27;http:&#x2F;&#x2F;security.ubuntu.com&#x2F;ubuntu artful-security Release&#x27; does not have a Release file.‌
E: The repository 'http://archive.ubuntu.com/ubuntu artful Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu artful-updates Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu artful-backports Release' does not have a Release file.
The command &#x27;&#x2F;bin&#x2F;sh -c apt-get update     &amp;&amp; apt-get install -y wget     &amp;&amp; echo &quot;deb http:&#x2F;&#x2F;llvm.org&#x2F;apt&#x2F;artful&#x2F; llvm-toolchain-artful main&quot; | tee &#x2F;etc&#x2F;apt&#x2F;sources.list.d&#x2F;llvm.list     &amp;&amp; echo &quot;deb http:&#x2F;&#x2F;llvm.org&#x2F;apt&#x2F;artful&#x2F; llvm-toolchain-artful-3.9 main&quot; | tee -a &#x2F;etc&#x2F;apt&#x2F;sources.list.d&#x2F;llvm.list     &amp;&amp; wget -O - http:&#x2F;&#x2F;llvm.org&#x2F;apt&#x2F;llvm-snapshot.gpg.key | apt-key add -     &amp;&amp; apt-get update     &amp;&amp; apt-get install -y         clang-3.9         cmake         gdb         liblldb-3.9-dev         lldb-3.9         llvm-3.9         make         python-lldb-3.9         sudo     &amp;&amp; rm -rf &#x2F;var&#x2F;lib&#x2F;apt&#x2F;lists&#x2F;*&#x27; returned a non-zero code: 100‌

opensuse CI is failing

Failure snippet

11:00:18 -- EXECUTING: docker pull opensuse:42.1
11:00:19 Retry 1/5, retrying in 1 seconds...
11:00:19 Error response from daemon: manifest for opensuse:42.1 not found

caused from https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/opensuse/42.1/Dockerfile#L1

see https://hub.docker.com/_/opensuse

This image has been deprecated in favor of the opensuse/leap and opensuse/tumbleweed images provided and maintained by the openSUSE Project release team.

opensuse:42.3 (= opensuse:latest) will receive security relevant fixes until the EOL of Leap 42.3. Newer openSUSE Leap releases such as 15.x are only available at opensuse/leap.

The opensuse:tumbleweed image is no longer updated, please use opensuse/tumbleweed instead. opensuse/tumbleweed is updated on every snapshot release.

An archive for images of unsupported versions can be found at opensuse/archive.

Remove customized fix within the eng/common infrastructure for #187

When the changes to onboard this repo onto the common Docker build infra were brought in, the fix for #144 was lost - see https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/156/files#diff-fbb8382709c1eeda44e133299b934e0dL34.

A temporary fix was made to unblock the builds with #188. A long term solution needs to be made so that next time new eng/common infra is brought merged in, this fix doesn't get overridden again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.