This is related to aspnet/aspnet-docker#259. This functionality is need to support breaking apart the Windows builds into separate legs in order to decrease the overall build times.

Update ImageBuilder to pickup estesp/manifest-tool#27

As part of this work, ImageBuilder should pin which version of the manifest-tool it uses. This will help ensure stabilization and unexpected behavior changes.

https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build?_a=summary&buildId=823706

2017-06-23T21:48:35.6701850Z -- EXECUTING: manifest-tool --username msimons --password ******** push from-spec manifest.yml
2017-06-23T21:48:42.6187630Z time="2017-06-23T21:48:42Z" level=fatal msg="Inspect of image \"microsoft/dotnet-nightly:1.1.2-sdk-1.0.4-nanoserver\" failed with error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 8.8.4.4:53: dial udp 8.8.4.4:53: i/o timeout" 
2017-06-23T21:48:42.7191960Z System.InvalidOperationException: Failed to execute manifest-tool --username msimons --password ******** push from-spec manifest.yml
2017-06-23T21:48:42.7210060Z    at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Func`2 executor, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /Microsoft.DotNet.ImageBuilder/ExecuteHelper.cs:line 69
2017-06-23T21:48:42.7226720Z    at Microsoft.DotNet.ImageBuilder.ImageBuilder.PublishManifest() in /Microsoft.DotNet.ImageBuilder/ImageBuilder.cs:line 124
2017-06-23T21:48:42.7241140Z    at Microsoft.DotNet.ImageBuilder.ImageBuilder.Main(String[] args) in /Microsoft.DotNet.ImageBuilder/ImageBuilder.cs:line 45
2017-06-23T21:48:43.2048220Z ##[error]/usr/bin/docker failed with return code: 1
2017-06-23T21:48:43.2098080Z ##[error]/usr/bin/docker failed with error: /usr/bin/docker failed with return code: 1

Perhaps retry logic should be added to the image-builder. Should also consider if the manifest-tool should be more robust.

Rather than invoking docker CLI directly, ImageBuilder could take a dependency on Docker.DotNet and execute all the Docker client operations in-proc. This should make things more robust and probably improve performance as well.

Currently there is no validation logic that runs to ensure username and password are specified when performing operations on a Docker Registry (e.g. pushing images/manifests). This validation logic should run up front in order to fail early.

Image-Builder has logic to pull all external image dependencies before the build is started. The logic that determines if a dependency is external does not handle multi-stage Dockerfiles where a stage is based on a previous stage. When this happens, the logic tries to pull the stage images.

FROM microsoft/dotnet:2.0-sdk AS build
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.sln .
COPY dotnetapp/*.csproj ./dotnetapp/
COPY utils/*.csproj ./utils/
COPY tests/*.csproj ./tests/
RUN dotnet restore

# copy everything else and build app
COPY . .
WORKDIR /app/dotnetapp
RUN dotnet build


FROM build AS test
WORKDIR /app/tests
RUN dotnet test

Within ImageBuilder the same tag nomenclature is used for the Model/ViewModel members however they behave slightly different which causes some confusion. Within the Model, the tag is the short name while within the ViewModel the tag is the fully qualified name. The members should be renamed within the ViewModel to reflect this difference.

This issue started happening in the Windows build legs.

Example - https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=1849508&view=logs

2018-07-11T11:13:41.3769506Z -- EXECUTING: docker logout dotnetdocker.azurecr.io
2018-07-11T11:13:41.7324372Z System.ComponentModel.Win32Exception (5): Access is denied
2018-07-11T11:13:41.7325586Z at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2018-07-11T11:13:41.7326722Z at System.Diagnostics.Process.Start()
2018-07-11T11:13:41.7327510Z at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
2018-07-11T11:13:41.7328778Z at Microsoft.DotNet.ImageBuilder.ExecuteHelper.ExecuteProcess(ProcessStartInfo info) in C:\image-builder\src\ExecuteHelper.cs:line 78
2018-07-11T11:13:41.7330236Z at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Func`2 executor, Boolean isDryRun, String errorMessage, String executeMessageOverride) in C:\image-builder\src\ExecuteHelper.cs:line 65
2018-07-11T11:13:41.7332001Z at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Boolean isDryRun, String errorMessage, String executeMessageOverride) in C:\image-builder\src\ExecuteHelper.cs:line 29
2018-07-11T11:13:41.7333190Z at Microsoft.DotNet.ImageBuilder.DockerHelper.Logout(String server, Boolean isDryRun) in C:\image-builder\src\DockerHelper.cs:line 144
2018-07-11T11:13:41.7334457Z at Microsoft.DotNet.ImageBuilder.DockerHelper.ExecuteWithUser(Action action, String username, String password, String server, Boolean isDryRun) in C:\image-builder\src\DockerHelper.cs:line 49
2018-07-11T11:13:41.7335700Z at Microsoft.DotNet.ImageBuilder.Commands.CopyImagesCommand.ExecuteAsync() in C:\image-builder\src\Commands\CopyImagesCommand.cs:line 43
2018-07-11T11:13:41.7336839Z at Microsoft.DotNet.ImageBuilder.ImageBuilder.Main(String[] args) in C:\image-builder\src\ImageBuilder.cs:line 54

Image-Builder has logic that will explicitly pull the FROM image when building. This ensures the images get built with the latest base image. There is a potential problem with this logic.

Dockerfile1 -> FROM official-image-xyz
Dockerfile2 -> FROM official-image-xyz

Currently if you use Image-Builder to build these the following happens

docker pull official-image-xyz
docker build Dockerfile1
docker pull official-image-xyz
docker build Dockerfile2

Normally the docker pull resolves to the same image but it is possible that official-image-xyz could get updated during the build in which the second pull would get a different image. This is bad as a single build should get built with a common set of dependencies.

Image-Builder should be modified to compute all dependent images into a unique list and pull them once.

I tried to add a repo with no owner (instead of microsoft/dotnet-buildtools-prereqs, just dotnet-buildtools-prereqs, and I think it failed here:

I'm not completely sure because I only got this exception and stack:

READING MANIFEST
----------------
System.ArgumentOutOfRangeException: Length cannot be less than zero.
Parameter name: length
   at System.String.Substring(Int32 startIndex, Int32 length)
   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
   at System.Linq.Set`1.UnionWith(IEnumerable`1 other)
   at System.Linq.Enumerable.DistinctIterator`1.FillSet()
   at System.Linq.Enumerable.DistinctIterator`1.GetCount(Boolean onlyIfCheap)
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.ValidateCommonRepoOwner(Manifest manifest) in /image-builder/src/ViewModel/ModelExtensions.cs:line 29
   at Microsoft.DotNet.ImageBuilder.ViewModel.ModelExtensions.Validate(Manifest manifest) in /image-builder/src/ViewModel/ModelExtensions.cs:line 16
   at Microsoft.DotNet.ImageBuilder.Commands.Command`1.LoadManifest() in /image-builder/src/Commands/Command.TOptions.cs:line 37
   at Microsoft.DotNet.ImageBuilder.ImageBuilder.Main(String[] args) in /image-builder/src/ImageBuilder.cs:line 53

But I don't see any reason for this to happen other than that method, given the caller:

Throwing a more meaningful error on -1 seems useful.

The scenario for this is that you developing a new feature and would like to build and push the images to a private Docker registry. In order to do this today, you have to change the manifest as well as all of the runtime images which reference a intra-registry image (e.g. runtime-deps). It would be convenient if you didn't have to make an source changes to do this as it would make it easier to test PRs.

Currently the official builds always build the entire repo when triggered. This is inefficient and hurts the ability to have shared images/layers in production. Tooling should be created that will detect which Dockerfiles should get rebuilt whenever Dockerfiles are changed as well as base images.

This is needed to support dotnet/dotnet-docker#223

ImageBuilder has a dependency on System.CommandLine for handling the parsing of the command line args. This library is unsupported at this time. We should find an alternative library to use.

With the following Dockerfile, the parsing logic incorrectly detects SOURCE as an external dependency and tries to pull it when building.

FROM microsoft/dotnet-buildtools-prereqs:alpine-3.6
 RUN apk update
 ENV NODE_VERSION 10.13.0
 ### BUILD NODE FROM SOURCE

1. Will help ensure the tag section of the various readmes are the same.
2. Will help ensure all of the tags are documented and the links are valid.
3. Could be automated to run as part of the build to ensure the readme is always up to date.

Currently the microsoft/dotnet images have tags that include the Nano Server version. This version number gets updated every patch Tuesday. Ensuring that each tag gets updated correctly is prone to error. It would be nice if the version number was specified in one location and referenced several everywhere else.

The image-builder tool should provide functionality to update the Full Description in Docker Hub. Without this the Full Description is surely going to get out of sync with the README.md stored in the GH repo.

All of the .NET core repos are using ImageBuilder already. It should be utilized by the various full framework repos as well. Would be useful to also inform others the benefits it provides.

Currently the ImageBuilder command line parsing is pretty simple. As a consequence the options are not linked to the commands they are for. For example the --skip-test option is only valid for the build command.

dotnet run -- -h
Docker Image Builder

Summary:  Builds all Dockerfiles detected in the current folder and sub-folders in the correct order to satisfy cross dependencies.

Usage:  image-builder [options]

Options:
      --command                         Build command to execute (Build/PublishManifest/UpdateReadme)
      --dry-run                         Dry run of what images get built and order they would get built in
  -h, --help                            Show help information
      --manifest                        path to json file which describes the repo
      --password                        Password for the Docker registry the images are pushed to
      --push                            Push built images to Docker registry
      --skip-pulling                    Skip explicitly pulling the base images of the Dockerfiles
      --skip-test                       Skip running the tests
      --username                        Username for the Docker registry the images are pushed to

Using System.CommandLine (CoreFxLab) or Microsoft.Extensions.CommandLine (ASPNET common) would help solve this problem.

Consider adding checked in definitions that are used for detecting changes to base images.

https://devdiv.visualstudio.com/DevDiv/_build/index?context=mine&path=%5Cdotnet-docker&definitionId=7154&_a=completed
https://devdiv.visualstudio.com/DevDiv/_build/index?context=mine&path=%5Cdotnet-docker&definitionId=7116&_a=completed

Consider a scenario where a user would like to improve the traceability of images. User would like the
images built using ImageBuilder to be tagged with a specific time-stamp. User can update the manifest.json to include a time-stamp variable in image tag. An example of such tag is alpine-3.6--$(timeStamp). ImageBuilder should be able to replace all such $(timeStamp) variables with a time-stamp. This means the example tag should be like alpine-3.6-20173413043446, using a yyyymmddhhmmss DateTime format. All images built during the ImageBuilder session would have the same "global" time-stamp.

Another scenario to further improve the traceability of images is to include the commit SHA of the Dockerfile that was used to build the image. For this purpose, say, (dockerfileSha) variable is included in image tag. An example of such tag is alpine-3.6-$(dockerfileSha)-$(timeStamp). ImageBuilder should replace both the variables so that the example tag would be like alpine-3.6-3148f11-20173413043446, where 3148f11 is the commit SHA of the Dockerfile.

Currently the GenerateTagsReadme command emits the tags documentation section of the readme to the console. Logic should be added to optionally update the actual readme. This would make the workflow for updating the readme easier/quicker.

The only reason this wasn't done originally was due to time constraints. This functionality has proven to be pretty useful therefore it would be nice to enhance it a little more.

Each build in the Docker repositories uses the following pattern.

• Build Phase - build images and push to ACR staging location
• Test Phase - pull images from staging location and run tests
• Publish Phase - published images from staging location to MCR/DH

We don't have logic that cleans up the ACR staging location. This location filled up recently and I ran a local script to cleanup old repos. We should have automated tooling that cleans this up.

I don't think this is as simple as adding a cleanup phase to the build. I think we may want to keep the images around for a time period in case there are issues downstream. We also have scenarios where we want to support publishing to internal MCR and then release at a later time. Being able to rerun the publish build phase to publish a specific build to public MCR would be desirable in this scenario.

This make sense for nightly repos. Official repos would pickup changes only when nightly is migrated to official.

The updateReadme command has regressed and fails to run within the Linux image produced by the official builds. The following exception is thrown when run.

System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.
IOException: Unable to write data to the transport connection: The lazily-initialized type does not have a public, parameter
less constructor.. ---> System.MissingMemberException: The lazily-initialized type does not have a public, parameterless con
structor.
   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target)
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToke
n cancellationToken)
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationTok
en)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationTok
en)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncReques
t)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions,
 CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_0(SslClientAuthenticationOptions arg1, Cancellation
Token arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAc
tion, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions,
 CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions
, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions
, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationTok
en)
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationT
oken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSourc
e cts, Boolean disposeCts)
   at Microsoft.DotNet.ImageBuilder.Commands.UpdateReadmeCommand.ExecuteAsync() in /image-builder/src/Commands/UpdateReadmeC
ommand.cs:line 41
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.DotNet.ImageBuilder.ImageBuilder.Main(String[] args) in /image-builder/src/ImageBuilder.cs:line 53
---> (Inner Exception #0) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner excep
tion. ---> System.IO.IOException: Unable to write data to the transport connection: The lazily-initialized type does not hav
e a public, parameterless constructor.. ---> System.MissingMemberException: The lazily-initialized type does not have a publ
ic, parameterless constructor.
   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target)
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToke
n cancellationToken)
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationTok
en)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationTok
en)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncReques
t)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions,
 CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_0(SslClientAuthenticationOptions arg1, Cancellation
Token arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAc
tion, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions,
 CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions
, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions
, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationTok
en)
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationT
oken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSourc
e cts, Boolean disposeCts)
   at Microsoft.DotNet.ImageBuilder.Commands.UpdateReadmeCommand.ExecuteAsync() in /image-builder/src/Commands/UpdateReadmeC
ommand.cs:line 41

@MichaelSimons commented on Mon Feb 25 2019

Today if you build an image on hardware who's architecture does not align with the image, the architecture in the image's metadata will not be set correctly. For example if you build an arm32v7 image on an aarch64, the resulting image's architecture will be arm64. This is related to moby/moby#35158.

This is going to become an issue as there are plans to onboard aarch64 machines to build the .NET arm images with.

I read about how people update the metadata and the consensus seems to save the image, edit the metadata file, and then load.  e.g. https://kok-tips.blogspot.com/2018/05/how-to-edit-existing-docker-image.html  I tried it out and it works alright.  The downside is it takes a while to save given the size of the Windows images.  Loading is very quick given the layers already exist on disk.

It seems like a workaround is inevitable therefore it would be logical to add logic to handle this within the image-builder tool so that all the .NET repos can easily make use of it.

Steps to reproduce the issue

1. git clone https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_git/DotNetCore
2. cd dockerfiles
3. ./build.ps1 -ImageBuilderCustomArgs --dry-run

Expected behavior

Do not show a heading if there is no content in that section. User might think that the testing was skipped because the correct argument was not passed to the script, until the user opens manifest.json and finds that there are no test commands.

Another option is show the heading but include an appropriate message. For example, "No test command was found in manifest". Same applies to other headers. For example, if no image was pushed to a registry, then PUSH IMAGES can display a message indicating the same. This saves investigation time and avoids confusion for users.

Actual behavior

TESTING IMAGES section is blank. See the console output below

TESTING IMAGES
--------------

IMAGES BUILT
------------

Additional information (e.g. issue happens only occasionally)

Issue happens consistently. Can be reproduced with any other repositories where manifest.json does not have a test command.

Output of docker version

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:45:38 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.837GiB
Name: moby
ID: 2U3X:7GXP:Y6JF:3NV3:3PIT:K5MQ:RWUT:KANE:YNCT:YWXR:NXKE:A4FK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 26
 System Time: 2018-01-19T03:19:54.8106644Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

This is needed so that the updateVersions command can be used to update the linux arm32 versions when running on a Windows 10 machine running Linux Containers.

The current Docker login logic is using the password option which is insecure. At the time the logic was written, it was the only supported option. The password-stdin option is now the preferred way.

In order to gain performance during publishManifest, start using the tags element added as part of the fix for estesp/manifest-tool#23

The the introduction of the Windows 1709 release, a decision was made to put the 1709 related tags under a separate Supported Windows Server 2016 Version 1709 (Fall Creators Update) amd64 tags heading. The current GenerateTagsReadmeCommand does not support this. It places all of the Windows tags within a single section.

Currently the manifest's Dockerfile property must be set to a directory containing a Dockerfile. This property could be changed to either be a directory that contains a Dockerfile file or a full path to a Dockerfile. It is important that the implementation is backwards compatible so that existing manifests still work.

This is needed for dotnet/dotnet-docker-samples#55. The samples build the same app with multiple Dockerfiles. The various dockerfiles have different suffixes - e.g. .arm32v7

Acquisition of ImageBuilder (Microsoft.DotNet.ImageBuilder) would be simpler if the built assemblies are available for download. Similar to dotnet CLI, compress the built assemblies to a zip file and host in a Azure storage account such that the latest file is a static URL.

For comparison, current acquisition mechanisms are -

• Pull the Docker image from (https://hub.docker.com/r/microsoft/dotnet-buildtools-prereqs/), and copy the assemblies off of that image.
• Clone or Download the source from this repo, and build.

Add unit tests to help detect regressions.

Validation logic should be added to ensure all of the tags in the manifest are unique.

Create documentation that describes the purpose of the manifest and details of the schema.

Detailed docs should be placed in a new /Documentation folder.

A hook is a script that can be run either before or after a Docker command. Hook name is convention based, which means pre-build is the name of the script that runs before Docker build command, while post-build is the one that runs after build command. Thus, this enables running a custom script before and/or after the build command.

Once #8 is implemented, CI should be enabled on the repo to prevent regressions

Currently the readme tag order is determined by the readmeOrder property if specified else the order of the images in the manifest. Using the readmeOrder property can get clunky to maintain. For example adding a new image can require all readmeOrder properties to need to be incremented by one.

The current functionality doesn't have flexibility in grouping. With the introduction of the .NET Core 2.1 preview images, there is discussion of wanting a preview section. Creating built in sort/grouping logic to handle all these changing scenarios is not feasible/maintainable nor is this something the tool should be responsible for.

A more general solution should be implemented in which the individual repos have complete control and still benefit from having a tool generate the tag information. Without this, the tag information is prone to errors such as incorrect/missing images and tags as well as broken hyperlinks, etc.

All Microsoft images are to be hosted on MCR. The repo for the image-builder should be

mcr.microsoft.com/dotnet-buildtools/image-builder

Since these images are engineering infrastructure for the dotnet open source project, they should be hosted in a community repo on Docker Hub versus a verified publisher repo.

@MichaelSimons commented on Thu Jul 06 2017

The logic within image-builder that explicitly pulls the FROM images during a build needs to be updated to support multi-stage builds. Multi-stage build Dockerfiles may have multiple FROM commands. image-builder should be explicitly pulling each one of the base images.

This will greatly reduce the size of the image because it is currently a an 'sdk' based image.

Remove unused project Microsoft.DotNet.ImageBuilder.Tests from Microsoft.DotNet.ImageBuilder.sln

Refer to https://github.com/dotnet/docker-tools/blob/master/Microsoft.DotNet.ImageBuilder/Microsoft.DotNet.ImageBuilder.sln#L7

The current header that gets generated for Windows Server, version 1803 tags is

Windows Server 2016 amd64 tags

It should be

Windows Server, version 1803 amd64 tags

The Docker Hub autobuild infrastructure has functionality that will rebuild an image whenever a base image is changed. Unfortunately Docker Hub autobuilds cannot be used for repos that want mutli-arch support. This is partially why ImageBuilder was created. Infrastructure should be created that will kick off a build whenever a dependent image is changed.

This should not perform any type of build, just run the tests.

Port CI to Azure DevOps. The benefits of this work include:

1. Ability to share build definitions (all or parts) between CI and official builds.
2. Ability to make CI definition changes and easily test them within the same PR. This is a limitation with the current Jenkins implementation.

Currently in the dotnet-docker repo, there are runtime-deps images that are shared between multiple versions (e.g. 2.1 & 2.2). There is a requirement have separate entries in the tag documentation for each version. The reason being is it makes it easier for users to find the image they are looking for. If a user is looking for a 2.2-runtime-deps image, it will be easier to find the 2.2 tags if they are grouped.

To achieve this today, the TagDocList variable is used in the readme template and the individual tags are listed.

$(TagDocList:2.1.5-runtime-deps-stretch-slim|2.1-runtime-deps-stretch-slim|2.1.5-runtime-deps|2.1-runtime-deps|runtime-deps)
...
$(TagDocList:2.2.0-preview3-runtime-deps-stretch-slim|2.2-runtime-deps-stretch-slim|2.2.0-preview3-runtime-deps|2.2-runtime-deps)

This design is not ideal because of the maintenance of using this pattern and having to list out the tags explicitly. It would be desirable there was a way to capture this metadata within the manifest and then update the template to use the TagDoc variable.

$(TagDoc:2.1-runtime-deps-stretch-slim)
...
$(TagDoc:2.2-runtime-deps-stretch-slim)

This could be achieved by introducing a documentation grouping mechanism.

{
  "sharedTags": {
    "$(2.1-RuntimeVersion)-runtime-deps": {
      "documentationGroup": "2.1"
    },
    "2.1-runtime-deps": {
      "documentationGroup": "2.1"
    },
    "$(2.2-RuntimeVersion)-runtime-deps": {
      "documentationGroup": "2.2"
    },
    "2.2-runtime-deps": {
      "documentationGroup": "2.2"
    },
    "runtime-deps": {
      "documentationGroup": "2.1"
    },
    "2-runtime-deps": {
      "isUndocumented": true
    }
  },
  "platforms": [
    {
      "dockerfile": "2.1/runtime-deps/stretch-slim/amd64",
      "os": "linux",
      "tags": {
        "$(2.1-RuntimeVersion)-runtime-deps-stretch-slim": {
          "documentationGroup": "2.1"
        },
        "2.1-runtime-deps-stretch-slim": {
          "documentationGroup": "2.1"
        },
        "$(2.2-RuntimeVersion)-runtime-deps-stretch-slim": {
          "documentationGroup": "2.2"
        },
        "2.2-runtime-deps-stretch-slim": {
          "documentationGroup": "2.2"
        }
      }
    },
    {
      "architecture": "arm",
      "dockerfile": "2.1/runtime-deps/stretch-slim/arm32v7",
      "os": "linux",
      "tags": {
        "$(2.1-RuntimeVersion)-runtime-deps-stretch-slim-arm32v7": {
          "documentationGroup": "2.1"
        },
        "2.1-runtime-deps-stretch-slim-arm32v7": {
          "documentationGroup": "2.1"
        },
        "$(2.2-RuntimeVersion)-runtime-deps-stretch-slim-arm32v7": {
          "documentationGroup": "2.2"
        },
        "2.2-runtime-deps-stretch-slim-arm32v7": {
          "documentationGroup": "2.2"
        }
      },
      "variant": "v7"
    }
  ]
}

The way this would work is that the TagDoc variable would list all tags that are in the same grouping as the platform tag listed in the variable.

Steps to reproduce the issue

1. I would like to try out the .NET Core SDK Docker image. So, I visit https://hub.docker.com/r/microsoft/dotnet/
2. I try to get image details such as the Dockerfile that was used to assemble the image, the version of .NET Core SDK and Shared Framework available on the image.

Expected behavior

Detailed tag information is available. Information should include -

• Dockerfile. Not just a link to latest version. But the specific version (commit) that was used to build the image
• Platforms
• Layers
• Date-time when the image was created
• Docker version that was used to assemble the image

Similar to https://github.com/docker-library/repo-info/blob/master/repos/golang/remote/windowsservercore-1709.md

Actual behavior

Only a link to the latest Dockerfile is posted. For example, https://github.com/dotnet/dotnet-docker/blob/master/2.0/sdk/nanoserver-1709/amd64/Dockerfile