hylandsoftware / harbor.tagd Goto Github PK

View Code? Open in Web Editor NEW

39.0 39.0 5.0 101 KB

Automated tag cleanup for VMware Harbor

License: MIT License

Dockerfile 0.11% C# 90.64% PowerShell 3.88% Shell 4.11% HTML 1.26%

docker-registry dotnet-core harbor

harbor.tagd's People

Contributors

Stargazers

Watchers

Forkers

hypromgr zhouyuanyuana lf1029698952 krusty-krab-industries lesta1

harbor.tagd's Issues

The rules match only one project

Hey:
Need to clear only a project the image of the tag.
In the "defaultRule" and "Additional Rules to process", The specified project and repo. Will global match。

config.yml

# Projects, Repositories, and Tags to globally ignore. Must match exactly
ignoreGlobally:
  projects: []
  repos: []
  tags: ['latest']
# The default rule to process if no other rules match
defaultRule:
  project: 'test2'      # A regular expression matching a project
  repo: 'test2'         # A regular Expression matching a repository
  tag: '.*'          # A regular Expression matching a tag
  ignore: ['latest'] # Tags to always keep. Must match exactly
  keep: 1           # The number of tags matching this rule to keep, sorted by creation date
# Additional Rules to process
rules:
- project: 'test1'
  repo: 'test1'
  tag: 'dev.*'
  ignore: ['latest']
  keep: 1

In the "defaultRule" "tag" : Set a rule does not exist tag ,Can only match to a project. Only set up the project and repo not to take effect. But iterates through all project request.

config.yml

# Projects, Repositories, and Tags to globally ignore. Must match exactly
ignoreGlobally:
  projects: []
  repos: []
  tags: ['latest']
# The default rule to process if no other rules match
defaultRule:
  project: 'test2'      # A regular expression matching a project
  repo: 'test2'         # A regular Expression matching a repository
  tag: 'no.*'          # A regular Expression matching a tag
  ignore: ['latest'] # Tags to always keep. Must match exactly
  keep: 1           # The number of tags matching this rule to keep, sorted by creation date
# Additional Rules to process
rules:
- project: 'test1'
  repo: 'test1'
  tag: 'dev.*'
  ignore: ['latest']
  keep: 1

"Harbor.Tagd" for all projects match the request. This leads to the execution time is too long and the influence on the performance of harbor.Whether can according to the rule of the project only initiate the request.

How to match only one project, Match the request for only one project .Please help us to solve, thank you.

logon error ?

Hello .
ive got this problem when running own private harbor registry. used dotnet 2.1 on centos 7.5 username and password are correct (docker login .... ), url is accessible. self signed cert is used for nginx. insecure registry key is correct for https://reg1.xxx.zzz . Harbor an Tagd are on same machine.
any hints where problem is ?

same as bug "Improve logging around cert issues" ?

dotnet ./dist/Harbor.Tagd.dll --config-file config.yml --endpoint https://reg1.xxx.zzz --user zzzzz --password xxxxxxxx -v V
[22:46:04 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[22:46:04 WRN] The rule provider did not return any explicit tag rules
[22:46:04 VRB] Using default rule {'Project': '.', 'Repo': '.', 'Tag': '.*', 'Keep': 1, 'Ignore': [latest]}
[22:46:04 VRB] Ignoring the following projects: []
[22:46:04 VRB] Ignoring the following repos: []
[22:46:04 VRB] Ignoring the following tags: ["latest"]
[22:46:04 WRN] Running in report-only mode. To delete tags, run with --destructive
[22:46:04 INF] Connecting to https://reg1.xxxx.zz as zzzzzzzz
[22:46:04 INF] Running in report-only mode
[22:46:05 INF] Logging out
[22:46:05 FTL] An Error was encountered while processing tags
System.InvalidOperationException: Not logged in
at Harbor.Tagd.API.HarborClient.Logout() in /root/harbor.tagd/Harbor.Tagd-master/src/Harbor.Tagd/API/HarborClient.cs:line 61
at Harbor.Tagd.TagEngine.Process() in /root/harbor.tagd/Harbor.Tagd-master/src/Harbor.Tagd/TagEngine.cs:line 104
at Harbor.Tagd.Program.DoClean(ApplicationSettings appSettings, IRuleProvider rules) in /root/harbor.tagd/Harbor.Tagd-master/src/Harbor.Tagd/Program.cs:line 120
at Harbor.Tagd.Program.Main(String[] args) in /root/harbor.tagd/Harbor.Tagd-master/src/Harbor.Tagd/Program.cs:line 90

thanks

tags are not deleting

docker run -it --rm -v /root/Harbor.Tagd/config.yml:/config.yml hylandsoftware/tagd:1.0.26 --config-file /config.yml --endpoint https://harbor --insecure-disable-certificate-validation -u admin -p Harbor12345
[08:50:45 WRN] Ignoring Certificate Errors
[08:50:45 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[08:50:45 WRN] Running in report-only mode. To delete tags, run with --destructive
[08:50:45 INF] Connecting to https://harboras admin
[08:50:45 INF] Running in report-only mode
[08:50:46 INF] Processing project {"Id": 3, "Name": "infra"}
[08:50:46 INF] Processing project {"Id": 7, "Name": "mobitv"}
[08:50:46 INF] Processing project {"Id": 2, "Name": "os"}
[08:50:46 INF] Processing repository {"Id": 3, "Name": "os/alpine", "TagCount": 211}
[08:50:48 INF] Tag os/alpine:3.8 skipped because it was found in an ignore list that applies to os/alpine
[08:50:48 INF] Tag os/alpine:15.3 kept
[08:50:48 INF] Tag os/alpine:2.0 kept
[08:50:48 INF] Tag os/alpine:17.2 kept
[08:50:48 INF] Tag os/alpine:13.5 kept
[08:50:48 INF] Tag os/alpine:9.4 kept
[08:50:48 INF] Tag os/alpine:20.0 kept
[08:50:48 INF] Tag os/alpine:18.9 kept
[08:50:48 INF] Tag os/alpine:12.0 kept
[08:50:48 INF] Tag os/alpine:13.7 kept
[08:50:48 INF] Tag os/alpine:9.0 kept
[08:50:48 INF] Processing repository {"Id": 1, "Name": "os/centos", "TagCount": 2}
[08:50:49 INF] Tag os/centos:7.6.1 kept
[08:50:49 INF] Tag os/centos:6.9 kept
[08:50:49 INF] Processing project {"Id": 1, "Name": "library"}
[08:50:49 INF] Processing project {"Id": 5, "Name": "application"}
[08:50:49 INF] Processing repository {"Id": 11, "Name": "application/httpd", "TagCount": 1}
[08:50:49 INF] Tag application/httpd:2.2 kept
[08:50:49 INF] Processing project {"Id": 6, "Name": "snapshot"}
[08:50:49 INF] Keeping tags that refer to the same image as an explicitly kept or ignored tag
[08:50:49 INF] DRY RUN - Tag cleanup complete. Summary: {"RemovedTags": 0, "IgnoredTags": 14, "IgnoredRepos": 0, "IgnoredProjects": 1}
[08:50:49 INF] Logging out
[08:50:49 INF] Finished in 00:00:04.2407388

config.yaml

Projects, Repositories, and Tags to globally ignore. Must match exactly

ignoreGlobally:
projects: [buildagent]
repos: [maven-build-agent]
tags: ['latest']

The default rule to process if no other rules match

defaultRule:
project: '.' # A regular expression matching a project
repo: '.' # A regular Expression matching a repository
tag: '^(\d+.)?(\d+.)?(*|\d+)$' # A regular Expression matching a tag
ignore: ['latest','3.8'] # Tags to always keep. Must match exactly
keep: 10 # The number of tags matching this rule to keep, sorted by creation date

Additional Rules to process

rules:

project: '.'
repo: '.'
tag: '.*'
ignore: ['latest']
keep: 10

As alpine has 211 tag why its not removed and kept the tags which are shown in results.

Slack Notifier not called if config pulling throws an exception

The certificate on our config server expired recently, causing tagd to fail after some retries with an exception. I was unable to diagnose this until after I got back from kubecon and could trigger an ad-hoc job and watch the logs.

Publish to Docker Hub

Once legal approves our use of the Docker Hub, we should automatically build & publish an image.

SSL connect error in docker image

Hi,

I have the followin error when running tagd.
The same docker image used to work, so our Harbor provider probably upgraded its TLS requirements.

Is it possible to build a new version of tagd based on a newer base image with an upgraded openssl stack ?

Here is the error logs & stack trace:

[19:30:02 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[19:30:02 WRN] The rule provider did not return any explicit tag rules
[19:30:02 WRN] Not running in report-only mode. Tags will be deleted
[19:30:02 INF] Connecting to https://xxxxxx as xxxxxx
[19:30:02 WRN] Tags will be deleted when matched!
[19:30:02 DBG] Detecting Harbor Version
[19:30:03 ERR] Unable to detect the version of harbor. If your deployment blocks /api/systeminfo or you are running a non-release version you may need to adjust the --login-behavior depending on your version of harbor
Flurl.Http.FlurlHttpException: Call failed. An error occurred while sending the request. GET https://xxxxxx/api/systeminfo ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: SSL connect error
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at Flurl.Http.FlurlRequest.<HandleExceptionAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.HttpResponseMessageExtensions.<ReceiveJson>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<ProbeVersion>d__13.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 49
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<Login>d__14.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 62

Unable to run tagd harbor v.1.8.1 with oidc.

Hi, I'm trying to use tagd on harbor v.1.8.1, auth OIDC with Azure AD in kubernetes, using tagd:1.7.2.

We already tried multiple things, but always the same issue, with or without https:

[11:13:10 WRN] Ignoring Certificate Errors
[11:13:10 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[11:13:10 WRN] Not running in report-only mode. Tags will be deleted
[11:13:10 INF] Connecting to https://mydomain.com as harboruser
[11:13:10 WRN] Tags will be deleted when matched!
[11:13:10 FTL] An Error was encountered while processing tags
System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Harbor.Tagd.API.HarborClient.<Login>d__13.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 80
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/HarborTagd/TagEngine.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 128
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 98

On harbor core pod logs I see 403 | match| POST /c/login r:/c/login

I'm using the CLI secret from the UI -> user profile to connect from tagd, docker login works from my machine, but no success to run tagd.

Any idea?

Thank you!

Please add document about supporting Habror version range

Trigger online GC On Harbor 1.7+

Harbor 1.7.0+ supports online GC. If specified (and if the version of harbor we're talking to supports it), trigger a GC. This automatically puts the registry into read-only mode.

Parallel-ize tag processing

On repositories with many projects and repos, tag processing can take minutes to complete. We should investigate what performance improvements we can make. I see three options:

Process projects serially, repos in parallel
Process projects in parallel, repos serially
Process projects in paralle, then repos in parallel

clean up for Helm charts

Command/Docker container always return exit code 0 even on failure

To reproduce, just run the docker container with false info:

 docker run -it --rm -v config.yml:/config.yml hylandsoftware/tagd \
    --config-file /config.yml \
    --endpoint ar.ss.com \
    --user aa \
    --password 123
[13:39:16 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[13:39:16 FTL] An Error was encountered while processing tags
System.UnauthorizedAccessException: Access to the path '/config.yml' is denied. ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks)
   at System.IO.File.InternalReadAllText(String path, Encoding encoding)
   at System.IO.File.ReadAllText(String path)
   at Harbor.Tagd.Rules.FilesystemRuleProvider.Load() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Rules/FilesystemRuleProvider.cs:line 33
   at Harbor.Tagd.TagEngine.LoadRules() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 114
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 42
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 128
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 98





echo $?
0

This is causing containers/k8s pods(jobs) to show a COMPLETED status instead of ERROR

Better rule-validation mode

Currently, tagd can validate rules and exit, but you still have to specify a Harbor endpoint, username, and password (even if they're blank). These fields should not be required if --dump-rules is specified. Furthermore, we should probably make this an explicit --validate mode or similar.

Default project / repo / tag to `.*` if not specified

If you leave out project / repo / tag from a rule you get a nasty NullReferenceException. We should default these to .* if they're not specified or at least fix the log message.

Error with harbor 1.8 ?

Hi,
I'm trying Tagd with a hosted version of Harbor 1.8 by OVH

I run the following command:

docker run -it --rm -v /tmp/config.yml:/config.yml hylandsoftware/tagd --endpoint https://xxxxx.xxxx.container-registry.ovh.net --user xxxxxxx --password xxxxxx --config-file /config.yml -vdebug

And the config.yml file the example one provided in README.md
I have the following output:

[17:07:19 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[17:07:19 WRN] Running in report-only mode. To delete tags, run with --destructive
[17:07:19 INF] Connecting to https://xxxx.xxx.container-registry.ovh.net as xxxxxx
[17:07:19 INF] Running in report-only mode
[17:07:19 FTL] An Error was encountered while processing tags
System.ArgumentOutOfRangeException: Length cannot be less than zero.
Parameter name: length
   at System.String.Substring(Int32 startIndex, Int32 length)
   at Harbor.Tagd.API.HarborClient.<ProbeVersion>d__12.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<Login>d__13.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 51
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 128
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 98

Could this be an incompatibility with Harbor 1.8 ?

need support for https harbor

when i try to clean up from https harbor.
it gives me below error.

dotnet ./dist/Harbor.Tagd.dll --config-file rules.yaml --endpoint https://harbor -u admin -p Harbor12345
[14:37:19 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[14:37:20 WRN] Running in report-only mode. To delete tags, run with --destructive
[14:37:20 INF] Connecting to https://harbor as admin
[14:37:20 INF] Running in report-only mode
[14:37:21 FTL] An Error was encountered while processing tags
Flurl.Http.FlurlHttpException: Call failed. An error occurred while sending the request. GET https://harbor/api/systeminfo ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Peer certificate cannot be authenticated with given CA certificates
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at Flurl.Http.FlurlRequest.<HandleExceptionAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.HttpResponseMessageExtensions.<ReceiveJson>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<ProbeVersion>d__12.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<Login>d__13.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 51
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 128
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in /root/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 98

504 Timeout should skip to next repo and not crash

We have a few repo's that have grown quite large and return 504 timeouts when listing the tags, this crashes Tagd, it should rather log a warning, and continue to the next repo.

Error with harbor 1.10.1 : 422 (Unprocessable Entity).

Hi, after upgrade Harbor to version 1.10.1 I have this error

docker run -it --rm -v /etc/harbortagd/config:/config hylandsoftware/tagd \
        --config-file /config/config.yml \
        --endpoint https://HARBOR.ADDRESS/ \
        --insecure-disable-certificate-validation \
        --user USER --password PASSWORD \
        -vdebug

Unable to find image 'hylandsoftware/tagd:latest' locally
latest: Pulling from hylandsoftware/tagd
Digest: sha256:a360897f30eb81cc0e57bf295ef781833ed976e870a98809782404077269f35d
Status: Downloaded newer image for hylandsoftware/tagd:latest
[16:04:05 WRN] Ignoring Certificate Errors
[16:04:05 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[16:04:06 WRN] Running in report-only mode. To delete tags, run with --destructive
[16:04:06 INF] Connecting to https://HARBOR.ADDRESS/ as USER
[16:04:06 INF] Running in report-only mode
[16:04:06 DBG] Detecting Harbor Version
[16:04:06 DBG] Newer version of harbor found, using 1.7.0+ login route
[16:04:06 FTL] An Error was encountered while processing tags
System.Net.Http.HttpRequestException: Response status code does not indicate success: 422 (Unprocessable Entity).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Harbor.Tagd.API.HarborClient.<DoLoginFor>d__15.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 116
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<Login>d__14.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 82
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 131
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 98

/api/systeminfo:

with_notary	true
with_admiral	false
admiral_endpoint	""
auth_mode	"ldap_auth"
registry_url	"HARBOR.ADDRESS"
external_url	"https://HARBOR.ADDRESS"
project_creation_restriction	"everyone"
self_registration	true
has_ca_root	false
harbor_version	"v1.10.1-f3e11715"
registry_storage_provider_name	"filesystem"
read_only	false
with_chartmuseum	true
notification_enable	true

/var/log/harbor/proxy.log

Mar 31 14:35:37 192.168.64.1 proxy[99]: 192.168.64.1 - "GET /api/systeminfo HTTP/1.1" 200 277 "-" "-" 0.004 0.004 .
Mar 31 14:35:37 192.168.64.1 proxy[99]: 192.168.64.1 - "POST /c/login HTTP/1.1" 422 1934 "-" "-" 0.003 0.004 .

Improve logging around cert issues

[15:49:56 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[15:49:57 WRN] Running in report-only mode. To delete tags, run with --destructive
[15:49:57 INF] Connecting to https://hcr.io as harborsvc
[15:49:57 INF] Running in report-only mode
[15:49:57 FTL] An Error was encountered while processing tags
Flurl.Http.FlurlHttpException: Call failed. An error occurred while sending the request. POST https://hcr.io/login ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: SSL connect error
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at Flurl.Http.FlurlRequest.<HandleExceptionAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<Login>d__9.MoveNext() in C:\Users\nlowe\projects\harbor.tagd\src\Harbor.Tagd\API\HarborClient.cs:line 42
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in C:\Users\nlowe\projects\harbor.tagd\src\Harbor.Tagd\TagEngine.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in C:\Users\nlowe\projects\harbor.tagd\src\Harbor.Tagd\TagEngine.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in C:\Users\nlowe\projects\harbor.tagd\src\Harbor.Tagd\TagEngine.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in C:\Users\nlowe\projects\harbor.tagd\src\Harbor.Tagd\Program.cs:line 120
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in C:\Users\nlowe\projects\harbor.tagd\src\Harbor.Tagd\Program.cs:line 90

It now makes it look like harbor immediately logs out and then complains that it's not logged in:

[15:50:57 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[15:50:57 WRN] Running in report-only mode. To delete tags, run with --destructive
[15:50:57 INF] Connecting to https://hcr.io as harborsvc
[15:50:57 INF] Running in report-only mode
[15:50:58 INF] Logging out
[15:50:58 FTL] An Error was encountered while processing tags
System.InvalidOperationException: Not logged in
   at Harbor.Tagd.API.HarborClient.<Logout>d__10.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/API/HarborClient.cs:line 61
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/TagEngine.cs:line 104
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<DoClean>d__3.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 120
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.Program.<Main>d__2.MoveNext() in /home/travis/build/HylandSoftware/Harbor.Tagd/src/Harbor.Tagd/Program.cs:line 90

HTTP/405 on registry connect attempt

An HTTP/405 occurs when trying to connect to registry:

[17:15:16 INF] Loading rules using Harbor.Tagd.Rules.FilesystemRuleProvider
[17:15:16 VRB] Found rule {'Project': '.', 'Repo': '.', 'Tag': '.', 'Keep': 10, 'Ignore': [latest]}
[17:15:16 VRB] Using default rule {'Project': '.', 'Repo': '.', 'Tag': '.', 'Keep': 10, 'Ignore': [latest]}
[17:15:16 VRB] Ignoring the following projects: []
[17:15:16 VRB] Ignoring the following repos: []
[17:15:16 VRB] Ignoring the following tags: ["latest"]
[17:15:16 WRN] Running in report-only mode. To delete tags, run with --destructive
[17:15:16 INF] Connecting to https://docker-registry-test as admin
[17:15:16 INF] Running in report-only mode
[17:15:16 FTL] An Error was encountered while processing tags
System.Net.Http.HttpRequestException: Response status code does not indicate success: 405 (Not Allowed).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Harbor.Tagd.API.HarborClient.Login(String user, String password) in /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/API/HarborClient.cs:line 38
at Harbor.Tagd.TagEngine.Process() in /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/TagEngine.cs:line 64
at Harbor.Tagd.TagEngine.Process() in /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/TagEngine.cs:line 99
at Harbor.Tagd.TagEngine.Process() in /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/TagEngine.cs:line 106
at Harbor.Tagd.Program.DoClean(ApplicationSettings appSettings, IRuleProvider rules) in /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/Program.cs:line 126
at Harbor.Tagd.Program.Main(String[] args) in /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/Program.cs:line 96
[root@docker-registry-test Harbor.Tagd-master]# less /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/Program.cs
[root@docker-registry-test Harbor.Tagd-master]# vi /home/ivansa/dist/tagd/Harbor.Tagd-master/src/Harbor.Tagd/Program.cs

Parallelization changes sometimes throw FlurlHttpException stating the collection was modified

After merging #10 we got an exception this morning while tag processing. This didn't happen while testing, I'll look into it.

Flurl.Http.FlurlHttpException: Call failed. Collection was modified; enumeration operation may not execute. GET https://hcr.io/api/repositories/***/***/tags ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException_InvalidOperation_EnumFailedVersion()
   at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
   at Flurl.Http.FlurlRequest.WriteRequestCookies(HttpRequestMessage request)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at Flurl.Http.FlurlRequest.<HandleExceptionAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Flurl.Http.FlurlRequest.<SendAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.HttpResponseMessageExtensions.<ReceiveJson>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.API.HarborClient.<GetTags>d__13.MoveNext() in /home/jenkins/workspace/tagd-deployment_master-JF7YPWJZTOOXFXCSSMKKW3M2EI6YOS6ORV23RH3WIKLWAALDS2CA/tagd/src/Harbor.Tagd/API/HarborClient.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<ProcessRepository>d__12.MoveNext() in /home/jenkins/workspace/tagd-deployment_master-JF7YPWJZTOOXFXCSSMKKW3M2EI6YOS6ORV23RH3WIKLWAALDS2CA/tagd/src/Harbor.Tagd/TagEngine.cs:line 147
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<ProcessProject>d__11.MoveNext() in /home/jenkins/workspace/tagd-deployment_master-JF7YPWJZTOOXFXCSSMKKW3M2EI6YOS6ORV23RH3WIKLWAALDS2CA/tagd/src/Harbor.Tagd/TagEngine.cs:line 139
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Harbor.Tagd.TagEngine.<Process>d__9.MoveNext() in /home/jenkins/workspace/tagd-deployment_master-JF7YPWJZTOOXFXCSSMKKW3M2EI6YOS6ORV23RH3WIKLWAALDS2CA/tagd/src/Harbor.Tagd/TagEngine.cs:line 80

Specify absolute path to dotnet and Harbor.Tagd.dll in docker entrypoint

If you change the working directory in the container it doesn't start unless you override the entrypoint:

$ docker run -it --rm -v "$(pwd):/rules" -w /rules hylandsoftware/tagd check --config-file tagd.yml --verbosity verbose
Did you mean to run dotnet SDK commands? Please install dotnet SDK from:
  http://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409