ccremer / greposync Goto Github PK
View Code? Open in Web Editor NEWManaged Git repositories in bulk
Home Page: https://ccremer.github.io/greposync
License: Apache License 2.0
Managed Git repositories in bulk
Home Page: https://ccremer.github.io/greposync
License: Apache License 2.0
Git hosting services like GitHub have API rate limiting in place. Even when using a dedicated API token, GitHub may block requests if there's too much activity with the token (either by amount of requests or by bursting requests in short time). One can retry later since this rate limit gets reset after some time.
An idea to counter this is to randomize the order in which repositories get updated or synced.
The scenario works around cases where such a limit can be hit by always running the same order.
Consider a list repositories from A-Z. Repos at the beginning of the list will almost always be synced, while the ones at end never if the API limiting takes place in the middle of it.
Randomizing the list at the start should better distribute creating PRs. However it'll be only useful in cases where running the update happens on a schedule or without filters.
Currently, logs and messages to the logging facility output everything immediately.
If jobs run in parallel, there could be confusing output shown where it's not easily identifiable which messages belong to which repo.
The idea is to suppress the logging output only to a single "success" message or a summarized error, where each update error is listed.
Add a flag to show the log output of the failed updates in an ordered fashion after running (e.g. like Docker buildkit).
Sometimes, repositories benefit from a template, yet they have a slightly different structure than the template.
For example, some files in template/module/.gitignore
might actually end up in a subdir my-repo/.gitignore
of the target repo.
Sometimes there are cases where a file needs to be in every repository, but they're individual and should stay unmanaged.
However, to improve experience of onboarding a new repository there should be an initial version of this file in case it doesn't exist.
Basically: If file is missing -> create from template -> don't ever touch again.
we could do this with a new special flag: initOnly: true
There may be cases where some executable scripts are synced.
Ensure that same file permissions stay when rendering.
It could be useful in templating or configuring repositories to apply common variables to a whole bunch of files within a subdirectory.
.github:
defaultBranch: master
.github/workflows/release.yml:
# here `defaultBranch` property would be inherited by the `.github` key. Or overwritten.
Currently, .Metadata
exposes .Repository
of the Git repository that is being updated.
This idea is about adding the Repository details of the controlling/template repository itself.
Ideas:
.Metadata.Self
.Metadata.Control
, .Metadata.ControlRepository
.Metadata.Greposync
These metadata could be used in templates, but also PR body and title.
Possible fields:
While https://github.com/urfave/cli works, there are some issues that I see:
--verbose
is a global flag, then it has to be <bin> --verbose subcommand
, while <bin> subcommand --verbose
doesn't work. This is a bad user experience, but one that be accepted if the chosen lib cannot handle this.however, the CLI flag library does stand out in the --help
output. We see
--pr-bodyTemplate value
, where value
indicates that something has to be added. Not every CLI library does this.--help
(some CLI libraries add additional ones completely unusable in some contexts)Other libraries
Notes:
--include
that gets mapped to the project.include
struct. We can/have to somehow adapt them though since the YAML config files provide options that can be overridden by CLI flags. OTOH, there are settings available exclusively via CLI flags (e.g. --dry-run
)In ccremer/go-command-pipeline#16 the labels were not applied.
Logs
DEBUG Using config (config="{"project":{"rootDir":"repos","jobs":1},"log":{"level":"debug","showDiff":true,"showLog":true},"pr":{"create":true,"targetBranch":"","labels":["greposync"],"bodyTemplate":"This Pull request updates this repository with changes from a greposync template repository.","subject":"Update from greposync"},"template":{"rootDir":"template"},"git":{"skipReset":false,"skipCommit":false,"skipPush":false,"forcePush":false,"commitMessage":"Update from greposync","commitBranch":"greposync","defaultBranch":"","name":"","namespace":"ccremer"},"repositoryLabels":null}")
DEBUG Executing step (step="configure infrastructure")
DEBUG Executing step (step="fetch managed repos config")
DEBUG Loading config file (name="managed_repos.yml")
...
DEBUG (github.com/ccremer/go-command-pipeline) Executing step (step="push changes")
INFO (github.com/ccremer/go-command-pipeline) git push origin greposync
DEBUG (github.com/ccremer/go-command-pipeline) Executing step (step="find existing pull request")
DEBUG (github.com/ccremer/go-command-pipeline) No PR found
DEBUG (github.com/ccremer/go-command-pipeline) Executing step (step="ensure pull request")
INFO (github.com/ccremer/go-command-pipeline) PR created (url="https://github.com/ccremer/go-command-pipeline/pull/16")
SUCCESS (github.com/ccremer/go-command-pipeline) Update finished for repository
v0.2.0
Currently, only GitHub is supported when it comes to managing PRs and labels.
Add support for GitLab API
When updating multiple repositories, some of them may fail for various reasons: wrong permissions, network error, etc.
Add a flag that allows to skip a repository update error, so that others may continue.
This issue provides visibility into Renovate updates and their statuses. Learn more
These updates are awaiting their schedule. Click on a checkbox to get an update now.
golang.org/x/oauth2
, golang.org/x/sys
)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
The git diff
command can really bloat the console output.
Add a flag that disables the diff'ing.
There are use cases where a repository may need to customize a bigger part of a template, but keep everything managed by modulesync.
Those files would then be merged toghether.
Example 1: Add custom job to a GitHub workflow. One could provide such YAML via .sync.yml file, but maintaining GitHub actions in this file is not really the way to go.
Possible implementation:
v0.1.0 has the project-include
and project-exclude
flags to control which repositories get actually synced or ignored.
In v0.2.0-rc release they are missing, lost during DDD rewrite.
Add CLI flags like --filter
that takes a regex to filter for repositories. Only repo names that match the filter are upated.
Consequentially, add the negative like --negative-filter
to skip repos matching the regex.
This is the same feature that ModuleSync has.
Writing templates can be tedious with lots of try-and-error
It should be possible to render templates in a local non-Git directory and compare them with a given set of expected file contents.
With that, it's possible to write kind of "unit tests" of a template.
Proposed file structure:
.tests
└── case-1
└── README.adoc
tests
├── case-1
│ ├── README.adoc
│ └── .sync.yaml
└── tests.yaml
Test cases are put into arbitrary-named folders inside the tests
directory in a template repository.
Each of these folders all the files that are expected to be rendered.
The .sync.yaml
contains the template variables with the exact same syntax as .sync.yaml
for a managed repository (in fact it's treated as a repository).
The .tests
folder exists only after running gsync test
and contains the rendered files based on the files in tests
with the same directory structure.
The file tests/tests.yaml
may contain any global variables (similar to config_defaults.yaml
).
That way, the render process is almost exactly the same.
Currently, [email protected]
is hardcoded to be the Git base when fetching repositories.
Make this configurable.
Greposync acts like modulesync where it pushes the updated template files to the target repositories and optionally makes PRs.
This is called "push" mode.
This method doesn't scale well. Especially GitHub does rate limiting and there's only limited options to work with those.
Other projects (e.g. https://github.com/cruft/cruft) work pull-based. Meaning the execution is inversed: The template repository gets cloned into a local directory, then the templates get rendered and finally a git diff is made. This allows including such a "check" as a form of linting step into CI/CD of the target repository itself.
This method scales better with large ecosystems.
ModuleSync knows 2 kinds of special values:
delete
unmanaged
From a syntax perspective, it's easy to mistake them for variables that are being used in templates.
However, greposync shall make this more explicit and also make those free for use in templates.
Furthermore, greposync supports more special values:
targetPath
initOnly
(possibly, see #100 )Proposal:
targetPath
to :special:targetPath
delete
to :special:delete
*unmanaged
to :special:unmanaged
**This will cause a breaking change for all .sync.yml
files in all managed repos. To make migration easier, the following shall be done:
--migrate-special-values
that replaces delete
and unmanaged
with the new parameter names and includes the change in the commit. Note: We'd have to search-replace line-by-line and cannot read/write as YAML, as that likely removes any user comments.Currently, gsync
is expected to be called from the current workdir.
It should be possible to add a path as CLI flag where everything is relative from that path.
That way, users don't have to cd ..
into the dirs when managing multiple template repositories.
At the end of a run, the output might be cluttered with Diffs and Logs.
The repo update should provide a summary table.
This table could look like this:
Repo | Status | Changed files | Pull request |
---|---|---|---|
repo-1 | Success | README.adoc, .gitignore and 12 others | https://github.com/.../12 |
repo-2 | Failure | unknown | not found |
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.