Submodule not registered registered in destination about copybara HOT 9 CLOSED

Yes, git.destination doesn't have submodules support. The workaround would be to move each submodule independently (I know it is not ideal).

Is the submodule shared between the origin and the destination?

from copybara.

I've just tried with git.gerrit_destination and SQUASH and got the same result.

Yes the submodule is shared.

from copybara.

git.gerrit_destination doesn't have support for writting to submodules.

If I understand correctly you have two repos, and you want that when one of the repos updates the submodule (and other files in the repo itself), the change is copied to a destination (the submodule, shared in the destination is updated and the files copied). Right?

Maybe you can allow copying the .gitmodules file, you use --git-destination-path and --dry-run and then you can wrap copybara in a script that does git submodule update --remote && git push after copybara run.

from copybara.

Thank you for the suggestions!
I'm a little hesitant in wrapping copybara with a shell script that manages a submodule, mainly because I find submodules fragile enough as is.

You are correct in me wanting to keep the submodules on the same version in both repositories.
In fact, the purpose of having the submodule is to allow external non-bazel users to use the same version of proto files as (external and internal) bazel users, the required protos are in a third party repo not controlled by us.
Bumping the third party dependency is then limited to updating the bazel dependency and the submodule.

The option we are considering now is having a submodule in the internal repository and copybara the necessary files from that submodule to the external repository.
This allows us to easily bump and version check the dependency and not even having to init the submodule.
The downside here is that in the external repository the files will look vendored without there being a reference to a specific git commit (effectively disallowing version bumps from the public repository).

from copybara.

from copybara.

Definitely sounds like something we could use, how do you extract/generate the bazel_version in bazel.bzl from a copybara export?

from copybara.

Wow Copybara is a lot more extensive than I realized, I ended up with the following

def transform_test(ctx):
    if len(ctx.changes.current) > 0:
        changes = ctx.changes.current
        tip = changes[0]

        summary = ["Bump <repository> to %s" % tip.ref, "", "Imported changes:"]
        for change in changes:
            summary.append("* %s" % change.first_line_message)

        ctx.set_message("\n".join(summary))

        version_str = "version = \"%s\"  # Imported on %s\n" % (tip.ref, ctx.now_as_string("yyyy-MM-dd", "UTC"))
        ctx.write_path(ctx.new_path("third_party/<repository-name>/version.bzl"), version_str)

core.workflow(
    name = "test",
    origin = git.origin(url = "file:///<path-to-origin>", ref = "HEAD", submodules = "RECURSIVE"),
    destination = git.destination("file:///<path-to-destination>", "<ref>"),
    origin_files = glob(["**"]),
    destination_files = glob(["**"]),
    transformations = [
        transform_test,
   ],
    authoring = authoring.pass_thru(default = ""),
    migrate_noop_changes = True,
)

Note that migrate_noop_changes makes Copybara consider all revs, not just the revs that change files in origin_files.

@mikelalcon we will not be using submodules after all, feel free to close the issue.

from copybara.

Sorry for the late response, yes, that is what we do :)

Some things you can do as improvements:
Add experimental_custom_rev_id = "PROTO_VERSION_REV_ID" --> This will use this rev-id instead of the default GitOrigin-RevId. If you use Copybara for other cases or other dependencies, this avoid conflict between tags.

You can also use this for the commit message:

metadata.squash_notes(
"Update Foo dependency to latest version\n\nIncluded changes:\n\n",
show_author = False)

In our case we use this origin_files:

origin_files = glob(["WORKSPACE"], exclude = ["WORKSPACE"]),

What this does is basically end up with no files. Then we generate the version file (as you do) and we end up in the destination with a change of just the bzl file.

BTW I'll keep this issue in mind when we add features to make this workflow less hacky :)

from copybara.

Created #85 for making dependency updates better. Closing this for now (despite submodules not been solved. We can reopen if there is demand).

from copybara.