Trigger "UpdatePackageStorage" on repository update about integrations HOT 16 CLOSED

Hmm.. this is good argument and it may work! I will try this way :)

from integrations.

Will this issue be resolved in favor of elastic/package-storage#110 ?

this comment is outdated.

from integrations.

I want to trigger mage UpdatePackageStorage as additional step of the master job. I can adjust Jenkinsfile to execute it, but the script requires a Github token.

mage UpdatePackageStorage opens a PR against package-storage once it detects that there are changes that should be pushed. To push a PR it need a github token in file in ~/.elastic/github.token, but the path can be adjusted. I suppose it should be stored in a secret.

As @kuisathaverat is OOTO, can I ask for any guidance from the team? @cachedout @mdelapenya

from integrations.

To push a PR it need a github token in file in ~/.elastic/github.token, but the path can be adjusted. I suppose it should be stored in a secret.

Can it use an environment variable or is it required that ~/.elastic/github.token be written out?

from integrations.

I can modify the code to fetch from two places. This is not a problem at all.

I see an issue in preparing the sandbox for the user, I assume Elasticmachine? The user needs to have forked a repository, https://github.com/elastic/package-storage , then while executing the script, it will update the fork, checkout new branch with changes and open a PR. I understand I need to open an issue with infra team to fork the public repo? I hope it's not the problem...

BTW I don't need the Elasticmachine user. I can use any available one.

from integrations.

@mtojek Which pipeline is this? If you have access to the APM Pipeline Library, could you just use githubCreatePullRequest()

Example: https://github.com/elastic/apm/blob/master/.ci/Jenkinsfile#L113

from integrations.

Pipeline: https://beats-ci.elastic.co/job/Beats/job/integrations/job/master/

The UpdatePackageStorage script doesn't only open a PR, but it executes also some logic to determine whether the PR should be created and what is the scope, hence it's hard to simply replace it with bunch of groovy's functions (like githubCreatePullRequest()) or hub.

Docs for UpdatePackageStorage: https://github.com/elastic/integrations/tree/master/dev/update-package-storage#principle-of-operation

As you see the script belongs to the https://github.com/elastic/integrations , but it accesses and commits to the https://github.com/elastic/package-storage .

from integrations.

Actually with GITHUB_TOKEN, this would be relatively easy. I believe the missing link is where and how can I configure the forked package-storage, and which Github user can own this. In all cases I saw Elastic Machine, but I don't see it has any public repos available (and fork of package-storage must be public).

from integrations.

..how can I configure the forked package-storage, and which Github user can own this.

Hmm, this is an interesting question. We might need to bring in infra here to help advise us on the right course of action here. I will cc: @jonahbull as a starting point and see if we can figure this out.

from integrations.

@mtojek In thinking about this a bit more, is there a specific reason that a fork even needs to exist? Could you just submit a PR to the main package-storage repo?

from integrations.

Here is a sample PR: elastic/package-storage#206

We used to review changes in the PR and also involved some CI checks around. Actually it's not main, but snapshot branch. Also, we enabled some branch protection rules for prod and there are talks to enable them for all stages.

Later on, the package travels through different branches (snapshot->staging->production). I would keep this path aside from the context of the PR. Let's focus on the push from integrations to the package-storage.

from integrations.

Could you just submit a PR to the main package-storage repo?

Sorry I misread your comment, thought you propose to directly push to the branch without pull request. Actually, my above response is valid, but I will ask a followup question then: is it possible to submit a PR without a fork to a different repo (integrations vs package-storage)? or any repo?

from integrations.

is it possible to submit a PR without a fork to a different repo (integrations vs package-storage)? or any repo?

I believe you'll need write permissions to push to a temporary branch on the repo but then after that, nothing special is required to create a PR from that newly created branch.

from integrations.

@mtojek This sounds very promising. If we are already on CI opening a PR, could this CI potentially just run the tests for the specific branch + the changes in addition and if they pass push a commit directly? :-D BTW, should not block your progress on this one, happy to have the PR.

from integrations.

@cachedout

In the PR draft #224 I'm trying to store credentials in variable GITHUB_TOKEN (mtojek@7a018f9#diff-8db37e8fcea0f1a8f2f39667e94ebcc4), but unfortunately it fails the CI without any hints: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fintegrations/detail/PR-224/7/pipeline/122

I assume that it's a security mechanism and I need some permissions?

EDIT:

Found also this:

15:10:36  ERROR: Credentials 'f6c7695a-671e-4f4f-a331-acdce44ff9ba' is of type 'SSH Username with private key' where 'org.jenkinsci.plugins.plaincredentials.StringCredentials' was expected

from integrations.

I found different credentials used by Kibana (2a9602aa-ab9f-4e52-baf3-b71ca88469c7) and managed to open a PR: elastic/package-storage#302. I have no idea how to force Elasticmachine to sign CLA :)

from integrations.