Git Product home page Git Product logo

Comments (26)

dblock avatar dblock commented on August 16, 2024 1

It's in OpenSearch, search for no-jdk, e.g. https://github.com/opensearch-project/OpenSearch/blob/0ba0e7cc26060f964fcbf6ee45bae53b3a9941d0/qa/os/src/test/java/org/opensearch/packaging/util/Distribution.java#L60

I think the first step for this one is to produce the no-jdk distribution as part of the daily build. Probably the easiest is to always make a JDK and no-JDK build when calling gradlew assemble (a change in OpenSearch). A slightly hackier solution is to add a different gradlew assemble to https://github.com/opensearch-project/opensearch-build/blob/main/scripts/default/opensearch/build.sh#L68. We pickup everything in build/distributions, so that would just get included for the ride across all platforms/architectures.

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

@rursprung I think the next step is to add it to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile, so it starts appearing in the build output

These currently run in a private Jenkins instance, but are this close to being run in public https://build.ci.opensearch.org/, see opensearch-project/opensearch-ci#2.

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

@peterzhuamazon Could you help @rursprung make sense of this stuff please? This is all visible and testable, and even documented, but a bit dense ;) let's try to update the documentation as we go

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

I think the next step is to add a step to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile that would call these gradle tasks that exist.

from opensearch-build.

peterzhuamazon avatar peterzhuamazon commented on August 16, 2024 1

Notable:

distribution:archives:no-jdk-darwin-tar
distribution:archives:no-jdk-linux-tar
distribution:archives:no-jdk-windows-zip
distribution:packages:no-jdk-rpm

Basically covered all the distributions we are looking at.

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

So from the top:

                            buildManifest(
                                componentName: "${COMPONENT_NAME}",
                                inputManifest: "manifests/${INPUT_MANIFEST}",
                                distribution: 'tar',
                                snapshot: true
                            )

This says take a manifest, e.g. https://github.com/opensearch-project/opensearch-build/blob/main/manifests/3.0.0/opensearch-3.0.0.yml and build it with build.sh ... + bunch of arguments such as the above distribution and snapshot=true/false. That iterates over components in the manifest and for each locates a build.sh. So for opensearch core it will by default call https://github.com/opensearch-project/opensearch-build/blob/main/scripts/components/OpenSearch/build.sh in which you can see a big switch that ultimately turns this into a ./gradle ... command. I think you want to start modifying that build.sh to support nojdk-tar so it can be referenced from the above snippet.

Alternatively, and possibly easier, build.sh for opensearch could always produce both a JDK and no-JDK distribution, so you would add a second line below ./gradlew :distribution:$TYPE:$TARGET:assemble -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER in https://github.com/opensearch-project/opensearch-build/blob/main/scripts/components/OpenSearch/build.sh. Then that file would be automatically collected and published next to the min distribution that we have today.

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

is there a specific reason why these do not exist or could/should they be added?

I don't know of any specific reason, simply unused.

i'm also a bit surprised ...

That's because it is unused, so no "good" reason.

suggested next steps (if you agree with them):

  1. add the missing no-jdk targets (see list above; everything but no-jdk-integ-test-zip) to OpenSearch on main & 2.x
  2. update the build.sh line here as indicated on main
  3. optional, unrelated: remove the (probably unnecessary?) empty folders for the targets in OpenSearch on main to avoid confusion in the future

Sounds good!

does the build.sh of this repository need to be compatible with multiple OpenSearch releases (i presume so, based on the fact that it's pretty version-agnostic otherwise?). if so, would we have to add the missing no-jdk targets also to older OpenSearch releases (where applicable)?

We have been moving build.sh's into each repo so that it branches with the repo, and you can do the same for OpenSearch repo. The default in this repo can go away if all built branches in opensearch-project/OpenSearch have script/build.sh.

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

where exactly should the file go? directly as /script/build.sh or in another location? then i'd just copy it over from the current main branch of this repository here?

Yes.

and what about the other files in /scripts/components/OpenSearch? do they stay here or should they be moved as well? do you maybe have an example PR where you already did such a move to see what should be changed?

They can stay here until there's a build.sh in all active branches of OpenSearch. We should backport build.sh all over the place, and then delete here.

from opensearch-build.

dblock avatar dblock commented on August 16, 2024 1

@rursprung That list is here: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/check-for-build.jenkinsfile#L17

Yes. No other changes needed. The logic is documented in https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts and implemented in https://github.com/opensearch-project/opensearch-build/blob/main/src/paths/script_finder.py.

from opensearch-build.

peterzhuamazon avatar peterzhuamazon commented on August 16, 2024 1

@peterzhuamazon (or anyone else who knows this): is there anything else left to be done so that the files created by the build process are also published to artifacts.opensearch.org?

Based on the email exchange let's start some discussion on this.
I will sync up with you on the necessary steps.

cc: @setiah @bbarani

Thanks.

from opensearch-build.

peternied avatar peternied commented on August 16, 2024

[Triage] The build process can support this, it would need to be integrated and tested so it could be released.

from opensearch-build.

gaiksaya avatar gaiksaya commented on August 16, 2024

[Triage] @dblock @CEHENKLE Does the opensearch-min artifact generation process (gradle) support this today?

from opensearch-build.

dblock avatar dblock commented on August 16, 2024

[Triage] @dblock @CEHENKLE Does the opensearch-min artifact generation process (gradle) support this today?

AFAIK yes, but I am not familiar with details

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

out of interest: where's the code which packages the JDK with the ZIP file? is this here or in the opensearch repo? i couldn't find it in either with a cursory glance (plus i'm not certain if this is even something which could easily be contributed by somebody who doesn't have access to the complete build infrastructure as, after all, the no-jdk dist will have to be published and linked to in various places?)

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

i browsed around a bit and found the following relevant things:

what i haven't seen yet at first glance is where gradlew assemble would define what should be created during tests (then again, i've only been clicking around on the web UI and haven't even cloned the repo yet... i'll do that next year if nobody picks it up until then).

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

what is missing here? from what i can see there are various distribution:archives:no-jdk-* (see archives) which seem to work (i ran assemble on the windows ZIP (because i'm on windows right now) and it created a ZIP w/o the JDK in it, as expected.

based on that i'd somehow expect that now only (AWS-internal?) tooling is missing which grabs this during the release process and publishes it everywhere?

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

@rursprung I think the next step is to add it to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile, so it starts appearing in the build output

These currently run in a private Jenkins instance, but are this close to being run in public https://build.ci.opensearch.org/, see opensearch-project/opensearch-ci#2.

thanks, that's good news!

however, i'm not sure how much sense it'll make if i'll try to contribute to this as i can't test whether my changes to the script would work.
and, if i understood the script correctly, it'd need some changes to buildAssembleUpload (which might still be somewhere in the not-yet-opensourced part?) - but then again i admittedly don't quite see yet where the script decides which artifact to pull in (it e.g. never seems to explicitly say whether it wants x64 or arm64?).

from opensearch-build.

peterzhuamazon avatar peterzhuamazon commented on August 16, 2024

[Copy from another conversation]:

Hi All,

This is a rather complicated task.

Just from the top of my head you need to introduce a new param in python side to allow jdk/non-jdk input.
Then the input needs to be passed to the building script side in order to call the corresponding gradle task without jdk.
I have not check the gradle tasks yet but if I recall correctly there exists non-jdk task for tar, probably not for rpm I am not sure.

If we introduce non-jdk this needs to be discussed extensively as well as on an issue.
Since all the distributions must take this new settings in mind, not just tar.
Also, whether OSD needs nodejs bundled or not is also something to follow up, not just OS.

Adding @bbarani into the discussion on this.

Thanks,
Peter

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

Thanks a lot for your feedback @peterzhuamazon! That does sound more complicated than it sounded initially!

I ran a quick test (on the latest main branch and on Windows) and it does seem that the standard assemble call already builds both the non-jdk as well as the bundled package (both [..]\distribution\archives\windows-zip\build\distributions\opensearch-min-3.0.0-SNAPSHOT-windows-x64.zip and [..]\distribution\archives\no-jdk-windows-zip\build\distributions\opensearch-min-3.0.0-SNAPSHOT-no-jdk-windows-x64.zip were generated). So maybe there’s no need to call a new gradle target and instead this is already part of the target being called?
There also seem to be gradle targets for no-jdk deb & rpm packages: https://github.com/opensearch-project/OpenSearch/tree/main/distribution/packages

Note that I raised the ticket explicitly for OpenSearch, not OpenSearch Dashboards – we’re currently not using that, so it’s of no relevance for me. Though I can see that the same discussion can be had there, I think it should be split away (and be driven by somebody involved in / using OpenSearch Dashboards) and could potentially also result in a different decision (AFAIK Node.JS versions are more finicky than JDK versions?).

from opensearch-build.

peterzhuamazon avatar peterzhuamazon commented on August 16, 2024

@rursprung all the non-jdk task I have seen so far:

task logs 

distribution:archives:no-jdk-darwin-tar:assemble - Assembles the outputs of this project.
distribution:archives:no-jdk-freebsd-tar:assemble - Assembles the outputs of this project.
distribution:archives:no-jdk-linux-tar:assemble - Assembles the outputs of this project.
distribution:archives:no-jdk-windows-zip:assemble - Assembles the outputs of this project.
distribution:packages:no-jdk-deb:assemble - Assembles the outputs of this project.
distribution:packages:no-jdk-rpm:assemble - Assembles the outputs of this project.
distribution:archives:no-jdk-darwin-tar:build - Assembles and tests this project.
distribution:archives:no-jdk-freebsd-tar:build - Assembles and tests this project.
distribution:archives:no-jdk-linux-tar:build - Assembles and tests this project.
distribution:archives:no-jdk-windows-zip:build - Assembles and tests this project.
distribution:packages:no-jdk-deb:build - Assembles and tests this project.
distribution:packages:no-jdk-rpm:build - Assembles and tests this project.
distribution:archives:no-jdk-darwin-tar:clean - Deletes the build directory.
distribution:archives:no-jdk-freebsd-tar:clean - Deletes the build directory.
distribution:archives:no-jdk-linux-tar:clean - Deletes the build directory.
distribution:archives:no-jdk-windows-zip:clean - Deletes the build directory.
distribution:packages:no-jdk-deb:clean - Deletes the build directory.
distribution:packages:no-jdk-rpm:clean - Deletes the build directory.
distribution:archives:no-jdk-darwin-tar:assembleDist - Assembles the main distributions
distribution:archives:no-jdk-freebsd-tar:assembleDist - Assembles the main distributions
distribution:archives:no-jdk-linux-tar:assembleDist - Assembles the main distributions
distribution:archives:no-jdk-windows-zip:assembleDist - Assembles the main distributions
distribution:packages:no-jdk-deb:assembleDist - Assembles the main distributions
distribution:packages:no-jdk-rpm:assembleDist - Assembles the main distributions
distribution:archives:no-jdk-darwin-tar:distTar - Bundles the project as a distribution.
distribution:archives:no-jdk-freebsd-tar:distTar - Bundles the project as a distribution.
distribution:archives:no-jdk-linux-tar:distTar - Bundles the project as a distribution.
distribution:archives:no-jdk-windows-zip:distTar - Bundles the project as a distribution.
distribution:packages:no-jdk-deb:distTar - Bundles the project as a distribution.
distribution:packages:no-jdk-rpm:distTar - Bundles the project as a distribution.
distribution:archives:no-jdk-darwin-tar:distZip - Bundles the project as a distribution.
distribution:archives:no-jdk-freebsd-tar:distZip - Bundles the project as a distribution.
distribution:archives:no-jdk-linux-tar:distZip - Bundles the project as a distribution.
distribution:archives:no-jdk-windows-zip:distZip - Bundles the project as a distribution.
distribution:packages:no-jdk-deb:distZip - Bundles the project as a distribution.
distribution:packages:no-jdk-rpm:distZip - Bundles the project as a distribution.
distribution:archives:no-jdk-darwin-tar:installDist - Installs the project as a distribution as-is.
distribution:archives:no-jdk-freebsd-tar:installDist - Installs the project as a distribution as-is.
distribution:archives:no-jdk-linux-tar:installDist - Installs the project as a distribution as-is.
distribution:archives:no-jdk-windows-zip:installDist - Installs the project as a distribution as-is.
distribution:packages:no-jdk-deb:installDist - Installs the project as a distribution as-is.
distribution:packages:no-jdk-rpm:installDist - Installs the project as a distribution as-is.
distribution:archives:no-jdk-darwin-tar:buildEnvironment - Displays all buildscript dependencies declared in project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:buildEnvironment - Displays all buildscript dependencies declared in project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:buildEnvironment - Displays all buildscript dependencies declared in project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:buildEnvironment - Displays all buildscript dependencies declared in project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:buildEnvironment - Displays all buildscript dependencies declared in project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:buildEnvironment - Displays all buildscript dependencies declared in project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:components - Displays the components produced by project ':distribution:archives:no-jdk-darwin-tar'. [incubating]
distribution:archives:no-jdk-freebsd-tar:components - Displays the components produced by project ':distribution:archives:no-jdk-freebsd-tar'. [incubating]
distribution:archives:no-jdk-linux-tar:components - Displays the components produced by project ':distribution:archives:no-jdk-linux-tar'. [incubating]
distribution:archives:no-jdk-windows-zip:components - Displays the components produced by project ':distribution:archives:no-jdk-windows-zip'. [incubating]
distribution:packages:no-jdk-deb:components - Displays the components produced by project ':distribution:packages:no-jdk-deb'. [incubating]
distribution:packages:no-jdk-rpm:components - Displays the components produced by project ':distribution:packages:no-jdk-rpm'. [incubating]
distribution:archives:no-jdk-darwin-tar:dependencies - Displays all dependencies declared in project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:dependencies - Displays all dependencies declared in project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:dependencies - Displays all dependencies declared in project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:dependencies - Displays all dependencies declared in project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:dependencies - Displays all dependencies declared in project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:dependencies - Displays all dependencies declared in project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:dependencyInsight - Displays the insight into a specific dependency in project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:dependencyInsight - Displays the insight into a specific dependency in project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:dependencyInsight - Displays the insight into a specific dependency in project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:dependencyInsight - Displays the insight into a specific dependency in project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:dependencyInsight - Displays the insight into a specific dependency in project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:dependencyInsight - Displays the insight into a specific dependency in project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:dependentComponents - Displays the dependent components of components in project ':distribution:archives:no-jdk-darwin-tar'. [incubating]
distribution:archives:no-jdk-freebsd-tar:dependentComponents - Displays the dependent components of components in project ':distribution:archives:no-jdk-freebsd-tar'. [incubating]
distribution:archives:no-jdk-linux-tar:dependentComponents - Displays the dependent components of components in project ':distribution:archives:no-jdk-linux-tar'. [incubating]
distribution:archives:no-jdk-windows-zip:dependentComponents - Displays the dependent components of components in project ':distribution:archives:no-jdk-windows-zip'. [incubating]
distribution:packages:no-jdk-deb:dependentComponents - Displays the dependent components of components in project ':distribution:packages:no-jdk-deb'. [incubating]
distribution:packages:no-jdk-rpm:dependentComponents - Displays the dependent components of components in project ':distribution:packages:no-jdk-rpm'. [incubating]
distribution:archives:no-jdk-darwin-tar:help - Displays a help message.
distribution:archives:no-jdk-freebsd-tar:help - Displays a help message.
distribution:archives:no-jdk-linux-tar:help - Displays a help message.
distribution:archives:no-jdk-windows-zip:help - Displays a help message.
distribution:packages:no-jdk-deb:help - Displays a help message.
distribution:packages:no-jdk-rpm:help - Displays a help message.
distribution:archives:no-jdk-darwin-tar:model - Displays the configuration model of project ':distribution:archives:no-jdk-darwin-tar'. [incubating]
distribution:archives:no-jdk-freebsd-tar:model - Displays the configuration model of project ':distribution:archives:no-jdk-freebsd-tar'. [incubating]
distribution:archives:no-jdk-linux-tar:model - Displays the configuration model of project ':distribution:archives:no-jdk-linux-tar'. [incubating]
distribution:archives:no-jdk-windows-zip:model - Displays the configuration model of project ':distribution:archives:no-jdk-windows-zip'. [incubating]
distribution:packages:no-jdk-deb:model - Displays the configuration model of project ':distribution:packages:no-jdk-deb'. [incubating]
distribution:packages:no-jdk-rpm:model - Displays the configuration model of project ':distribution:packages:no-jdk-rpm'. [incubating]
distribution:archives:no-jdk-darwin-tar:outgoingVariants - Displays the outgoing variants of project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:outgoingVariants - Displays the outgoing variants of project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:outgoingVariants - Displays the outgoing variants of project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:outgoingVariants - Displays the outgoing variants of project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:outgoingVariants - Displays the outgoing variants of project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:outgoingVariants - Displays the outgoing variants of project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:projects - Displays the sub-projects of project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:projects - Displays the sub-projects of project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:projects - Displays the sub-projects of project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:projects - Displays the sub-projects of project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:projects - Displays the sub-projects of project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:projects - Displays the sub-projects of project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:properties - Displays the properties of project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:properties - Displays the properties of project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:properties - Displays the properties of project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:properties - Displays the properties of project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:properties - Displays the properties of project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:properties - Displays the properties of project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:tasks - Displays the tasks runnable from project ':distribution:archives:no-jdk-darwin-tar'.
distribution:archives:no-jdk-freebsd-tar:tasks - Displays the tasks runnable from project ':distribution:archives:no-jdk-freebsd-tar'.
distribution:archives:no-jdk-linux-tar:tasks - Displays the tasks runnable from project ':distribution:archives:no-jdk-linux-tar'.
distribution:archives:no-jdk-windows-zip:tasks - Displays the tasks runnable from project ':distribution:archives:no-jdk-windows-zip'.
distribution:packages:no-jdk-deb:tasks - Displays the tasks runnable from project ':distribution:packages:no-jdk-deb'.
distribution:packages:no-jdk-rpm:tasks - Displays the tasks runnable from project ':distribution:packages:no-jdk-rpm'.
distribution:archives:no-jdk-darwin-tar:cleanEclipse - Cleans all Eclipse files.
distribution:archives:no-jdk-freebsd-tar:cleanEclipse - Cleans all Eclipse files.
distribution:archives:no-jdk-linux-tar:cleanEclipse - Cleans all Eclipse files.
distribution:archives:no-jdk-windows-zip:cleanEclipse - Cleans all Eclipse files.
distribution:packages:no-jdk-deb:cleanEclipse - Cleans all Eclipse files.
distribution:packages:no-jdk-rpm:cleanEclipse - Cleans all Eclipse files.
distribution:archives:no-jdk-darwin-tar:cleanIdea - Cleans IDEA project files (IML, IPR)
distribution:archives:no-jdk-freebsd-tar:cleanIdea - Cleans IDEA project files (IML, IPR)
distribution:archives:no-jdk-linux-tar:cleanIdea - Cleans IDEA project files (IML, IPR)
distribution:archives:no-jdk-windows-zip:cleanIdea - Cleans IDEA project files (IML, IPR)
distribution:packages:no-jdk-deb:cleanIdea - Cleans IDEA project files (IML, IPR)
distribution:packages:no-jdk-rpm:cleanIdea - Cleans IDEA project files (IML, IPR)
distribution:archives:no-jdk-darwin-tar:eclipse - Generates all Eclipse files.
distribution:archives:no-jdk-freebsd-tar:eclipse - Generates all Eclipse files.
distribution:archives:no-jdk-linux-tar:eclipse - Generates all Eclipse files.
distribution:archives:no-jdk-windows-zip:eclipse - Generates all Eclipse files.
distribution:packages:no-jdk-deb:eclipse - Generates all Eclipse files.
distribution:packages:no-jdk-rpm:eclipse - Generates all Eclipse files.
distribution:archives:no-jdk-darwin-tar:idea - Generates IDEA project files (IML, IPR, IWS)
distribution:archives:no-jdk-freebsd-tar:idea - Generates IDEA project files (IML, IPR, IWS)
distribution:archives:no-jdk-linux-tar:idea - Generates IDEA project files (IML, IPR, IWS)
distribution:archives:no-jdk-windows-zip:idea - Generates IDEA project files (IML, IPR, IWS)
distribution:packages:no-jdk-deb:idea - Generates IDEA project files (IML, IPR, IWS)
distribution:packages:no-jdk-rpm:idea - Generates IDEA project files (IML, IPR, IWS)
distribution:archives:no-jdk-darwin-tar:check - Runs all checks.
distribution:archives:no-jdk-freebsd-tar:check - Runs all checks.
distribution:archives:no-jdk-linux-tar:check - Runs all checks.
distribution:archives:no-jdk-windows-zip:check - Runs all checks.
distribution:packages:no-jdk-deb:check - Runs all checks.
distribution:packages:no-jdk-rpm:check - Runs all checks.
distribution:archives:no-jdk-darwin-tar:checkExtraction
distribution:archives:no-jdk-freebsd-tar:checkExtraction
distribution:archives:no-jdk-linux-tar:checkExtraction
distribution:archives:no-jdk-windows-zip:checkExtraction
distribution:packages:no-jdk-deb:checkExtraction
distribution:packages:no-jdk-rpm:checkExtraction
distribution:archives:no-jdk-darwin-tar:checkLicense
distribution:archives:no-jdk-freebsd-tar:checkLicense
distribution:archives:no-jdk-linux-tar:checkLicense
distribution:archives:no-jdk-windows-zip:checkLicense
distribution:packages:no-jdk-deb:checkLicense
distribution:packages:no-jdk-rpm:checkLicense
distribution:packages:no-jdk-deb:checkLicenseMetadata
distribution:packages:no-jdk-rpm:checkLicenseMetadata
distribution:archives:no-jdk-darwin-tar:checkNotice
distribution:archives:no-jdk-freebsd-tar:checkNotice
distribution:archives:no-jdk-linux-tar:checkNotice
distribution:archives:no-jdk-windows-zip:checkNotice
distribution:packages:no-jdk-deb:checkNotice
distribution:packages:no-jdk-rpm:checkNotice
distribution:archives:no-jdk-darwin-tar:checkPart1
distribution:archives:no-jdk-freebsd-tar:checkPart1
distribution:archives:no-jdk-linux-tar:checkPart1
distribution:archives:no-jdk-windows-zip:checkPart1
distribution:packages:no-jdk-deb:checkPart1
distribution:packages:no-jdk-rpm:checkPart1
distribution:archives:no-jdk-darwin-tar:checkPart2
distribution:archives:no-jdk-freebsd-tar:checkPart2
distribution:archives:no-jdk-linux-tar:checkPart2
distribution:archives:no-jdk-windows-zip:checkPart2
distribution:packages:no-jdk-deb:checkPart2
distribution:packages:no-jdk-rpm:checkPart2
distribution:archives:no-jdk-darwin-tar:cleanEclipseProject
distribution:archives:no-jdk-freebsd-tar:cleanEclipseProject
distribution:archives:no-jdk-linux-tar:cleanEclipseProject
distribution:archives:no-jdk-windows-zip:cleanEclipseProject
distribution:packages:no-jdk-deb:cleanEclipseProject
distribution:packages:no-jdk-rpm:cleanEclipseProject
distribution:archives:no-jdk-darwin-tar:cleanIdeaModule
distribution:archives:no-jdk-freebsd-tar:cleanIdeaModule
distribution:archives:no-jdk-linux-tar:cleanIdeaModule
distribution:archives:no-jdk-windows-zip:cleanIdeaModule
distribution:packages:no-jdk-deb:cleanIdeaModule
distribution:packages:no-jdk-rpm:cleanIdeaModule
distribution:archives:no-jdk-darwin-tar:copyEclipseSettings
distribution:archives:no-jdk-freebsd-tar:copyEclipseSettings
distribution:archives:no-jdk-linux-tar:copyEclipseSettings
distribution:archives:no-jdk-windows-zip:copyEclipseSettings
distribution:packages:no-jdk-deb:copyEclipseSettings
distribution:packages:no-jdk-rpm:copyEclipseSettings
distribution:archives:no-jdk-darwin-tar:eclipseProject - Generates the Eclipse project file.
distribution:archives:no-jdk-freebsd-tar:eclipseProject - Generates the Eclipse project file.
distribution:archives:no-jdk-linux-tar:eclipseProject - Generates the Eclipse project file.
distribution:archives:no-jdk-windows-zip:eclipseProject - Generates the Eclipse project file.
distribution:packages:no-jdk-deb:eclipseProject - Generates the Eclipse project file.
distribution:packages:no-jdk-rpm:eclipseProject - Generates the Eclipse project file.
distribution:archives:no-jdk-darwin-tar:ideaModule - Generates IDEA module files (IML)
distribution:archives:no-jdk-freebsd-tar:ideaModule - Generates IDEA module files (IML)
distribution:archives:no-jdk-linux-tar:ideaModule - Generates IDEA module files (IML)
distribution:archives:no-jdk-windows-zip:ideaModule - Generates IDEA module files (IML)
distribution:packages:no-jdk-deb:ideaModule - Generates IDEA module files (IML)
distribution:packages:no-jdk-rpm:ideaModule - Generates IDEA module files (IML)
distribution:archives:no-jdk-darwin-tar:resolveAllDependencies
distribution:archives:no-jdk-freebsd-tar:resolveAllDependencies
distribution:archives:no-jdk-linux-tar:resolveAllDependencies
distribution:archives:no-jdk-windows-zip:resolveAllDependencies
distribution:packages:no-jdk-deb:resolveAllDependencies
distribution:packages:no-jdk-rpm:resolveAllDependencies
distribution:archives:no-jdk-darwin-tar:wipeEclipseSettings
distribution:archives:no-jdk-freebsd-tar:wipeEclipseSettings
distribution:archives:no-jdk-linux-tar:wipeEclipseSettings
distribution:archives:no-jdk-windows-zip:wipeEclipseSettings
distribution:packages:no-jdk-deb:wipeEclipseSettings
distribution:packages:no-jdk-rpm:wipeEclipseSettings

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

I think the next step is to add a step to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile that would call these gradle tasks that exist.

this is where i got stuck in may (see our comments from may 5 & 6) - i haven't seen how that build script decides what to actually build, i.e. i haven't understood what i'd have to add/change to get a no-jdk build. if you could point me in the right direction there (and tell me how to test it) that'd be great!

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

Alternatively, and possibly easier, build.sh for opensearch could always produce both a JDK and no-JDK distribution, so you would add a second line below ./gradlew :distribution:$TYPE:$TARGET:assemble -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER in https://github.com/opensearch-project/opensearch-build/blob/main/scripts/components/OpenSearch/build.sh. Then that file would be automatically collected and published next to the min distribution that we have today.

i've now added it on the same line even (makes the build much faster as it avoids running the other tasks twice):

./gradlew :distribution:$TYPE:$TARGET:assemble :distribution:$TYPE:no-jdk-$TARGET:assemble -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER

however, this does require that every target called this way also has a corresponding no-jdk target.
i've checked the targets in the OpenSearch repository and saw that the following are currently missing:

  • :distribution:archives:no-jdk-linux-arm64-tar
  • :distribution:archives:no-jdk-linux-s390x-tar (though the s390x platform is anyway currently unsupported by build.sh - was this intentional @pantovad (you recently introduced support for s390x with opensearch-project/OpenSearch#4001)) - EDIT: after further checking i just realised that s390x-tar is always a no-jdk release, i.e. the name is misleading. would it make sense to rename the target? or should there be a target with a JDK as well (= change of behaviour of the target added with the PR, so that it'll ship with a JDK and there'll be a new one without a JDK - which would be aligned with the namping pattern?)
  • :distribution:archives:no-jdk-integ-test-zip (probably not needed: integ-test-zip sounds like a special target and is unsupported in build.sh?)
  • :distribution:packages:no-jdk-arm64-deb
  • :distribution:packages:no-jdk-arm64-rpm

is there a specific reason why these do not exist or could/should they be added?

i'm also a bit surprised that for some targets there's a sub-folder with an empty build.gradle in archives and packages and for others (e.g. the s390x platform) there isn't. are these even still needed or could they be deleted to avoid confusion (i first didn't notice that platform because i looked at the folders and only looked at the actual gradle targets in a second step)? some initial tests seem to show that they're not needed (what's relevant is just the entry in settings.gradle.

suggested next steps (if you agree with them):

  1. add the missing no-jdk targets (see list above; everything but no-jdk-integ-test-zip) to OpenSearch on main & 2.x
  2. update the build.sh line here as indicated on main
  3. optional, unrelated: remove the (probably unnecessary?) empty folders for the targets in OpenSearch on main to avoid confusion in the future

does the build.sh of this repository need to be compatible with multiple OpenSearch releases (i presume so, based on the fact that it's pretty version-agnostic otherwise?). if so, would we have to add the missing no-jdk targets also to older OpenSearch releases (where applicable)?

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

i've done the first and third step in the PRs opensearch-project/OpenSearch#4722 and opensearch-project/OpenSearch#4723 (the latter isn't actually relevant for this topic here, just a cleanup).

We have been moving build.sh's into each repo so that it branches with the repo, and you can do the same for OpenSearch repo. The default in this repo can go away if all built branches in opensearch-project/OpenSearch have script/build.sh.

where exactly should the file go? directly as /script/build.sh or in another location? then i'd just copy it over from the current main branch of this repository here?
and what about the other files in /scripts/components/OpenSearch? do they stay here or should they be moved as well?
do you maybe have an example PR where you already did such a move to see what should be changed?

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

all active branches of OpenSearch

is there a full list of those? my presumption would be that it's the following:

  • main
  • 2.x
  • 1.x

and maybe also the following?

  • 2.3
  • 2.2
  • 2.1
  • 2.0
  • 1.3

though with pure semver i don't see why they'd still be maintained (or why they exist as branches at all), as e.g. (random example) 2.4.0 with a bugfix would be a perfectly valid upgrade path for a user on 2.1.0, so no 2.1.1 would be needed. but that'd then be a discussion for elsewhere 😄

and then delete here.

does it afterwards automatically look for the correct script in the other repo or are more changes needed here (or elsewhere) than just deleting the files?

from opensearch-build.

dblock avatar dblock commented on August 16, 2024

@peterzhuamazon Am I correct for the list of active branches? I don't see 1.3.x being built in that list, but I know we just released 1.3.6. Do you typically add that on demand? Should @rursprung backport build.sh changes to 1.x and 1.3 in opensearch?

from opensearch-build.

rursprung avatar rursprung commented on August 16, 2024

@peterzhuamazon (or anyone else who knows this): is there anything else left to be done so that the files created by the build process are also published to artifacts.opensearch.org?

the other question is: where should these files be linked to? i'd suggest to just add them in the package dropdown of OpenSearch on the downloads page.
EDIT: just had a look at how the page is set up and if i saw it correctly it'd be enough to then add an additional file along the lines of this one, just specific for no-jdk? and then some site generator magic would pick it up and it'd end up in the dropdown?

from opensearch-build.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.