[darc/maestro++] Darc should be buildable from source about arcade-services HOT 17 CLOSED

Doing a "proper" build from source (no prebuilts) isn't a blocker for 3.0 preview, so no.

That said, the sooner we can build Darc from source rather than downloading it, the better, to avoid building up debt that ends up delaying the delivery of a fully source-built .NET Core. (And yeah, Darc is part of the required toolset, so it's included.) When I try building Arcade from source I'm planning to file issues to get this on the radar.

from arcade-services.

I agree. Does darc even need to be buildable from source? What darc functionality would be used in the source-build scenario?

from arcade-services.

source-build will use darc for dotnet/arcade#427 "Create a product source build layout from a repo sha" but even if that's done "before" the build from source, I'm pretty sure we'll be using darc to flow dependencies between repos.
/cc @mmitche

from arcade-services.

@dagood I know you mentioned that you'll try building Arcade from source in the following two weeks, I assume this includes darc. Would you consider any potential issues in the process to be preview (12/14) blockers?

from arcade-services.

With Darc in particular, I don't see any source-buildability issues yet beyond the dependency stuff I already mentioned in the issue description, so I just added some more info and more detail on the design suggestion.

from arcade-services.

@dagood - do you feel this requirement is fulfilled yet? If not, what do you see is missing?

from arcade-services.

AFAIK none of the work discussed in this thread has been done.

This was discussed in a email thread ~11/29 "Source-buildability of Darc's local capabilities" with @ChadNedzlek, @mmitche, and @jcagme, so the stuff to do is still known.

from arcade-services.

@mmitche - what's needed for Preview II?

from arcade-services.

From the source-build perspective, this is not needed for preview 2. We'll need it to be able to deliver source-build on new distros without prebuilts, which I don't think is tied to any specific 3.0.0 milestones.

from arcade-services.

Is "darc" necessary to do anything more than fetching the correct commits/source blobs? Presumably that could be run on something other than the target computer (since it's not actually "the build"... it's just getting the source code), then that thing that's created would be either shipped off to distros like RedHat or built on the target computer... but at that point nothing in darc would be necessary. If there is no prebuilt stuff, darc won't be able to run anyway, so it doesn't help in that scenario either (since it requires a full functioning dotnet core to run).

Darc is a pretty intense mess of dependencies and pulling it apart is going to be painful and awkward and difficult to maintain. I'm hoping we can find a more sustainable solution that doesn't drag the web service into some dependency cycle.

from arcade-services.

Is "darc" necessary to do anything more than fetching the correct commits/source blobs?

Yes, when I mentioned "I'm pretty sure we'll be using darc to flow dependencies between repos." I'm referring to this:

• Darc is meant to be the only way developers and other tools like Maestro++ interact and alter version/dependency files as well as bootstrapping files and scripts in arcade participating repos.
• Darc's operations range from altering version/dependency files to creating PRs in specified repos.

Currently we use the Repo API (specifically Package Version Props) to do this, and from the above it sounds like using Package Version Props won't be supported. (And we might still need Darc for a standard way to flow zips through?)

Presumably that could be run on something other than the target computer (since it's not actually "the build"... it's just getting the source code), then that thing that's created would be either shipped off to distros like RedHat or built on the target computer... but at that point nothing in darc would be necessary.

That's correct, to me the reliance on Microsoft-prebuilt bits to do the trivial-seeming thing of gathering the source makes community maintenance less sustainable, but I don't have justification for this put together right now.

If there is no prebuilt stuff, darc won't be able to run anyway, so it doesn't help in that scenario either (since it requires a full functioning dotnet core to run).

This isn't the case: the N-1 SDK and Runtime aren't prebuilts. When there is no N-1, first-time bootstrapping has specific exceptions, but we can only use stuff we can build, not e.g. Octokit and Azure packages. If we can bring those into source-build, sweet, but that seems like a lot more maintenance cost than cutting them out of the build.

If you're interested, EngineeringPlanLinuxDistroRepos.md should clarify the requirements and N-1/bootstrap process. (It looks like it hasn't been updated for the switch from 2.1 => master yet.)

Darc is a pretty intense mess of dependencies and pulling it apart is going to be painful and awkward and difficult to maintain. I'm hoping we can find a more sustainable solution that doesn't drag the web service into some dependency cycle.

From what I've seen so far, I think the concerns are a bit overstated, but I think it's reasonable to wait for me to give it a try or something. Keep in mind that we aren't asking for this for preview2 or RTM.

from arcade-services.

@dagood I don't think this is required for preview 2, right?

from arcade-services.

Correct, not needed for preview2.

from arcade-services.

@dagood is this required for .net 5 rtm?

from arcade-services.

is this required for .net 5 rtm?

Nope.

(I think we (source-build) should actually consider extracting the subset of darc clone functionality we actually need and put it in a dotnet/arcade build task, which would end up resolving this. More info at dotnet/source-build#1571.)

from arcade-services.

Thanks @dagood. The idea seems interesting. Let me know what you decide to see if we can close this or if we should re-examine for future releases.

from arcade-services.

Is "darc" necessary to do anything more than fetching the correct commits/source blobs?

Yes, when I mentioned "I'm pretty sure we'll be using darc to flow dependencies between repos." I'm referring to this:

• Darc is meant to be the only way developers and other tools like Maestro++ interact and alter version/dependency files as well as bootstrapping files and scripts in arcade participating repos.
• Darc's operations range from altering version/dependency files to creating PRs in specified repos.

Currently we use the Repo API (specifically Package Version Props) to do this, and from the above it sounds like using Package Version Props won't be supported. (And we might still need Darc for a standard way to flow zips through?)

I noticed this when reading this thread back and I think I should give an update. It turns out that this didn't happen: we kept on using Package Version Props, and that infra has been maintained. We only use darc clone when creating a source tarball, not when building it, so building darc from source is not strictly necessary to build .NET from source without prebuilts. But I think this is an ugly technicality. Having to use a proprietary tool to create a usable source tarball from a Git tag isn't in the spirit of building from source, IMO.

Let me know what you decide to see if we can close this or if we should re-examine for future releases.

I think we can close it when we no longer depend on the prebuilt darc, whether that's via a build task or the original plan. I think it's worth keeping it open until then to keep track of this.

from arcade-services.