jerrymarino / xcbuildkit Goto Github PK

View Code? Open in Web Editor NEW

212.0 10.0 17.0 610 KB

xcbuildkit is a framework to extend or replace Xcode's build system

License: Other

Objective-C 1.35% Makefile 7.71% C 0.50% Python 13.59% Shell 7.70% Swift 56.01% Starlark 13.14%

xcbuildkit's Introduction

xcbuildkit

xcbuildkit is a framework to extend or replace Xcode's build system. No plugins needed!

Usage

Checkout the examples and Makefile.

Bazel Progress Bar Only

To install the progress bar for Bazel in Xcode, the example BazelBuildService provides all functionality.

tools/bazelwrapper build :BazelBuildServiceInstaller
sudo installer -pkg bazel-bin/BazelBuildServiceInstaller.pkg -target /

The installer requires sudo to setup an environment variable override. For testing only use make open_xcode

Additonally set Bazel options ( e.g. in ~/.bazelrc ) :

setup Bazel to write a binary BEP to /tmp/ --build_event_binary_file=/tmp/bep.bep This can be improved in the future
optionally crank up the BEP updates: --progress_report_interval=1 --build_event_publish_all_actions

What is this used for?

Generally, integrating third party build systems like Bazel.

Xcode can't render updates ( progress, errors, and warnings ) for external build systems, e.g. Bazel, because it isn't intrinsically aware of these systems. There is no supported way for run script build phases to communicate their state to Xcode. As a result, during long running tasks, like downloading a dependency or compiling an iOS application, Xcode's UI feels stuck. This is problematic as it can make the ad-hoc tasks "feel slow" due to the perception of no progress.

Additionally, for other features like testing, Xcode needs source files loaded into test targets. Currently, Xcode requires mocking out the toolchain with stub linkers and compilers so XCBuild can run ( e.g. XCHammer ). This adds extra, unavoidable overhead to each build. xcbuildkit enables replacing Xcode's build system entirely to improves the user experience of external build systems and remove the hacks.

Build system architecture

To build applications, Xcode runs tools like compilers and linkers via a build service daemon. Xcode and the build service communicate via a binary protocol. For example, to create a build, Xcode sends a create build message. The build service creates the build, and when the build is done, it sends a message back to Xcode know it's done. Throughout the lifecycle of the build, diagnostics and other information are also exchanged via this protocol. Xcode's UI is driven by this protocol as well.

xcbuildkit simply implements this protocol to enable extending or replacing default behavior. No plugins or hacks necessary!

Injecting external build system progress messages via a Proxy

A common path to integrate external build systems like buck or Bazel is to run them via a runscript. To preserve all functionality of Xcode's build system and inject progress messages and build diagnostics xcbuildkit provides a proxy build service. The main difference between the default architecture is that progress messages are injected within XCBBuildService's message.

See examples/BazelBuildService for an example implementation.

Replacing Xcode's build system with an external build system

It may be desirable to replace Xcode's default build system with an external one. This approach allows Xcode to run external build systems transparently to the user. Additionally, it removes the need to have ad-hoc integration via runscripts, which require stubbing out Xcode's toolchain with mock tools.

See examples/BSBuildService for an example implementation.

xcbuildkit's People

Contributors

Stargazers

Watchers

Forkers

nanwng longjohncoder ob jimmyhoran omarzl kaanbiryol tamerlx woshimaliang 0x1306a94 zkldream codelife2012 thiagohmcruz dierksen bazel-ios kuustudio zhaishuai

xcbuildkit's Issues

Fatal error: unexpected failure starting build:

This crash happens when button mashing the play button in Xcode and a build is in progress and emitting errors

Application Specific Information:
ProductBuildVersion: 11B500
Fatal error: unexpected failure starting build: file /Library/Caches/com.apple.xbs/Sources/XCBuild/XCBuild-15509/XCBuild/XCBBuildServiceSession.swift, line 110


Thread 0:: Dispatch queue: CGImageProviderCallbackQueue
0   libsystem_kernel.dylib              0x00007fff70bc8976 kevent_id + 10
1   libdispatch.dylib                   0x00007fff70a48844 _dispatch_kq_poll + 247
2   libdispatch.dylib                   0x00007fff70a492fe _dispatch_event_loop_wait_for_ownership + 498
3   libdispatch.dylib                   0x00007fff70a397ab __DISPATCH_WAIT_FOR_QUEUE__ + 270
4   libdispatch.dylib                   0x00007fff70a3940e _dispatch_sync_f_slow + 171
5   com.apple.CoreImage                 0x00007fff39ee57a3 CI::MetalContext::compute_quad(unsigned int, CI::MetalMainProgram const*, CGSize const&, void const**, unsigned long) + 293
6   com.apple.CoreImage                 0x00007fff39edfa78 CI::MetalContext::render_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, CGRect const&, void const**, __IOSurface**, unsigned long) + 932
7   com.apple.CoreImage                 0x00007fff39edf487 CI::MetalContext::render_root_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, void () block_pointer) + 243
8   com.apple.CoreImage                 0x00007fff39ede8b8 CI::Context::recursive_render(CI::TileTask*, CI::Node*, CGRect const&, CI::Node*, bool) + 1612
9   com.apple.CoreImage                 0x00007fff39ede09c CI::Context::render(CI::ProgramNode*, CGRect const&) + 98
10  com.apple.CoreImage                 0x00007fff3a12a2f6 invocation function for block in CI::image_get_bitmap(CI::Context*, CI::Image*, CGRect, CGColorSpace*, CI::Bitmap*, CI::RenderDestination const*) + 1378
11  com.apple.CoreImage                 0x00007fff39ed3ec8 CI::recursive_tile(CI::RenderTask*, CI::Context*, CI::RenderDestination const*, char const*, CI::Node*, CGRect const&, CI::PixelFormat, CI::swizzle_info const&, CI::TileTask* (CI::ProgramNode*, CGRect) block_pointer) + 3247

Support for ad-hoc diagnostic and fixits

Consider adding a way to emit ad-hoc diagnostics and fixits for swift and clang. There's a different codepath for these in Xcode's UI with xcbuild. It's likely possible to add this support in just a SWIFT_EXEC stub, however it might interface nicely inside of this build system.

Prototype

Prototype impl

I've created a prototype of this by stubbing out SWIFT_EXEC, and attached a demo project with the relevant details

# Example of stubbing out `SWIFT_EXEC` with a program that can transmit arbitrary diags
def _main():
    # type: () -> None
    if sys.argv[1:] == ["-v"]:
        os.system("swiftc -v")
        return

    _touch_deps_files(sys.argv)
    _touch_swiftmodule_files(sys.argv)


    os.system("mkdir -p  /Users/jmarino/Library/Developer/Xcode/DerivedData/Some-hettvdxoaohocfempirmoljvflkq/Build/Intermediates.noindex/Some.build/Debug/Some.build/Objects-normal/x86_64/")

    # During xcbuild, the build system seems to read in a dia. This is a
    # contrived diagnostic file, but xcbuild reads it in once swiftc emits it.
    # /tmp/main.dia resides in-tree at TMP_main.dia
    os.system("cp /tmp/main.dia /Users/jmarino/Library/Developer/Xcode/DerivedData/Some-hettvdxoaohocfempirmoljvflkq/Build/Intermediates.noindex/Some.build/Debug/Some.build/Objects-normal/x86_64/main.dia")

    os.system("echo xxxxx > /Users/jmarino/Library/Developer/Xcode/DerivedData/Some-hettvdxoaohocfempirmoljvflkq/Build/Intermediates.noindex/Some.build/Debug/Some.build/Objects-normal/x86_64/main.o")

    # Under the xcbuild, it runs the swift compiler with -parseable output. XCBuild
    # parses this is and dumps message pack material over the stdout file descriptor 
    # which Xcode further processes.
    # NOTE: for swift compilation, the .dias must be on the file system in this file.
    # /tmp/out resides in-tree at TMP_out
    os.system("cat /tmp/out >&2")

I have attached an example of the swiftc stub working - DiagReversingExample.zip

[Question] About XCBuildKit features

Hi!
I'm in the middle of integrating Bazel into Xcode for our developers to use.
I saw this repo and I think it is amazing! Some of developers feel the Xcode stuck and not doing anything because the UI is not showing any progress.

Some developers also complaining using Bazel, Xcode doesn't show error while they typing or show any auto-fix error (where we can click the error in Xcode and click button Fix-it)

I wonder if xcbuildkit adds that capability too.

Issues with make open_xcode / development workflow

This has gone through some Xcode updates - and now, it looks like there are a few issues. I looked at this briefly and it seems like stub.sh is no longer in the runfiles dir when calling Bazel build on 5.0.0.

There should be a series of PRs to make this work consistently again

Synthesize swift interface for existing logic

It is often hard and unapproachable to reason about the output of XCBBuildService and translate that into a higher level language. The other approach to a custom message pack implementation and related code is to generate an interface and call existing primitives in provided build system frameworks.

Now that the Swift ABI has become more stable we may have better luck calling it via ABI assumptions. Finally, the existence of a stable ABI also doesn't guarantee that this a sustainable approach, or allow us to guarantee existing frameworks will be callable from a snythetic interface or different compiler version. After all, Xcode, and macOS require updates annually and are subject to change.

Allow specifying binary BEP in another location than /tmp/bep.bep

The BazelBuildService example currently requires hardcoding by adding the param to their project's .bazelrc or ~/.bazelrc and it doesn't support multiple bazel builds at a time.

It needs more effort to set the param and or run Bazel directly. This is a good time to consider core features including:

Calling Bazel directly from here and dropping XCBuild's logic. Users currently have a runscript phase that invoke Bazel. Currently Tulsi and XCHammer both pass a temp JSON BEP path to Bazel.
BEP vs build event service.

Support for Indexing

It looks like an indexing message type is used in BSBuildService, but that the encoding/decoding for that type doesn’t have much. Replacing Xcode’s indexing with something that used a Bazel-generated compilation database seems very useful.