Building Buck on Illumos about buck2 HOT 13 CLOSED

A PR to fix compilation of fs4 on illumos is here: al8n/fs4-rs#3

from buck2.

FWIW the psutil dep is probably only needed for straight-up Linux and macOS and can be gated to those two platforms only. Prior to the OSS release it was easiest to gate to Unix, but sounds like we should be a bit more precise with the conditional imports now.

I can update this later this weekend.

from buck2.

All my diffs are approved and running through CI - expect them to land in 2-5 hours.

from buck2.

Fantastic!

For info, the upgrade to fs4 had to get reverted because it pulls in an old rustix that has problems on Windows. If the fs4 diffs you have up get merged, then we should be good to do it again. If they don't get merged, I guess someone needs to fork to fs6...

I'll work on an upgrade of rustyline. I see 11.0 is the latest, so will upgrade to that.

You can ignore the future incompatibilities things - we'll take care of them when they become real issues, and they show up for everyone.

from buck2.

I've got an internal diff moving to fs4. There is your protoc which I'm importing. For jemalloc I've got a diff to use that on linux/mac, rather than generic unix, which seems like the only flavours of unix it will actually work. I'm pushing those through the review/CI process internally and then they'll be in the repo (tomorrow if I get a diff review tonight, otherwise likely Monday). Then we'll be on to new and exciting errors :)

from buck2.

44b28fd should remove the psutil dep for non-linux/mac.

from buck2.

With dd685b5 we've upgraded to rustyline 11.0.

Unfortunately we can't upgrade to fs4 until al8n/fs4-rs#2 lands.

from buck2.

All these sound like things we should clean up. Moving to fs4. Fixes to protobuf stuff. We'll aim to reduce the delta you are maintaining, and hopefully eventually get to success.

from buck2.

<3.

I'm gonna send a PR with the protobuff bits right now, as it's very small and self-contained.

from buck2.

I notice you upgraded rustyline and removed psutil - are both of those required too?

from buck2.

ah yes, the dangers of reconstructing a story after the fact, I forgot about those too :)

So, psutil has a bunch of errors if you try to compile it, and is only being used in-tree for one tiny thing, which on some platforms returns empty results already anyway, so my idea was to try and remove it for now, just to get things building.

Rustyline also has vague illumos issues (I know I've run into them and hit a corner case, but I can't off the top of my head remember what that is) and I was upgrading it in-tree because it also failed to build for some reason; personally I tend to use reedline because I love nu shell, and it does seem to work well on illumos, but I'm not even sure whatever that edge case was is relevant here, so I wasn't suggesting removing it. I also didn't have a chance yet to dig into how integrated it is, but I assume it's fairly important, as readline often is.

from buck2.

Catching up on this, very cool to see all this activity!

  thread 'main' panicked at 'PROTOC not available for platform Error { os: "illumos", arch: "x86_64" }, set $BUCK2_BUILD_PROTOC to override', app/buck2_protoc_dev/src/lib.rs:30:17

This error is way nicer now too :)

buuuuuut!

success! Well, with this patch anyway:

diff --git a/Cargo.toml b/Cargo.toml
index 22e2f2d70c..7bbc9117be 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -128,7 +128,7 @@ faccess = "0.2.3"
 fancy-regex = "0.10.0"
 flate2 = "1.0.22"
 fnv = "1.0.7"
-fs2 = "0.4.3"
+fs4 = { git = "https://github.com/sunfishcode/fs4-rs", branch = "sunfishcode/illumos", features = ["sync"] }
 futures = { version = "0.3.24", features = ["async-await", "compat"] }
 futures-intrusive = "0.4"
 glob = "0.3.0"
diff --git a/app/buck2_client_ctx/Cargo.toml b/app/buck2_client_ctx/Cargo.toml
index 7edd24b340..25bef4ba3c 100644
--- a/app/buck2_client_ctx/Cargo.toml
+++ b/app/buck2_client_ctx/Cargo.toml
@@ -16,7 +16,7 @@ chrono = { workspace = true }
 crossterm = { workspace = true }
 derivative = { workspace = true }
 derive_more = { workspace = true }
-fs2 = { workspace = true }
+fs4 = { workspace = true }
 futures = { workspace = true }
 hex = { workspace = true }
 httparse = { workspace = true }
diff --git a/app/buck2_client_ctx/src/build_count.rs b/app/buck2_client_ctx/src/build_count.rs
index 0c3bf71804..e18eda173f 100644
--- a/app/buck2_client_ctx/src/build_count.rs
+++ b/app/buck2_client_ctx/src/build_count.rs
@@ -15,7 +15,7 @@ use anyhow::Context;
 use buck2_common::client_utils;
 use buck2_core::fs::paths::abs_norm_path::AbsNormPathBuf;
 use buck2_core::fs::paths::file_name::FileName;
-use fs2::FileExt;
+use fs4::FileExt;
 use serde::Deserialize;
 use serde::Serialize;
 use tokio::io::AsyncReadExt;
diff --git a/app/buck2_client_ctx/src/daemon/client/mod.rs b/app/buck2_client_ctx/src/daemon/client/mod.rs
index 72b4201efe..21b64c9a18 100644
--- a/app/buck2_client_ctx/src/daemon/client/mod.rs
+++ b/app/buck2_client_ctx/src/daemon/client/mod.rs
@@ -19,7 +19,7 @@ use buck2_cli_proto::*;
 use buck2_common::daemon_dir::DaemonDir;
 use buck2_core::fs::fs_util;
 use buck2_core::fs::paths::abs_norm_path::AbsNormPathBuf;
-use fs2::FileExt;
+use fs4::FileExt;
 use futures::future::BoxFuture;
 use futures::pin_mut;
 use futures::stream;
diff --git a/starlark-rust/starlark/Cargo.toml b/starlark-rust/starlark/Cargo.toml
index 4f4906933c..6f063d7b7a 100644
--- a/starlark-rust/starlark/Cargo.toml
+++ b/starlark-rust/starlark/Cargo.toml
@@ -41,7 +41,7 @@ walkdir = "2.3"
 serde = { version = "1.0", features = ["derive"] }
 logos = "0.12"
 serde_json = "1.0"
-rustyline = "7.1"
+rustyline = "9.1"
 maplit = "1.0.2"
 lsp-server = "0.5"
 lsp-types = "0.93.0"

and then

$ cargo update -p rustix
$ cargo update -p nix
$ cargo update -p rustyline

(these operations not reflected in the diff because of Cargo.lock not being in version control.)

Once @sunfishcode 's PR lands and a new release is cut, I can send in a PR to actually update this.

from buck2.

fs4 0.6.4 is now released, which has the fixes for building on illumos.

from buck2.