Git Product home page Git Product logo

large-monorepo's Introduction

Benchmarking Nx, Turbo, and Lage

Recording:

nx-turbo-recording

Repo contains:

  1. 5 shared buildable packages/libraries with 250 components each
  2. 5 Next.js applications built out of 20 app-specific libraries. Each app-specific lib has 250 components each. Each library uses the shared components.

Combined there are about 26k components. It's a lot of components, but they are very small. This corresponds to a medium size enterprise repo. A lot of our clients have repos that are 10x bigger than this, so this repo isn't something out or ordinary. And, the bigger the repo, the bigger the difference in performance between Nx and Turbo.

The repo has Nx, Turbo, and Lage enabled. They don't affect each other. You can remove one without affecting the other one.

Bencharmk & Results (Dec 04)

npm run benchmark runs the benchmark. The following numbers produced by an M1Max MBP on macOS 13 (Ventura). On a Windows machine all the tools will get slower, and the delta between Nx and Turbo/Lage will get bigger.

  • average lage time is: 1183.9
  • average turbo time is: 1153.8
  • average nx time is: 166.83
  • nx 7.1x faster than lage
  • nx is 6.9x faster than turbo

No daemon

Another performance mark that we're going to start tracking is commands run without their respective daemon. This would represent running the tools in an CI environment. These can be run with npm run benchmark-no-daemon

  • average lage time is: 1169.6
  • average turbo time is: 1197.3
  • average nx time is: 1195.1

Benchmark & Results June 28

  • average nx time is: 142.7
  • average turbo time is: 900.6
  • average lage time is: 1378.4
  • nx is 6.31x faster than turbo
  • nx is 9.66x faster than lage
No daemon
  • average nx time is: 1258.2
  • average turbo time is: 1285.6
  • average lage time is: 1213.5

Numbers from May 31 (M1 Pro Ventura):

  • average nx time is: 149.3
  • average turbo time is: 907.3
  • average lage time is: 1084.4

Numbers from May 19:

  • average nx time is: 172.8
  • average turbo time is: 1134.1
  • average lage time is: 1109.9

Why is Nx faster than Turbo

Nx uses several optimizations to minimize the amount of computation required. For instance, it stores information about the repository on disk to be able to recompute only what is needed. It runs a daemon process that gets all the necessary data structures ready before the developer invokes a command. And the data structures are updated incrementally, usually in just a few milliseconds.

Is Nx always faster? No. The performance sensitive parts of Nx are written in Rust, but it is all wrapped into a Node.js process. Loading Node.js takes about 70ms (on a mac), regardless of what you do. You build 1000 projects, takes 70ms. You build 1 project, it takes 70ms. If you have a repo with say 10 files in it, running Turbo will likely be faster because it boots faster.

Yarn, npm, pnpm have a similar boot time to Nx, and folks don't mind. And, of course, it's worth asking whether a high-performance build tool is even required for a repo with 10 files in it.

Does this performance difference matter in practice?

The cache restoration Turborepo provides is likely to be fast enough for a lot of small and mid-size repos. What matters more is the ability to distribute any command across say 50 machines while preserving the dev ergonomics of running it on a single machine. Nx can do it. Bazel can do it (which Nx borrows some ideas from). Turbo can't. This is where the perf gains are for larger repos. See this benchmark to learn more.

Dev ergonomics & Staying out of your way

When some folks compare Nx and Turborepo, they say something like "Nx may do all of those things well, and may be faster, but Turbo is built to stay out of you way". Let's talk about staying out of your way:

Run nx build crew --skip-nx-cache and turbo run build --scope=crew --force:

terminal outputs

Nx doesn't change your terminal output. Spinners, animations, colors are the same whether you use Nx or not (we instrument Node.js to get this result). What is also important is that when you restore things from cache, Nx will replay the terminal output identical to the one you would have had you run the command.

Examine Turbo's output: no spinners, no animations, no colors. Pretty much anything you run with Turbo looks different ( and a lot worse, to be honest) from running the same command without Turbo.

A lot of Nx users don't even know they use Nx, or even what Nx is. Things they run look the same, they just got faster.

Lerna and Nx

Lerna 5.1 adds the ability to use Nx for task orchestration and computation caching (in addition to p-map and p-queue, which it had before). Given that Lerna uses Nx to run tasks, unsurprisingly, the numbers for Lerna and Nx will be very similar--it's the same powerful task orchestrator under the hood. This also means that Lerna supports distributed tasks execution (see above) and that it captures terminal output correctly.

Found an issue? Send a PR.

If you find any issue with the repo, with the benchmark or the setup, please send a PR. The intention isn't to cherry pick some example where Turbo doesn't do well because of some weird edge case. If it happens that the repo surfaces some edge case with how turbo works, send a PR, and let's fix it. We tried to select the setup that Turbo should handle well (e.g., Next.js apps). The repo doesn't use any incrementality which Nx is very good at. We did our best to make it fair.

large-monorepo's People

Contributors

cammisuli avatar isaacplmann avatar jameshenry avatar kenotron avatar nartc avatar vsavkin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

large-monorepo's Issues

nx is slower than turbo (windows machine) July 16

Not sure I read this correctly but this is my results by running npm run benchmark

I ran this twice and got similar results

The command ran in 13375ms
------------------------
results
------------------------
average lage time is: 13584.7
average turbo time is: 4748.2
average lerna (powered by nx) time is: 5246.8
average nx time is: 4893.4
nx is 2.776127028242122x faster than lage
nx is 0.9703273797359709x faster than turbo
nx is 1.0722197245269138x faster than lerna (powered by nx)

npx envinfo

System:
  OS: Windows 10 10.0.22000
  CPU: (16) x64 AMD Ryzen 7 5800H with Radeon Graphics
  Memory: 4.01 GB / 13.86 GB
Binaries:
  Node: 16.13.2 - C:\Program Files\nodejs\node.EXE
  Yarn: 1.22.19 - ~\AppData\Roaming\npm\yarn.CMD
  npm: 8.1.2 - C:\Program Files\nodejs\npm.CMD
Managers:
  Cargo: 1.61.0 - ~\.cargo\bin\cargo.EXE
Utilities:
  Git: 2.37.0. - /mingw64/bin/git
IDEs:
  VSCode: 1.69.1 - C:\Users\bryan\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD
Languages:
  Bash: 4.4.23 - C:\Program Files\Git\usr\bin\bash.EXE
  Perl: 5.34.0 - C:\Program Files\Git\usr\bin\perl.EXE
  Rust: 1.61.0 - /c/Users/bryan/.cargo/bin/rustc
Browsers:
  Edge: Spartan (44.22000.120.0), Chromium (103.0.1264.62)
  Internet Explorer: 11.0.22000.120
Monorepos:
  Yarn Workspaces: 1.22.19
  Lerna: 5.1.0

Running benchmark showing issues

Steps to reproduce:

Step 1 : npm install
after install, script postinstall run and giving following issues
image

quick help would be appreciated

NX about 1.6x faster than Turbo

Here my results:

Versions:
"nx": "17.0.3",
"lage": "2.7.8",
"turbo": "1.10.16",

Platform: AMD 7945HX on Ubuntu 22 LTS

Run 1:
average lage time is: 906
average turbo time is: 1260.6
average nx time is: 764.1
nx is 1.1857086768747547x faster than lage
nx is 1.6497840596780524x faster than turbo

Run 2:
average lage time is: 916.8
average turbo time is: 1270.3
average nx time is: 749.9
nx is 1.22256300840112x faster than lage
nx is 1.6939591945592745x faster than turbo

Run 3:
average lage time is: 923.5
average turbo time is: 1284.6
average nx time is: 754.4
nx is 1.22415164369035x faster than lage
nx is 1.7028101802757156x faster than turbo

Testing methodology does not take IO behavior into account

My relevant setup is a Ryzen 7950X3D, and a Sabrent Rocket NVMe SSD.

Initially I got that Turbo is almost twice as fast as NX.
image

The benchmark itself showed me a very interesting thing: Turbo consistently ran the cached output around ~380ms with very little deviance, and NX did ~880ms and ~420ms runs one after the other. ~880ms, ~420ms, ~880ms, ~420ms, ~880ms, ~420ms... and so on. That seems odd.

But since these tests are running back-to-back and hit the filesystem, I figured it's something IO related, so I modified the tests a tiny bit and added a sleep between each run to let everything settle. I had to increase this sleep all the way to 5 seconds to get NX to give me as consistent results as Turbo. But after this NX managed to finish around ~430ms.
image

Anyway, I hope no one wants to settle on a tool based on minute performance differences. The whole experience around it is much more important. For example I don't like that for the NX example every package has to have a nx entry in the package.json file that contains workspace relative paths instead of package relative paths, it's just one more thing to update when you want to move/rename a package.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.