Comments (4)
I'm not sure if we've exposed enough of the runtime metrics for those caches to make decisions from any externally observable measurements.
Pathologically, you want that cache to be large enough to keep the hottest values in without thrashing, and small enough where you only seek to populate when you will get a decent reuse benefit - i.e. it should follow the trends of your builds to incorporate or discard novel or old metadata sets, respectively.
FWIW there is not a lot of additional memory use on the server anyway, so if you're not using it for this, what else do you need it for?
from bazel-buildfarm.
Currently, I set -Xmx256g
and I've noticed that sometimes jvm_memory_bytes_used / jvm_memory_bytes_max
goes up to 100% for a short period of time, and I'm not sure what's causing it yet.
Then, I use jmap -dump:live,format=b,file=heap.hprof {pid}
dump heap file and MAT analyse it, and MAT shows that ShardInstance、directoryCache and commandCache occupy a large amount of memory. So I'm guessing it's because I set the size too high. Do you think I should reduce the size?
from bazel-buildfarm.
Or do you have any suggestions for locating OOM-related problems?
from bazel-buildfarm.
I say, yes, reduce the size (of the caches). I bet that a good portion of them decay quickly (but again, I would need to have/add stats to say that for sure), and that a useful comparison will be a sentinel build of some sort with full execution, on a fresh server (twice, to see the impact of population), against one that has a reduced size. If your time is unaffected (and the builds are representative), then you're likely unaffected by thrash.
Dealing with OOMs is a matter of exactly what you're doing. jmap, object counts over time, perturbation of likely causes, and measuring the effects and timing of GC runs.
from bazel-buildfarm.
Related Issues (20)
- Work directory should be added automatically as an input directory HOT 3
- Many `WARNING: DEADLINE_EXCEEDED: write` logs from buildfarm-server. HOT 2
- [Bazel CI] gaussian_distribution_test is failing at Bazel@HEAD HOT 1
- No logs HOT 4
- Running a server on Linux and build on MacOS HOT 3
- buildfarm-worker on Windows Server 2022 fails to clean up operation files HOT 3
- if redisUri is empty, java.lang.NullPointerException error occurred HOT 1
- [Bazel CI] Build Error: No repository for `@com_google_protobuf_javalite` in module `grpc-java` without bazel_dep or use_repo HOT 6
- Do deployments via Helm work as RBE ? HOT 5
- make execution-policy=as-nobody the default on Linux
- [ZstdCompression] Read request with offset incorrectly fails with not_found error and removes blob
- [Bazel CI] apt-get and integration tests are failing with Bazel@HEAD HOT 1
- Tips on learning buildfarm? HOT 2
- Remote builds stuck on Buildfarm that is deployed with Helm. HOT 6
- Implement Fetch 'Push' Service
- Support http_header: prefix in Fetch Qualifiers
- bazel.canonical_id unsupported in Asset Fetch API HOT 1
- ERROR: error running 'git fetch origin refs/heads/*:refs/remotes/origin/* refs/tags/*:refs/tags/*' while working with @build_buildfarm~: HOT 2
- Updated from java17 to java21 to use the new server and worker causes JVM to shutdown
- Fetch service does not properly handle missing content-length HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazel-buildfarm.