This project forked from mojodna/heroku-buildpack-jemalloc
Heroku buildpack that installs the Jemalloc memory allocator
License: BSD 3-Clause "New" or "Revised" License
It has come up a couple times that people are surprised that just changing JEMALLOC_VERSION isn't enough. Today, changing the environment variable also requires a code push because it is copied in during the build phase. This helps avoid applications issues if the version of jemalloc changes or isn't available during a dyno reboot. It also means we're much better citizens to GitHub since we make one request per build instead of one request per restart per dyno.
To make this work we would need to do a version check during the boot process. Heroku offers some storage for buildpacks where we could stash downloaded versions instead of always requesting them from GitHub, or only need to request them once per version.
Using this buildpack caused our Rails app on Heroku-20 stack crashed with error: bundle command not found.
stack:
decide to revert to heroku/ruby buildpack for now
Since the change of version of jemalloc, we've started getting failures in the release phase of our heroku deploys that look like this
rake aborted!
JSON::ParserError: 765: unexpected token at 'node[29]: pthread_create: Invalid argument
["ok"]'
/app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/external_runtime.rb:68:in `extract_result'
/app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/external_runtime.rb:39:in `exec'
/app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/external_runtime.rb:14:in `initialize'
/app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/runtime.rb:57:in `new'
/app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/runtime.rb:57:in `compile'
/app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/module.rb:27:in `compile'
/app/vendor/bundle/ruby/2.5.0/gems/uglifier-4.1.10/lib/uglifier.rb:154:in `initialize'
/app/config/environments/staging.rb:21:in `new'
/app/config/environments/staging.rb:21:in `block in <main>'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/railtie.rb:209:in `instance_eval'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/railtie.rb:209:in `configure'
/app/config/environments/staging.rb:1:in `<main>'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `require'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `block in require_with_bootsnap_lfi'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:18:in `require_with_bootsnap_lfi'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in `require'
/app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `block in require'
/app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:259:in `load_dependency'
/app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `require'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/engine.rb:600:in `block (2 levels) in <class:Engine>'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/engine.rb:599:in `each'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/engine.rb:599:in `block in <class:Engine>'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:30:in `instance_exec'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:30:in `run'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:55:in `block in run_initializers'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:44:in `each'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:44:in `tsort_each_child'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:54:in `run_initializers'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/application.rb:352:in `initialize!'
/app/config/environment.rb:5:in `<main>'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `require'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `block in require_with_bootsnap_lfi'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:18:in `require_with_bootsnap_lfi'
/app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in `require'
/app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `block in require'
/app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:259:in `load_dependency'
/app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `require'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/application.rb:328:in `require_environment!'
/app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/application.rb:448:in `block in run_tasks_blocks'
/app/vendor/bundle/ruby/2.5.0/gems/bugsnag-5.0.1/lib/bugsnag/rake.rb:12:in `execute_with_bugsnag'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)
The line in question in staging.rb that seems to fail is this one
config.assets.js_compressor = Uglifier.new(harmony: true)
All our deploys with the jemalloc buildpack removed work just fine. (Figured this because our review apps don't use jemalloc, but it seemed that we had to run
heroku plugins:install heroku-repo
heroku repo:purge_cache
To get it to work.
I am not sure why, but my memory usage exploded a while ago and from what I can tell jemalloc is no longer working. No action was performed on our side.
Here are some excerpts from my configuration:
> heroku run bash
> ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm> jemalloc.sh ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm> heroku config:get JEMALLOC_ENABLED
true
Jemalloc 5.2.1 was released on 2019-08-05. See the release notes for details. This is a tracking ticket for switching the default from 5.1.0 to 5.2.1.
Release target is 2019-07-07 2019-09-30 2020-01-21
To try out the new version run the below command then push a code change.
heroku config:set JEMALLOC_VERSION=5.2.1 -a [app]
If you run into any issues with 5.2.1 please post details on this issue. Please include your stack[1] and list of buildpacks[2]. Extra points go to anyone who creates a reproducible example.
1: heroku apps:stacks -a [app]
2: heroku buildpacks -a [app]
On heroku-16 before setting JEMALLOC_ENABLED bash kicks out an error
/app/.profile.d/jemalloc.sh: line 5: [: =: unary operator expected
Hi!
Thank you for adding support for Heroku-20 so fast (1e33801).
I noticed that currently the buildpack doesn't invalidate the cache when the app's $STACK changes, which would presumably cause issues depending on whether it's a stack upgrade or downgrade, and what ABI differences exist between stacks.
A possible pattern for solving this would be:
https://github.com/heroku/heroku-buildpack-apt/blob/0357ec1e8863a803ada6c2d57f0850f2c5fa0366/bin/compile#L32-L41
Many thanks :-)
I'm trying to deploy to Heroku using Ruby 2.5.1, but getting the following error:
remote: -----> Detecting rake tasks
remote:
remote: !
remote: ! Could not detect rake tasks
remote: ! ensure you can run `$ bundle exec rake -P` against your app
remote: ! and using the production group of your Gemfile.
remote: ! rake aborted!
remote: ! LoadError: /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/jemalloc-1.0.1/lib/jemalloc.so: undefined symbol: Init_jemalloc - /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/jemalloc-1.0.1/lib/jemalloc.so
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `block in require_with_bootsnap_lfi'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require_with_bootsnap_lfi'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bundler-2.0.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bundler-2.0.1/lib/bundler/runtime.rb:76:in `each'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bundler-2.0.1/lib/bundler/runtime.rb:76:in `block in require'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bundler-2.0.1/lib/bundler/runtime.rb:65:in `each'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bundler-2.0.1/lib/bundler/runtime.rb:65:in `require'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/bundler-2.0.1/lib/bundler.rb:114:in `require'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/config/application.rb:5:in `<top (required)>'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/Rakefile:4:in `require_relative'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/Rakefile:4:in `<top (required)>'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/rake_module.rb:29:in `load'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/rake_module.rb:29:in `load_rakefile'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:703:in `raw_load_rakefile'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:104:in `block in load_rakefile'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:103:in `load_rakefile'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:82:in `block in run'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
remote: ! /tmp/build_35b27e3a334f16038d45bd4e05d69cce/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
remote: ! vendor/bundle/bin/rake:29:in `load'
remote: ! vendor/bundle/bin/rake:29:in `<main>'
remote: ! Push rejected, failed to compile Ruby app.
I followed the instructions on https://elements.heroku.com/buildpacks/gaffneyc/heroku-buildpack-jemalloc, any ideas?
If this buildpack is used in an app that uses Heroku Pipelines would there by any extra steps needed? I use a pipeline like this at work:
QA -> Staging -> Production
The app is built on QA then promoted to staging and finally to production.
If QA has the buildpack do staging and production also need the buildpack? What about the environment variables like JEMALLOC_ENABLED and JEMALLOC_VERSION?
Thanks
Thanks for Maintaining this buildpack
I got a question, this variable LD_PRELOAD, being set here
heroku-buildpack-jemalloc/bin/compile
Line 86 in 8ce845e
adds an extra space at the end of the file name, so when I check File.exists?( ENV['LD_PRELOAD'] ), it fails
However, when I check File.exists?( ENV['LD_PRELOAD'].strip ), it works
checking: ENV['LD_PRELOAD']
=> "/app/vendor/jemalloc/lib/libjemalloc.so "
so I was wondering, why is it like that?, and should it be conditioned if $LD_PRELOAD doesn't exist to be the string without this extra space?
also, might this affect Jemalloc being used in Ruby 2.4.10/Rails 4.2.8/Heroku 18?
I only used the variable JEMALLOC_ENABLED as true, and the buildpack was: https://github.com/gaffneyc/heroku-buildpack-jemalloc.git, which I assume pulls the latest
finally, is there anyway to verify that Jemalloc is working as it should?
I removed JEMALLOC_ENABLED, consiquently LD_PRELOAD was removed, and I added jemalloc.sh before the bundle call in my Procfile for both puma and sidekiq worker
The newest version of jemalloc was released on 2018-05-08. It's a recommended upgrade by the authors (see the release notes). This is a tracking ticket for switching the default from 5.0.1 to 5.1.0.
Rough target is 2018-07-02.
To try out the new version run the below command then push a code change.
heroku config:set JEMALLOC_VERSION=5.1.0 -a [app]
If you run into any issues with 5.1.0 please post details on this issue.
jemalloc 5.3.0 was released today. I plan to cut builds for it since it's a quick change but it's a good reminder to get #28 merged as well.
Hello :)
Thanks for this buildpack!
I'm just wondering why the default JEMALLOC_VERSION is still 5.2.1
Are there any issues with jemalloc 5.3.0?
add support to install this buildpack in different paas providers than just heroku (im trying to set this buildpack in railway currently).
Thanks for a great project!
Have you considered providing, or at least suggesting, a MALLOC_CONF?
This might be a good choice - it's used by https://github.com/fly-apps/dockerfile-rails/ which will likely be in rails 7.1
MALLOC_CONF="dirty_decay_ms:1000,narenas:2,background_thread:true"
more info:
https://gist.github.com/jjb/9ff0d3f622c8bbe904fe7a82e35152fc
I have setup a fresh rails app on a Heroku to try out the affects of changing the allocator to jemalloc however adding the buildpack and setting the env var does not seem to have any effect.
Here is the buildpack setup:
=== secure-ravine-91290 Buildpack URLs
1. https://github.com/gaffneyc/heroku-buildpack-jemalloc.git
2. heroku/ruby
Here are the ENV vars:
JEMALLOC_ENABLED: true
LANG: en_US.UTF-8
RACK_ENV: production
RAILS_ENV: production
RAILS_LOG_TO_STDOUT: enabled
RAILS_SERVE_STATIC_FILES: enabled
SECRET_KEY_BASE: ...
DATABASE_URL: ...
The stack is: heroku-18.
I have been checking with puts RbConfig::CONFIG['LIBS'] and this outputs -lpthread -lgmp -ldl -lcrypt -lm. I believe it should be something more like -lpthread -ljemalloc -ldl -lobjc which would indicate that jemalloc is in use.
Do I need to recompile ruby with support for jemalloc or is there something else I might be doing wrong?
We've been using this buildback with no problems for some time now.
Today, our builds started failing with the below message.
-----> jemalloc app detected
-----> Vendoring jemalloc 5.2.1
Fetching https://github.com/gaffneyc/heroku-buildpack-jemalloc/releases/download/heroku-18/jemalloc-5.2.1.tar.bz2
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now
! Push rejected, failed to compile jemalloc app.
! Push failed
There have been no buildpack/configuration changes and we did not have this problem as recently as yesterday.
After removing the buildpack and retrying, the build was successful.
Any ideas on what could be the issue?
Thanks
This is a full write-up of my unsuccessful attempt today at adding Jemalloc to my Rails app on Heroku. I'd greatly appreciate any suggestions and am happy to answer questions and try stuff. One idea floating in my mind is to maybe try a different version of Jemalloc, ie. 3.6.0.
My app is https://pullreminders.com and it uses Ruby 2.5.0, Rails 5.2.1, and Sidekiq. I run a single 2x Heroku dyno with 6 Puma workers using 5 threads each. This is on the heroku-18 stack. I was interested in jemalloc to see if I could reduce memory usage of my web server (so I could maybe run more Puma workers), scheduled jobs, and Sidekiq.
The steps I took to add jemalloc to my production app were:
JEMALLOC_ENABLED=true and JEMALLOC_VERSION=5.0.1heroku buildpacks:add --index 1 https://github.com/gaffneyc/heroku-buildpack-jemalloc.gitgit commit --allow-empty -m "empty commit"I did this at at ~4:45 PST. Just mentioning this because it was after Heroku's incident that day was resolved.
Almost immediately after adding jemalloc, I noticed an uptick in my app's response times. In the chart below, the bars represent response time, and the yellow line is throughput (which was low relative to earlier in the day).
I double checked my Heroku metrics which also showed an approximate doubling of both my 50th percentile and 95th percentile average response times. My mean response time had gone from ~150ms to ~300ms. My 95th percentile response times had gone from ~350ms to ~800ms and were a bit spikey. There were no timeouts or errors.
Memory usage appeared to be creeping back up to its pre-jemalloc level:
After a little while I removed jemalloc and deployed another empty commit to rebuild my app. Sure enough, response times fell back to their pre-jemalloc levels:
I followed the instructions and cannot see any hint of Jemalloc running in the app. On my local machine I was able to use jemalloc with rvm reinstall 2.6.4 --with-jemalloc and I can print to the console that it is in fact running. I am unable to do so with this setup. Either what do you suggest I do to get it running or what command will enable me to verify it is installed?
Hi, not totally clear (and I'm in a bit over my head here). If we want to use this with our current Rails app - do we need to install the ruby Jemalloc gem as well?
It would be good if the readme could specify that.
(PS - was not sure where else to ask, so posting this here).
Hi!
I was installing the buildpack and pinning it over to our stack version of heroku-20 and used the same tag on this repo but noticed that all the tags point to the same commit SHA.
Should I just pin it to the commit SHA instead?Or are the tags intended to point to the actual commit?
Thanks!
Thank you for this very helpful buildpack! Would you mind including a license file to be explicit about the terms under which your contributions are licensed? mojodna/heroku-buildpack-jemalloc is currently BSD licensed.
https://github.com/mojodna/heroku-buildpack-jemalloc/blob/master/LICENSE
Just a quick question how to validate the buildpack is in effect. I tried what's suggested in some places:
ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"
But this only gives me:
-lpthread -lgmp -ldl -lcrypt -lm
Although the buildpack seems to do something, as I saw a pthread error similar to the one in #3 and had to revert to version 5.0.1 as suggested to make it go away.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.