Comments (13)
Thanks for pinpointing the version. That is really helpful.
There were a lot of moving parts under the hood when comparing v10 to v11, so more investigation work is required.
- Do you use the default Airbrake config or do you have custom filters? Can you share it?
- Do you see this error locally?
- What does the line of code that crashes do?
When you say there might be something wrong on our side, I can try and fix it but would need more guidance about what to look for
Since the backtrace shows that the error occurs inside a model and you use bullet
, it makes me think that bullet
does something funky. I can see that the gem monkey-patches a lot of Rails internals. This frequently opens a can of worms. That said, at a cursory glance, their approach looks harmless.
Is there anything I can check or disable to change the behavior of that call to rack via airbrake?
You've mentioned the following:
> Tried removing some we can get away with not using like bullet.
What I understood is that you removed the gem, tried to run the app, and yet saw the error. Is this correct?
from airbrake.
I tried disabling the remote configs but the same issue happens.
After searching the other suggestion I did see some Net::HTTP.class_eval
block on our
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/CUSTOM_MIDDLEWARE.rb:10:in `call'
So I will try and coordinate to issue a fix patch for that custom middleware to use prepend (separate team and project
from airbrake.
Hey, Daniel! Do you have an example backtrace with a SystemStackError? It's a bit hard to debug this without knowing where exactly this happens.
from airbrake.
Yes, sorry for the delay on that here is the latest stack I got. Although this issue happens in multiple places with different stacks each time. Were there certain errors ignored before and now airbrake tries to catch them too?
Airbrake entry point:
...
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:34:in `call!'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:23:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-contrib-2.3.0/lib/rack/contrib/response_headers.rb:19:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/headers.rb:18:in `call'
...
Full printed stack:
/MY_REPO_FOLDER/app/MY_MODEL_CLASS:237:in `fetch_profile_attributes'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:517:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:517:in `invoke_after'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:825:in `_run_find_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/core.rb:372:in `init_with_attributes'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/persistence.rb:403:in `instantiate_instance_of'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/persistence.rb:258:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency/join_part.rb:66:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:264:in `construct_model'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:146:in `construct_model'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:252:in `block in construct'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:229:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:229:in `construct'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:141:in `construct'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:131:in `block (2 levels) in instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/result.rb:62:in `block in each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/result.rb:62:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/result.rb:62:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:128:in `block in instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:127:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:109:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:820:in `block (2 levels) in exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation/finder_methods.rb:396:in `apply_join_dependency'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:814:in `block in exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:842:in `skip_query_cache_if_necessary'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:811:in `exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/association_relation.rb:36:in `exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:626:in `load'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:250:in `records'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:46:in `records'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation/delegation.rb:88:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model/serializer/collection_serializer.rb:80:in `map'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model/serializer/collection_serializer.rb:80:in `serializers_from_resources'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model/serializer/collection_serializer.rb:15:in `initialize'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:50:in `new'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:50:in `serializer_instance'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:44:in `block in find_adapter'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:43:in `catch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:43:in `find_adapter'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:37:in `adapter'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/action_controller/serialization.rb:58:in `get_serializer'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/action_controller/serialization.rb:71:in `block (2 levels) in <module:Serialization>'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/renderers.rb:150:in `block in _render_to_body_with_renderer'
/.rbenv/versions/2.7.6/lib/ruby/2.7.0/set.rb:328:in `each_key'
/.rbenv/versions/2.7.6/lib/ruby/2.7.0/set.rb:328:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/renderers.rb:146:in `_render_to_body_with_renderer'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/rendering.rb:25:in `render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/rendering.rb:36:in `render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
/.rbenv/versions/2.7.6/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:43:in `render'
/MY_REPO_FOLDER/app/controllers/HELPERS/MY_HELPER:70:in `render_SOMETHING'
/MY_REPO_FOLDER/app/controllers/MY_CONTROLLER.rb:26:in `block (2 levels) in index'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
/MY_REPO_FOLDER/app/controllers/MY_CONTROLLER.rb:24:in `index'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/base.rb:195:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
/MY_REPO_FOLDER/app/controllers/ROOT_CONTROLLER.rb:208:in `init_user_time_zone'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actiontext-6.0.5.1/lib/action_text/rendering.rb:19:in `with_renderer'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actiontext-6.0.5.1/lib/action_text/engine.rb:55:in `block (4 levels) in <class:Engine>'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:121:in `instance_exec'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications.rb:180:in `block in instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications.rb:180:in `instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/base.rb:136:in `process'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionview-6.0.5.1/lib/action_view/rendering.rb:39:in `process'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal.rb:190:in `dispatch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal.rb:254:in `dispatch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/journey/router.rb:32:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/app_request_handler.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/meta_request_handler.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/CUSTOM_MIDDLEWARE.rb:10:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/http_accept_language-2.0.5/lib/http_accept_language/middleware.rb:14:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/rack.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/flipper-0.7.5/lib/flipper/middleware/memoizer.rb:40:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/warden-1.2.9/lib/warden/manager.rb:36:in `block in call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `catch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/conditional_get.rb:27:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/cookies.rb:654:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:34:in `call!'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:23:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-contrib-2.3.0/lib/rack/contrib/response_headers.rb:19:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/headers.rb:18:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/rack/logger.rb:37:in `call_app'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/rack/logger.rb:26:in `block in call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/tagged_logging.rb:80:in `block in tagged'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/tagged_logging.rb:28:in `tagged'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/tagged_logging.rb:80:in `tagged'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/rack/logger.rb:26:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/host_authorization.rb:97:in `call'
/MY_REPO_FOLDER/lib/middleware/strict_transport_security.rb:8:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-cors-1.1.1/lib/rack/cors.rb:100:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/engine.rb:527:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/railtie.rb:190:in `public_send'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/railtie.rb:190:in `method_missing'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/lint.rb:50:in `_call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/lint.rb:38:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/show_exceptions.rb:23:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/common_logger.rb:38:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/sinatra-2.2.0/lib/sinatra/base.rb:255:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/chunked.rb:98:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/content_length.rb:17:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/MY_REPO_FOLDER/bin/unicorn:16:in `load'
/MY_REPO_FOLDER/bin/unicorn:16:in `<main>'
Things tried already given the previously suggested solutions, but all these gave the same results:
- I've also tried updating
rack
,unicorn
,warden
andactionpack
via rails to the latest version we can safely update without refactors. - Tried removing some we can get away with not using like
bullet
. - Tried moving airbrake as the first or last gem in the Gemfile to play with the order change suggested.
from airbrake.
Thanks for the extra details.
The reordering trick is necessary if you use other tools that mess with Rails internals or hook into their APIs. If you don't use anything like that, then it has no effect.
I think it may be something with your setup or code. If there was a SystemStackError that had been raised consistently, more users would complain. So far, I haven't heard any similar reports.
The backtrace doesn't seem to be useful to debug this issue. Is there any chance for you to upgrade the gem gradually? From 10 to 11, from 11 to 12. I might be able to pinpoint the messy version.
The failing line is this one:
https://github.com/airbrake/airbrake/blob/6ca4b6023d56dba214b55f649743b41fa4934edf/lib/airbrake/rack/middleware.rb#LL34
Perhaps, there's something wrong inside your Rack env object that we try to read.
from airbrake.
I've tried this approach and the error starts happening from 10 to 11 upgrade.
Is there anything I can check or disable to change the behavior of that call to rack via airbrake?
When you say there might be something wrong on our side, I can try and fix it but would need more guidance about what to look for. As I said the only change we have made is the upgrade and v10 used to work, so it points at something new on the airbrake monitoring or features. Hopefully, with the details we can find the underlying problem, thanks again for all your help :)
from airbrake.
- Do you use the default Airbrake config or do you have custom filters? Can you share it?
I can't share code snippets unfortunately :/ but I can try and describe them in as much detail as possible.
- We have the project id and key defined in the environment via env variables loaded by
airbrake.yml
file. - The
airbrake.rb
initializer only sets one config that is a list of blocklist_keys - We add a filter from airbrake rack (Airbrake::Rack::HttpHeadersFilter.new) with no custom behavior
- We have a custom filter that checks for
notice[:errors]
comparing different types and callsnotice.ignore!
if we want to skip the errors.
- Do you see this error locally?
Yes, this stack is an error we see locally.
But... I've also gotten a different stack with the same error on the CI test setup. I will share that one at the bottom of the answers too hopefully it helps narrow down more.
- What does the line of code that crashes do?
The line of code in the model is an after_find callback that adds attributes to the model via self.attributes = ...
What I understood is that you removed the gem, tried to run the app, and yet saw the error. Is this correct?
This is correct I have tried it and even after removing the bullet gem entirely we still see the error locally.
The other CI stack:
I know it is a different stack but also ends in SystemStackError:stack level too deep
so it might help narrow down the issue:
# this last 4 lines loop in repeat until the error
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/bundler/gems/INTERNAL_GEM/request_id_propagation.rb:45:in `request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rails/net_http.rb:11:in `block in request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack.rb:25:in `capture_timing'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rails/net_http.rb:10:in `request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/connection_pool.rb:348:in `request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:79:in `block in transmit'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:133:in `block in session'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:128:in `session'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:50:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/content_length.rb:24:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/request_callback.rb:85:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.60.0/lib/aws-sdk-dynamodb/plugins/crc32_validation.rb:32:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/json/error_handler.rb:10:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/signature_v4.rb:58:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/json/handler.rb:13:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.60.0/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:119:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/request.rb:72:in `send_request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.60.0/lib/aws-sdk-dynamodb/client.rb:2730:in `describe_table'
/home/semaphore/accounts/app/services/aws/dynamodb_service.rb:63:in `table_exists?'
/home/semaphore/accounts/lib/MY_TASK:10:in `block in create_user_profiles_tables'
/home/semaphore/accounts/lib/MY_TASK.rake:8:in `each'
/home/semaphore/accounts/lib/MY_TASK.rake:8:in `create_user_profiles_tables'
/home/semaphore/accounts/lib/MY_TASK:59:in `block (3 levels) in <top (required)>'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rake.rb:17:in `execute'
/home/semaphore/accounts/lib/MY_TASK.rake:70:in `block in <top (required)>'
from airbrake.
Thanks for the extra info! I can see that your new backtrace includes aws-sdk-core-3.130.1/lib/seahorse/client/net_http
.
I checked how they monkey-patched Net::HTTP
and it looks safe. Could you bump aws-sdk-core
to the latest version and see if that helps? Maybe the older version contained a bug? My gut tells me some dependencies of yours cause this issue, not the Airbrake gem, but I don't have data to prove this.
The airbrake.rb initializer only sets one config that is a list of blocklist_keys
Since you can reproduce the problem locally, could you set blocklist_keys
to []
so that the library doesn't filter out anything?
We add a filter from airbrake rack (Airbrake::Rack::HttpHeadersFilter.new) with no custom behavior
Could you comment out that line and run the app without that filter?
We have a custom filter that checks for notice[:errors] comparing different types and calls notice.ignore! if we want to skip the errors.
Could you comment out that code?
It would be perfect if you could test these scenarios separately. I am not sure if this will help, but so far I don't have any good clues.
from airbrake.
I tried the suggested changes one at a time, commenting out each section, and then removing all configurations at once. in all cases, the local issue still happens.
I also tried updating aws-sdk-core
latest 3.168.3
and that updates a few of the individual service packages, but I see the same result.
My gut tells me some dependencies of yours cause this issue, not the Airbrake gem, but I don't have data to prove this.
This is hard because in our changes the only one is updating airbrake, so it's definitely airbrake's interaction with any of the other gems, but I can't update all dependencies blindly until things work again, it's a higher risk that other things would break this way.
I checked how they monkey-patched Net::HTTP and it looks safe
Could you tell me what you look for here specifically? maybe I can search in the local gem installation for "unsafe" usages and try to identify the issue that way...
from airbrake.
Could you tell me what you look for here specifically?
I look for eval
calls or "unsafe" class reopenings (reopening a class and redefining a method). That kind of stuff messes up our expectations when multiple gems monkey-patch the same thing. Here's an older reply of mine: #1080 (comment)
People should be using prepend
for monkey-patching, so that everyone can hook into the existing API and not break the code of others.
Your particular case doesn't look like it's the same issue, though. Even though the second backtrace indicates it could be (because Net::HTTP doesn't behave like we expected), the error from the first backtrace is new to me. I can't remember seeing an error coming from that class. It survived without bugs for 2 years.
from airbrake.
This is completely unrelated, but we were developing the "remote config" feature during the airbrake-ruby
v5.0.0 times. I would try to disable it to see if it changes anything. I doubt it, but it's worth a shot because, at the moment, I have no better options.
Airbrake.configure do |c|
c.remote_config = false
end
from airbrake.
When testing these changes locally, they solve the issue. I will continue testing on my CI and other environments but we can close this for now since the fix is not on the airbrake side. Thanks @kyrylo for guiding me in investigating this and helping us find the underlying issue we also needed to fix on our side.
The problem seems to be that the Net::HTTP.class_eval
in our custom middleware had an alias_method
definition for the class' request
method and it was calling it under the parent class request
block causing the recursive loop call and triggering the SystemStackError:stack level too deep
error.
# This was the problematic code, do not use it...
::Net::HTTP.class_eval do
alias_method :some_alias_method, :request
# ...
def request(req, *args, &block)
# some custom logic
# ....
some_alias_method(req, *args, &block)
end
end
It's still unclear to me why this is only an issue with the airbrake upgrade and goes unnoticed otherwise, I'm guessing airbrake 10+ added a prepend
of their own, and that interfered with the previous code we had. We fixed it on our side since prepend is the recommended way to do these patches.
from airbrake.
Thanks Daniel! Just ping me if you still experience any issues.
from airbrake.
Related Issues (20)
- Ability to limit APM queries HOT 3
- Add filter causing memory to increase HOT 1
- airbrake gem don't work with 7.0.0.alpha2 release HOT 8
- How to add exception specific context? HOT 3
- Airbrake: HTTP error: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed HOT 3
- InstrumentationSubscriberError HOT 6
- airbrake issues with lack of details HOT 1
- Airbrake does not track DB queries with MongoDB HOT 1
- Some people can get the alarm email, some people can't HOT 2
- APM only showing root paths HOT 1
- Airbrake Gem Attempts to Connect to Database in Databaseless App in Rails 6 HOT 4
- Airbrake 500 Errors HOT 31
- Airbrake and ruby 3.1 versions, change display of ErrorHighlight errors HOT 3
- Premature notification in Airbrake::Sidekiq::RetryableJobsFilter HOT 10
- Rails 7.0.3 with ruby 3.0.2 undefined method `new' for "Airbrake::Rack::Middleware":String (NoMethodError) HOT 2
- Mention options on how to disable performance/query/job stats HOT 1
- Airbrake seems to conflict with Scout and NR
- Rails 3.2 LTS error: uninitialized constant `Airbrake::Rails::ActiveRecord::Base`
- No clear documentation in terms of information being collected by default which includes PII
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airbrake.