Git Product home page Git Product logo

Comments (13)

kyrylo avatar kyrylo commented on June 12, 2024 1

Thanks for pinpointing the version. That is really helpful.

There were a lot of moving parts under the hood when comparing v10 to v11, so more investigation work is required.

  • Do you use the default Airbrake config or do you have custom filters? Can you share it?
  • Do you see this error locally?
  • What does the line of code that crashes do?

When you say there might be something wrong on our side, I can try and fix it but would need more guidance about what to look for

Since the backtrace shows that the error occurs inside a model and you use bullet, it makes me think that bullet does something funky. I can see that the gem monkey-patches a lot of Rails internals. This frequently opens a can of worms. That said, at a cursory glance, their approach looks harmless.

Is there anything I can check or disable to change the behavior of that call to rack via airbrake?

You've mentioned the following:
> Tried removing some we can get away with not using like bullet.

What I understood is that you removed the gem, tried to run the app, and yet saw the error. Is this correct?

from airbrake.

Nandez89 avatar Nandez89 commented on June 12, 2024 1

I tried disabling the remote configs but the same issue happens.

After searching the other suggestion I did see some Net::HTTP.class_eval block on our

/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/CUSTOM_MIDDLEWARE.rb:10:in `call'

So I will try and coordinate to issue a fix patch for that custom middleware to use prepend (separate team and project 😅) and update the status here if that solves the issue. If you think of anything else I can try in the meantime please let me know 👍

from airbrake.

kyrylo avatar kyrylo commented on June 12, 2024

Hey, Daniel! Do you have an example backtrace with a SystemStackError? It's a bit hard to debug this without knowing where exactly this happens.

from airbrake.

Nandez89 avatar Nandez89 commented on June 12, 2024

Yes, sorry for the delay on that here is the latest stack I got. Although this issue happens in multiple places with different stacks each time. Were there certain errors ignored before and now airbrake tries to catch them too?

Airbrake entry point:

...
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:34:in `call!'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:23:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-contrib-2.3.0/lib/rack/contrib/response_headers.rb:19:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/headers.rb:18:in `call'
...

Full printed stack:

/MY_REPO_FOLDER/app/MY_MODEL_CLASS:237:in `fetch_profile_attributes'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:517:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:517:in `invoke_after'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:825:in `_run_find_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/core.rb:372:in `init_with_attributes'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/persistence.rb:403:in `instantiate_instance_of'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/persistence.rb:258:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency/join_part.rb:66:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:264:in `construct_model'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:146:in `construct_model'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:252:in `block in construct'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:229:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:229:in `construct'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:141:in `construct'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:131:in `block (2 levels) in instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/result.rb:62:in `block in each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/result.rb:62:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/result.rb:62:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:128:in `block in instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/associations/join_dependency.rb:127:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:109:in `instantiate'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:820:in `block (2 levels) in exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation/finder_methods.rb:396:in `apply_join_dependency'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:814:in `block in exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:842:in `skip_query_cache_if_necessary'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:811:in `exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/association_relation.rb:36:in `exec_queries'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:626:in `load'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation.rb:250:in `records'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/active_record60.rb:46:in `records'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/relation/delegation.rb:88:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model/serializer/collection_serializer.rb:80:in `map'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model/serializer/collection_serializer.rb:80:in `serializers_from_resources'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model/serializer/collection_serializer.rb:15:in `initialize'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:50:in `new'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:50:in `serializer_instance'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:44:in `block in find_adapter'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:43:in `catch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:43:in `find_adapter'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/active_model_serializers/serializable_resource.rb:37:in `adapter'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/action_controller/serialization.rb:58:in `get_serializer'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/active_model_serializers-0fbe0fad0dec/lib/action_controller/serialization.rb:71:in `block (2 levels) in <module:Serialization>'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/renderers.rb:150:in `block in _render_to_body_with_renderer'
/.rbenv/versions/2.7.6/lib/ruby/2.7.0/set.rb:328:in `each_key'
/.rbenv/versions/2.7.6/lib/ruby/2.7.0/set.rb:328:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/renderers.rb:146:in `_render_to_body_with_renderer'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/rendering.rb:25:in `render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/rendering.rb:36:in `render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
/.rbenv/versions/2.7.6/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:43:in `render'
/MY_REPO_FOLDER/app/controllers/HELPERS/MY_HELPER:70:in `render_SOMETHING'
/MY_REPO_FOLDER/app/controllers/MY_CONTROLLER.rb:26:in `block (2 levels) in index'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
/MY_REPO_FOLDER/app/controllers/MY_CONTROLLER.rb:24:in `index'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/base.rb:195:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
/MY_REPO_FOLDER/app/controllers/ROOT_CONTROLLER.rb:208:in `init_user_time_zone'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actiontext-6.0.5.1/lib/action_text/rendering.rb:19:in `with_renderer'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actiontext-6.0.5.1/lib/action_text/engine.rb:55:in `block (4 levels) in <class:Engine>'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:121:in `instance_exec'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications.rb:180:in `block in instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/notifications.rb:180:in `instrument'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activerecord-6.0.5.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/abstract_controller/base.rb:136:in `process'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionview-6.0.5.1/lib/action_view/rendering.rb:39:in `process'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal.rb:190:in `dispatch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_controller/metal.rb:254:in `dispatch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/journey/router.rb:32:in `each'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/app_request_handler.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/meta_request_handler.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/bundler/gems/CUSTOM_MIDDLEWARE.rb:10:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/http_accept_language-2.0.5/lib/http_accept_language/middleware.rb:14:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/bullet-7.0.1/lib/bullet/rack.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/flipper-0.7.5/lib/flipper/middleware/memoizer.rb:40:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/warden-1.2.9/lib/warden/manager.rb:36:in `block in call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `catch'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/conditional_get.rb:27:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/cookies.rb:654:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:34:in `call!'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack/middleware.rb:23:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-contrib-2.3.0/lib/rack/contrib/response_headers.rb:19:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/meta_request-0.7.3/lib/meta_request/middlewares/headers.rb:18:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/rack/logger.rb:37:in `call_app'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/rack/logger.rb:26:in `block in call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/tagged_logging.rb:80:in `block in tagged'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/tagged_logging.rb:28:in `tagged'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/tagged_logging.rb:80:in `tagged'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/rack/logger.rb:26:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/activesupport-6.0.5.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/actionpack-6.0.5.1/lib/action_dispatch/middleware/host_authorization.rb:97:in `call'
/MY_REPO_FOLDER/lib/middleware/strict_transport_security.rb:8:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-cors-1.1.1/lib/rack/cors.rb:100:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/engine.rb:527:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/railtie.rb:190:in `public_send'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/railties-6.0.5.1/lib/rails/railtie.rb:190:in `method_missing'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/lint.rb:50:in `_call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/lint.rb:38:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/show_exceptions.rb:23:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/common_logger.rb:38:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/sinatra-2.2.0/lib/sinatra/base.rb:255:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/chunked.rb:98:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/content_length.rb:17:in `call'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/MY_REPO_FOLDER/.bundle/ruby/2.7.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/MY_REPO_FOLDER/bin/unicorn:16:in `load'
/MY_REPO_FOLDER/bin/unicorn:16:in `<main>'

Things tried already given the previously suggested solutions, but all these gave the same results:

  • I've also tried updating rack, unicorn, warden and actionpack via rails to the latest version we can safely update without refactors.
  • Tried removing some we can get away with not using like bullet.
  • Tried moving airbrake as the first or last gem in the Gemfile to play with the order change suggested.

from airbrake.

kyrylo avatar kyrylo commented on June 12, 2024

Thanks for the extra details.

The reordering trick is necessary if you use other tools that mess with Rails internals or hook into their APIs. If you don't use anything like that, then it has no effect.

I think it may be something with your setup or code. If there was a SystemStackError that had been raised consistently, more users would complain. So far, I haven't heard any similar reports.

The backtrace doesn't seem to be useful to debug this issue. Is there any chance for you to upgrade the gem gradually? From 10 to 11, from 11 to 12. I might be able to pinpoint the messy version.

The failing line is this one:
https://github.com/airbrake/airbrake/blob/6ca4b6023d56dba214b55f649743b41fa4934edf/lib/airbrake/rack/middleware.rb#LL34

Perhaps, there's something wrong inside your Rack env object that we try to read.

from airbrake.

Nandez89 avatar Nandez89 commented on June 12, 2024

I've tried this approach and the error starts happening from 10 to 11 upgrade.

Is there anything I can check or disable to change the behavior of that call to rack via airbrake?
When you say there might be something wrong on our side, I can try and fix it but would need more guidance about what to look for. As I said the only change we have made is the upgrade and v10 used to work, so it points at something new on the airbrake monitoring or features. Hopefully, with the details we can find the underlying problem, thanks again for all your help :)

from airbrake.

Nandez89 avatar Nandez89 commented on June 12, 2024
  • Do you use the default Airbrake config or do you have custom filters? Can you share it?

I can't share code snippets unfortunately :/ but I can try and describe them in as much detail as possible.

  • We have the project id and key defined in the environment via env variables loaded by airbrake.yml file.
  • The airbrake.rb initializer only sets one config that is a list of blocklist_keys
  • We add a filter from airbrake rack (Airbrake::Rack::HttpHeadersFilter.new) with no custom behavior
  • We have a custom filter that checks for notice[:errors] comparing different types and calls notice.ignore! if we want to skip the errors.
  • Do you see this error locally?

Yes, this stack is an error we see locally.
But... I've also gotten a different stack with the same error on the CI test setup. I will share that one at the bottom of the answers too hopefully it helps narrow down more.

  • What does the line of code that crashes do?

The line of code in the model is an after_find callback that adds attributes to the model via self.attributes = ...

What I understood is that you removed the gem, tried to run the app, and yet saw the error. Is this correct?

This is correct I have tried it and even after removing the bullet gem entirely we still see the error locally.

The other CI stack:
I know it is a different stack but also ends in SystemStackError:stack level too deep so it might help narrow down the issue:

# this last 4 lines loop in repeat until the error
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/bundler/gems/INTERNAL_GEM/request_id_propagation.rb:45:in `request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rails/net_http.rb:11:in `block in request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rack.rb:25:in `capture_timing'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rails/net_http.rb:10:in `request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/connection_pool.rb:348:in `request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:79:in `block in transmit'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:133:in `block in session'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:128:in `session'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/net_http/handler.rb:50:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/content_length.rb:24:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/request_callback.rb:85:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.60.0/lib/aws-sdk-dynamodb/plugins/crc32_validation.rb:32:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/json/error_handler.rb:10:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/signature_v4.rb:58:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/json/handler.rb:13:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.60.0/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:119:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.130.1/lib/seahorse/client/request.rb:72:in `send_request'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.60.0/lib/aws-sdk-dynamodb/client.rb:2730:in `describe_table'
/home/semaphore/accounts/app/services/aws/dynamodb_service.rb:63:in `table_exists?'
/home/semaphore/accounts/lib/MY_TASK:10:in `block in create_user_profiles_tables'
/home/semaphore/accounts/lib/MY_TASK.rake:8:in `each'
/home/semaphore/accounts/lib/MY_TASK.rake:8:in `create_user_profiles_tables'
/home/semaphore/accounts/lib/MY_TASK:59:in `block (3 levels) in <top (required)>'
/home/semaphore/accounts/vendor/bundle/ruby/2.7.0/gems/airbrake-13.0.3/lib/airbrake/rake.rb:17:in `execute'
/home/semaphore/accounts/lib/MY_TASK.rake:70:in `block in <top (required)>'

from airbrake.

kyrylo avatar kyrylo commented on June 12, 2024

Thanks for the extra info! I can see that your new backtrace includes aws-sdk-core-3.130.1/lib/seahorse/client/net_http.

I checked how they monkey-patched Net::HTTP and it looks safe. Could you bump aws-sdk-core to the latest version and see if that helps? Maybe the older version contained a bug? My gut tells me some dependencies of yours cause this issue, not the Airbrake gem, but I don't have data to prove this.

The airbrake.rb initializer only sets one config that is a list of blocklist_keys

Since you can reproduce the problem locally, could you set blocklist_keys to [] so that the library doesn't filter out anything?

We add a filter from airbrake rack (Airbrake::Rack::HttpHeadersFilter.new) with no custom behavior

Could you comment out that line and run the app without that filter?

We have a custom filter that checks for notice[:errors] comparing different types and calls notice.ignore! if we want to skip the errors.

Could you comment out that code?

It would be perfect if you could test these scenarios separately. I am not sure if this will help, but so far I don't have any good clues.

from airbrake.

Nandez89 avatar Nandez89 commented on June 12, 2024

I tried the suggested changes one at a time, commenting out each section, and then removing all configurations at once. in all cases, the local issue still happens.

I also tried updating aws-sdk-core latest 3.168.3 and that updates a few of the individual service packages, but I see the same result.

My gut tells me some dependencies of yours cause this issue, not the Airbrake gem, but I don't have data to prove this.

This is hard because in our changes the only one is updating airbrake, so it's definitely airbrake's interaction with any of the other gems, but I can't update all dependencies blindly until things work again, it's a higher risk that other things would break this way.

I checked how they monkey-patched Net::HTTP and it looks safe

Could you tell me what you look for here specifically? maybe I can search in the local gem installation for "unsafe" usages and try to identify the issue that way...

from airbrake.

kyrylo avatar kyrylo commented on June 12, 2024

Could you tell me what you look for here specifically?

I look for eval calls or "unsafe" class reopenings (reopening a class and redefining a method). That kind of stuff messes up our expectations when multiple gems monkey-patch the same thing. Here's an older reply of mine: #1080 (comment)

People should be using prepend for monkey-patching, so that everyone can hook into the existing API and not break the code of others.

Your particular case doesn't look like it's the same issue, though. Even though the second backtrace indicates it could be (because Net::HTTP doesn't behave like we expected), the error from the first backtrace is new to me. I can't remember seeing an error coming from that class. It survived without bugs for 2 years.

from airbrake.

kyrylo avatar kyrylo commented on June 12, 2024

This is completely unrelated, but we were developing the "remote config" feature during the airbrake-ruby v5.0.0 times. I would try to disable it to see if it changes anything. I doubt it, but it's worth a shot because, at the moment, I have no better options.

Airbrake.configure do |c|
  c.remote_config = false
end

from airbrake.

Nandez89 avatar Nandez89 commented on June 12, 2024

When testing these changes locally, they solve the issue. I will continue testing on my CI and other environments but we can close this for now since the fix is not on the airbrake side. Thanks @kyrylo for guiding me in investigating this and helping us find the underlying issue we also needed to fix on our side.

The problem seems to be that the Net::HTTP.class_eval in our custom middleware had an alias_method definition for the class' request method and it was calling it under the parent class request block causing the recursive loop call and triggering the SystemStackError:stack level too deep error.

# This was the problematic code, do not use it... 
::Net::HTTP.class_eval do
  alias_method :some_alias_method, :request
  # ...
  def request(req, *args, &block)
    # some custom logic
    # ....
    some_alias_method(req, *args, &block)
  end
end

It's still unclear to me why this is only an issue with the airbrake upgrade and goes unnoticed otherwise, I'm guessing airbrake 10+ added a prepend of their own, and that interfered with the previous code we had. We fixed it on our side since prepend is the recommended way to do these patches.

from airbrake.

kyrylo avatar kyrylo commented on June 12, 2024

Thanks Daniel! Just ping me if you still experience any issues.

from airbrake.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.