Sidekiq Web: Unhandled can't convert String into an exact number (TypeError) about sidekiq HOT 15 OPEN

Your job payload has enqueued_at as a String rather than a Float as Sidekiq expects it. How did you created the job?

from sidekiq.

@mperham here's how

SlackPostMessageWorker.perform_async({ channel: '#server-events', blocks: }.deep_stringify_keys)

Sidekiq config

# Web Dashboard
require 'sidekiq/pro/web'
require 'sidekiq-scheduler'
require 'sidekiq-scheduler/web'

Sidekiq.configure_client do |config|
  config.redis = { url: Settings.redis.url, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
  config.logger.level = Rails.env.development? ? Logger::INFO : Logger::WARN

  # Sidekiq Pro options
  Sidekiq::Client.reliable_push! unless Rails.env.test?

  # Sidekiq Unique Jobs options
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
end

Sidekiq.configure_server do |config|
  config.redis = { url: Settings.redis.url, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
  config.logger.level = Rails.env.development? ? Logger::INFO : Logger::WARN

  # Default is 5
  config.average_scheduled_poll_interval = 2

  config.on(:startup) do
    Sidekiq.schedule = YAML.safe_load(ERB.new(Rails.root.join('config/schedule.yml').read).result)
    SidekiqScheduler::Scheduler.instance.reload_schedule!
  end

  # Sidekiq Pro options
  Sidekiq::Client.reliable_push! unless Rails.env.test?
  config.super_fetch!
  config.reliable_scheduler!

  # Sidekiq Unique Jobs options
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end

  config.server_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  SidekiqUniqueJobs::Server.configure(config)
  SidekiqUniqueJobs.config.lock_info = true
end

if Rails.env.staging? || Rails.env.production?
  Sidekiq::Web.use Rack::Auth::Basic do |username, password|
    username == Settings.sidekiq.admin.username && password == Settings.sidekiq.admin.password
  end
end

from sidekiq.

I think you'll need to show us the exact job payload. Can you get it out of Redis with redis-cli?

from sidekiq.

@mperham these are the arguments "gid://tradespace/SomeModel/725", "some_field_name" and its a basic Sidekiq worker (Sidekiq::Job).

how can I get the "exact payload"?

from sidekiq.

If the job is enqueued in the default queue, you'd use redis-cli like this:

redis-cli lindex queue:default 0

If the job is a retry, you'd need to use a more complex ZRANGE command.

from sidekiq.

@mperham we are seeing these weird errors with ActionMailbox::RoutingJob as well. (see screenshots below)

This happens during retries - and after a retry we loose the original error context, and start seeing re-queueing errors instead. Happening for ActiveJobs, and jobs based on Sidekiq::Job.

So I ran redis-cli -h <<MY_REDIS_SERVER>> -p <<REDIS_PORT>> -a <<MY_REDIS_PASSWORD>> lindex queue:default 0

and got the following response...

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(nil)

from sidekiq.

@mperham this is happening a lot for us. Anything we can do on our side to fix this issue?

Whenever a job fails, it lands in Retries and then you click on a link in Next Retry column, we see an error. (stack trace - see above).

from sidekiq.

The issue is that you haven't shown me how to reproduce the problem. I can create both native Sidekiq::Jobs and ActiveJobs which crash and render fine when I go to their retry page. I can't really understand how best to fix the problem if I don't understand how the unexpected data format is getting into Redis in the first place.

from sidekiq.

@mperham what kind of data can I provide to help you replicate this?

from sidekiq.

I've already asked for the job payload. Your initializer would also be helpful.

from sidekiq.

@mperham already shared my Sidekiq config/initializer (see above).

Is it possible to share the ZRANGE command? I am not well versed with Redis or Sidekiq internals.

from sidekiq.

If you can give me the exact URL of a retry which crashes, I can give you the ZRANGE command to pull out that data.

from sidekiq.

@mperham here - */sidekiq/retries/1717471278.092402-0aa5dcc8678e3bc4bec102b0

from sidekiq.

redis-cli "zrange" "retry" "1717471278.092402" "1717471278.092402" "BYSCORE" "withscores"

Obviously you'll need to add the parameters to connect to your production instance.

from sidekiq.

If you run redis-cli monitor in a local Terminal, you can see the redis commands that Sidekiq runs as you use it on your machine; you can click around the Web UI and see the corresponding commands.

from sidekiq.