This gem/code https://github.com/oldmoe/neverblock uses a similar approach but completely modifies the Ruby core classes to be non-blocking. It looks pretty old, but it might be interesting to take a closer look and see how it works.

I've been thinking it would be nice to have some kind of per-fiber annotation - e.g. what it's currently doing (e.g. :sleeping, :wait_readable, etc), like a process title. it would help with debugging. I'm just not sure if it's a good idea from the POV of overhead. Perhaps having a mixin which provides debugging functionality which is only loaded if $DEBUG is set..?

I've been surprised when using Task.current? and, instead of receiving an object instead of a boolean. I'm probably not the only one used to some conventions when providing methods with and without ?, so I propose:

• Renaming Task.current to Task.current!, since it could return the Task object or raise a RuntimeError
• Renaming Task.current? to Task.current, since it could return the Task object or nil
• Removing Task.current?, since it doesn't return a boolean.

The initial design of async, didn't track nested tasks. This means that when you stop the parent task, async blocks started in that task wouldn't also be stopped, they'd keep on running. I found in async-dns this wasn't really great - it made it cumbersome to keep track of execution state and stop things.

Ideally, when stopping a task, you kill that task, and any child async blocks. The implications of this, is that if you have a TCP server, and it has spawned a number of children tasks, when you kill the server you also kill all the children. I think this is the right design, but in theory you could also detach a child task from it's parent so it would continue on unimpeded. However, the machinery for stopping tasks needs to be robust in the face of complex user code, without the user having to track a lot of state.

We're looking to use https://github.com/guard/rb-inotify as a wrapped class that will monitor a path on the filesystem and also invoke other work. We're seeing the monitor get blocked on startup of the Reactor.

Here's a simplified example

Async::Reactor.run do |task|
  task.async do |subtask|
    SyslogMonitor.run
  end

  task.async do |subtask|
    while true
      periodic_work
      subtask.sleep 300
    end
  end
end

class SyslogMonitor
  def self.run
    Async do |task|
      File.open(LOG_PATH, "r") do |f|
        f.seek(0, IO::SEEK_END)

        queue = INotify::Notifier.new

        queue.watch(LOG_PATH, :modify) do
          change = f.read

          Async do |task|
            task.sleep 2
            process_change
          end
        end

        queue.run
      end
    end
  end
end

Would we have to use rb-inotify differently for it to be compatible with the reactor?

BTW - the whole family of async gems have been incredibly helpful. Thanks so much for your time and effort on them!

In some RSpec tests, I expect an async task to timeout and raise Async::TimeoutError. It works, but the task exception prints to the terminal, which I would rather avoid in this case, since it interferes with the RSpec output

code/rsmp % be rspec spec/archive_spec.rb
..... 0.81s    error: Async::Task [oid=0x1518] [pid=52153] [2020-04-05 22:43:49 +0200]
               |   Async::TimeoutError: execution expired
               |   → /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/task.rb:64 in `yield'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/condition.rb:38 in `wait'
               |     /Users/emiltin/Documents/code/rsmp/lib/rsmp/probe.rb:51 in `block in capture'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/reactor.rb:266 in `with_timeout'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/2.7.0/forwardable.rb:235 in `with_timeout'
               |     /Users/emiltin/Documents/code/rsmp/lib/rsmp/probe.rb:50 in `capture'
               |     /Users/emiltin/Documents/code/rsmp/lib/rsmp/archive.rb:50 in `capture'
               |     /Users/emiltin/Documents/code/rsmp/spec/archive_spec.rb:122 in `block (4 levels) in <top (required)>'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/task.rb:255 in `block in make_fiber'
.

Finished in 0.18939 seconds (files took 0.7485 seconds to load)
6 examples, 0 failures

Is there a way to suppress the error output?

If Async::Reactor.run(&block) happens within an existing reactor, it will schedule an asynchronous task and return. If Async::Reactor.run(&block) happens outside of an existing reactor, it will create a reactor, schedule the asynchronous task, and block until it completes. The task is scheduled by calling Async::Reactor.async(&block)

last sentence:

The task is scheduled by calling Async::Reactor.async(&block)

I'm not sure what this means as there is no .async

Want a logo? Perhaps we could adapt the one from https://github.com/socketry/socketry-async somehow

I have a server class based on async-io. This server class is used to test external clients, using RSpec.

Because the site connects to the sever (not the other way around) and this includes a delay, I want to keep the server running across tests. But unless I stop the reactor at the end of each test, it blocks and no more tests are run. The way I've tried to solve this is to start and stop the reactor for each test, essentially:

reactor.run do |task|
  # run the test
  reactor.stop
end

But it seems reactor.stop is rather slow. Is there a better way to achieve what I want?

There are occasions where any one of several Task activities might produce a result and only the first result matters.

Ideally there's a method like #wait_first which can return the result of the first task to finish.

In some cases it may be necessary to select the first non-nil result, but that could be handled with an Async::Condition instead.

The keyword argument causes the hash to be interpreted incorrectly by Ruby:

require 'async'
require 'async/notification'

Async do
  notification = Async::Notification.new

  hash = {
    async: true
  }

  notification.signal(hash)
  # => ArgumentError: unknown keyword: :async
end

It may be better to go all keyword args, like signal(value:, task:) to avoid ambiguity here, though that would be a breaking change.

How would you use async to wait for a condition, with a timeout, like ConditionVariable?
E.g. make a task sleep until a counter reach a certain value, or the until the timeout is reached?

It would be interesting to explore something which can check the sanity of operations, e.g.

expect do
	user_code
end.to_not modify_globals(except: @cache)

So, with the branch I just made, I'm attempting to add some inline documentation following the YARD style -- because I think that it's awesome like a possum.

Some Lil Bumps In Da Road

I don't know, like, everything that is going on under the hood for this gem obviously. In general, this is some of my first exposure to IO related concepts. Plus, I get terribly distracted with other things I end up working on ( like a lot of us, I'm sure ).

So, wherever I obviously am falling short on explanations, I'd appreciate you cool cats ( or dogs ) in helping contribute to the documentation effort.

plz

So, please be sure to go over ma' mistakes and help correct them.

Why YardDoc Ahoy?

Because pirates, of course.

It would be nice if something like Reactor.run was composable and provided some basic synchronisation when nested. This would be useful for having async APIs nest their own reactors, but use the parent's reactor if one was available.

Following on from oracle/truffleruby#1436 ...

We've migrated timers to us Process.clock_gettime. There might be other issues too.

You can run a large suite of tests by checking out the async source code and running:

rake external.

It will run specs for the following code bases:

cc @jjyr @eregon

Hi, while trying to run examples getting this results

root@a42ebebf32a8:/app/async/examples# ruby ./aio.rb 
Creating server
/usr/local/bundle/gems/async-0.13.0/lib/async/reactor.rb:87:in `wrap': undefined method `[]' for IO:Class (NoMethodError)
	from /usr/local/bundle/gems/async-0.13.0/lib/async/task.rb:69:in `block in initialize'
	from /usr/local/bundle/gems/async-0.13.0/lib/async/task.rb:69:in `collect'
	from /usr/local/bundle/gems/async-0.13.0/lib/async/task.rb:69:in `initialize'
	from /usr/local/bundle/gems/async-0.13.0/lib/async/reactor.rb:98:in `new'
	from /usr/local/bundle/gems/async-0.13.0/lib/async/reactor.rb:98:in `async'
	from ./aio.rb:17:in `<main>'

root@a42ebebf32a8:/app/async/examples# ruby ./echo.rb 
D, [2017-05-01T20:05:41.933171 #263] DEBUG -- : [#<Async::Reactor:0x00558802e0bca0> Ensure] Exiting run-loop (stopped: false exception: undefined method `connect' for Async::TCPServer:Class)...
D, [2017-05-01T20:05:41.933868 #263] DEBUG -- : [["#<Async::Task:0x00558802e0b930>[failed]", false]]
./echo.rb:25:in `block in echo_client': undefined method `connect' for Async::TCPServer:Class (NoMethodError)
	from /usr/local/bundle/gems/async-0.13.0/lib/async/task.rb:83:in `block in initialize'

root@a42ebebf32a8:/app/async/examples# ruby --version
ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux]

I note that there are two CI configured, and that perhaps Travis has more elements to its matrix. GitHub Actions is faster.

• Is there overlap?
• Can some elements be dropped from Travis?

• will use https://github.com/ruby/setup-ruby

how to replicate:

Simple WS server:

require 'em-websocket'
channel = EM::Channel.new
Thread.new do
  EM.run do
    EM::WebSocket.run(host: '0.0.0.0', port: 8080) do |ws|
      ws.onopen do |_handshake|
        client_connected = true
        channel.subscribe { |msg| ws.send msg }
        puts 'WebSocket connection open'
      end
      ws.onclose { puts 'Connection closed' }
      ws.onmessage do |msg|
        puts "Recieved message: #{msg}"
      end
    end
  end
end

Simple ws-client:

require 'async/io/stream'
require 'async/http/endpoint'
require 'async/websocket/client'

@reactor = Async::Reactor.new
url = 'ws://127.0.0.1:8080'
endpoint = Async::HTTP::Endpoint.parse(url, alpn_protocols: Async::HTTP::Protocol::HTTP11.names)

Thread.new do
  @reactor.run do |task|
    Async::WebSocket::Client.connect(endpoint) do |connection|
      puts 'Connected.'
      input_task = task.async do
        connection.write({ test: 'ok' }.to_json)
        connection.flush
      end

      while (message = connection.read)
        $logger.router.info message
      end
    ensure
      puts 'Stopping task...'
      input_task&.stop
      connection.close
      @reactor.interrupt
    end
  ensure
    puts "Stopping reactor"
  end
end

According to documentation: "Async::Reactor#interrupt can be called safely from a different thread (or signal handler) and will cause the reactor to invoke #stop." but @reactor.interrupt has no effect.

This is not something I'd personally want to do, but I'm not against it completely either.

Let's have a discussion here and see where it ends up.

Fiber.yield is used by Enumerator, so if enumerator will call to some io, it will return to calling fiber instead of being scheduled.
But if scheduler will use fiber.transfer, than it will play nicely with Enumerator.

During converting async gem to deb package, I've found that papers/*.pdf license is unclear.
Is it under OSS license?

If not, remove them is fine for me since it's kind of blocker for uploading Debian repository.

The documentation string for Barrier is copied exactly from Semaphore. Did that happen by mistake?

Uncaught exceptions in tasks is shown in a format starting with a line like this:

  0.0s    error: Async::Task [oid=0x3c] [pid=9016] [2020-12-16 21:01:17 +0100]

It would be useful if the task annotation was shown, to quickly identify the failed task.

Need to review, spec and test this case. PRs welcome.

I am looking for the equivalent of every in Celluloid. It was suggested to loop and task.sleep, however it doesn't behave the same way.

  ::Async::Reactor.run do |task|
    3.times do |i|
      puts "1: #{i}"
      task.sleep 3
    end
  end

  ::Async::Reactor.run do |task|
    3.times do |i|
      puts "2: #{i}"
      task.sleep 3
    end
  end

1: 0
1: 1
1: 2
2: 0
2: 1
2: 2

that said, it works if both reactor runs are inside of a reactor

::Async::Reactor.run do
  ::Async::Reactor.run do |task|
    3.times do |i|
      puts "1: #{i}"
      task.sleep 3
    end
  end

  ::Async::Reactor.run do |task|
    3.times do |i|
      puts "2: #{i}"
      task.sleep 3
    end
  end
end

1: 0
2: 0
1: 1
2: 1
1: 2
2: 2

Is the latter the correct way?

Maybe implementing a higher level timers API could be valuable? It would ensure to start a "parent" reactor, always behave like the second version, and have predictable (eg. not bail out on error) error behavior?

I also see

require 'async'

i = 0
j = 0

reactor = Async::Reactor.new

reactor.run do |task|
  reactor.every 3 do
    puts "1: #{i}"
    i += 1
  end

  reactor.every 3 do
    puts "2: #{j}"
    j += 1
  end
end

Works as expected. Does nothing without reactor.run. Maybe it just needs documentation and/or being wrapped and error out if not running under a reactor?

I've been playing around with chaining together constructs within async.

For example:

barrier = Async::Barrier.new
semaphore = Async::Semaphore.new(10, parent: barrier)
queue = Async::Queue.new

# This queue will process up to 10 items at once:
queue.async(parent: semaphore) do |item|
	# Do something with item...
end

# Wait for all outstanding tasks.
barrier.wait

I haven't released this yet, but the code is in master. Welcome feedback.

I'd like to use a custom logger throughout the async library. Currently doing this:

Async.instance_variable_set(:@logger, @logger)

Can/Should this be supported in a better way?

Probably triggered by incorrect use of async gem somehow, but should async throw a runtime error?

 0.14s    error: Async::Task [oid=0x1180] [pid=39201] [2020-12-09 09:34:28 +0100]
               |   RuntimeError: can't add a new key into hash during iteration
               |   → /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/2.7.0/set.rb:339 in `add'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/node.rb:99 in `add_child'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/node.rb:91 in `parent='
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/node.rb:36 in `initialize'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/task.rb:74 in `initialize'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/task.rb:125 in `new'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/task.rb:125 in `async'
               |     /Users/emiltin/Documents/code/rsmp/lib/rsmp/site_proxy.rb:361 in `wait_for_command_responses'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/command_helpers.rb:5 in `block in send_command_and_confirm'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/log_helpers.rb:5 in `log_confirmation'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/command_helpers.rb:3 in `send_command_and_confirm'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/command_helpers.rb:88 in `set_functional_position'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/command_helpers.rb:306 in `switch_normal_control'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/site/command_spec.rb:13 in `block (3 levels) in <top (required)>'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/test_site.rb:80 in `block in connected'
               |     /Users/emiltin/Documents/code/rsmp_validator/spec/support/test_site.rb:24 in `block in within_reactor'
               |     /Users/emiltin/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/async-1.23.0/lib/async/task.rb:255 in `block in make_fiber'

I'm trying to figure out why this is failing:

38.87s: <Async::Task:0x6df8 failed> [pid=8615]
      |   NameError: no method to_str
      |   → /home/travis/.rvm/rubies/truffleruby-1.0.0-rc13/lib/truffle/truffle/cext.rb:941 in `special_form'
      |     /home/travis/.rvm/rubies/truffleruby-1.0.0-rc13/lib/truffle/truffle/cext.rb:941 in `rb_funcall'
      |     /home/travis/build/socketry/async/vendor/bundle/truffleruby/2.4.0/gems/nio4r-2.3.1/ext/nio4r/selector.c:282 in `NIO_Selector_synchronize'
      |     /home/travis/build/socketry/async/vendor/bundle/truffleruby/2.4.0/gems/nio4r-2.3.1/ext/nio4r/selector.c:310 in `NIO_Selector_register'

Here is the C code:

https://github.com/socketry/nio4r/blob/master/ext/nio4r/selector.c#L273-L291

Here is TruffleRuby code:

https://github.com/oracle/truffleruby/blob/master/lib/truffle/truffle/cext.rb#L937-L944

cc @eregon

Hi,
Is it possible to have async tasks run in the background in irb? For example, a server that handles connections in the background, and which you can interact with:

server = Server.new
=> #<Server...>
server.start
=> "Server running"
server.connections
=> 0
server.connections # a bit later
=> 6

Try this:

a = [1,2,3,4,5,6,7,8]
a.each{|i| a<<i}

I was requested to open an issue for the async io section: forgive me if this is not the correct place.

I am playing around with the gems and the async-http example on the read me page, shows the following errors:
For the async-http, when I run the async-http example.
Traceback (most recent call last):
11: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-1.10.0/lib/async/task.rb:74:in block in initialize' 10: from ./async-http.rb:20:inblock (2 levels) in
'
9: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-http-0.24.3/lib/async/http/server.rb:69:in run' 8: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-io-1.12.2/lib/async/io/endpoint.rb:46:inaccept'
7: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-http-0.24.3/lib/async/http/url_endpoint.rb:118:in bind' 6: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-io-1.12.2/lib/async/io/host_endpoint.rb:55:inbind'
5: from C:/Ruby25/lib/ruby/2.5.0/socket.rb:227:in foreach' 4: from C:/Ruby25/lib/ruby/2.5.0/socket.rb:227:ineach'
3: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-io-1.12.2/lib/async/io/host_endpoint.rb:56:in block in bind' 2: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-io-1.12.2/lib/async/io/socket.rb:166:inbind'
1: from C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-io-1.12.2/lib/async/io/socket.rb:110:in build' C:/Ruby25/lib/ruby/gems/2.5.0/gems/async-io-1.12.2/lib/async/io/socket.rb:168:inblock in bind': uninitialized constant Socket::SO_REUSEPORT (NameError)
Did you mean? Socket::SO_REUSEADDR

Requested to aske about the above line.

Under the hood, ruby often calls getaddrinfo to turn things like localhost into a struct sockaddr. This is often going to be a blocking operation, which will block the entire reactor.

I'd like to enumerate cases where blocking behaviour might be experienced (e.g. Async::TCPSocket.connect with a hostname), and figure out a solution.

Hello everyone in no particular order: @halorgium @digitalextremist @e2 @benlangfeld @tarcieri @grantr

I reimplemented here, what I wanted from celluloid-io. I bet everyone has moved on in one way or another, but I just wanted to stay thanks for everyone's effort on celluloid, and going forward I think there is an opportunity to take the best ideas from celluloid, and perhaps build on them here.

A few things on the agenda to figure out:

• I'm going to be honest, based on where things have ended up for celluloid, I don't know if it's possible to recover. I think we need to consider what strategy we adopt going forward for that org/gem - do we mark it as deprecated?
• Tabs/Spaces. I'm firmly in the tabs camp for my own code, but I'm happy to consider compromise if it's a deal breaker for other developers who would be actively willing to contribute.
• Directions going forward. I'm planning to retain full control of this project for the foreseeable future. I'm going to be a dictator to start with. The core async gem is almost feature complete IMHO, and doesn't require a lot of work, but it DOES require documentation, testing, and momentum. I'd like to keep the async-* namespace really tight to start with, but I do think we need some additional bits to make usage easier.

Projects on the table that I can think of:

• async-http - light-weight HTTP client and server.
• async-wrap (became async-io) - light-weight wrappers to support systems that allow socket_class: option.
• async-dns - working and almost done.
• async-actor - light-weight actor implementation with message passing, ala celluloid.

Is it possible to wrap up the code with tasks and make it look synchronized? Something similar to synchrony? Or await in JavaScript? Let's say my server gets a request, execute some async io , release his fiber to other consumers, when get result from async it publish a domain event , the controller need to do a non blocking wait to this domain event. Is it something this library can do ? I'm looking for something similar to "run a command , subscribe to possible events that command should fire, without blocking"

@ganmacs Looks like this is all good now: https://travis-ci.org/socketry/async-http/builds/556647690

Bearing in mind, I want to establish some policy on deprecation of EOL Ruby.

Can you give me some ideas since this seems important to you.

I tried using tcp_socket.rb as an example, but don't fully undesrtand how to add support for SSL sockets. Namely, it looks like tasks assume there is a fileno method, but OpenSSL::SSL::Server doesn't have that. The base TCPServer does though.

I've tried various combinations, but can't seem to get a fileno method that actually proxies down to whatever is supposed to have it. Calling fileno on an Async::TCPServer doesn't work, but when I try to proxy fileno from the internal IO, it seems to be trying to call it on the host parameter string.

Any thoughts on how to implement an SSLServer/SSLSocket for async?

Right now, tasks are resumed directly, but perhaps it would make sense for a fiber to capture the "return value" as it were. I'm not sure if this is possible or how much work would be required.

However, as part of this, handling "return" within an async block also needs to be checked - does the reactor blow up?

^ Title. Is it possible? Based on the code I assume not, but wanted to ask. If my analysis is correct (and it's not possible), would monkey-patching reactor to provide three new methods like (basically splitting Reactor#run into three methods)

def setup(*args, &block)
  raise RuntimeError, 'Reactor has been closed' if @selector.nil?
  
  @stopped = false
  
  # Allow the user to kick of the initial async tasks.
  @initial_task = async(*args, &block) if block_given?
end

def loop_iter
  return if @stopped

  # running used to correctly answer on `finished?`, and to reuse Array object.
  @running, @ready = @ready, @running
  if @running.any?
    @running.each do |fiber|
      fiber.resume if fiber.alive?
    end
    @running.clear

    # if there are tasks ready to execute, don't sleep.
    if @ready.any?
      interval = 0
    else
      # The above tasks may schedule, cancel or affect timers in some way. We need to compute a new wait interval for the blocking selector call below:
      interval = @timers.wait_interval
    end
  end
  
  # - nil: no timers
  # - -ve: timers expired already
  # -   0: timers ready to fire
  # - +ve: timers waiting to fire
  if interval && interval < 0
    interval = 0
  end
  
  # Async.logger.debug(self) {"Updating #{@children.count} children..."}
  # As timeouts may have been updated, and caused fibers to complete, we should check this.
  
  # If there is nothing to do, then finish:
  if !interval && self.finished?
    return @initial_task
  end
  
  # Async.logger.debug(self) {"Selecting with #{@children.count} fibers interval = #{interval.inspect}..."}
  if monitors = @selector.select(interval)
    monitors.each do |monitor|
      monitor.value.resume
    end
  end
end

def teardown
  return @initial_task
ensure
  Async.logger.debug(self) {"Exiting run-loop because #{$! ? $!.inspect : 'finished'}."}
  @stopped = true
end

do mostly what I want? Calls would look something like (taken from README.md):

begin
  reactor = Async::Reactor.new
  reactor.setup do
    socket = connect(remote_address) # May raise Async::Stop
    begin
      socket.write(...) # May raise Async::Stop
      socket.read(...) # May raise Async::Stop
    ensure
      socket.close
    end
  end
  loop do
    reactor.loop_iter
    # Do some of my own work
    break unless should_stop?
  end
ensure
  reactor.teardown
end

Would this work? I guess my main question is, will the #loop_iter here be noop if nothing is ready?

I've been thinking about how we can improve this API.

I like the idea of something like TCPServer.accept_each do |peer| ....

Wondering what your though might be @tarcieri

I think it's more canonical ruby than the pythonic task.with(server.accept) do |peer| ... end while true

What could cause these double resume errors when calling stop() on a task?
It seems to be related to having a subtask that's waiting for a Async::Condition and using yield, but I haven't been able to isolate an example. I'm on ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18].

 0.07s    error: <Async::Task:0x3ffc3c89b0e8 failed> [pid=61267] [2019-08-15 15:34:44 +0200]
               |   FiberError: double resume
               |   → /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:149 in `resume'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:149 in `stop'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/2.6.0/set.rb:338 in `each_key'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/2.6.0/set.rb:338 in `each'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:153 in `ensure in stop'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:153 in `stop'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/2.6.0/set.rb:338 in `each_key'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/2.6.0/set.rb:338 in `each'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:153 in `stop'
               |     /Users/emiltin/Documents/code/rsmp/node.rb:45 in `stop'
               |     /Users/emiltin/Documents/code/rsmp/supervisor.rb:81 in `stop'
               |     /Users/emiltin/Documents/code/rsmp/spec/command.rb:22 in `block in up'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:228 in `block in make_fiber'
               |   Caused by FiberError: double resume
               |   → /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:149 in `resume'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:149 in `stop'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/2.6.0/set.rb:338 in `each_key'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/2.6.0/set.rb:338 in `each'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:153 in `stop'
               |     /Users/emiltin/Documents/code/rsmp/node.rb:45 in `stop'
               |     /Users/emiltin/Documents/code/rsmp/supervisor.rb:81 in `stop'
               |     /Users/emiltin/Documents/code/rsmp/spec/command.rb:22 in `block in up'
               |     /Users/emiltin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/async-1.20.1/lib/async/task.rb:228 in `block in make_fiber'

Hello - We are trying to use this gem but are coming up against a issue. Please see code below:

c = Cloudflare.connect(key: cf_key, email: cf_email)

c.zones
=> #<Cloudflare::Zones #<Async::REST::Resource #<Protocol::HTTP::Reference:0x00005583b26b9168 @path="/client/v4/zones", @query_string=nil, @fragment=nil, @parameters=nil> #<Protocol::HTTP::Headers [["X-Auth-Key", "OUR AUTH KEY"], ["X-Auth-Email", "OUR AUTH EMAIL"]]>>: value=nil>

c.zones.first
Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):10:in `first'
ArgumentError (unknown keyword: transient)

We manage to access a specific zone directly using:

z = c.zones.find_by_id(ZONE_ID)
#<Cloudflare::Zone #<Async::REST::Resource #<Protocol::HTTP::Reference:0x0000557076d6d5d0 @path="/client/v4/zones/c3475d024182ec43da34191c1283159a", @query_string=nil, @fragment=nil, @parameters=nil> #<Protocol::HTTP::Headers [["X-Auth-Key", "2e35c2015790e34a172b566b504afd49fa9b6"], ["X-Auth-Email", "[email protected]"]]>>: value=nil>

But we hit the same error when we request the DNS Records for a zone.

z.dns_records
=> #<Cloudflare::DNS::Records #<Async::REST::Resource #<Protocol::HTTP::Reference:0x0000557076d4ae68 @path="/client/v4/zones/c3475d024182ec43da34191c1283159a/dns_records", @query_string=nil, @fragment=nil, @parameters=nil> #<Protocol::HTTP::Headers [["X-Auth-Key", "2e35c2015790e34a172b566b504afd49fa9b6"], ["X-Auth-Email", "[email protected]"]]>>: value=nil>
irb(main):010:0> z.dns_records.first
Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):10:in `first'
ArgumentError (unknown keyword: transient)

I sometimes get "deadlock; recursive locking (ThreadError)" on ctrl-c in the terminal. I'm on MacOS 10.14.6.

Emils-iMac:rsmp emiltin$ ruby ignore/test.rb 
^CTraceback (most recent call last):
	6: from ignore/test.rb:3:in `<main>'
	5: from /Users/emiltin/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/async-1.20.1/lib/async.rb:34:in `Async'
	4: from /Users/emiltin/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/async-1.20.1/lib/async/reactor.rb:56:in `run'
	3: from /Users/emiltin/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/async-1.20.1/lib/async/reactor.rb:56:in `ensure in run'
	2: from /Users/emiltin/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/async-1.20.1/lib/async/reactor.rb:226:in `close'
	1: from /Users/emiltin/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/async-1.20.1/lib/async/reactor.rb:226:in `close'
/Users/emiltin/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/async-1.20.1/lib/async/reactor.rb:226:in `lock': deadlock; recursive locking (ThreadError)

Emils-iMac:rsmp emiltin$ cat ignore/test.rb
require 'async'

Async do |task|
  task.async do |subtask|
    subtask.sleep 100
  end
end

Emils-iMac:rsmp emiltin$ gem list async

*** LOCAL GEMS ***

async (1.20.1)
async-await (0.3.0)
async-io (1.24.0)
async-rspec (1.13.0)

Emils-iMac:rsmp emiltin$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin17]

@soutaro What is the process to create/add RBS to this project? I would love to add it and expose users downstream to it, and maybe catch bugs with it too!

I am trying to catch some exceptions that might occur inside a task, this is related to this issue on async-redis. What I was trying to do essentially boils down to this

Async.run do |main_task|
	begin
		Async.run do
			w = Async::Wrapper.new(File.open mytemp.path, 'r')

			w.wait_readable
			puts w.io.gets

			w.close

			raise ProtocolError
		end.wait
		# without #wait ProtocolError doesn't get caught at all
	rescue ProtocolError
		# This catches the exception, but it gets raised again. Somehow...
		puts "Rescuing from ProtocolError"
	end
end

Note ProtocolError is just a subclass of StandardError. I expected the output of this to be,

the first line from the file
Rescuing from ProtocolError

However I got,

the first line from the file
Rescuing from ProtocolError
Traceback (most recent call last):
        1: from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/task.rb:74:in `block in initialize'
./test.rb:24:in `block (2 levels) in <main>': ProtocolError (ProtocolError)

The exception was caught and handled, yet it still makes the program exit with an error? This happens no matter where I raise ProtocolError inside the child task after the file is opened and wrapped.

This gets weirder when I start more sub-tasks with io in the same reactor. In light of what I described so far I would expect it to crash before finishing the first one, yet that's not what happens.

Async.run do |main_task|
	3.times do |n|
		begin
			puts "Read #{n}"
			Async.run do
				puts "Task #{n}"
				w = Async::Wrapper.new(File.open mytemp.path, 'r')

				w.wait_readable
				puts w.io.gets

				w.close

				raise ProtocolError
			end.wait
			# without #wait ProtocolError doesn't get caught at all
		rescue ProtocolError
			# This catches the exception, but it gets raised again. Somehow...
			puts "Rescuing from ProtocolError"
		end
	end
end

The result is,

Read 0
Task 0
first line of the file
Rescuing from ProtocolError
Read 1
Task 1
Read 2
/usr/lib/ruby/2.5.0/set.rb:349:in `add': can't add a new key into hash during iteration (RuntimeError)
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/node.rb:87:in `parent='
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/node.rb:36:in `initialize'
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/task.rb:61:in `initialize'
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/reactor.rb:95:in `new'
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/reactor.rb:95:in `async'
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/reactor.rb:49:in `run'
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async.rb:28:in `run'
	from ./test.rb:18:in `block (2 levels) in <main>'
	from ./test.rb:15:in `times'
	from ./test.rb:15:in `block in <main>'
	from /home/huba/.gem/ruby/2.5.0/gems/async-1.12.0/lib/async/task.rb:74:in `block in initialize'

Again, this does not happen if the io is removed from the inner task. My next step I suppose is to enable debug output and/or step through with a debugger.

Hi,

@ioquatix as we talked on the call the other day, here's my list of questions about async that I wish I could have read somewhere so I can get up to speed faster. I'll add question for any async-* gem, not just core async.

Anyone can add their own questions. These will be used when building the docs for async gems.

Note: I don't expect answers to these questions. I just think it would be useful to document this stuff.

Related: here's another ruby async project that has decent webpage and docs https://digital-fabric.github.io/polyphony/

async gem

• What is a "task"? How is it different from fiber? Why does async manipulate tasks and not raw fibers?

• What are transient tasks?

• What really are fibers?

  • The answer "a cooperative scheduling concurrency mechanism" is really not helpful.
  • This talk from C++ community really helped me better understand this concept https://www.youtube.com/watch?v=_fu0gx-xseY. Maybe we can draw notes from that talk when explaining fibers?

• What is a selector?

• What is a reactor? How is it different from selector?

• When using async should I make any method be asynchronous? Example, should I wrap any method in async block like thid: Async::Task.current.async { print_to_logger }. Also related to async-await gem: should I define all the methods with async def my_method.

• The readme shows using barrier and semaphore together. Do I really always need to use these two together?

• Question related to semaphore and barrier use:

barrier = Async::Barrier.new
semaphore = Async::Semaphore.new(2)

# Example 1
semaphore.async(parent: barrier) do
  # ...
end

# Example 2
barrier.async(parent: semaphore) do
  # ...
end

Which of the two examples above is right? If both are right, when do I use one and when do I use the other?

• (maybe bug?) I would like more info on how #every, #after and #sleep methods work. Here are a couple examples I found to be contradictory:

require 'async'

Async do |task| # reactor exits immediately, not what I expected
  task.reactor.after(1) do
    puts "Example 1" # never runs, not sure why
  end
end

Async do |task|
  task.reactor.sleep(1)
  puts "Example 2" # this one runs, all good
end

Async do |task|
  task.async do |subtask|
    subtask.sleep 1
    puts "Example 3" # this one runs, all good
  end
end

Async do |task| # reactor exits immediately, not what I expected
  task.reactor.every(1) do
    puts "Example 4" # never runs, I expect this to run forever
  end
end

• (maybe an enhancement) Can we have Async::Task#every and Async::Task#after method?

async-io

• What's the difference between a io, socket, endpoint and a stream? They're all similar concepts, but they're used differently.

• Why don't you just monkey patch core ruby classes? Why did you go with wrappers? What are the monkey patching pros and cons?

async-http

• Why is Async::HTTP::Internet the main class for making requests? How does it differ from Async::HTTP::Client? Other http libraries use "client" for making requests, why doesn't async do that?

Questions related to all async gems

• Can I use async with rails?

• I'm at the point where I know how to use async suite of gems productively. But I'm curious about the magic behind making async http requests. How can you trigger a http request and continue running other ruby code? Isn't ruby supposed to block after a http request is made?
  Answer suggestion: it took me a lot of time to figure this one out. Reading these methods was helpful in understanding how the magic works:

  1. https://github.com/socketry/async-io/blob/6835c9853f370c7e8a190a1baea4990c3aef7043/lib/async/io/generic.rb#L57-L72 (also see how and where wrap_blocking_method is used)
  2. https://github.com/socketry/async-io/blob/cfbf3891379240871cd91db2be442948b5a05c6a/lib/async/io/generic.rb#L214-L227

  3. async/lib/async/wrapper.rb

     Lines 117 to 129 in 17ef44f

     	def wait_readable(timeout = nil)
     	raise WaitError if @readable
     	self.reactor = Task.current.reactor
     	begin
     	@readable = Fiber.current
     	wait_for(timeout)
     	ensure
     	@readable = nil
     	@monitor.interests = interests if @monitor
     	end
     	end

  4. async/lib/async/wrapper.rb

     Lines 132 to 144 in 17ef44f

     	def wait_writable(timeout = nil)
     	raise WaitError if @writable
     	self.reactor = Task.current.reactor
     	begin
     	@writable = Fiber.current
     	wait_for(timeout)
     	ensure
     	@writable = nil
     	@monitor.interests = interests if @monitor
     	end
     	end

@tarcieri has raised the good point that Fibers might be expensive. At the very least, we should investigate the memory overhead and performance cost of using Fibers vs another approach.

If it's hard to do a comparison, perhaps we could simply provide benchmarks - number of connections per second/memory usage.