ezmobius / nanite Goto Github PK

See http://github.com/flori/json/issues/issue/31

Why is json/add/rails required?

Two questions

1. When a nanite agent timeouts - Is there a way to detect it with code? Also sometimes the agents come back alive after a long running task. We want to strike a balance between number of agents vs timeouts. Any suggestions/ideas?
2. Also is there an inbuilt way to gracefully stop agents? Something like nanite-agent --token "test" stop --force-after 60 seconds

I did read an email from mattmatt a while ago. Thoughts?

I have been struggling for several days with a problem in my agents. Randomly, they will stall and use 100% of the CPU. strace reveals the agents are just context switching and doing nothing:

--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 40001616
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 0
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 0
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 1
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 1
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 0
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 101
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0) = 0

I have tried everything: modified agents to use epoll rather than select, tried ruby enterprise edition and ruby1.9 (they remove the syscalls in strace, but agents still lock). I cannot discern a pattern or reason the agents lock specifically, meaning the job they lock on isn't consistent ASIDE from happening during a job that utilizes net/http to pull down some images and stitch them together.

I thought it might be an issue with calling sleep() inside the agents, but that didn't solve anything. I really have no idea where to go from here.

Pastie to my agent code: http://pastie.org/702881
Pastie to image fetch/stitch code: http://pastie.org/702895

On the plus side, I'll be able to give you a quick modification to nanite that causes it to use epoll, which dropped my CPU utilization a hair while performing a large amount of jobs! Any ideas on where to start even looking from here would be appreciated, otherwise I am going to just start commenting out code until something changes (the worst way to debug!).

Hello,

I submitted a few patches to make nanite examples work out-of-the-box using amqp 0.8.x gem.
The pull request is found here : olegshnitko#1

The main difficulty was that using amq.queue() will raise loudly if the parameters passed do not match the ones provided when the queue was first declared.
Since the fork is a bit far from the original project, I post an issue here.

Sincerely,
Julien

System configurations:
ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0]
amqp (0.6.0)
eventmachine (0.12.10)
rabbitmq 1.7.0
erlang 5.6.5

I follow the steps given by the "Test Nanite (finally)",

First, I created a new agent called bob, http://gist.github.com/223056

Second, I created a new mapper in another shell, http://gist.github.com/223058

I patched nanite to make it work with RabbitMQ 1.8.
The 1.8 version doesn't allow the declaration of an exchange with different parameters anymore.

See "Exchange equivalence" at:
http://lists.rabbitmq.com/pipermail/rabbitmq-announce/2010-June/000025.html

Please review my commit at:
bernd/nanite@747359b

I'm not sure if my changes will do something bad when running with an older RabbitMQ version. Especially the change in lib/nanite/mapper.rb. At least the simpleagent example runs again and doesn't throw stuff like the following anymore.

NOT_ALLOWED - cannot redeclare exchange 'registration' in vhost '/nanite' with different
type, durable or autodelete value in AMQP::Protocol::Exchange::Declare

Thanks,
Bernd

I met some promblem when using nanite. I think it may caused by gem environment.
Can you tell me the gem version information which you guys running.

One problem I met is.
After I start up agent and mapper using the simple-agent example which is in the source code, the agent side log stopped at :
[Fri, 06 Nov 2009 17:07:01 +0800] INFO: SEND [register] d72993a0de1aed09f42dd291e5046d3d, services: /simple/echo, /simple/time, /simple/gems, /simple/yielding, /simple/delayed, tags:

And the mapper side the log stopped at:
[Fri, 06 Nov 2009 17:07:11 +0800] INFO: [setup] starting mapper

after a while the mapper down, cause this error
/opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:811:in connect_server': no connection (RuntimeError) from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:811:inreconnect'
from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:172:in reconnect' from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:85:incall'
from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:85:in unbind' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:incall'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in run_deferred_callbacks' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:intimes'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in run_deferred_callbacks' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:inrun_machine'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:in `run'
from simpleagent/cli.rb:21

My environment is EventMachine 0.12.8 and amqp 0.6.0, rabbitmq 1.7.1

I just think whether it is rabbitmq permission,I try to re-install rabbitmq and run the "rabbitconf.rb" , then I the problem still.
here is the log when I run "rabbitconf.rb"

Setting permissions for user "mapper" in vhost "/nanite" ...
...done. Setting permissions for user "nanite" in vhost "/nanite" ...
...done. Listing users ...
guest
mapper
nanite
...done. Listing vhosts ...
/ /nanite ...done. Listing permissions in vhost "/nanite" ...
mapper . . .
nanite . . .
...done.

Supposing cli-mapper that sends a push to a client-agent:

push('/client_agent/foo', "hi")

And this client-agent sends a push (or request) to a main-agent:

push('/main_agent/foo', ["bar"] )

Main agent receives 2 requests instead of one. If I have for instance, 8 thins running, main agent receives 10 exaclty equal requests.
Looks like the requests are getting multiplicated by each mapper online on that time.

The prefetch option that is a savior when you have high volumes of intense tasks is only available on mappers. The mapper doesn't really benefit from having a prefetch limit, as the mapper receives mostly pings and registrations, which can be processed really fast. It is the agent that is going to predominantly be affected, and should also have the option.

I went ahead and forked nanite, made the changes and am using it in production. This finally fixed my issue with overloading my agents to the point they were unresponsive and had to be killed. The fork is available on my github page if you are interested in bringing the change upstream.

In the fork, I simply added a configuration option for the agent, including adding the command line options for the nanite-agent script. I also documented the new config option in the init method, so aside from some review, it should be ready to go.

I have started the mapper in config/init.rb but Nanite.request() simply returns false without any error message. The agent does not receive the request, either.

But it works fine if I test it with nanite-mapper !

Here's the code: http://gist.github.com/147641

ruby - 1.8.7
mongrel - 1.1.5
nanite - 0.4.1 (edge)
amqp - 0.6.0
rabbitmq - 1.5.5
Arch - x64
OS - Cent OS 5.1 (virtual box)

A week ago or so I did a bundle install --relock on my Rails 3 app in development which uses a nanite mapper to talk to other agents. I cant be sure if something happened back then but now I can't Control-C the rails s(webrick) process any longer so now I have to kill -9 the process every time., I get this error:

^C[2010-07-01 12:02:24] ERROR SystemExit: exit
/Library/Ruby/Gems/1.8/gems/nanite-0.4.1.17/lib/nanite/mapper.rb:313:in `exit'

Im on a Mac with Snow Leopard 10.6.4, running system ruby and rails 3 betas.

No problems when running on Thin.

With RabbitMQ 1.5.4, some of the commands (permissioning) in the `examples/rabbitconf.rb` fail. These failures don’t affect the functioning of the examples, but they don’t exactly inspire user confidence, either…

There are also some places in the examples where hardcoded paths are still in place, e.g., `examples/crew.rb`.

Getting a weird SSL error with ruby 1.9r129 on osx - any help / ideas would be appreciated!

$ rake spec
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31: [BUG] Bus Error
ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-darwin9.7.0]

-- control frame ----------
c:0029 p:---- s:0080 b:0080 l:000079 d:000079 CFUNC  :initialize
c:0028 p:---- s:0078 b:0078 l:000077 d:000077 CFUNC  :new
c:0027 p:0063 s:0075 b:0075 l:000074 d:000074 CLASS  /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31
c:0026 p:0011 s:0073 b:0073 l:000072 d:000072 CLASS  /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:23
c:0025 p:0011 s:0071 b:0071 l:000070 d:000070 CLASS  /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:22
c:0024 p:0045 s:0069 b:0069 l:000068 d:000068 TOP    /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:21
c:0023 p:---- s:0067 b:0067 l:000066 d:000066 FINISH
c:0022 p:---- s:0065 b:0065 l:000064 d:000064 CFUNC  :require
c:0021 p:0059 s:0061 b:0061 l:000060 d:000060 TOP    /usr/local/lib/ruby19/site_ruby/1.9.1/openssl.rb:22
c:0020 p:---- s:0059 b:0059 l:000058 d:000058 FINISH
c:0019 p:---- s:0057 b:0057 l:000056 d:000056 CFUNC  :require
c:0018 p:0083 s:0053 b:0053 l:000052 d:000052 TOP    /Users/ippy04/Code/Samples/nanite/lib/nanite.rb:7
c:0017 p:---- s:0051 b:0051 l:000050 d:000050 FINISH
c:0016 p:---- s:0049 b:0049 l:000048 d:000048 CFUNC  :require
c:0015 p:0084 s:0045 b:0045 l:000044 d:000044 TOP    /Users/ippy04/Code/Samples/nanite/spec/spec_helper.rb:6
c:0014 p:---- s:0043 b:0043 l:000042 d:000042 FINISH
c:0013 p:---- s:0041 b:0041 l:000040 d:000040 CFUNC  :require
c:0012 p:0039 s:0037 b:0037 l:000036 d:000036 TOP    /Users/ippy04/Code/Samples/nanite/spec/actor_registry_spec.rb:1
c:0011 p:---- s:0035 b:0035 l:000034 d:000034 FINISH
c:0010 p:---- s:0033 b:0033 l:000032 d:000032 CFUNC  :load
c:0009 p:0012 s:0029 b:0029 l:000020 d:000028 BLOCK  /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:15
c:0008 p:---- s:0026 b:0026 l:000025 d:000025 FINISH
c:0007 p:---- s:0024 b:0024 l:000023 d:000023 CFUNC  :each
c:0006 p:0036 s:0021 b:0021 l:000020 d:000020 METHOD /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:14
c:0005 p:0097 s:0017 b:0017 l:000016 d:000016 METHOD /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/options.rb:107
c:0004 p:0068 s:0012 b:0012 l:000011 d:000011 METHOD /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/command_line.rb:9
c:0003 p:0077 s:0007 b:0006 l:002614 d:002494 EVAL   /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/bin/spec:4
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:002614 d:002614 TOP    :43044
---------------------------
-- Ruby level backtrace information-----------------------------------------
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31:in `initialize'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31:in `new'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31:in `'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:23:in `'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:22:in `'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:21:in `'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl.rb:22:in `require'
/usr/local/lib/ruby19/site_ruby/1.9.1/openssl.rb:22:in `'
/Users/ippy04/Code/Samples/nanite/lib/nanite.rb:7:in `require'
/Users/ippy04/Code/Samples/nanite/lib/nanite.rb:7:in `'
/Users/ippy04/Code/Samples/nanite/spec/spec_helper.rb:6:in `require'
/Users/ippy04/Code/Samples/nanite/spec/spec_helper.rb:6:in `'
/Users/ippy04/Code/Samples/nanite/spec/actor_registry_spec.rb:1:in `require'
/Users/ippy04/Code/Samples/nanite/spec/actor_registry_spec.rb:1:in `'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:15:in `load'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:15:in `block in load_files'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:14:in `each'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:14:in `load_files'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/options.rb:107:in `run_examples'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/command_line.rb:9:in `run'
/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/bin/spec:4:in `'

-- C level backtrace information -------------------------------------------
0x117042 0   ruby19                              0x00117042 rb_vm_bugreport + 82
0x2c21c 1   ruby19                              0x0002c21c rb_warning + 444
0x2c27b 2   ruby19                              0x0002c27b rb_bug + 43
0xbd37b 3   ruby19                              0x000bd37b rb_enable_interrupt + 75
0x91e0b2bb 4   libSystem.B.dylib                   0x91e0b2bb _sigtramp + 43
0xffffffff 5   ???                                 0xffffffff 0x0 + 4294967295

[NOTE]
You may encounter a bug of Ruby interpreter. Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

rake aborted!
Command /usr/local/bin/ruby19 -I"/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib"  "/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/bin/spec" "spec/actor_registry_spec.rb" "spec/actor_spec.rb" "spec/agent_spec.rb" "spec/cached_certificate_store_proxy_spec.rb" "spec/certificate_cache_spec.rb" "spec/certificate_spec.rb" "spec/cluster_spec.rb" "spec/dispatcher_spec.rb" "spec/distinguished_name_spec.rb" "spec/encrypted_document_spec.rb" "spec/job_spec.rb" "spec/local_state_spec.rb" "spec/log_spec.rb" "spec/mapper_spec.rb" "spec/packet_spec.rb" "spec/rsa_key_pair_spec.rb" "spec/secure_serializer_spec.rb" "spec/serializer_spec.rb" "spec/signature_spec.rb" "spec/static_certificate_store_spec.rb" "spec/util_spec.rb" --format specdoc --colour failed

Nanite mapper seems to log all requests to agents as INFO:

[Sun, 15 Nov 2009 01:34:35 -0500] INFO: SEND [result] to TEST_I
[Sun, 15 Nov 2009 01:34:35 -0500] INFO: RECV [result] from TEST_II

Is there any way to control this? The log files grow like crazy when every request is logged. Not sure if this is a logging issue or the way we are using it.

Looks like the file "nanite/examples/cli.rb" got moved to "nanite/examples/simpleagent/cli.rb". This causes two problems:

First, it breaks a path in cli.rb:

"require File.dirname(FILE) + '/../lib/nanite'" should become
"require File.dirname(FILE) + '/../../lib/nanite'"

Second, the tutorial code on the main webpage in the section "Test Nanite (finally)" needs to change from "cd examples; ./cli.rb;" to "cd examples/simpleagent; ./cli.rb"

In nanite/state.rb, line 88 has an 'sdelete', which is thrown as an invalid function call when passed into redis. Replacing the sdelete with 'srem' eliminates this problem. It looks like an oversight made when moving from the earlier version of redis, am I missing something? Or is this a safe fix?

When using redis the nanite- key gets set with the load_average when the node registers but it is not updated after that by heartbeat messages. It seems that the intention is that the heartbeat/ping sends the load average so that the load average will be put in Redis. This is not happening because the code in cluster.rb (handle_ping) is doing:

if nanite = nanites[ping.identity]
    nanite[:status] = ping.status

but nanites[ping.identity] returns an anonymous Hash, so updating it here does nothing. As such this value is never sent to Redis. I have confirmed that hacking in a update_status function to the Nanite::State class (which just updates the nanite- key in redis) and then calling it in the handle_ping as:

nanites.update_status(ping.identity, ping.status)

causes the value in Redis to be updated at every heartbeat. Was it the intended behavior for this function (handle_ping) to update Redis? The comment seems to indicate that is the case.

nanite wont startup in ruby1.9 without some code changes. nanite uses FileUtils which does not seem to be available by default in 1.9. I have sent a pull request with a fix to the issue but never got a response (probably with good reason). in any case, with the change nanite runs flawlessly on 1.9. we have it running on several servers without any issues once that change is made.

I am getting a seemingly random and intermittent error (about once a day, so around 300 jobs) where inside of cluster.rb, line 169 (inside the block passed to nanites_for), one of the returned nanites has nil for its attributes. I cannot seem to duplicate this issue manually. It fails when, inside that block, the nanite is passed to the 'timed_out' function.

NoMethodError: undefined method []' for nil:NilClass /usr/local/lib/ruby/gems/1.8/gems/nanite-0.4.1.13/lib/nanite/cluster.rb:162 /usr/local/lib/ruby/gems/1.8/gems/nanite-0.4.1.13/lib/nanite/cluster.rb:169:innanites_providing'

I am not familiar enough with the code to determine if there is someplace where a race condition is possible? I don't know where to go for further troubleshooting, but, I have implemented a bandaid and am waiting to see if the issue crops up again.

I just added a check for nil before calling timed_out?
if nanite_attributes.nil? or timed_out?(nanite_attributes)

If this fixes the issue, I'll commit to my branch here on github...Just hesitant in bandaiding a problem I don't understand!

Nanite depends on the json gem, this should be defined in the gemspec.

Hi,

First: I'm a noob using Nanite.. I'm sorry

I want to use Nanite in a distributed environment with multiple machines and I don't know how can I achieve it. I can't find any examples in google about it. Imagine, for example, that we are using Amazon EC2, but in reality I'll use physical servers

Could you could give me some guidelines, please?
Where I can get information?

Thanks in advance!

In mapper.rb the request function looks like:

request(type, payload = '', opts = {}, &blk)

and then passes the first 3 arguments in the same order to:

build_deliverable(deliverable_type, type, payload, opts)

with deliverable_type == Request. Then build_deliverable passes these arguments directly to:

deliverable_type.new(type, payload, opts)

but the problem is that Request has an initialize function that looks like:

initialize(type, payload, size=nil, opts={})

so the options that were passed in to the original request get passed in as the 'size' parameter, which obviously doesn't work. This causes any :selector or :target to be ignored for example.

In our environment, this was causing problems when we passed the :target option because it was being ignored. Changing the build_deliverable function to make the call:

deliverable_type.new(type, payload, nil, opts)

fixed the issue.

I got the following error when trying to install nanite with rspec 2.0.0

mrt@drone02:~/nanite$ rake gem
(in /media/production/users/mrt/nanite)
rake aborted!
no such file to load -- spec/rake/spectask
/media/production/users/mrt/nanite/Rakefile:3
(See full trace by running task with --trace)

Apparently rspec 2.0.0 does not package spec/rake/spectask.

The solution for me was to uninstall rspec 2.0.0 and install rspec 1.1.12 and also json. e.g.

gem uninstall rspec
gem install rspec --version '1.1.12'
gem install json

Not sure if this is an amqp error or a nanite error, I have posted it on amqp as well.

/vendor/gems/gems/amqp-0.6.5/lib/amqp/buffer.rb:252:in min': comparison of Array with Array failed (ArgumentError) from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:137:ineach'
from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:137:in min' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:137:inleast_loaded'
from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:23:in __send__' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:23:intargets_for'
from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/mapper.rb:198:in send_request' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/mapper.rb:191:inrequest'
from base_prog.rb:58:in start' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/em/timers.rb:51:incall'
from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/em/timers.rb:51:in fire' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:incall'
from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in run_machine' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:inrun'
from base_prog.rb:41:in `start'
from base_prog.rb:70

This happens a lot. And when it happens it continues to happen repeatedly every couple of minutes till a restart is done. Wondering if this has to do with rabbitmq/amqp or the state of the nanite.

Any thoughts would be greatly appreciated.
Thanks!

I'm working on a test for our Nanite agent, but the request initiated by Nanite.request are all async, which means, i can get nothing back from my agent before the test finishes.

Can I initiate a sync request? Thank you