liveh2o / active_remote Goto Github PK

View Code? Open in Web Editor NEW

62.0 62.0 23.0 536 KB

Active Remote provides Active Record-like object-relational mapping over RPC. It's Active Record for your platform.

License: MIT License

Ruby 100.00%

active_remote's People

Contributors

Stargazers

Watchers

active_remote's Issues

Attribute values aren't always coerced on read

If you access an attribute through the [] method (e.g. self[:name]), the raw attribute value is returned. If you access an attribute through it's getter (e.g. self.name) or through the attributes hash (e.g. self.attributes['name']), it is coerced. Not exactly sure where the breakdown is, but these values should either always be raw or always be coerced (leaning towards coerced).

Add support for Strong Params

Active Record now enforces attribute whitelisting through Strong Params, but ARem doesn't support it. It should.

calling "reload" results in method missing to_hash

https://github.com/liveh2o/active_remote/blob/master/lib/active_remote/search.rb#L139

self.class.find returns an instance of an ActiveRemote object, not a response. We need to call attributes instead of to_hash.

Support `update` as an alias for `update_attributes`

Rails 4 added update as an alias for update_attributes, and generators will now use update instead of update_attributes. It seems that update is now the preferred method.

http://apidock.com/rails/v4.2.1/ActiveRecord/Persistence/update

Make rdp_adapter pluggable

I propose adding the following to the README and implementing the adapter described:

RPC Adapter

An RPC Adapter is responsible for gathering the attributes from your model and marshalling them to the RPC call. Marshalling behaviors can be injected or modified at this point. The default RPC adapter is ::ActiveRemote::RPCAdapters::ProtobufAdapter.

You can, of course override it if need be:

  # If you have a custom rpc adapter to use globally:
  ::ActiveRemote::RPC.default_adapter = ::MyCustom::RPCAdapter

  # If you have a custom rpc adapter for an individual class:
  class Product < ActiveRemote::Base
    rpc_adapter ::MyCustom::RPCAdapter
  end

  # If you have a custom rpc adapter that has special setup needs:
  class Product < ActiveRemote::Base
    rpc_adapter :build_custom_rpc_adapter

    private

    def build_custom_rpc_adapter(model)
      ::MyCustom::RPCAdapter.new(@rpc_settings)
    end
  end

  # Or, with a block
  class Product < ActiveRemote::Base
    rpc_adapter do |model|
      ::MyCustom::RPCAdapter.new(model.rpc_settings)
    end
  end

@liveh2o @brianstien

belongs_to does not work with namespaces

It seems like #belongs_to https://github.com/liveh2o/active_remote/blob/master/lib/active_remote/association.rb#L40 does not work with namespaces.

Say a in a model structure

module Service
  class User 
  end
end

module Data
  class Address
    # This does not work
    belongs_to :user
    # could be like this 
    #belongs_to :user, :class_name => ::Service::User

  end
end

When no primary key is specified and no `guid` field exists, `to_key` raises an error

If using an ActiveRemote object in a form and using form_for in Rails for the model, ActiveRemote causes an undefined method 'guid' for error to be raised.

Enforce validations

Currently, ARem supports the validation DSL methods provided by Active Model, but doesn't actually call valid? before saving a record. 👎

leaking method sites on activemodel attributes

Not sure where the leak is, but this method defines a new method when a proxy is created for the attributes that are defined https://github.com/rails/rails/blob/4-2-stable/activemodel/lib/active_model/attribute_methods.rb#L383

the value of __FILE__ is stored > 14,000 times in a single running process of synchronicity-subscriber ... this is the only spot this is called; may be because of JRuby (and not active_remote), but my assumption is active_remote because we recently refactored the methods that influence if a method is generated ... looking for additional evidence

@liveh2o @brianstien

uninitialized constant UnknownAttributeError

When updating a field that doesn't exist, instead of raising an error indicating the field is unknown, it fails with this error:

NameError: uninitialized constant ActiveRemote::Attributes::UnknownAttributeError
from ~/active_remote-1.3.3/lib/active_remote/attributes.rb:11:in `read_attribute'

bulk.parse_records does not support active_remote objects

This record expects objects to respond_to :to_hash, but not :attributes.

As a result you cannot call bulk operations on active remote objects

https://github.com/liveh2o/active_remote/blob/master/lib/active_remote/bulk.rb#L105-L113

Type Casting Problems

We know there are some type casting issues that we've resolved internally, but still need to either be fixed here or in attr_model.

I noticed we are using additional type casting on the Field class which is undoing some of our internal optimizations:

active_remote/lib/active_remote/serializers/protobuf.rb

Lines 129 to 134 in db2d3ef

 def typecaster 

 @typecaster ||= begin 

 typecaster = TYPECASTER_MAP[field.type_class] 

 typecaster.new if typecaster 

 end 

 end

This is called for every repeated? field on every search request. Aka, a lot.

cc @liveh2o @abrandoned

Working example

I'm working with an integration with Active_remote, do you have any sample or tutorial for the library?

I have a scenario running Gruf, and I implemented some abstraction to deal with "remote models".
After that solution I found ActiveRemote gem, but I was unable to make it work.

The thing is, I am using GRUF, both server and client sides, to handle grpc calls, and I do need to make it work with Active_remote on the client.

My service is something like that

  require 'grpc'
  require 'app/proto/Clients_pb'

  module RemoteProto
    module Clients
      class Service

        include GRPC::GenericService

        self.marshal_class_method = :encode
        self.unmarshal_class_method = :decode
        self.service_name = 'remote_proto.Clients'

        rpc :Index, ::RemoteProto::IndexReq, stream(::RemoteProto::ClientModel)
        rpc :Show, ::RemoteProto::ShowReq, ::RemoteProto::ShowResp
      end

      Stub = Service.rpc_stub_class
    end
  end

and my proto is

  syntax = "proto3";

  import "AddressModel.proto";
  import "Filter.proto";

  package remote_proto;

  service Clients {
    rpc Index(IndexReq) returns (stream ClientModel) {}
    rpc Show(ShowReq) returns (ShowResp) { }
  }
  
  message ClientModel {
    uint32 id = 1;
    string name = 2;
    float price = 3;
    repeated Address addresses = 4;
  }

  message IndexReq {
    uint32 limit = 1;
    uint32 page = 2;
    Filter filter = 3;
  }

  message ShowReq {
    uint64 id = 1;
  }

  message ShowResp {
    ClientModel client = 1;
  }

Reading your lib, I think that I have to change the rpc_adapter, but I don have no idea how to work on that based on my service, or if I am right.

Is there where I really have to change? Do you have any sugestions?
It would be awesome, trully, if you could share some similar working example

Note: BTW, I just started to work with RPC

Thanks a lot...

ActiveRemote 3.0 feature set

I wanted to start working on the 3.0 release.

Items that will be included

`write_attribute` is removed in Rails 6.1

This gem calls write_attribute which was removed from ActiveModel in Rails 6.1. We need to find a good alternative implementation.

Association attribute writer does not work when association name doesn't match association class

When defining an association with a name that doesn't match the class, the writer method does not work:

class Job < ActiveRemote::Base
end

class Member < ActiveRemote::Base
  belongs_to :recent_job, :class_name => "Job"
end

2.6.10 :002 > m = Member.new
2.6.10 :002 > m.recent_job = Job.new
 =>  #<Job ... >
2.6.10 :003 > m.recent_job
 => nil

Raise more granular errors in the Protobuf adapter

The Protobuf library returns errors with specific reasons when requests fail:

// Server-side errors
BAD_REQUEST_DATA = 0;                         // Server received bad request data
BAD_REQUEST_PROTO = 1;                        // Server received bad request proto
SERVICE_NOT_FOUND = 2;                        // Service not found on server
METHOD_NOT_FOUND = 3;                         // Method not found on server
RPC_ERROR = 4;                                // Rpc threw exception on server
RPC_FAILED = 5;                               // Rpc failed on server

// Client-side errors (these are returned by the client-side code)
INVALID_REQUEST_PROTO = 6;                    // Rpc was called with invalid request proto
BAD_RESPONSE_PROTO = 7;                       // Server returned a bad response proto
UNKNOWN_HOST = 8;                             // Could not find supplied host
IO_ERROR = 9;                                 // I/O error while communicating with server

The Protobuf adapter should raise specific errors for each of these reasons instead of a single generic error.

Date/DateTime conversions

Date and DateTimes no longer get converted into integers when serializing (since AR 2) for the protobuf adapter

pretty sure we didn't mean to remove that functionality, can't run client side code that ran under previous AR if Date/DateTime searching is to be used

@liveh2o

Add support for eager loading

Consider an API that publishes Transaction#user_identifer and Transaction#account_identifer. These fields are delegated to the associated user and account (respectively) because they are not part of the transaction model. This works, but results in RPC calls to get the user and account for each record returned when searching for transactions (i.e., 2N+1 RPC calls). This is especially impactful when searching for large sets of transactions (i.e., 1000), which this API also supports.

Potential solution

Active Record solves this problem with eager loading: after loading records, associations are eager loaded with a single query per associated record (i.e., Transaction#account, etc.). With two associations, this results in 3 queries rather than 2N+1 queries.

For API under consideration, we added an optimized search method to the transaction model that eagerly loads the user and accounts when searching for transactions:

def self.optimized_search(args = nil)
  # Eager load user and account to avoid 2N + 1 RPC calls
  search(args).tap do |records|
    return records if records.empty?

    # Requests must be scoped to a single user, so we can assume all transactions have the same :user_guid
    user = User.find(:guid => records.first.user_guid)

    account_guids = records.map(&:account_guid)
    account_guids.uniq!
    accounts = Account.search(:guid => account_guids, :user_guid => user.guid)
    accounts_by_guid = accounts.inject({}) do |hash, account|
      hash[account[:guid]] = account
      hash
    end

    records.each do |record|
      record.account = accounts_by_guid[record.account_guid]
      record.user = user
    end
  end
end

This is clearly specific to the API setup (i.e., assuming all transactions belong to the same user), but the pattern of tapping the results and setting the association records this way should be possible to generalize.

For the API under consideration, using the optimized search is nearly 38 times faster than search:

Running transaction search benchmarks:
Warming up --------------------------------------
              search     1.000  i/100ms
    optimized_search     1.000  i/100ms
Calculating -------------------------------------
              search      0.046  (± 0.0%) i/s -      1.000  in  21.954028s
    optimized_search      1.712  (± 0.0%) i/s -     35.000  in  20.482257s

Comparison:
    optimized_search:        1.7 i/s
              search:        0.0 i/s - 37.60x  (± 0.00) slower

Considerations

Ideally, this would not require an additional method to be created in order to tap the results and eager load the associations. Something like Active Record's:

::Transaction.search(:user_guid => "USR-123").eager_load(:user, :account)

That's not currently possible with the results being a simple Ruby array. Extending array or creating a special result set that provides this eager loading could work.

Additionally, eager loading will need to respect defined scope_keys of the association being eager loaded (which will be tricky if the scope key values are not available on the main object (i.e., Transaction would need both #account_guid and #user_guid fields in the above example).

Investigate compatibility with Rails 4.2

ARem has been compatible with Rails 4 so far, but we haven't done any thing to explicitly check compatibility. We should do that.

Associations do not provide attribute writers

For some reason, defining an association (i.e. belongs_to) only defines an attribute reader. It should also provide an attribute writer.

Support for Rails 6

Currently active_remote pinned on activesupport 5.2. What would it take to be able to support activesupport 6?

Getting `NoMethodError` when calling `super` in overridden association method

I have a model with a has_many association and a custom reader method of the same name. I would expect that calling super would execute the original method, but it does not and I get a no such method exception instead. Removing everything from my model expect for the association and method, I am left with the following:

class Credential < ::ActiveRemote::Base
  has_many :credential_options

  def credential_options
    binding.pry
    @credential_options ||= super
  end
end

This is what I get when I run my code:

irb(main):001:0> Credential.new(:guid => "1").credential_options

From: /myapp/app/models/credential.rb @ line 5 Credential#credential_options:

    4: def credential_options
 => 5:   binding.pry
    6:   @credential_options ||= super
    7: end

[1] pry(#<Credential>)> defined? super
=> nil
[2] pry(#<Credential>)> exit
NoMethodError: super: no superclass method `credential_options' for #<Credential:0x482549c0>
Did you mean?  credential_options=
        from org/jruby/RubyBasicObject.java:1657:in `method_missing'
        from /home/.rbenv/versions/jruby-9.1.12.0/lib/ruby/gems/shared/gems/activemodel-4.2.7.1/lib/active_model/attribute_methods.rb:430:in `method_missing'
        from /myapp/app/models/credential.rb:6:in `credential_options'
        from (irb):1:in `<eval>'
        from org/jruby/RubyKernel.java:1000:in `eval'
        from org/jruby/RubyKernel.java:1298:in `loop'
        from org/jruby/RubyKernel.java:1120:in `catch'
        from org/jruby/RubyKernel.java:1120:in `catch'
        from /home/.rbenv/versions/jruby-9.1.12.0/lib/ruby/gems/shared/gems/railties-4.2.7.1/lib/rails/commands/console.rb:110:in `start'
        from /home/.rbenv/versions/jruby-9.1.12.0/lib/ruby/gems/shared/gems/railties-4.2.7.1/lib/rails/commands/console.rb:9:in `start'
        from /home/.rbenv/versions/jruby-9.1.12.0/lib/ruby/gems/shared/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /home/.rbenv/versions/jruby-9.1.12.0/lib/ruby/gems/shared/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /home/.rbenv/versions/jruby-9.1.12.0/lib/ruby/gems/shared/gems/railties-4.2.7.1/lib/rails/commands.rb:17:in `<main>'
        from org/jruby/RubyKernel.java:961:in `require'
        from script/rails:6:in `<main>'
irb(main):002:0>

Commenting out the credential_options method allows the original method to work as expected:

irb(main):001:0> Credential.new(:guid => "1").credential_options
I, [2018-10-18T16:21:34.084808 #76011]  INFO -- : [CLT] - ...
=> []
irb(main):002:0>

Environment

$ ruby --version
jruby 9.1.12.0 (2.3.3) 2017-06-15 33c6439 Java HotSpot(TM) 64-Bit Server VM 25.171-b11 on 1.8.0_171-b11 +jit [darwin-x86_64]

$ cat Gemfile.lock
...
    active_remote (3.1.3)
      activemodel (>= 4.0, < 5)
      activesupport (>= 4.0)
      protobuf (>= 3.0)
    active_remote-bulk (0.1.0)
      active_remote (>= 2.4)
    active_remote-cached (0.1.4)
      active_remote
      activesupport
    active_remote-nullify (2.0.0)
      active_remote
    active_remote-pagination (2.3.0)
      active_remote (>= 2.1.0)
...

Attributes may be stomped on when update results in an error

ccp = MyClass.find(guid: '123abc')
ccp.guid # '123abc'
ccp.update_attributes(other_field: 'xyz789') # returns false due to constraint failure
ccp.guid # nil

When the call to update returns an object with all empty fields due to an error, the attributes on the original object get stomped on. Rather than update the attributes with whatever is returned when an error happens, it would seem like perhaps the original attributes should stay exactly the same and only have any errors get appended to the object. Then the caller can make any decisions on what to do next.

Calling `save` appears to always make an RPC call

Unless the behavior changed in Active Record, calling save should be a no-op on a record that has not changed. The same should be true for calling save in Active Remote, but that doesn't appear to be the case:

     Failure/Error: member.save

     Errno::ECONNREFUSED:
       Connection refused - connect(2) for "127.0.0.1" port 9399
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/connectors/socket.rb:30:in `initialize'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/connectors/socket.rb:30:in `new'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/connectors/socket.rb:30:in `connect_to_rpc_server'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/connectors/socket.rb:12:in `block in send_request'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/connectors/base.rb:191:in `timeout_wrap'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/connectors/socket.rb:10:in `send_request'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-opentracing-1.0.7/lib/protobuf/rpc/extensions/client.rb:8:in `send_request'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/protobuf-3.10.5/lib/protobuf/rpc/client.rb:129:in `method_missing'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/rpc_adapters/protobuf_adapter.rb:28:in `execute'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/rpc.rb:27:in `remote_call'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/rpc.rb:61:in `remote_call'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/persistence.rb:168:in `remote'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/dirty.rb:34:in `remote'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/persistence.rb:282:in `block in remote_update'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/persistence.rb:276:in `remote_update'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/dirty.rb:84:in `remote_update'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/persistence.rb:268:in `create_or_update'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/persistence.rb:186:in `block in save'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/persistence.rb:185:in `save'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/dirty.rb:42:in `save'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/active_remote-6.0.1/lib/active_remote/validations.rb:17:in `save'
     # ./app/controllers/api/platform/experian/jobs_controller.rb:72:in `block in set_member'
     # ./app/controllers/api/platform/experian/jobs_controller.rb:68:in `tap'
     # ./app/controllers/api/platform/experian/jobs_controller.rb:68:in `set_member'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/md-logstasher-1.5.0/lib/logstasher/context_wrapper.rb:9:in `process_action'
     # ./app/controllers/api/platform/api_controller.rb:21:in `process_action'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/rails-controller-testing-1.0.5/lib/rails/controller/testing/template_assertions.rb:62:in `process'
     # /Users/adam.hutchison/.rvm/gems/ruby-2.7.5@harvey/gems/rails-controller-testing-1.0.5/lib/rails/controller/testing/integration.rb:16:in `block (2 levels) in <module:Integration>'
     # ./spec/controllers/api/platform/experian/jobs_controller_spec.rb:69:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:99:in `block (2 levels) in <top (required)>'

	def typecaster
	@typecaster \|\|= begin
	typecaster = TYPECASTER_MAP[field.type_class]
	typecaster.new if typecaster
	end
	end