dmitrytsepelev / ar_lazy_preload Goto Github PK
View Code? Open in Web Editor NEWLazy loading associations for the ActiveRecord models
License: MIT License
Lazy loading associations for the ActiveRecord models
License: MIT License
Hi,
I see that some N+1 improvements are made on the master branch and version is pushed to 0.3.0 but there is no release for that version. Can you build a new release?
Thanks.
Hi @DmitryTsepelev,
Could you please help with this case?
require 'bundler/inline'
gemfile(true) do
source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
gem 'activerecord', '= 7.0.6'
require "active_record"
gem 'ar_lazy_preload'
gem 'sqlite3'
end
require "ar_lazy_preload"
ArLazyPreload.install_hooks
require 'minitest/autorun'
require 'logger'
ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
ActiveRecord::Base.logger = Logger.new($stdout)
ActiveRecord::Schema.define do
create_table :users, force: true do |t|
end
create_table :invitations, force: true do |t|
t.integer :user_id
t.integer :company_id
end
create_table :companies, force: true do |t|
end
end
class User < ActiveRecord::Base
has_many :invitations, dependent: :destroy
has_many :companies, through: :invitations
accepts_nested_attributes_for :invitations, allow_destroy: true
end
class Invitation < ActiveRecord::Base
belongs_to :company
belongs_to :user
validate :something
def something
# do something
# if we comment out this but keep ArLazyPreload on the test passes.
user.invitations.each(&:itself)
end
end
class Company < ActiveRecord::Base
has_many :invitations, dependent: :destroy
has_many :users, through: :invitations
end
ArLazyPreload.config.auto_preload = true
class BugTest < ActiveSupport::TestCase
def test_nested_preloading
user = User.create
company = Company.create
# This test is failing but if we comment out ArLazyPreload.config.auto_preload = true it will pass as expected
assert_difference "Invitation.count" do
user.update(invitations_attributes: [{company_id: company.id}])
# => true regardless of ArLazyPreload mode
# It is funny that if we do update twice (with ArLazyPreload on), the test passes by creating only single record
# user.update(invitations_attributes: [{company_id: company.id}])
end
end
end
P.S. I apologize for the messy code above, but it should show the problem.
I've been using https://github.com/salsify/goldiloader for a while so I was curious how this project is different or if it offers any advantages ?
Thanks for your gem and your work! π
Introduction
Firstly to properly describe our problem, I want to show our use cases and where it comes from.
Since we extensively started using this gem, we faced the issue when we need to create a context for a particular set of objects which was not loaded by relation. For instance, some smart service returns as 10 objects, which were loaded by code which we cannot control. I've come up with the following solution:
def register(objects)
objects = Array.wrap(objects)
context = ArLazyPreload::Context.register(
records: Array.wrap(objects),
association_tree: [],
auto_preload: true
)
# This always rewrites the context to have predictable behaviour
objects.each { |object| object.lazy_preload_context = context }
context
end
It works pretty well for now and solves most of our problems.
Problem
Imagine we have something similar, but we need to tie up a different sets of objects into the same context.
for instance, we have the following GQL query
users {
skills {
jobs { id }
}
}
and skills
is not an association with users, but generated by some smart ML service. But I want able to register those skills into the same context to make preload work properly
find_skills_for_user - returns array of skills
users = User.all.preload_associations_lazily
skills = users.flat_map{|user| find_skills_for_user(user) }
skills.each{|skill| skill.jobs } - this is going to generate N+1, because they are not bounded to
the same context. I could use function defined above, but it's going to have less queries, but still it's N+1
I hope I could explain the problem.
I really appreciate if you could guide/help how I could solve this. Thanks in advance.
Is a full Rails dependency actually necessary for this gem to function? Would love to use it but I'm in the simpler Rack-app land.
Please help me understand this is expected behavior or a bug? And if itβs expected, is it possible to somehow make it more predictable?
Gems
activerecord (7.1.3.4)
ar_lazy_preload (2.0.0)
pg (1.5.6)
Preload schema and seeds, we have 2 tables and has_many
association
ActiveRecord::Schema.define do
create_table :teams, force: true do |t|
end
create_table :users, force: true do |t|
t.string :name
t.integer :team_id
end
end
class Team < ActiveRecord::Base
has_many :users
end
class User < ActiveRecord::Base
belongs_to :team
end
team = Team.create!
User.create!(name: 'A', team: team)
User.create!(name: 'B', team: team)
User.create!(name: 'C', team: team)
The issue is
team = Team.lazy_preload(:users).find(team.id)
team.users.build(name: 'D')
p 'just call team users somewhere', team.users
team.save!
p '===' * 5
team = Team.lazy_preload(:users).find(team.id)
team.users.build(name: 'D')
p 'without call team users works fine'
team.save!
p '===' * 5
team = Team.lazy_preload(:users).find(team.id)
team.users.load.build(name: 'D')
p 'with force load also works fine'
team.save!
In the first example, user D
will not be created because if we call association after build
we replace object to new one from database, is this expected behavior? I didn't find this in the description, many thanks!
logs
D, [2024-06-06T00:44:43.238636 #51420] DEBUG -- : TRANSACTION (0.5ms) BEGIN
D, [2024-06-06T00:44:43.239524 #51420] DEBUG -- : Team Create (1.4ms) INSERT INTO "teams" DEFAULT VALUES RETURNING "id"
D, [2024-06-06T00:44:43.240386 #51420] DEBUG -- : TRANSACTION (0.7ms) COMMIT
D, [2024-06-06T00:44:43.246105 #51420] DEBUG -- : TRANSACTION (0.6ms) BEGIN
D, [2024-06-06T00:44:43.247101 #51420] DEBUG -- : User Create (1.6ms) INSERT INTO "users" ("name", "team_id") VALUES ($1, $2) RETURNING "id" [["name", "A"], ["team_id", 1]]
D, [2024-06-06T00:44:43.247902 #51420] DEBUG -- : TRANSACTION (0.7ms) COMMIT
D, [2024-06-06T00:44:43.248577 #51420] DEBUG -- : TRANSACTION (0.4ms) BEGIN
D, [2024-06-06T00:44:43.249080 #51420] DEBUG -- : User Create (0.9ms) INSERT INTO "users" ("name", "team_id") VALUES ($1, $2) RETURNING "id" [["name", "B"], ["team_id", 1]]
D, [2024-06-06T00:44:43.250079 #51420] DEBUG -- : TRANSACTION (0.9ms) COMMIT
D, [2024-06-06T00:44:43.250910 #51420] DEBUG -- : TRANSACTION (0.6ms) BEGIN
D, [2024-06-06T00:44:43.251557 #51420] DEBUG -- : User Create (1.3ms) INSERT INTO "users" ("name", "team_id") VALUES ($1, $2) RETURNING "id" [["name", "C"], ["team_id", 1]]
D, [2024-06-06T00:44:43.252366 #51420] DEBUG -- : TRANSACTION (0.7ms) COMMIT
"========================"
D, [2024-06-06T00:44:43.257612 #51420] DEBUG -- : Team Load (0.6ms) SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
"just call team users somewhere"
D, [2024-06-06T00:44:43.265162 #51420] DEBUG -- : User Load (0.5ms) SELECT "users".* FROM "users" WHERE "users"."team_id" = $1 [["team_id", 1]]
#<ActiveRecord::Associations::CollectionProxy [#<User id: 1, name: "A", team_id: 1>, #<User id: 2, name: "B", team_id: 1>, #<User id: 3, name: "C", team_id: 1>]>
"========================"
D, [2024-06-06T00:51:33.203851 #52175] DEBUG -- : Team Load (1.0ms) SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
"without call team users works fine"
D, [2024-06-06T00:51:33.205138 #52175] DEBUG -- : TRANSACTION (0.7ms) BEGIN
D, [2024-06-06T00:51:33.205848 #52175] DEBUG -- : User Create (1.4ms) INSERT INTO "users" ("name", "team_id") VALUES ($1, $2) RETURNING "id" [["name", "D"], ["team_id", 1]]
D, [2024-06-06T00:51:33.207215 #52175] DEBUG -- : TRANSACTION (1.2ms) COMMIT
"========================"
D, [2024-06-06T00:44:43.269206 #51420] DEBUG -- : Team Load (0.5ms) SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
D, [2024-06-06T00:44:43.270050 #51420] DEBUG -- : User Load (0.4ms) SELECT "users".* FROM "users" WHERE "users"."team_id" = $1 [["team_id", 1]]
"with force load also works fine"
D, [2024-06-06T00:44:43.271183 #51420] DEBUG -- : TRANSACTION (0.4ms) BEGIN
D, [2024-06-06T00:44:43.271787 #51420] DEBUG -- : User Create (1.0ms) INSERT INTO "users" ("name", "team_id") VALUES ($1, $2) RETURNING "id" [["name", "D"], ["team_id", 1]]
D, [2024-06-06T00:44:43.272553 #51420] DEBUG -- : TRANSACTION (0.7ms) COMMIT
how it works with includes
, ofc because we preload users before build
team = Team.includes(:users).find(team.id)
team.users.build(name: 'D')
p 'just call team users somewhere', team.users
team.save!
D, [2024-06-06T00:48:27.639756 #51807] DEBUG -- : Team Load (0.5ms) SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
D, [2024-06-06T00:48:27.645837 #51807] DEBUG -- : User Load (0.5ms) SELECT "users".* FROM "users" WHERE "users"."team_id" = $1 [["team_id", 1]]
"just call team users somewhere"
#<ActiveRecord::Associations::CollectionProxy [#<User id: 1, name: "A", team_id: 1>, #<User id: 2, name: "B", team_id: 1>, #<User id: 3, name: "C", team_id: 1>, #<User id: nil, name: "D", team_id: 1>]>
D, [2024-06-06T00:48:27.649317 #51807] DEBUG -- : TRANSACTION (1.0ms) BEGIN
D, [2024-06-06T00:48:27.650261 #51807] DEBUG -- : User Create (2.0ms) INSERT INTO "users" ("name", "team_id") VALUES ($1, $2) RETURNING "id" [["name", "D"], ["team_id", 1]]
D, [2024-06-06T00:48:27.651341 #51807] DEBUG -- : TRANSACTION (0.9ms) COMMIT
Hello Dmitry π
I noticed an unexpected behavior in a context of has_one ... through
associations.
Imagine a model:
class Alice < ApplicationRecord
has_one :bob
has_one :charlie, through: :bob
end
and then
record = Alice.preload_associations_lazily.find(some_id)
assert record.lazy_preload_context.present?, "lazy context is missing for alice"
# at this point record has no associations loaded
record.charlie
# at this point record has both bob and charlie associations loaded
assert record.charlie.lazy_preload_context.present?, "lazy context is missing for charlie"
assert record.bob.lazy_preload_context.present?, "lazy context is missing for bob"
Observed behavior: the last assertion fails
Expected behavior: all assertions succeed
Steps to reproduce:
docker compose up -d && bin/setup && bin/rails test
Output:
% bin/rails test
Running 1 tests in a single process (parallelization threshold is 50)
Run options: --seed 56792
# Running:
F
Failure:
AliceTest#test_the_truth [test/models/alice_test.rb:15]:
lazy context is missing for bob
rails test test/models/alice_test.rb:4
Finished in 0.127767s, 7.8267 runs/s, 46.9605 assertions/s.
1 runs, 6 assertions, 1 failures, 0 errors, 0 skips
Let's say User
has many posts
. User.lazy_preload(:posts).map(&:posts)
works perfect. But if I want ids with User.lazy_preload(:posts).map(&:post_ids)
it does N+1 queries.
I have tried to add support for association_ids
but I could not manage to implement it. Could you give some hints so that I can implement it and send a pull request?
Model example:
require_dependency 'board'
class Board
class Member < ::ApplicationRecord
belongs_to :board
belongs_to :user
ADMIN_ROLES = %i[owner admin].freeze
scope :admins, -> { where(role: ADMIN_ROLES) }
enum role: { owner: 0, admin: 1, contributor: 2 }
end
end
Error example:
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
/home/ruslan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/ar_lazy_preload-0.6.1/lib/ar_lazy_preload/active_record/association_relation.rb:13:in `lazy_preload_context'
I can see it's now only tested against rails 4.2 and master
Why not test against 5.x?
I can create PR if you want
ArLazyPreload.config.auto_preload = true
Should this line go in the config block in application.rb
?
Issue can be reproduced with the following spec:
it "can lazy preload associations on association collection" do
user = User.first
expect(user.posts).to be_a(ActiveRecord::Associations::CollectionProxy)
expect { user.posts.load }.not_to raise_exception
end
Like how hamlit
does it
Hi. Thanks for the wonderful gem! I faced the following issue by just plugging the gem into my project that has chained has_many through
association:
class Employee
belongs_to :company
has_many :products, foreign_key: 'created_by_id'
end
class Company
has_many :employees
has_many :products, through: :employees
end
class Subproduct
belongs_to :product
end
class ChannelSubproduct
belongs_to :subproduct
end
class ChannelProduct
has_many :channel_subproducts
has_many :subproducts, through: :channel_subproducts
has_many :products, -> { uniq }, through: :subproducts
end
class Product
has_many :subproducts
has_many :channel_subproducts, through: :subproducts
has_many :channel_products, through: :channel_subproducts
belongs_to :created_by, class_name: 'Employee'
end
When I try to call Company.first.products
, the gem fails here:
NoMethodError: undefined method `klass' for nil:NilClass
/bundler/gems/rails-6f310f798eeb/activerecord/lib/active_record/reflection.rb:671:in `source_reflection'
/bundler/gems/rails-6f310f798eeb/activerecord/lib/active_record/reflection.rb:883:in `derive_class_name'
/bundler/gems/rails-6f310f798eeb/activerecord/lib/active_record/reflection.rb:157:in `class_name'
/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/merger.rb:31:in `block in reflect_and_merge_lazy_preloads'
/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/merger.rb:30:in `each'
/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/merger.rb:30:in `find'
/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/merger.rb:30:in `reflect_and_merge_lazy_preloads'
/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/merger.rb:15:in `merge'
I believe this is because the nested through
association has delegate_reflection
instead (assoc.class_name
- fails, assoc.delegate_reflection.class_name
- works):
[2] DEVELOPMENT(#<ActiveRecord::Relation::Merger>)> assoc = relation.klass.reflect_on_all_associations.find { |a| !a.class_name rescue a }
=> #<ActiveRecord::Reflection::ThroughReflection:0x00007fb03bdc38e8
@delegate_reflection=
#<ActiveRecord::Reflection::HasManyReflection:0x00007fb03bdc3b40
@active_record=
Product(...),
@association_scope_cache={},
@automatic_inverse_of=nil,
@constructable=true,
@foreign_type="channel_products_type",
@klass=nil,
@name=:channel_products,
@options={:through=>:channel_sellables},
@plural_name="channel_products",
@scope=nil,
@scope_lock=#<Thread::Mutex:0x00007fb03bdc3910>,
@type=nil>,
@klass=nil,
@source_reflection_name=nil>
Rails 4.2.10
/ Ruby 2.5.5
Hope it helps!
https://github.com/rails/rails/blob/v7.0.4/activerecord/lib/active_record/persistence.rb#L943
Rails definition has arguments
https://github.com/DmitryTsepelev/ar_lazy_preload/blob/v1.1.0/lib/ar_lazy_preload/active_record/base.rb#L16
1.1.0 of this gem doesn't
Hi @DmitryTsepelev,
I'm using the global auto preload feature (ArLazyPreload.config.auto_preload = true
) in a large project, and am very happy with the performance gain that it provides.
Nevertheless, since I enabled this feature, I'm fixing bugs related to auto preloading quite often, most of which are because of some interactions with other gems. I think that I can count the time spent in fixing those bugs in days.
For now I think that the balance between the performance / maintenance gain is still good to keep the feature enabled, but I'm starting to think about disabling it and going the manual way if I still find such issues.
I was thinking that it could be a good thing to add a warning in the README about this feature to inform people about the risks of using the feature, as it can create hard to trace bugs.
What do you think about it ?
https://www.reddit.com/r/ruby/comments/jt2uub/deep_dive_moving_ruby_projects_from_travis_to/
Travis will no longer be free for Open Source projects (And it's already slow for OS projects now)
This is to move to GitHub Action for CI
Example: https://github.com/PikachuEXE/where_lower/blob/master/.github/workflows/tests.yaml
This example contains multiple ruby versions & gemfiles
But since this project's Travis config matrix is quite complex
I am too lazy to convert it :P
Having both ar_lazy_preload
and bullet
gems in the project, it ends up with a circular dependency when you try to load any association:
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/bundler/gems/ar_lazy_preload-e7fa4004e70c/lib/ar_lazy_preload/active_record/association.rb:8:in `load_target'
/Users/nattfodd/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/bullet-6.1.0/lib/bullet/active_record42.rb:165:in `load_target'
I'm not sure wether it should be fixed in bullet or here, had to create the issue somewhere at least:
Bullet patches load_target
this way:
https://github.com/flyerhzm/bullet/blob/master/lib/bullet/active_record42.rb#L163-L165
ArLazyPreload patches load_target
this way:
https://github.com/DmitryTsepelev/ar_lazy_preload/blob/master/lib/ar_lazy_preload/active_record/association.rb#L6-L9
master...PikachuEXE:fix-deep-asso-loading
Specifying deep associations in array of hashes + symbols make the spec fail in rails 4.2 - 5.1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.