pacific73 / gorm-cache Goto Github PK
View Code? Open in Web Editor NEWgorm v2的即插即用、无需修改代码的旁路缓存。An easy-to-use look-aside cache solution for gorm v2 users.
License: Apache License 2.0
gorm v2的即插即用、无需修改代码的旁路缓存。An easy-to-use look-aside cache solution for gorm v2 users.
License: Apache License 2.0
gorm-cache/cache/after_query.go
Lines 23 to 26 in 229ea3b
看了下 sql 和 vars 好像只用在打印日志和生成 key 上,如果直接在 before_query 拼接好 key 然后传过去呢,个人想法哈
在Redis集群模式下会得到出现Lua脚本加载错误
ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS array, and KEYS should not be in expression
InvalidateWhenUpdate: true, // when you create/update/delete objects, invalidate cache
例如我表结构是(id主键 uid索引)
1 101
2 101
当我查询 where uid = 101后, update 了 id=1的数据, uid=101这条缓存也会清理么
Hi @Pacific73
Gorm-cache is a great project, but it can't support Raw SQL operations, may I get the reason?
Is there any plan to support Raw SQL operations in feature? Thank you.
I believe that you can read Chinese without barriers, if you have any difficulty in understanding the following, please let me know
通常golang的习惯,json tag和gorm tag都会写在同一个struct里
我希望给用户交付的Json需要忽略某些字段的时候就会在struct里设置json:"-"的tag来忽略这个字段,但是同时这个字段又是要参与逻辑的,比如password,salt之类的
同时这个struct也是gorm要返回的,这时由于cache的序列化是基于encoding/json的,就会同时忽略json:"-"的字段,也就是逻辑判断字段会少内容
解决方案:我目前的做法是引入jsoniter来替换掉encoding/json,因为jsoniter允许指定tagkey,并且jsoniter号称性能也比encoding/json要快,询问一下这个变更是否适合合并入主线
different process use diffrent instanceId, is it a waste of memory?
Count操作会异常,原因是gorm的Count的返回类型是int64,会提示反射类型是uint, 直接异常
panic: reflect: call of reflect.Value.Field on uint Value
goroutine 29 [running]:
reflect.Value.Field({0xdd5ce0, 0xc0004b4600, 0x187}, 0x0)
C:/Program Files/Go/src/reflect/value.go:1278 +0x269
gorm.io/gorm/schema.(*Field).setupValuerAndSetter.func2({0xf95ba8, 0x157aca0}, {0xdd5ce0, 0xc0004b4600, 0x187})
C:/Users/Administrator/go/pkg/mod/gorm.io/[email protected]/schema/field.go:454 +0x117
github.com/Pacific73/gorm-cache/cache.getObjectsAfterLoad(0xc000212690)
C:/Users/Administrator/go/pkg/mod/github.com/!pacific73/[email protected]/cache/helpers.go:249 +0x3d6
github.com/Pacific73/gorm-cache/cache.AfterQuery.func1(0xc000212690)
C:/Users/Administrator/go/pkg/mod/github.com/!pacific73/[email protected]/cache/after_query.go:31 +0x314
Hi @Pacific73
I noticed that there are four Cache level:
const (
CacheLevelOff CacheLevel = 0
CacheLevelOnlyPrimary CacheLevel = 1
CacheLevelOnlySearch CacheLevel = 2
CacheLevelAll CacheLevel = 3
)
What's the difference between Search cache and Primary cache, or All cache? Thanks a lot.
gorm-cache/cache/after_query.go
Line 101 in 1d0b034
缓存有过期时间,如果碰巧在 after_query 前过期,那就会查不到
因为所有的数据混杂在同个Redis实例里面,所以希望可以指定Redis的Key前缀
// CacheMaxItemCnt for given query, if objects retrieved are more than this cnt,
// then we choose not to cache for this query. 0 represents caching all queries.
CacheMaxItemCnt int64
注释里写着0表示缓存所有,但是判断的时候好像没特殊判断0。
// cache search data
if int64(len(objects)) > cache.Config.CacheMaxItemCnt {
return
}
Line 120 in 00b422b
我们这边系统是分布式的,存在一个,其他服务更改了缓存数据,广播通知其他服务更新缓存
请问怎么主动删除某个表的缓存数据呢
版本v1.1.0
使用内存缓存,如下配置
cache.NewGorm2Cache(&config.CacheConfig{
CacheLevel: config.CacheLevelAll,
CacheStorage: config.CacheStorageMemory,
InvalidateWhenUpdate: true, // when you create/update/delete objects, invalidate cache
CacheTTL: 10000, // 10s
CacheSize: 5000,
// exceeds this number, then don't cache
}
AfterUpdate方法耗时会慢慢累积,最终会导致数据读写操作均超时
以下是go trace看到的结果
这里为什么采用deletables chan *Item这样的设计?
创建测试数据:
CREATE TABLE `testdb`.`users` (
`id` int NOT NULL,
`name` varchar(255) NOT NULL,
`value` int NOT NULL,
PRIMARY KEY (`id`, `id`)
);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (1, 'hello', 100);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (2, 'abc', 123);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (3, 'xyz', 8);
复现代码:
package main
import (
"fmt"
"github.com/Pacific73/gorm-cache/cache"
"github.com/Pacific73/gorm-cache/config"
"github.com/go-redis/redis/v8"
"gorm.io/driver/mysql"
"gorm.io/gorm"
)
type User struct {
ID int
Name string
Value int
}
func main() {
dsn := "user:pass@tcp(127.0.0.1:3306)/testdb?charset=utf8mb4"
db, _ := gorm.Open(mysql.Open(dsn), &gorm.Config{})
redisClient := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
})
redisCache, err := cache.NewGorm2Cache(&config.CacheConfig{
CacheLevel: config.CacheLevelAll,
CacheStorage: config.CacheStorageRedis,
RedisConfig: cache.NewRedisConfigWithClient(redisClient),
InvalidateWhenUpdate: true, // when you create/update/delete objects, invalidate cache
CacheTTL: 5000, // 5000 ms
CacheMaxItemCnt: 5, // if length of objects retrieved one single time
// exceeds this number, then don't cache
})
if err != nil {
panic(err)
}
// More options in `config.config.go`
db.Use(redisCache) // use gorm plugin
// cache.AttachToDB(db)
var users []User
db.Where("value > ?", 123).Find(&users) // search cache not hit, objects cached
db.Where("value > ?", 123).Find(&users) // search cache hit
db.Where("id IN (?)", []int{1, 2, 3}).Find(&users) // primary key cache not hit, users cached
db.Where("id IN (?)", []int{1, 3}).Find(&users) // primary key cache hit
var values = make([]int, 0)
db.Model(&User{}).Where("value >= ?", 100).Pluck("value", &values)
fmt.Println("values:", values)
}
复现代码说明:和README例子几乎一样,只是使用了Pluck,查询结果是int切片。
报错内容:
panic: reflect: call of reflect.Value.Field on int Value
reflect.Value.Field({0x100c1fb40?, 0x14000232000?, 0x147d2e3b0?}, 0x101140a68?)
.../go/go1.21.0/src/reflect/value.go:1278 +0xec
gorm.io/gorm/schema.(*Field).setupValuerAndSetter.func2({0x14000216198?, 0x14000092528?}, {0x100c1fb40?, 0x14000232000?, 0x100ca1c20?})
.../go/pkg/mod/gorm.io/[email protected]/schema/field.go:454 +0xb8
github.com/Pacific73/gorm-cache/cache.getObjectsAfterLoad(0x1400009ae10)
.../go/pkg/mod/github.com/!pacific73/[email protected]/cache/helpers.go:245 +0x280
github.com/Pacific73/gorm-cache/cache.(*Gorm2Cache).Initialize.AfterQuery.func5(0x1400009ae10)
.../go/pkg/mod/github.com/!pacific73/[email protected]/cache/after_query.go:31 +0x100
gorm.io/gorm.(*processor).Execute(0x140001e41e0, 0x140001f04b0?)
.../go/pkg/mod/gorm.io/[email protected]/callbacks.go:130 +0x300
gorm.io/gorm.(*DB).Pluck(0x140001f0480?, {0x100b504ea, 0x5}, {0x100c18fc0?, 0x14000092528})
.../go/pkg/mod/gorm.io/[email protected]/finisher_api.go:568 +0x260
main.main()
.../main.go:51 +0x4f0
错误原因说明:
尝试对一个int使用反射来获取字段,但整数是一个基础类型,没有字段。使用Pluck取所有基础类型都会panic。
Hi @Pacific73
I am studying gorm-cache, I noticed there are two tier caches: one is remote Redis, another is local memory.
Why design a two-tier cache? This makes the design more complex, will it improve performance?
Can I use the remote Redis only ? or use the local memory only?
Thank you.
Hi @Pacific73
Would you like to upgrade some dependencies to latest version, they are a bit old. Thank you.
go-redis: v6-->v8
ccache: v2-->v3
time.Duration(r.ttl)*time.Microsecond
应该是 Millisecond 而不是 Microsecond
Hi @Pacific73
Can gorm-chche support Tenant?
For example: there are many tenant in my system, they can access the same Table:task with different product Id. When Tenant A list task list A with product A Id, if Tenant B update his task B, this operation will not remove task list A in Cache.
In another word, A tenant Create/Update/Delete operations should only affect cached data associated with that tenant, can't affect other tenant cache data.
Thank you.
gorm-cache/cache/before_query.go
Line 28 in 1d0b034
现在的代码实现里,在 before_query 查 key 是否存在,在 after_query 里查 value,感觉会出现一种情况,查 key 的时候查到了,然后缓存过期了,再查 value 的时候失败了。
为什么不在 before_query 的时候直接查,放到 context 里传下去呢,还可以少一次查询操作,是我哪里没考虑到吗
实际使用情况是分库使用的
库1 : db_1.test表
库2 : db_2.test表
...
这个会不断扩展
但是都是从同一个gorm的对象去操作的,使用之前,根据业务,db.Table(tableName),去设置哪个库和表名
问题1:当我查询db_1和db_2的表数据,缓存的,是分开2份缓存吗?不会串数据吧
问题2:当我要InvalidateSearchCache去删除缓存的时候,我怎么精准的删除db_2的test表缓存?而不影响db_1.test表?
查询不存在记录时,会一直无法命中缓存。建议加一个选项开启后。不存在数据时,也缓存一下。
使用内存缓存,分页查询缓存不生效,追进去发现SearchKeyExists总视返回false,查询结果也相同
cache, _ := cache.NewGorm2Cache(&cacheConfig.CacheConfig{
CacheLevel: cacheConfig.CacheLevelAll,
CacheStorage: cacheConfig.CacheStorageMemory,
CacheSize: 5000,
InvalidateWhenUpdate: true, // when you create/update/delete objects, invalidate cache
CacheTTL: 60000, // 60000 ms
CacheMaxItemCnt: 60000, // if length of objects retrieved one single time
DebugMode: true,
// exceeds this number, then don't cache
})
2022-09-09 05:44:32.635 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:32.635 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:32.907 [INFO] [AfterQuery] start to set primary cache for kvs: []
2022-09-09 05:44:32.907 [INFO] [AfterQuery] start to set search cache for sql: SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL
2022-09-09 05:44:32.907 [INFO] [AfterQuery] set cache: 37
2022-09-09 05:44:32.907 [INFO] [AfterQuery] sql SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL cached
22-09-09 05:44:32 [DEBU] SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL [272.238116ms]
2022-09-09 05:44:32.907 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:32.907 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:33.318 [INFO] [AfterQuery] start to set search cache for sql: SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc
2022-09-09 05:44:33.319 [INFO] [AfterQuery] start to set primary cache for kvs: [{Key:0x40000b7258 Value:{响应内容}]
2022-09-09 05:44:33.319 [INFO] [AfterQuery] sql SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc cached
22-09-09 05:44:33 [DEBU] SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc [411.897497ms]
22-09-09 05:44:33 [INFO] 223.104.103.141 "POST /api/p/order/page" 200 (691ms)
2022-09-09 05:44:40.891 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:40.891 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:40.95 [INFO] [AfterQuery] start to set primary cache for kvs: []
2022-09-09 05:44:40.95 [INFO] [AfterQuery] start to set search cache for sql: SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL
2022-09-09 05:44:40.95 [INFO] [AfterQuery] set cache: 37
2022-09-09 05:44:40.95 [INFO] [AfterQuery] sql SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL cached
22-09-09 05:44:40 [DEBU] SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL [59.785883ms]
2022-09-09 05:44:40.95 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:40.95 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:41.091 [INFO] [AfterQuery] start to set search cache for sql: SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc
2022-09-09 05:44:41.092 [INFO] [AfterQuery] start to set primary cache for kvs: [{Key:0x400050b2a8 Value:{响应内容}}]
2022-09-09 05:44:41.092 [INFO] [AfterQuery] sql SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc cached
22-09-09 05:44:41 [DEBU] SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc [141.459062ms]
22-09-09 05:44:41 [INFO] 223.104.103.141 "POST /api/p/order/page" 200 (203ms)
gorm.io/gorm v1.23.8
github.com/Pacific73/gorm-cache v1.0.1-0.20220620032759-a6797d3696a5
v1.0.0 没有实现Plugin接口,用hash直接加载的合并了Plugin接口的pr
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.