pacific73 / gorm-cache Goto Github PK

View Code? Open in Web Editor NEW

99.0 99.0 26.0 57 KB

gorm v2的即插即用、无需修改代码的旁路缓存。An easy-to-use look-aside cache solution for gorm v2 users.

License: Apache License 2.0

Go 99.89% Shell 0.11%

cache golang gorm

gorm-cache's People

Contributors

Stargazers

Watchers

gorm-cache's Issues

Redis SetKey 单位错了

time.Duration(r.ttl)*time.Microsecond
应该是 Millisecond 而不是 Microsecond

并发情况下处理耗时慢慢累积，最终超时

版本v1.1.0

使用内存缓存，如下配置

       cache.NewGorm2Cache(&config.CacheConfig{
		CacheLevel:           config.CacheLevelAll,
		CacheStorage:         config.CacheStorageMemory,
		InvalidateWhenUpdate: true,  // when you create/update/delete objects, invalidate cache
		CacheTTL:             10000, // 10s
		CacheSize:            5000,
		// exceeds this number, then don't cache
	}

AfterUpdate方法耗时会慢慢累积，最终会导致数据读写操作均超时

以下是go trace看到的结果

首先可以看到AfterUpdate是耗时大户

大量的goroutine在AfterUpdate

scheduler wait

可以看到阻塞在写channel

对应的代码

这里为什么采用deletables chan *Item这样的设计？

before_query 查到缓存 key 存在，可能会在 after_query 前过期

gorm-cache/cache/after_query.go

Line 101 in 1d0b034

cacheValue, err := cache.GetSearchCache(ctx, tableName, sql, vars...)

缓存有过期时间，如果碰巧在 after_query 前过期，那就会查不到

严重问题：使用Pluck查询会导致panic

创建测试数据：

CREATE TABLE `testdb`.`users`  (
  `id` int NOT NULL,
  `name` varchar(255) NOT NULL,
  `value` int NOT NULL,
  PRIMARY KEY (`id`, `id`)
);

INSERT INTO `users` (`id`, `name`, `value`) VALUES (1, 'hello', 100);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (2, 'abc', 123);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (3, 'xyz', 8);

复现代码：

package main

import (
	"fmt"
	"github.com/Pacific73/gorm-cache/cache"
	"github.com/Pacific73/gorm-cache/config"
	"github.com/go-redis/redis/v8"
	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

type User struct {
	ID    int
	Name  string
	Value int
}

func main() {
	dsn := "user:pass@tcp(127.0.0.1:3306)/testdb?charset=utf8mb4"
	db, _ := gorm.Open(mysql.Open(dsn), &gorm.Config{})

	redisClient := redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})

	redisCache, err := cache.NewGorm2Cache(&config.CacheConfig{
		CacheLevel:           config.CacheLevelAll,
		CacheStorage:         config.CacheStorageRedis,
		RedisConfig:          cache.NewRedisConfigWithClient(redisClient),
		InvalidateWhenUpdate: true,  // when you create/update/delete objects, invalidate cache
		CacheTTL:             5000, // 5000 ms
		CacheMaxItemCnt:      5,     // if length of objects retrieved one single time
		// exceeds this number, then don't cache
	})
	if err != nil {
		panic(err)
	}
	// More options in `config.config.go`
	db.Use(redisCache) // use gorm plugin
	// cache.AttachToDB(db)

	var users []User

	db.Where("value > ?", 123).Find(&users) // search cache not hit, objects cached
	db.Where("value > ?", 123).Find(&users) // search cache hit

	db.Where("id IN (?)", []int{1, 2, 3}).Find(&users) // primary key cache not hit, users cached
	db.Where("id IN (?)", []int{1, 3}).Find(&users)    // primary key cache hit

	var values = make([]int, 0)
	db.Model(&User{}).Where("value >= ?", 100).Pluck("value", &values)

	fmt.Println("values:", values)
}

复现代码说明：和README例子几乎一样，只是使用了Pluck，查询结果是int切片。

报错内容：

panic: reflect: call of reflect.Value.Field on int Value

reflect.Value.Field({0x100c1fb40?, 0x14000232000?, 0x147d2e3b0?}, 0x101140a68?)
        .../go/go1.21.0/src/reflect/value.go:1278 +0xec
gorm.io/gorm/schema.(*Field).setupValuerAndSetter.func2({0x14000216198?, 0x14000092528?}, {0x100c1fb40?, 0x14000232000?, 0x100ca1c20?})
        .../go/pkg/mod/gorm.io/[email protected]/schema/field.go:454 +0xb8
github.com/Pacific73/gorm-cache/cache.getObjectsAfterLoad(0x1400009ae10)
        .../go/pkg/mod/github.com/!pacific73/[email protected]/cache/helpers.go:245 +0x280
github.com/Pacific73/gorm-cache/cache.(*Gorm2Cache).Initialize.AfterQuery.func5(0x1400009ae10)
        .../go/pkg/mod/github.com/!pacific73/[email protected]/cache/after_query.go:31 +0x100
gorm.io/gorm.(*processor).Execute(0x140001e41e0, 0x140001f04b0?)
        .../go/pkg/mod/gorm.io/[email protected]/callbacks.go:130 +0x300
gorm.io/gorm.(*DB).Pluck(0x140001f0480?, {0x100b504ea, 0x5}, {0x100c18fc0?, 0x14000092528})
        .../go/pkg/mod/gorm.io/[email protected]/finisher_api.go:568 +0x260
main.main()
        .../main.go:51 +0x4f0

错误原因说明：

尝试对一个int使用反射来获取字段，但整数是一个基础类型，没有字段。使用Pluck取所有基础类型都会panic。

Can gorm-cache support Raw SQL operations?

Hi @Pacific73
Gorm-cache is a great project, but it can't support Raw SQL operations, may I get the reason?

Is there any plan to support Raw SQL operations in feature? Thank you.

有没有什么办法可以指定缓存Key前缀

因为所有的数据混杂在同个Redis实例里面，所以希望可以指定Redis的Key前缀

redis为啥不共享存储空间嘞以及redis client 可否直接支持传入 go-redis

实际使用情况是分库使用的
库1 ： db_1.test表
库2 ： db_2.test表
...
这个会不断扩展
但是都是从同一个gorm的对象去操作的，使用之前，根据业务，db.Table(tableName)，去设置哪个库和表名
问题1:当我查询db_1和db_2的表数据，缓存的，是分开2份缓存吗？不会串数据吧
问题2:当我要InvalidateSearchCache去删除缓存的时候，我怎么精准的删除db_2的test表缓存？而不影响db_1.test表？

如果直接传 cache key 是不是也行

gorm-cache/cache/after_query.go

Lines 23 to 26 in 229ea3b

 sqlObj, _ := db.InstanceGet("gorm:cache:sql") 

 sql := sqlObj.(string) 

 varObj, _ := db.InstanceGet("gorm:cache:vars") 

 vars := varObj.([]interface{})

看了下 sql 和 vars 好像只用在打印日志和生成 key 上，如果直接在 before_query 拼接好 key 然后传过去呢，个人想法哈

Upgrade dependencies: go-redis and ccache

Hi @Pacific73
Would you like to upgrade some dependencies to latest version, they are a bit old. Thank you.

go-redis: v6-->v8
ccache: v2-->v3

请问怎么主动删除某个缓存数据

我们这边系统是分布式的，存在一个，其他服务更改了缓存数据，广播通知其他服务更新缓存

请问怎么主动删除某个表的缓存数据呢

why use intanceId?

different process use diffrent instanceId, is it a waste of memory?

Why there are two tiers for cache?

Hi @Pacific73
I am studying gorm-cache, I noticed there are two tier caches: one is remote Redis, another is local memory.
Why design a two-tier cache? This makes the design more complex, will it improve performance?

Can I use the remote Redis only ? or use the local memory only?

Thank you.

获取缓存的疑问

gorm-cache/cache/before_query.go

Line 28 in 1d0b034

 keyExists, err := cache.SearchKeyExists(ctx, tableName, sql, db.Statement.Vars...) 

现在的代码实现里，在 before_query 查 key 是否存在，在 after_query 里查 value，感觉会出现一种情况，查 key 的时候查到了，然后缓存过期了，再查 value 的时候失败了。

为什么不在 before_query 的时候直接查，放到 context 里传下去呢，还可以少一次查询操作，是我哪里没考虑到吗

How to support Tenant?

Hi @Pacific73
Can gorm-chche support Tenant?

For example: there are many tenant in my system, they can access the same Table:task with different product Id. When Tenant A list task list A with product A Id, if Tenant B update his task B, this operation will not remove task list A in Cache.

In another word, A tenant Create/Update/Delete operations should only affect cached data associated with that tenant, can't affect other tenant cache data.

Thank you.

使用内存缓存，分页查询中search key exists总是返回false

问题描述

使用内存缓存，分页查询缓存不生效，追进去发现SearchKeyExists总视返回false，查询结果也相同

缓存配置

cache, _ := cache.NewGorm2Cache(&cacheConfig.CacheConfig{
	CacheLevel:           cacheConfig.CacheLevelAll,
	CacheStorage:         cacheConfig.CacheStorageMemory,
	CacheSize:            5000,
	InvalidateWhenUpdate: true,  // when you create/update/delete objects, invalidate cache
	CacheTTL:             60000, // 60000 ms
	CacheMaxItemCnt:      60000, // if length of objects retrieved one single time
	DebugMode:            true,
	// exceeds this number, then don't cache
	})

日志

2022-09-09 05:44:32.635 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:32.635 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:32.907 [INFO] [AfterQuery] start to set primary cache for kvs: []
2022-09-09 05:44:32.907 [INFO] [AfterQuery] start to set search cache for sql: SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL
2022-09-09 05:44:32.907 [INFO] [AfterQuery] set cache: 37
2022-09-09 05:44:32.907 [INFO] [AfterQuery] sql SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL cached
22-09-09 05:44:32 [DEBU] SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL [272.238116ms]
2022-09-09 05:44:32.907 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:32.907 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:33.318 [INFO] [AfterQuery] start to set search cache for sql: SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc
2022-09-09 05:44:33.319 [INFO] [AfterQuery] start to set primary cache for kvs: [{Key:0x40000b7258 Value:{响应内容}]
2022-09-09 05:44:33.319 [INFO] [AfterQuery] sql SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  cached
22-09-09 05:44:33 [DEBU] SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  [411.897497ms]
22-09-09 05:44:33 [INFO] 223.104.103.141 "POST /api/p/order/page" 200 (691ms)
2022-09-09 05:44:40.891 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:40.891 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:40.95 [INFO] [AfterQuery] start to set primary cache for kvs: []
2022-09-09 05:44:40.95 [INFO] [AfterQuery] start to set search cache for sql: SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL
2022-09-09 05:44:40.95 [INFO] [AfterQuery] set cache: 37
2022-09-09 05:44:40.95 [INFO] [AfterQuery] sql SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL cached
22-09-09 05:44:40 [DEBU] SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL [59.785883ms]
2022-09-09 05:44:40.95 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:40.95 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:41.091 [INFO] [AfterQuery] start to set search cache for sql: SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc
2022-09-09 05:44:41.092 [INFO] [AfterQuery] start to set primary cache for kvs: [{Key:0x400050b2a8 Value:{响应内容}}]
2022-09-09 05:44:41.092 [INFO] [AfterQuery] sql SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  cached
22-09-09 05:44:41 [DEBU] SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  [141.459062ms]
22-09-09 05:44:41 [INFO] 223.104.103.141 "POST /api/p/order/page" 200 (203ms)

基本信息

gorm.io/gorm v1.23.8
github.com/Pacific73/gorm-cache v1.0.1-0.20220620032759-a6797d3696a5

v1.0.0 没有实现Plugin接口，用hash直接加载的合并了Plugin接口的pr

为什么这里需要删两遍呢

gorm-cache/cache/cache.go

Line 120 in 00b422b

return c.cache.CleanCache(ctx)

Json tag冲突的问题

I believe that you can read Chinese without barriers, if you have any difficulty in understanding the following, please let me know

通常golang的习惯，json tag和gorm tag都会写在同一个struct里

我希望给用户交付的Json需要忽略某些字段的时候就会在struct里设置json:"-"的tag来忽略这个字段，但是同时这个字段又是要参与逻辑的，比如password，salt之类的

同时这个struct也是gorm要返回的，这时由于cache的序列化是基于encoding/json的，就会同时忽略json:"-"的字段，也就是逻辑判断字段会少内容

解决方案：我目前的做法是引入jsoniter来替换掉encoding/json，因为jsoniter允许指定tagkey，并且jsoniter号称性能也比encoding/json要快，询问一下这个变更是否适合合并入主线

查询不存在记录时，会一直无法命中缓存。建议加一个选项。

查询不存在记录时，会一直无法命中缓存。建议加一个选项开启后。不存在数据时，也缓存一下。

请问下，如果查询表中其中几列也会走缓存么？

What's the difference between Search cache and Primary cache?

Hi @Pacific73
I noticed that there are four Cache level:

const (
	CacheLevelOff         CacheLevel = 0
	CacheLevelOnlyPrimary CacheLevel = 1
	CacheLevelOnlySearch  CacheLevel = 2
	CacheLevelAll         CacheLevel = 3
)

What's the difference between Search cache and Primary cache, or All cache? Thanks a lot.

关于CacheMaxItemCnt

// CacheMaxItemCnt for given query, if objects retrieved are more than this cnt,
// then we choose not to cache for this query. 0 represents caching all queries.
CacheMaxItemCnt int64

注释里写着0表示缓存所有，但是判断的时候好像没特殊判断0。
// cache search data
if int64(len(objects)) > cache.Config.CacheMaxItemCnt {
	return
}

使用缓存后Count操作会异常

Count操作会异常，原因是gorm的Count的返回类型是int64，会提示反射类型是uint，直接异常

panic: reflect: call of reflect.Value.Field on uint Value

goroutine 29 [running]:
reflect.Value.Field({0xdd5ce0, 0xc0004b4600, 0x187}, 0x0)
C:/Program Files/Go/src/reflect/value.go:1278 +0x269
gorm.io/gorm/schema.(*Field).setupValuerAndSetter.func2({0xf95ba8, 0x157aca0}, {0xdd5ce0, 0xc0004b4600, 0x187})
C:/Users/Administrator/go/pkg/mod/gorm.io/[email protected]/schema/field.go:454 +0x117
github.com/Pacific73/gorm-cache/cache.getObjectsAfterLoad(0xc000212690)
C:/Users/Administrator/go/pkg/mod/github.com/!pacific73/[email protected]/cache/helpers.go:249 +0x3d6
github.com/Pacific73/gorm-cache/cache.AfterQuery.func1(0xc000212690)
C:/Users/Administrator/go/pkg/mod/github.com/!pacific73/[email protected]/cache/after_query.go:31 +0x314

	sqlObj, _ := db.InstanceGet("gorm:cache:sql")
	sql := sqlObj.(string)
	varObj, _ := db.InstanceGet("gorm:cache:vars")
	vars := varObj.([]interface{})

pacific73 / gorm-cache Goto Github PK

gorm-cache's People

Contributors

Stargazers

Watchers

Forkers

gorm-cache's Issues

问题描述

缓存配置

基本信息

Recommend Projects

Recommend Topics

Recommend Org