pacific73 / gorm-cache Goto Github PK

gorm v2的即插即用、无需修改代码的旁路缓存。An easy-to-use look-aside cache solution for gorm v2 users.

License: Apache License 2.0

Go 99.89% Shell 0.11%

gorm-cache's Issues

Why there are two tiers for cache?

Hi @Pacific73
I am studying gorm-cache, I noticed there are two tier caches: one is remote Redis, another is local memory.
Why design a two-tier cache? This makes the design more complex, will it improve performance?

Can I use the remote Redis only ? or use the local memory only?

Thank you.

并发情况下处理耗时慢慢累积，最终超时

版本v1.1.0

使用内存缓存，如下配置

       cache.NewGorm2Cache(&config.CacheConfig{
		CacheLevel:           config.CacheLevelAll,
		CacheStorage:         config.CacheStorageMemory,
		InvalidateWhenUpdate: true,  // when you create/update/delete objects, invalidate cache
		CacheTTL:             10000, // 10s
		CacheSize:            5000,
		// exceeds this number, then don't cache
	}

AfterUpdate方法耗时会慢慢累积，最终会导致数据读写操作均超时

以下是go trace看到的结果

首先可以看到AfterUpdate是耗时大户

大量的goroutine在AfterUpdate

scheduler wait

可以看到阻塞在写channel

对应的代码

这里为什么采用deletables chan *Item这样的设计？

What's the difference between Search cache and Primary cache?

Hi @Pacific73
I noticed that there are four Cache level:

const (
	CacheLevelOff         CacheLevel = 0
	CacheLevelOnlyPrimary CacheLevel = 1
	CacheLevelOnlySearch  CacheLevel = 2
	CacheLevelAll         CacheLevel = 3
)

What's the difference between Search cache and Primary cache, or All cache? Thanks a lot.

获取缓存的疑问

gorm-cache/cache/before_query.go

Line 28 in 1d0b034

 keyExists, err := cache.SearchKeyExists(ctx, tableName, sql, db.Statement.Vars...) 

现在的代码实现里，在 before_query 查 key 是否存在，在 after_query 里查 value，感觉会出现一种情况，查 key 的时候查到了，然后缓存过期了，再查 value 的时候失败了。

为什么不在 before_query 的时候直接查，放到 context 里传下去呢，还可以少一次查询操作，是我哪里没考虑到吗

Upgrade dependencies: go-redis and ccache

Hi @Pacific73
Would you like to upgrade some dependencies to latest version, they are a bit old. Thank you.

go-redis: v6-->v8
ccache: v2-->v3

请问怎么主动删除某个缓存数据

我们这边系统是分布式的，存在一个，其他服务更改了缓存数据，广播通知其他服务更新缓存

请问怎么主动删除某个表的缓存数据呢

How to support Tenant?

Hi @Pacific73
Can gorm-chche support Tenant?

For example: there are many tenant in my system, they can access the same Table:task with different product Id. When Tenant A list task list A with product A Id, if Tenant B update his task B, this operation will not remove task list A in Cache.

In another word, A tenant Create/Update/Delete operations should only affect cached data associated with that tenant, can't affect other tenant cache data.

Thank you.

为什么这里需要删两遍呢

gorm-cache/cache/cache.go

Line 120 in 00b422b

return c.cache.CleanCache(ctx)

严重问题：使用Pluck查询会导致panic

创建测试数据：

CREATE TABLE `testdb`.`users`  (
  `id` int NOT NULL,
  `name` varchar(255) NOT NULL,
  `value` int NOT NULL,
  PRIMARY KEY (`id`, `id`)
);

INSERT INTO `users` (`id`, `name`, `value`) VALUES (1, 'hello', 100);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (2, 'abc', 123);
INSERT INTO `users` (`id`, `name`, `value`) VALUES (3, 'xyz', 8);

复现代码：

package main

import (
	"fmt"
	"github.com/Pacific73/gorm-cache/cache"
	"github.com/Pacific73/gorm-cache/config"
	"github.com/go-redis/redis/v8"
	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

type User struct {
	ID    int
	Name  string
	Value int
}

func main() {
	dsn := "user:pass@tcp(127.0.0.1:3306)/testdb?charset=utf8mb4"
	db, _ := gorm.Open(mysql.Open(dsn), &gorm.Config{})

	redisClient := redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})

	redisCache, err := cache.NewGorm2Cache(&config.CacheConfig{
		CacheLevel:           config.CacheLevelAll,
		CacheStorage:         config.CacheStorageRedis,
		RedisConfig:          cache.NewRedisConfigWithClient(redisClient),
		InvalidateWhenUpdate: true,  // when you create/update/delete objects, invalidate cache
		CacheTTL:             5000, // 5000 ms
		CacheMaxItemCnt:      5,     // if length of objects retrieved one single time
		// exceeds this number, then don't cache
	})
	if err != nil {
		panic(err)
	}
	// More options in `config.config.go`
	db.Use(redisCache) // use gorm plugin
	// cache.AttachToDB(db)

	var users []User

	db.Where("value > ?", 123).Find(&users) // search cache not hit, objects cached
	db.Where("value > ?", 123).Find(&users) // search cache hit

	db.Where("id IN (?)", []int{1, 2, 3}).Find(&users) // primary key cache not hit, users cached
	db.Where("id IN (?)", []int{1, 3}).Find(&users)    // primary key cache hit

	var values = make([]int, 0)
	db.Model(&User{}).Where("value >= ?", 100).Pluck("value", &values)

	fmt.Println("values:", values)
}

复现代码说明：和README例子几乎一样，只是使用了Pluck，查询结果是int切片。

报错内容：

panic: reflect: call of reflect.Value.Field on int Value

reflect.Value.Field({0x100c1fb40?, 0x14000232000?, 0x147d2e3b0?}, 0x101140a68?)
        .../go/go1.21.0/src/reflect/value.go:1278 +0xec
gorm.io/gorm/schema.(*Field).setupValuerAndSetter.func2({0x14000216198?, 0x14000092528?}, {0x100c1fb40?, 0x14000232000?, 0x100ca1c20?})
        .../go/pkg/mod/gorm.io/[email protected]/schema/field.go:454 +0xb8
github.com/Pacific73/gorm-cache/cache.getObjectsAfterLoad(0x1400009ae10)
        .../go/pkg/mod/github.com/!pacific73/[email protected]/cache/helpers.go:245 +0x280
github.com/Pacific73/gorm-cache/cache.(*Gorm2Cache).Initialize.AfterQuery.func5(0x1400009ae10)
        .../go/pkg/mod/github.com/!pacific73/[email protected]/cache/after_query.go:31 +0x100
gorm.io/gorm.(*processor).Execute(0x140001e41e0, 0x140001f04b0?)
        .../go/pkg/mod/gorm.io/[email protected]/callbacks.go:130 +0x300
gorm.io/gorm.(*DB).Pluck(0x140001f0480?, {0x100b504ea, 0x5}, {0x100c18fc0?, 0x14000092528})
        .../go/pkg/mod/gorm.io/[email protected]/finisher_api.go:568 +0x260
main.main()
        .../main.go:51 +0x4f0

错误原因说明：

尝试对一个int使用反射来获取字段，但整数是一个基础类型，没有字段。使用Pluck取所有基础类型都会panic。

Can gorm-cache support Raw SQL operations?

Hi @Pacific73
Gorm-cache is a great project, but it can't support Raw SQL operations, may I get the reason?

Is there any plan to support Raw SQL operations in feature? Thank you.

如果直接传 cache key 是不是也行

gorm-cache/cache/after_query.go

Lines 23 to 26 in 229ea3b

 sqlObj, _ := db.InstanceGet("gorm:cache:sql") 

 sql := sqlObj.(string) 

 varObj, _ := db.InstanceGet("gorm:cache:vars") 

 vars := varObj.([]interface{})

看了下 sql 和 vars 好像只用在打印日志和生成 key 上，如果直接在 before_query 拼接好 key 然后传过去呢，个人想法哈

Json tag冲突的问题

I believe that you can read Chinese without barriers, if you have any difficulty in understanding the following, please let me know

通常golang的习惯，json tag和gorm tag都会写在同一个struct里

我希望给用户交付的Json需要忽略某些字段的时候就会在struct里设置json:"-"的tag来忽略这个字段，但是同时这个字段又是要参与逻辑的，比如password，salt之类的

同时这个struct也是gorm要返回的，这时由于cache的序列化是基于encoding/json的，就会同时忽略json:"-"的字段，也就是逻辑判断字段会少内容

解决方案：我目前的做法是引入jsoniter来替换掉encoding/json，因为jsoniter允许指定tagkey，并且jsoniter号称性能也比encoding/json要快，询问一下这个变更是否适合合并入主线

关于CacheMaxItemCnt

// CacheMaxItemCnt for given query, if objects retrieved are more than this cnt,
// then we choose not to cache for this query. 0 represents caching all queries.
CacheMaxItemCnt int64

注释里写着0表示缓存所有，但是判断的时候好像没特殊判断0。
// cache search data
if int64(len(objects)) > cache.Config.CacheMaxItemCnt {
	return
}

before_query 查到缓存 key 存在，可能会在 after_query 前过期

gorm-cache/cache/after_query.go

Line 101 in 1d0b034

cacheValue, err := cache.GetSearchCache(ctx, tableName, sql, vars...)

缓存有过期时间，如果碰巧在 after_query 前过期，那就会查不到

如果分库的情况下如何删除某个具体的库的具体的表的缓存呢？

实际使用情况是分库使用的
库1 ： db_1.test表
库2 ： db_2.test表
...
这个会不断扩展
但是都是从同一个gorm的对象去操作的，使用之前，根据业务，db.Table(tableName)，去设置哪个库和表名
问题1:当我查询db_1和db_2的表数据，缓存的，是分开2份缓存吗？不会串数据吧
问题2:当我要InvalidateSearchCache去删除缓存的时候，我怎么精准的删除db_2的test表缓存？而不影响db_1.test表？

在Redis集群模式下会得到出现Lua脚本加载错误

ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS array, and KEYS should not be in expression

Redis SetKey 单位错了

time.Duration(r.ttl)*time.Microsecond
应该是 Millisecond 而不是 Microsecond

why use intanceId?

different process use diffrent instanceId, is it a waste of memory?

查询不存在记录时，会一直无法命中缓存。建议加一个选项。

查询不存在记录时，会一直无法命中缓存。建议加一个选项开启后。不存在数据时，也缓存一下。

redis为啥不共享存储空间嘞以及redis client 可否直接支持传入 go-redis

两个表链接查询缓存吗还是要单表操作

使用内存缓存，分页查询中search key exists总是返回false

问题描述

使用内存缓存，分页查询缓存不生效，追进去发现SearchKeyExists总视返回false，查询结果也相同

缓存配置

cache, _ := cache.NewGorm2Cache(&cacheConfig.CacheConfig{
	CacheLevel:           cacheConfig.CacheLevelAll,
	CacheStorage:         cacheConfig.CacheStorageMemory,
	CacheSize:            5000,
	InvalidateWhenUpdate: true,  // when you create/update/delete objects, invalidate cache
	CacheTTL:             60000, // 60000 ms
	CacheMaxItemCnt:      60000, // if length of objects retrieved one single time
	DebugMode:            true,
	// exceeds this number, then don't cache
	})

日志

2022-09-09 05:44:32.635 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:32.635 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:32.907 [INFO] [AfterQuery] start to set primary cache for kvs: []
2022-09-09 05:44:32.907 [INFO] [AfterQuery] start to set search cache for sql: SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL
2022-09-09 05:44:32.907 [INFO] [AfterQuery] set cache: 37
2022-09-09 05:44:32.907 [INFO] [AfterQuery] sql SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL cached
22-09-09 05:44:32 [DEBU] SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL [272.238116ms]
2022-09-09 05:44:32.907 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:32.907 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:33.318 [INFO] [AfterQuery] start to set search cache for sql: SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc
2022-09-09 05:44:33.319 [INFO] [AfterQuery] start to set primary cache for kvs: [{Key:0x40000b7258 Value:{响应内容}]
2022-09-09 05:44:33.319 [INFO] [AfterQuery] sql SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  cached
22-09-09 05:44:33 [DEBU] SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  [411.897497ms]
22-09-09 05:44:33 [INFO] 223.104.103.141 "POST /api/p/order/page" 200 (691ms)
2022-09-09 05:44:40.891 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:40.891 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:40.95 [INFO] [AfterQuery] start to set primary cache for kvs: []
2022-09-09 05:44:40.95 [INFO] [AfterQuery] start to set search cache for sql: SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL
2022-09-09 05:44:40.95 [INFO] [AfterQuery] set cache: 37
2022-09-09 05:44:40.95 [INFO] [AfterQuery] sql SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL cached
22-09-09 05:44:40 [DEBU] SELECT count(*) FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL [59.785883ms]
2022-09-09 05:44:40.95 [INFO] [BeforeQuery] search key exists ? false
2022-09-09 05:44:40.95 [INFO] [BeforeQuery] parse primary keys = []
2022-09-09 05:44:41.091 [INFO] [AfterQuery] start to set search cache for sql: SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc
2022-09-09 05:44:41.092 [INFO] [AfterQuery] start to set primary cache for kvs: [{Key:0x400050b2a8 Value:{响应内容}}]
2022-09-09 05:44:41.092 [INFO] [AfterQuery] sql SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = $1 AND "status" NOT IN ($2,$3,$4) AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  cached
22-09-09 05:44:41 [DEBU] SELECT * FROM "peiz"."p_order" WHERE "p_order"."mc_id" = 337644723584954368 AND "status" NOT IN ('start','deleted','panic') AND "p_order"."deleted_at" IS NULL ORDER BY create_time desc  [141.459062ms]
22-09-09 05:44:41 [INFO] 223.104.103.141 "POST /api/p/order/page" 200 (203ms)

基本信息

gorm.io/gorm v1.23.8
github.com/Pacific73/gorm-cache v1.0.1-0.20220620032759-a6797d3696a5

v1.0.0 没有实现Plugin接口，用hash直接加载的合并了Plugin接口的pr

使用缓存后Count操作会异常

Count操作会异常，原因是gorm的Count的返回类型是int64，会提示反射类型是uint，直接异常

panic: reflect: call of reflect.Value.Field on uint Value

goroutine 29 [running]:
reflect.Value.Field({0xdd5ce0, 0xc0004b4600, 0x187}, 0x0)
C:/Program Files/Go/src/reflect/value.go:1278 +0x269
gorm.io/gorm/schema.(*Field).setupValuerAndSetter.func2({0xf95ba8, 0x157aca0}, {0xdd5ce0, 0xc0004b4600, 0x187})
C:/Users/Administrator/go/pkg/mod/gorm.io/[email protected]/schema/field.go:454 +0x117
github.com/Pacific73/gorm-cache/cache.getObjectsAfterLoad(0xc000212690)
C:/Users/Administrator/go/pkg/mod/github.com/!pacific73/[email protected]/cache/helpers.go:249 +0x3d6
github.com/Pacific73/gorm-cache/cache.AfterQuery.func1(0xc000212690)
C:/Users/Administrator/go/pkg/mod/github.com/!pacific73/[email protected]/cache/after_query.go:31 +0x314

有没有什么办法可以指定缓存Key前缀

因为所有的数据混杂在同个Redis实例里面，所以希望可以指定Redis的Key前缀

	sqlObj, _ := db.InstanceGet("gorm:cache:sql")
	sql := sqlObj.(string)
	varObj, _ := db.InstanceGet("gorm:cache:vars")
	vars := varObj.([]interface{})

pacific73 / gorm-cache Goto Github PK

gorm-cache's Issues

问题描述

缓存配置

基本信息

Recommend Projects

Recommend Topics

Recommend Org