Git Product home page Git Product logo

ai-cache-ospp's Introduction

AI-Cache-OSPP

Code for AI Cache Plugin

0716 思路:利用Redis作为缓存,把textEmbeddingProvider和vectorStoreProvider做为可替换部分

  1. 在main函数中实现OnHttpRequestHeaders、OnHttpRequestBody、OnHttpResponseHeaders、OnHttpResponseBody这四个函数,其中只有OnHttpRequestBody需要根据不同的配置进行不同的处理
  2. 具体来说,在OnHttpRequestBody函数里拿到请求的body(query的字符串)后,执行cache.go中的缓存逻辑,首先逐字匹配query和缓存中的key,如果匹配成功则直接返回缓存中的value,否则首先调用textEmbeddingProvider的GetEmbedding方法得到query的embedding,然后调用vectorStoreProvider的QueryEmbedding方法得到query的vector和对应的score
  3. 判断分数后,如果相似度分数小于阈值,则直接返回缓存中的value,否则将query的embedding和vector存入缓存中并resume请求。
  4. 最后在OnHttpResponseBody函数中,将大模型返回的结果存入redis缓存中。

代码框架:

├── cache.go //缓存逻辑,在这里调用textEmbeddingProvider和vectorStoreProvider
├── config
│   └── config.go
├── go.mod
├── go.sum
├── main.go // 主要四个函数的逻辑,缓存部分主要实现还在cache.go中
├── option.yaml
├── README.md
├── textEmbeddingProvider // 类似AI-proxy中的相应配置,主要在provider.go中暴露接口,其他具体实现可以单独写一个go文件
│   ├── dashscope.go
│   └── provider.go
└── vectorStoreProvider // 类似AI-proxy中的相应配置,主要在provider.go中暴露接口,其他具体实现可以单独写一个go文件
    ├── dashvector.go
    └── provider.go

ai-cache-ospp's People

Contributors

suchun-sv avatar

Watchers

Lucian avatar  avatar

ai-cache-ospp's Issues

20240718 Review Notes

  1. 接受后端响应和主动返回响应时要考虑流式请求的场景
  2. 根据 Golang 官方的 Package names 规范:Good package names are short and clear. They are lower case, with no under_scores or mixedCaps. They are often simple nouns.
  3. QueryEmbeddingProvider 是两个独立的接口。那么 CreateProvider(ProviderConfig) (Provider, error) 返回的 Provider 实例理论上是不一定实现了 QueryEmbedding 接口的。这会不会有问题?

TBC

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.