Would it be possible to support customised AI model and vector database? about ai-town HOT 2 OPEN

Currently, in order to use a model in local container (or any other model online), you'd have to overwrite the functions here:

https://github.com/a16z-infra/ai-town/blob/main/convex/lib/openai.ts

These are the only LLM entrypoints in the repo, so customising them to use your API is enough.

Similarly with the vector DB, seems like all the relevant functions are here:

https://github.com/a16z-infra/ai-town/blob/main/convex/lib/pinecone.ts

So you can customise queryVectors to query your database instead of pinecone and so on.

You change these two files and I think that's basically it. Now the question is, what can be done to generalise support for custom 3rd parties, so that it's easier and allows people to add integrations?

from ai-town.

Would folks be ok mapping from the OpenAI request type to a custom request, or change all the callsites to use a more generic interface? A few things I'm worried would be lost with the latter:

• separating context into distinct messages, vs. one big prompt
• streaming support
• providing functions (afaik an openai-specific feature)

Anyone have an API they want to propose? Just default LangChain stuff? I've been wary of pulling in LangChain, since I find things more understandable when there's fewer layers of indirection. Even the openai library is hard to find the actual input and output types. I wrote the openai.ts to not have any dependencies or imports, so it's portable and lightweight, but it's also not standardized like LangChain is

from ai-town.