psychic-api / psychic Goto Github PK

Data integration platform for LLMs. Connect to SaaS tools with turnkey auth and sync documents from N data sources with only one integration

License: GNU Affero General Public License v3.0

Python 33.51% HTML 0.43% TypeScript 64.40% Dockerfile 0.26% Makefile 0.06% JavaScript 0.46% CSS 0.88%

chatbot conversational-ai openai chatgpt etl-pipelines langchain weaviate

psychic's Introduction

Data integration platform for LLMs

Psychic cloud is deprecated as of October 1, 2023 due to lack of demand, and the original authors are moving onto a new project. If you are interested in helping maintain or improve the open source project, email [email protected]. If you need support, try the public slack channel.

Psychic is an open source integration platform to extract and transform unstructured data from SaaS applications like Notion, Slack, Zendesk, Confluence, and Google Drive. Instead of building one integration for each data sources, you can build one integration that works for all data sources. Psychic is designed for startups that use LLMs and need to retrieve documents to use as context.

There are 3 parts of the platform

🪄 Psychic Link: A modal that lets end users to connect their data sources with a point and click interface, using OAuth. Available as an NPM package for React projects, or as a magic link.
🪢 Universal API: Abstractions to make it easy to ingest Document and Conversation data from any connected source through a single API endpoint. Filter based on connector_id and account_id. New integrations can be added by implementing a new DataConnector in the server-side code.
🎩 Psychic Dashboard: A front-end for the Psychic platform that makes it easy to manage connections and connect data sources without having to add Link to an existing app.

Read the docs

Demo

Using Psychic to connect data from a Notion workspace with a LangChain question and answer app.

Psychic.Twitter.Demo.May.21.2023.1225.PM.mp4

Get an API key to test out the cloud version by creating an account on the Psychic dashboard.

If you have any questions on how to get started, come join our Slack community!.

Roadmap

✅ LangChain DataLoader
✅ Webhook support to push data to an endpoint at predetermined intervals
🚧 CRM integrations (Salesforce, HubSpot, etc)
🚧 Support for write scopes for each connector, in addition to read scopes

Getting Started - 15 min

Check out the quickstart tutorial to get started.

Contributing

See CONTRIBUTING.md

psychic's People

Contributors

Stargazers

Watchers

Forkers

aspenmayer mmirman teddarific osub imomin itskrishna5 touhonoob kurtseifried neha-nupoor aaronbesson tmanager22 johnwojewoda dolaradesara arcreative echo719 touristshaun ml-sketch gitkaran send2cloud ghbook epicsandals tribe-health cav71 vijaykramesh deniska83 rsohlot itsharex pablongo woakes070048 mrtnbm etarley zakorainc dev1ender davidnhani kyrolabs wdshin andrewkuo t8840 ensoul-labs psychic-api aicodehunt sorokinvld alexanderdaly gardner kannangce sammindinventory jaytoday jgitsol stvhanna jamesthesnake inscapist fran-cois oytuntez jjhw skywolfdreamer rio100 petermathews linecode probhavesh rossman22590 adrianwedd neria05 sangkyunyoon positioner jfontestad cczhgit ibrahimmohammed47 jcontre905 meryemchafry typefrost javelo ambardhesi adam-s-tech 2251162071 orderofmagnitude1 kylebrussell a2o-ai kolorowemozgiai sweepai-dev gavinchen1314 ramstorage kkpan11 demandos selalipop williamtran29 princerumi shaowentian bmedi codehornets elischleifer sycomix polya20 raunaqkapoor radientbrain lgs cosmoras reanimatedmanx rootint bellyfat tomchapin

psychic's Issues

Add an integration for Linear

add a connector to ingest Issues from Linear as Conversations in Psyhic

Add a connector to server/connectors
Test locally, no need to make changes to the Link modal

Create a query page in the dashboard

The dashboard should have a page that allows users to query the vector store without using FastAPI. FastAPI's UI is janky and confusing.

Users should be able to query the vector store directly using the /query endpoint to get back matching DocumentChunks.
Users should be able to query the /ask-llm endpoint
The bearer token of each request should be fetched from supabase based on the user's Clerk ID. This is because we use a single Weaviate instance to host all customers data on the cloud version, and we need to be able to filter by the tenant.

Add Google Drive Connector

Add a connector for Google Drive.

Features for release v0.01:

Connect to Google Drive via OAuth
Get all Google Docs in a given folder, chunk them, and upsert into vectorstore

Integrate with Litellm

I believe, it's a great idea to incorporate the Litellm as it already supports Intergrations with 100+ LLMs.

https://github.com/BerriAI/litellm

LLM as drop-in replacement for gpt-3.5-turbo. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs)

cc @ishaan-jaff

Feature request: Supabase/Postgres datastore

one of the most common issues we see at supabase is developers getting stuck figuring out how to convert their content to embeddings (so you're working on a valuable problem!)

Would you be interested in adding support for Supabase as a datastore? I think it would help our community a lot. Happy to collaborate on this too.

More context:

Postgres supports vectors through pgvector.
Supabase has support for pgvector

Connectors allows users to click "Connect" without authorizing first

The "Connect" button for all Connectors is enabled as soon as the config fields are provided, even if the user hasn't authorized with the source application yet. This always errors out.

Correct behavior: the Connect button should be disabled until the user successfully authorizes with the source application AND provides all the configuration fields.

Is there a way to use OpenSource LLMs with Sidekick ?

How to obtain GDRIVE_CLIENT_SECRETS ?

In the google-docs data connector, the GDRIVE_CLIENT_SECRETS is read in as environment variable but I am not sure where it's set or to what should a self-hosting user set it to?

Also, as of now, to authenticate using OAuth2, user has to click on "Advanced" and then proceed. But once when you register your google-doc OAuth access endpoint with Google (after which we needn't use the "Advanced" option), will the self-hosting users also get the same benefit but with full data privacy (which is a matter of great concern when dealing with google-drive contents)?

Pass server-side error messages through to the API response

Right now any exceptions that occur server-side are not passed through to the API response. Instead, all server-side errors return error code 500 with no additional details. This can be frustrating for errors that are easily resolved, for example expired API keys or invalid values for certain parameters.

We should pass all server-side error messages through to the API response, and use the appropriate error codes depending on the nature of the exception.

Make google drive connector accept a URL instead of a path

Right now, the Google Drive connector requires users input a path to the directory they want to ingest. This is unintuitive and prone to error.

Instead, it should accept a URL to the directory. This is much more intuitive for users since they probably already have that folder open in a tab and can simply copy/paste the URL instead of figuring out what the path should be.

We can then parse the URL to determine the ID of the folder.

supabase error?

i run the server in my .env i set :
DATASTORE=weaviate
BEARER_TOKEN=***************************************************
OPENAI_API_KEY=sk-***************************************************
WEAVIATE_HOST=https://sidekickasasasasasas.weaviate.network
WEAVIATE_PORT=443
WEAVIATE_INDEX=sveltekitandother

and i got this errors :

This package has been deprecated and is no longer maintained, poetry-dotenv has moved to poetry-plugin-dotenv (https://pypi.org/project/poetry-plugin-dotenv)
2023-04-26 19:36:24.458 | DEBUG | datastore.providers.weaviate_datastore:init:126 - Connecting to weaviate instance at https://sidekick2-qdgbaiux.weaviate.network:443 with credential type NoneType
2023-04-26 19:36:26.175 | DEBUG | datastore.providers.weaviate_datastore:init:141 - Found index sveltekitandother with properties {'source_id', 'source_type', 'text', 'tenant_id', 'url', 'document_id', 'title'}
2023-04-26 19:36:26.175 | DEBUG | datastore.providers.weaviate_datastore:init:144 - Will reuse this schema
INFO: Will watch for changes in these directories: ['/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server']
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: Started reloader process [73594] using StatReload
2023-04-26 19:36:28.183 | DEBUG | datastore.providers.weaviate_datastore:init:126 - Connecting to weaviate instance at https://sidekick2-qdgbaiux.weaviate.network:443 with credential type NoneType
2023-04-26 19:36:29.611 | DEBUG | datastore.providers.weaviate_datastore:init:141 - Found index sveltekitandother with properties {'text', 'title', 'document_id', 'url', 'source_type', 'source_id', 'tenant_id'}
2023-04-26 19:36:29.612 | DEBUG | datastore.providers.weaviate_datastore:init:144 - Will reuse this schema
INFO: Started server process [73606]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 127.0.0.1:51289 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51292 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51315 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51316 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51317 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51318 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51319 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51320 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51322 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51345 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51346 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51348 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51349 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51350 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51399 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51400 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51402 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51405 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51406 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51411 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51412 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51420 - "GET /docs HTTP/1.1" 200 OK
INFO: 127.0.0.1:51420 - "GET /openapi.json HTTP/1.1" 200 OK
INFO: 127.0.0.1:51426 - "POST /query HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/applications.py", line 271, in call
await super().call(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 118, in call
await self.middleware_stack(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
solved_result = await solve_dependencies(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 545, in solve_dependencies
solved = await run_in_threadpool(call, **sub_values)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/server/main.py", line 78, in validate_token
app_config = StateStore().get_config(credentials.credentials)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/appstatestore/statestore.py", line 20, in init
self.supabase = create_client(supabase_url, supabase_key)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 248, in create_client
return Client(supabase_url=supabase_url, supabase_key=supabase_key, options=options)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 45, in init
raise SupabaseException("supabase_url is required")
supabase.client.SupabaseException: supabase_url is required
INFO: 127.0.0.1:51427 - "POST /query HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/applications.py", line 271, in call
await super().call(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 118, in call
await self.middleware_stack(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
solved_result = await solve_dependencies(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 545, in solve_dependencies
solved = await run_in_threadpool(call, **sub_values)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/server/main.py", line 78, in validate_token
app_config = StateStore().get_config(credentials.credentials)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/appstatestore/statestore.py", line 20, in init
self.supabase = create_client(supabase_url, supabase_key)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 248, in create_client
return Client(supabase_url=supabase_url, supabase_key=supabase_key, options=options)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 45, in init
raise SupabaseException("supabase_url is required")
supabase.client.SupabaseException: supabase_url is required
INFO: 127.0.0.1:51480 - "POST /upsert-web-data HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/applications.py", line 271, in call
await super().call(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 118, in call
await self.middleware_stack(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
solved_result = await solve_dependencies(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 545, in solve_dependencies
solved = await run_in_threadpool(call, **sub_values)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/server/main.py", line 78, in validate_token
app_config = StateStore().get_config(credentials.credentials)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/appstatestore/statestore.py", line 20, in init
self.supabase = create_client(supabase_url, supabase_key)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 248, in create_client
return Client(supabase_url=supabase_url, supabase_key=supabase_key, options=options)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 45, in init
raise SupabaseException("supabase_url is required")
supabase.client.SupabaseException: supabase_url is required
INFO: 127.0.0.1:51481 - "POST /upsert-web-data HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/applications.py", line 271, in call
await super().call(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 118, in call
await self.middleware_stack(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
solved_result = await solve_dependencies(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 545, in solve_dependencies
solved = await run_in_threadpool(call, **sub_values)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/server/main.py", line 78, in validate_token
app_config = StateStore().get_config(credentials.credentials)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/appstatestore/statestore.py", line 20, in init
self.supabase = create_client(supabase_url, supabase_key)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 248, in create_client
return Client(supabase_url=supabase_url, supabase_key=supabase_key, options=options)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 45, in init
raise SupabaseException("supabase_url is required")
supabase.client.SupabaseException: supabase_url is required
INFO: 127.0.0.1:51482 - "GET / HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:51488 - "POST /upsert-web-data HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/applications.py", line 271, in call
await super().call(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 118, in call
await self.middleware_stack(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 92, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
solved_result = await solve_dependencies(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 545, in solve_dependencies
solved = await run_in_threadpool(call, **sub_values)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/server/main.py", line 78, in validate_token
app_config = StateStore().get_config(credentials.credentials)
File "/Users/ivanvilches/Documents/proyectos/AI/varios openai/9/sidekick-server/appstatestore/statestore.py", line 20, in init
self.supabase = create_client(supabase_url, supabase_key)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 248, in create_client
return Client(supabase_url=supabase_url, supabase_key=supabase_key, options=options)
File "/Users/ivanvilches/Library/Caches/pypoetry/virtualenvs/sidekick-server-lDiM8zSg-py3.10/lib/python3.10/site-packages/supabase/client.py", line 45, in init
raise SupabaseException("supabase_url is required")
supabase.client.SupabaseException: supabase_url is required

Add Support for Pinecone

Enable users to connect Sidekick to their Pinecone instance via the dashboard.

On the cloud version:

The default vectorstore is the Sidekick managed Weaviate instance
Users can go to the destinations section in the dashboard to add their own destinations: either a Pinecone or Weaviate instance.

On the self-hosted version:

Users set env variables with their Pinecone API keys to use pinecone.

Steps:

Add a PineconeDataStore class that conforms to the DataStore interface.

Issue running the install instructions - Python version incompatibilities

$poetry self add poetry-dotenv
Using version ^0.3.0 for poetry-dotenv

Updating dependencies
Resolving dependencies... (0.0s)

The current project's Python requirement (3.7.0) is not compatible with some of the required packages Python requirement:
  - poetry-dotenv requires Python >=3.8,<4.0, so it will not be satisfied for Python 3.7.0

Because no versions of poetry-dotenv match >0.3.0,<0.4.0
 and poetry-dotenv (0.3.0) requires Python >=3.8,<4.0, poetry-dotenv is forbidden.
So, because poetry-instance depends on poetry-dotenv (^0.3.0), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties

    For poetry-dotenv, a possible solution would be to set the `python` property to "<empty>"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers`

I am running Python 3.10 inside a virtual env which is active.

Add Intercom integration

Add support for connecting to Intercom with OAuth auth type.

add new intercom connector to server/connectors
test new connector locally without making changes to the UI

Create Notion DataConnector

Create a Notion connector class that inherits from DataConnector that:

Accepts a Notion access token and workspace_id
Connects to Notion and fetches all pages the access token provides access to
Returns a list of Documents representing Notion pages, with text set to a string blob that contains the page contents as HTML

Self-hosted install

Hello,

I would like to test psychic-api locally, on my computer, but I couldn't find a way to create the necessary tables on the Supabase server in the source. How should I go about doing this?

Additionally, what are the environment variables that I have to configure in addition to those needed by Supabase?

Thanks.

Add obsidian

Cannot sign-up to the Slack

The "Community" link on the doc page (https://sidekick-public.slack.com/) doesn't seem to allow signing up (only signing in).

The "Support" link on the front page (https://join.slack.com/t/sidekick-public/shared_invite/zt-1ra86qug3-~UWNCISLWpNj55Im6C6OaQ) is "no longer active".

Adding Chroma vector DB?

https://www.trychroma.com/ - open source, self hosted, persists to a parquet file. does that help? happy to try to send PR if yes

Add Readme integration

Readme is commonly used to host developer docs. Instead of scraping these documents, it's more effective to fetch them from the API.

If possible, use basic auth instead of API key since it's lower friction. https://docs.readme.com/main/reference/intro/authentication

Implement Qdrant support

Do you plan to implement Qdrant as another data store? We'd be happy to support that.

Create Zendesk connector

Create a Zendesk connector class that inherits from DataConnector that:

Accepts a Zendesk API key and admin email in the constructor
Connects to Zendesk via the API via a load method
Returns a list of Documents representing support tickets (1 support ticket = 1 document)

Enforce token limits for chunkers

Currently, HTMLChunker, MarkdownChunker, and DefaultChunker all accept a token_limit argument that is not used. These chunkers should be modified so that the token limit is respected, and a new chunk is created if the current one exceeds the token limit.

This is not straightforward since it involves deciding how to split a single chunk that exceeds the token limit into 2 chunks. E.g.

How much overlap should there be, if at all, between chunks?
Where should a chunk be split? Between lines/words/sentences?

Google drive refuses to connect

I get this error when connecting my account.

This is using Chrome. In Safari I get no error at all, it just hangs while connecting.

No explanation about guardrails implemented

I came across this article: https://jfan001.medium.com/how-we-cut-the-rate-of-gpt-hallucinations-from-20-to-less-than-2-f3bfcc10e4ec, where you mention about techniques to reduce hallucinations. In the readme.md, you also mention that some guardrails are in place. These are amazing engineering feats to tame GPT3+ but on looking at the code base, I couldn't find the implementation of techniques to reduce hallucinations and the guardrails in place. Please point them!

Docs issue ...

Works great so far, but there is one issue in the docs ... if you try to iterate or access the result from psychic_docs = psychic.get_documents(account_id="account_id", chunked=True) ... you will get this error (in Python SDK at least) ... TypeError: 'GetDocumentsResponse' object is not iterable. The solution is to access the "documents" property on the response instead. psychic_response = psychic.get_documents(account_id="account_id", chunked=True), psychic_docs = psychic_reponse.documents.

Add Apify integration for web scraping

Use Apify's APIs to create connections with we pages and schedule scrapes.

implement a new connector under server/connectors
accept an Apify API key as the credential
No need to implement the UI for this in Link yet

Add dashboard UI

Add a dashboard for no-code usage. v0.01 features:

Authentication and user account management (via Clerk)
Generate API keys for Sidekick

Add Github connector

Add a Github connector that

connects via API key or OAuth
traverses through all markdown files in the repo and returns a list of Document objects

This should be a class in the connectors folder that conforms to DataConnector

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.