mtybadger / chromaviz Goto Github PK

A package for visualising Chroma vector collections in 3D

License: MIT License

Python 21.97% HTML 4.12% JavaScript 63.41% CSS 10.51%

chromaviz's Introduction

ChromaViz

A package for visualising vector embedding collections as part of the Chroma vector database.

Uses Flask, Vite, and react-three-fiber to host a live 3D view of the data in a web browser, should perform well up to 10k+ documents. Dimensional reduction is performed using PCA for colors down to 50 dimensions, followed by tSNE down to 3.

How to Use

pip install chromaviz or pip install git+https://github.com/mtybadger/chromaviz/. After installing from pip, simply call visualize_collection with a valid ChromaDB collection, and chromaviz will do the rest.

from chromaviz import visualize_collection
visualize_collection(chromadb.Collection)

It also works with Langchain+Chroma, as in:

from langchain.vectorstores import Chroma
vectordb = Chroma.from_documents(data, embeddings, ids)

from chromaviz import visualize_collection
visualize_collection(vectordb._collection)

Screenshots

To-Do

More dimensional reduction options and flexibility
Refactor extremely shoddy React code
Improve UX

chromaviz's People

Contributors

Stargazers

Watchers

chromaviz's Issues

pyproject.toml setup tools out of date

When working in docker, I get the following error message "[_MissingDynamic: license defined outside of pyproject.toml is ignored

If you change "requires = ["setuptools>=61.0"]" to "requires = ["setuptools==68.2.2"]" in pyproject.toml, the error is fixed.

See: https://stackoverflow.com/questions/77523055/missingdynamic-license-defined-outside-of-pyproject-toml-is-ignored

403 Forbidden

I've loaded up the server it seems without error:

However when I hit that url I get 403 forbidden any ideas what I'm doing wrong here?

Unable to install chroma viz package windows 11

Error when trying to install chroma viz package

ERROR: Could not find a version that satisfies the requirement chromaviz (from versions: none)
ERROR: No matching distribution found for chromaviz

Getting requirements to build wheel did not run successfully.

Compatibility Issue with Latest Version of ChromaDB in ChromaViz Library

Dear mtybadger,

I am writing to bring to your attention an issue I encountered while using your ChromaViz library alongside the latest version of ChromaDB.

As you may be aware, ChromaDB has recently released an update that involves changes to its client construction and data persistence methods. After updating to this new version, I found that the visualize_collection function in ChromaViz is still using a deprecated configuration of ChromaDB, causing a ValueError to be thrown.

The error message suggests visiting the ChromaDB migration documentation (https://docs.trychroma.com/migration) to understand how to change the client construction. After following these guidelines, the issue still persists.

Here's a snippet of the code causing the issue:
from chromaviz import visualize_collection visualize_collection(chromadb.Collection)

And the error message:
ValueError: You are using a deprecated configuration of Chroma. ...

I would greatly appreciate it if you could look into this matter and possibly release an update to make ChromaViz compatible with the latest version of ChromaDB.

Thank you for your time and consideration. I look forward to your response.

Best regards, Ander

Can't download from pip?

I can't seem to download this from pip.

> python -m pip install chromaviz 
ERROR: Could not find a version that satisfies the requirement chromaviz (from versions: none)
ERROR: No matching distribution found for chromaviz

I made sure it wasn't an error on my end tried the same on this online terminal and got the same result.

$ pip3 install chromaviz
Defaulting to user installation because normal site-packages is not writeable
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f42cb584520>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/chromaviz/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f42cb584850>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/chromaviz/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f42cb584a00>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/chromaviz/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f42cb584bb0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/chromaviz/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f42cb584d60>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/chromaviz/
ERROR: Could not find a version that satisfies the requirement chromaviz (from versions: none)
ERROR: No matching distribution found for chromaviz

http://127.0.0.1:5000/data returns error code 500 INTERNAL SERVER ERROR

When i try to visualize my collection, the /data endpoint causes a 500 server error:

INFO:werkzeug:127.0.0.1 - - [25/Oct/2023 12:58:07] "GET [/data](https://file+.vscode-resource.vscode-cdn.net/data) HTTP/1.1" 500 -
         0         1         2         3         4         5         6    \
0   0.037332 -0.023781  0.012198 -0.005240  0.076604  0.001155 -0.014276   
1   0.030470 -0.003900 -0.015485 -0.023972  0.063941 -0.060111 -0.014092   
2  -0.053416 -0.008176 -0.002372  0.051473  0.060097  0.035009  0.016631   
3  -0.056681 -0.026865 -0.034603  0.030002  0.042702 -0.028928  0.020197   
4   0.013042 -0.042856 -0.013761  0.036834  0.030365  0.038543 -0.010759   
5   0.005877 -0.042724 -0.027274  0.010114  0.023459 -0.004595 -0.005909   
6  -0.066772  0.006261 -0.047721  0.008414  0.056853 -0.047925 -0.010367   
7  -0.059567 -0.039042  0.000922 -0.001134  0.023288 -0.046869 -0.016602   
8  -0.042325 -0.032608 -0.000407  0.018782  0.053377 -0.056314  0.030714   
9  -0.028975  0.032526 -0.045487  0.055819  0.010403  0.034379  0.063932   
10 -0.048563 -0.029803 -0.039211  0.046758  0.066782 -0.002995  0.059443   
11 -0.078533 -0.034406 -0.018979  0.073978  0.067306  0.023627 -0.016573   
12 -0.040124  0.056453 -0.016754 -0.019866  0.001475 -0.035780 -0.007667   
13  0.018375  0.076316 -0.025173 -0.010658 -0.025128 -0.022667  0.035033   
14  0.030359  0.016650  0.029888 -0.049801 -0.043548  0.065044  0.029868   
15  0.013895  0.032976 -0.025223  0.032685  0.038733  0.007347  0.035505   
16 -0.032058  0.079850  0.034362 -0.065866 -0.036254 -0.070180  0.023393   
17 -0.050679  0.043993 -0.074905 -0.004738 -0.025914 -0.052690  0.034850   
18  0.001749  0.004550 -0.036564 -0.070386 -0.052163 -0.057619  0.035614   
19  0.036063  0.070261 -0.053000 -0.037017 -0.036630  0.010571  0.066405   
20  0.024235  0.063356 -0.054793 -0.029787 -0.020596 -0.071459  0.033064   
21 -0.059592  0.041362 -0.036292  0.007127  0.012279  0.014098 -0.052634   
22 -0.060635  0.073746 -0.043353  0.046178  0.000276 -0.031570  0.018131   
23  0.025379 -0.029765  0.002895 -0.017975 -0.038120  0.061182  0.008354   
24 -0.065203  0.037062 -0.025169 -0.043031 -0.002735 -0.039980  0.004093   
25  0.004894  0.055953  0.011643  0.046046  0.015779  0.065883  0.038230   
26 -0.024362 -0.035757 -0.019241  0.026651  0.039099 -0.065665 -0.016519   
27 -0.027676  0.026229 -0.044375 -0.000928 -0.007377 -0.066107  0.018930   
28 -0.072507  0.018927  0.003620  0.060673  0.030222  0.052429  0.030121   
29  0.054735  0.068607 -0.073256 -0.002494  0.046521 -0.029290  0.022471   
30 -0.037575  0.011722 -0.030505  0.072252  0.074436 -0.019824  0.038547   
31  0.046324  0.012548 -0.041335  0.034979  0.030446  0.023014 -0.023115   
32  0.009296 -0.021602 -0.057132  0.026589 -0.027671 -0.023843 -0.054125   
33 -0.047493 -0.036450 -0.013627  0.052395  0.047732 -0.058966 -0.037612   

         7         8         9    ...       502       503       504       505  \
0   0.008024  0.060044  0.029361  ... -0.009279 -0.023715 -0.015751 -0.013304   
1  -0.010978  0.043185  0.020901  ... -0.005410  0.003037 -0.011462 -0.064343   
2  -0.015547  0.042151  0.037624  ...  0.023804  0.029791  0.024964 -0.040885   
3  -0.013639  0.034665  0.020015  ...  0.006872  0.048773  0.039655 -0.069227   
4  -0.016177  0.068578  0.003459  ... -0.024571  0.062359 -0.006766 -0.036526   
5  -0.027050  0.058815  0.005194  ... -0.025979  0.069981  0.013728 -0.056290   
6  -0.033108 -0.060129  0.007087  ... -0.023777  0.030056  0.005010 -0.052337   
7  -0.017220  0.020616  0.005380  ...  0.014116  0.054109 -0.018877 -0.018004   
8  -0.058906 -0.071288 -0.084879  ...  0.043875  0.021019 -0.010141 -0.073011   
9  -0.015690 -0.089544  0.004032  ...  0.008158  0.077191  0.005024 -0.069949   
10 -0.051377 -0.079748 -0.077363  ... -0.047476  0.024338  0.006369 -0.085023   
11 -0.007470 -0.038817  0.026817  ... -0.071988  0.055395  0.064981  0.030653   
12  0.020591  0.080325  0.060433  ...  0.053331  0.065969 -0.036978 -0.083878   
13 -0.056395  0.054601  0.033762  ...  0.049224  0.047197 -0.053725 -0.028474   
14  0.033852  0.057624 -0.045598  ...  0.000804  0.065920 -0.030025 -0.081059   
15 -0.026954  0.043701  0.016701  ...  0.002854  0.059009 -0.072334 -0.034390   
16 -0.051204  0.077887 -0.048077  ...  0.039129  0.037951 -0.033659  0.012854   
17 -0.011362  0.036235 -0.058835  ...  0.034847  0.072305  0.048777 -0.075487   
18 -0.015401 -0.053733 -0.057567  ...  0.017474  0.069098 -0.049530 -0.075420   
19 -0.022096  0.072510 -0.056408  ...  0.057827  0.062493 -0.038635 -0.046401   
20  0.015191 -0.047425  0.009777  ...  0.044122  0.072265 -0.074245 -0.077606   
21  0.062831  0.032591  0.037691  ...  0.021291  0.060686 -0.021363 -0.023141   
22 -0.019911 -0.069257  0.016237  ...  0.029327  0.066845  0.009949 -0.086336   
23 -0.038988 -0.093563 -0.036053  ...  0.003514  0.070978  0.034271  0.022625   
24 -0.021587 -0.072259 -0.044717  ...  0.030433  0.072023 -0.037791 -0.081257   
25 -0.020064 -0.060841 -0.055939  ...  0.038341  0.010689 -0.084277 -0.079162   
26  0.021258 -0.053852 -0.038993  ... -0.038840  0.024913 -0.050933 -0.074633   
27  0.041727 -0.074194 -0.064412  ...  0.043329  0.064479  0.049853 -0.077119   
28 -0.003505 -0.096430  0.035436  ... -0.002518  0.068902 -0.013336 -0.038727   
29 -0.011793 -0.039794 -0.016975  ... -0.017931  0.049746 -0.030473 -0.059766   
30  0.010207 -0.044357 -0.054522  ...  0.064027  0.024848  0.054487 -0.070881   
31 -0.014810 -0.084757  0.027470  ... -0.087155  0.026196 -0.083433 -0.053757   
32 -0.061350 -0.076431  0.007814  ...  0.056633  0.053737  0.016689  0.026026   
33 -0.050325 -0.074288 -0.044579  ...  0.071013 -0.040185  0.022621 -0.061165   

         506       507       508       509       510       511  
0   0.007775 -0.095998 -0.046793  0.030200  0.045731 -0.027019  
1   0.016610 -0.087086 -0.051602  0.030389  0.050576 -0.020754  
2   0.039993 -0.092495 -0.058261  0.019242  0.042531 -0.030493  
3   0.041741 -0.090846 -0.063663  0.012026  0.042810 -0.027029  
4   0.015731 -0.099377 -0.049119  0.021458  0.022096  0.000982  
5   0.017304 -0.092860 -0.059497  0.024467  0.036667  0.002914  
6   0.026571 -0.086814  0.005377  0.014170  0.040233 -0.012169  
7   0.031023 -0.082871  0.003966  0.007463  0.036262  0.039472  
8   0.026039 -0.086807 -0.044998  0.007627  0.055528 -0.036907  
9   0.052685 -0.088993 -0.023056  0.025650  0.039750  0.005575  
10  0.022355 -0.081196 -0.000355  0.011888  0.037407  0.002402  
11  0.007430 -0.080597  0.009811  0.044425  0.006977 -0.010428  
12  0.058908 -0.076054 -0.000748 -0.018091  0.070045  0.058598  
13  0.047309 -0.060535 -0.035036  0.030780  0.076246  0.051501  
14  0.048645  0.073527  0.032925  0.029199  0.020705  0.075084  
15 -0.025052 -0.071407 -0.048819  0.026808  0.000149  0.037452  
16 -0.083757 -0.071528  0.044818  0.013089 -0.021773  0.052537  
17  0.015151 -0.082091 -0.002808  0.005803  0.048171 -0.060211  
18 -0.006031  0.071764 -0.020130 -0.015293  0.015983  0.026699  
19  0.049587 -0.072828 -0.028956  0.002941  0.022244  0.012457  
20  0.071769 -0.058493 -0.025794  0.005382  0.058536  0.047511  
21  0.000378 -0.085529  0.000841  0.046993  0.046224 -0.028370  
22  0.035358 -0.084977 -0.026532  0.015352  0.052120  0.036663  
23 -0.003215  0.004681  0.068580 -0.013297  0.009633 -0.045675  
24 -0.029970 -0.087881  0.022975  0.003078  0.021798  0.013759  
25  0.073087  0.000363  0.032176  0.015431 -0.035607  0.041946  
26  0.051575 -0.059862  0.026390  0.048598  0.036532  0.041015  
27  0.028671 -0.078497  0.024266  0.048963  0.017075  0.020607  
28 -0.067752 -0.092406  0.032862  0.055852 -0.001208  0.055984  
29  0.043003 -0.068191  0.020234  0.057081 -0.039051  0.028884  
30  0.038409 -0.076699  0.053538  0.017457  0.037722  0.002966  
31  0.017005 -0.087661 -0.003598 -0.023067 -0.009526  0.014693  
32 -0.081903 -0.078386 -0.031118  0.056724 -0.039457  0.038010  
33  0.001368 -0.077690  0.047063  0.054346  0.051049 -0.025174  

[34 rows x 512 columns]
Size of the dataframe: (34, 512)
ERROR:chromaviz.visualize:Exception on [/data](https://file+.vscode-resource.vscode-cdn.net/data) [GET]
Traceback (most recent call last):
  File "/home/linus/.local/lib/python3.10/site-packages/flask/app.py", line 1455, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/linus/.local/lib/python3.10/site-packages/flask/app.py", line 869, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/linus/.local/lib/python3.10/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/linus/.local/lib/python3.10/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/linus/.local/lib/python3.10/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/linus/.local/lib/python3.10/site-packages/chromaviz/visualize.py", line 59, in data_api
    pca_result_50 = pca_50.fit_transform(df)
  File "/home/linus/.local/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/linus/.local/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/linus/.local/lib/python3.10/site-packages/sklearn/decomposition/_pca.py", line 460, in fit_transform
    U, S, Vt = self._fit(X)
  File "/home/linus/.local/lib/python3.10/site-packages/sklearn/decomposition/_pca.py", line 510, in _fit
    return self._fit_full(X, n_components)
  File "/home/linus/.local/lib/python3.10/site-packages/sklearn/decomposition/_pca.py", line 524, in _fit_full
    raise ValueError(
ValueError: n_components=50 must be between 0 and min(n_samples, n_features)=34 with svd_solver='full'

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.