opencsgs / csghub-server Goto Github PK

CSGHub Server is the backend server for CSGHub which helps user to manage datasets, model files, codes and more.　CSGHub Server是开源大模型资产管理平台CSGHub的服务端部分的开源项目，提供基于REST API的模型和数据集等大模型资产管理功能。欢迎关注反馈和Star⭐️

Home Page: https://opencsg.com/models

License: Apache License 2.0

Dockerfile 0.05% Go 95.60% Shell 0.81% PLpgSQL 0.39% Python 3.15%

ai huggingface llm platform datasets golang models

csghub-server's Introduction

English ∙ 简体中文

CSGHub Server is a part of the open source and reliable large model assets management platform - CSGHub. It focuses on management of models、datasets and other LLM assets through REST API。

Key Features：

Creation and Management of users and orgnizations
Auto-tagging of model and dataset labels
Search for users, organizations, models, and data
Online preview of dataset files, like .parquet file
Content moderation for both text and image
Download of individual files, including LFS files
Tracking of model and dataset activity data, such as downloads and likes volume

Demo

In order to help users to quickly understand the features and usage of CSGHub, we have recorded a demo video. You can watch this video to get a quick understanding of the main features and operation procedures of this program.

CSGHub Demo video is as blew，you can also check it at YouTube or Bilibili

csghub-demo-1080p.mp4

Please visit the OpenCSG website to experience the powerful management features.

Quick Start

System resource requirements: 4c CPU/8GB memory

Please install Docker yourself. This project has been tested in Ubuntu22 environment.

You can quickly deploy the localized CSGHub Server service through docker-compose:

# The API token should be at least 128 characters long, and HTTP requests to csghub-server require the API token to be sent as a Bearer token for authentication.
export STARHUB_SERVER_API_TOKEN=<API token>
mkdir -m 777 gitea minio_data
curl -L https://raw.githubusercontent.com/OpenCSGs/csghub-server/main/docker-compose.yml -o docker-compose.yml
docker-compose -f docker-compose.yml up -d

Technical Architecture

Extensible and customizable

Supports different git servers, such as Gitea, GitLab, etc.
Supports flexible configuration of the LFS storage system, and you can choose to use local or any third-party cloud storage service that is compatible with the S3 protocol.
Enable content moderation on demand, and choose any third-party content moderation service.

Roadmap

Support more Git Servers: Currently supports Gitea, and plans to support mainstream Git repositories in the future.
Git LFS: Git LFS supports large files, and supports Git command operations and online download through the Web UI.
DataSet online viewer: Data set preview, supports the Top20/TopN loading preview of LFS format data sets.
Model/Dataset AutoTag: Supports custom metadata and automatic extraction of model/dataset tags.
S3 Protocol Support: Supports S3 (MinIO) storage protocol, providing higher reliability and storage cost-effectiveness.
Model format convert: Conversion of mainstream model formats.
Model oneclick deploy: Supports integration with OpenCSG llm-inference, one-click to start model inference.

License

We use the Apache 2.0 license, the content of which is detailed in the LICENSE file.

Contributing

If you wish to contribute, please follow the Contribution Guidelines. We are very excited about your contributions!

Acknowledgments

This project is based on open source projects such as Gin, DuckDB, minio, and Gitea. We would like to express our sincere gratitude to them for their open source contributions!

CONTACT WITH US

If you meet any problem during usage, you can contact with us by any following way:

initiate an issue in github
join our WeChat group by scaning wechat helper qrcode
join our offical discord channel: OpenCSG Discord Channel
join our slack workspace:OpenCSG Slack Channel

csghub-server's People

Contributors

Stargazers

Watchers

Forkers

crazypangpang kinglywayne ai-jie01 zhenrong-wang iqiuyu-0821 zhanglongchun seanhh86 wayneliu0019 celeskyking ganisback leipeizhao qauzy radiofpc buildfarmer besi1z wa1ke1 ohbabe zoya9

csghub-server's Issues

Support for Additional Git Servers

The current implementation of CSGHub Server only supports Gitea as the Git server. As mentioned in the roadmap, it would be beneficial to support more mainstream Git repositories in the future. I propose adding support for GitLab and GitHub as additional Git servers.

Motivation

This feature would allow users to integrate CSGHub Server with their existing Git workflows, making it more versatile and appealing to a wider range of users.

Expected Behavior

The CSGHub Server should be able to connect to and manage repositories on GitLab and GitHub, in addition to Gitea.

Acceptance Criteria

The CSGHub Server can successfully connect to GitLab and GitHub repositories.
The CSGHub Server can manage repositories on GitLab and GitHub, including creating, updating, and deleting repositories.
The CSGHub Server can handle authentication and authorization for GitLab and GitHub repositories.

Create inference by wrong access token

wrong access token for inference.

服务端镜像启动报错-无法加载 duckdb extension httpfs

在离线环境下运行服务端镜像，报错如下：
init logger, level: INFO, format: json
all_in_one-csghub_server-1 | {"time":"2024-04-27T06:49:16.456842199Z","level":"INFO","msg":"FIFOScheduler run started"}
all_in_one-csghub_server-1 | {"time":"2024-04-27T06:49:16.457924209Z","level":"ERROR","msg":"refresh status all failed","error":"Get "http://localhost:8082/status-all\": dial tcp [::1]:8082: connect: connection refused"}
all_in_one-csghub_server-1 | Error: failed to init router: error creating dataset viewer handler:failed to create parquet reader,cause:failed to setup s3 for duckdb, cause:IO Error: Failed to download extension "httpfs" at URL "http://extensions.duckdb.org/v0.9.2/linux_amd64/httpfs.duckdb_extension.gz"

cannot pass parameter endpoint by cli

endpoint can not be use by cli parameter
file_download error for repo type of dataset

Traceback (most recent call last):
  File "/Users/hhwang/code/jihulab/opencsg/csghub-sdk/test.py", line 21, in <module>
    result = snapshot_download(repo_id=repoid, cache_dir=cachedir, endpoint=ep, token=my_token, repo_type=repotype)
  File "/Users/hhwang/code/jihulab/opencsg/csghub-sdk/pycsghub/snapshot_download.py", line 61, in snapshot_download
    repo_info = utils.get_repo_info(repo_id,
  File "/Users/hhwang/code/jihulab/opencsg/csghub-sdk/pycsghub/utils.py", line 149, in get_repo_info
    return method(
  File "/Users/hhwang/code/jihulab/opencsg/csghub-sdk/pycsghub/utils.py", line 355, in model_info
    r.raise_for_status()
  File "/Users/hhwang/anaconda3/envs/hubsdk/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-stg.opencsg.com/hf/api/models/wanghh2003/myds1/revision/main