samagra-development / ai-tools Goto Github PK

View Code? Open in Web Editor NEW

41.0 41.0 109.0 2.68 MB

AI Tooling to bootstrap applications fast

Python 84.62% Shell 5.22% Dockerfile 10.15%

ai-tools's People

Contributors

Stargazers

Watchers

Forkers

gautamr-samagra ryanwalker277 djgupta sanjay-aurigait devayushdubey poonam1607 preetam1407 j-e-e-t nazi-pikachu wasup-yash itsrahulsehgal iamarjitgoyal parth1007 janeka1122 sanjaysinghrajpoot adityyax gupta-aryaman fykaa bruno-noir adityamisra01 psn0w apoorvgarg-creator rafey1104 shazy01 nitesh639 adv1k sarthakrw pyasma yb73 ajitg25 rishav-eulb harshinirhsv navneet-28 danishraza0912 itshmoh shrivastava95 anann99 cyber-machine goyalpramod ksgr5566 shradhayy posigangapavani uncanny69 asjad895 kaushikdey647 negrah-s unknownsentinel193 inkerton analystexplorer ysrastogi nishankjain1 jiya126 tejash11 sudeshna-s03 sathvikap paradise110302 animesh312 rohanhbtu aaryansahu utkarsh1210-tech killer-mohit sasank15 divyansh-tripathi7 abhinavarya7 dhruvsinghbora manas23601 ayush0x00 santhoshstark06 eeshnaugraiya shro-2002 wetleaf raksansuseendran smitjiwani arjunyaduvanshi02 pruntr sharma-satyam697 vishvjeet-thakur ajaman190 karun-development aakarshhhh prtkjakhar h4r5h1t-007 suyashchougale chatwithpdf shephinphilip amakrushai parth11791 bainaryglobe singhalkarun wassaf001 stophobia 35c4n0r xorsuyash prachipandey16 iamkrmayank soham-27 masterismail sanjay8602 tornado-hacker toshan07

ai-tools's Issues

Add prometheus server

Description

Add Prometheus server to the project and update the Docker Compose file to include the necessary configurations.

Goals

Install and configure Prometheus server for monitoring the project
Update the Docker Compose file to include the Prometheus service
Define appropriate configurations and targets for Prometheus scraping

Expected Outcome

By adding Prometheus server and updating the Docker Compose file, the project will have monitoring capabilities. Prometheus will be able to scrape metrics from the services defined in the Docker Compose file, providing valuable insights into the performance and health of the system.

Acceptance Criteria

Prometheus server is successfully installed and configured
Docker Compose file is updated to include the Prometheus service
Configurations and targets for Prometheus scraping are defined and working correctly

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

Docker, Prometheus

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

DevOps

Sub Category

Monitoring

Admin for AI Tools

Setup react admin for AI Tools. It should monitor

Repository

(Undercooked issues)

Create a fine grained access token for pushing docker images to github packages

Description

Create a fine-grained access token that allows pushing Docker images to GitHub Packages.

Goals

Generate a new access token with the necessary permissions for pushing Docker images to GitHub Packages
Configure the access token to have fine-grained permissions limited to the required repositories and packages
Document the access token and its usage for future reference

Expected Outcome

The access token will provide the necessary credentials to push Docker images to GitHub Packages. By creating a fine-grained access token, you can ensure that it only has the required permissions and restrict access to specific repositories and packages, enhancing security and access control.

Acceptance Criteria

A new access token is generated specifically for pushing Docker images to GitHub Packages
The access token is configured with fine-grained permissions limited to the necessary repositories and packages
The access token is documented, including its usage and any expiration or renewal instructions

Implementation Details

[List any technical details about the proposed implementation, including any specific technologies that will be used.]

Mockups / Wireframes

[N/A]

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

GitHub, Docker

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

DevOps

Sub Category

Credentials Management

Setup Integration Testing Suite

Use this - https://flask.palletsprojects.com/en/1.1.x/testing/#the-first-test

Sanjay

Figure out what to use other than `concept`

Add OpenAPI Spec

POST /useCase/model
POST /health
POST /embeddings/model

Samvit

Refactoring to include Vector Database

WarpSQL Embeddings

Document how to add data for a new `concept`

Graceful Degradation

Fix Basic Repo Structure

Abstract out and add an async similarity helper function

async def GPT3_farmer_prompt(data, context):
    # df1 = pd.read_csv('df1.csv')
    df1 = pd.read_csv('df_all_new_data.csv')
    content = df1['Content']
    actual_sentences = df1['sentence_constructed']
    encoded_string = df1['encoded_sentence']

    encoded = []
    for en_str in encoded_string:
        encoded.append(np.array(json.loads(en_str)))
    encoded = np.array(encoded)

    model = SentenceTransformer('bert-base-nli-mean-tokens')

    def get_prediction(sent):
        sim = cosine_similarity(
            model.encode([sent]),
            encoded
        )
        max_prob = np.argmax(sim)
        max_sim = np.max(sim)
        return actual_sentences[max_prob], content[max_prob], max_sim

We have to add this function as an async helper function (if possible) in the repo.

Deployment Scripts

Horizontally scalable deployment using Ansible on a GPU based inventory

Karun

Add CI using GitHub Workflows

The CI should

Installation
Run the integration test suite
Run the Unit tests suite

When

On a PR
Before merging

Outputs

Badges on coverage

Karun

Add Flake8 to force style guide

Docs: For new contributes and Revamp ReadMe docs for installation

It appears that the readme file of the project is currently outdated and lacks documentation for contributors. Assuming that the project team wants to improve the documentation, they may welcome contributions from interested parties who have the necessary expertise. It is also possible that they may have a plan in place to update the documentation themselves, but have not yet had the resources or time to do so. Regardless of the reason, it is essential that the documentation is kept up-to-date to ensure effective collaboration and understanding among the contributors. Interested parties may want to reach out to the project team to inquire about the status of the documentation and offer their assistance if possible.

Verify Response Format

Add response format to repository
Verify response format before sending it back here

Curl Requests for the following APIs

v1 - /translateX2Y/bhashini/remote - Input [(source, target, text)] => [text] @ChakshuGautam
v1 - /translateX2Y/bhashini/onprem (Indic to English, English to Indic) - Input [(source, target, text)] => [text] @djgupta
v1 - /llm/openai/remote - Input (description, examples, query) => text @djgupta
v1 - /detectLanguage/bhashini/remote - Input (text) => language @ChakshuGautam
v2 - /detectLanguage/bhashini/onprem - Input (text) => language @djgupta
v1 - /similarContent/BERT/onprem - Input (prompt, db, threshold) => [text] @GautamR-Samagra
v1 - /speech2text/bhashini/onprem - Input [(audio url, encoding)] => [text] @djgupta
v1 - /speech2text/bhashini/remote - Input [(audio url, encoding)] => [text] @ChakshuGautam
v1 - /embeddings/openai/remote @GautamR-Samagra @ChakshuGautam

[C4GT] Neural coreference for enabling conversational flow in bots

Project Details

AI Toolchain is a collection of tools for quickly building and deploying machine learning models for various use cases. Currently, the toolchain includes a text translation model, and more models may be added in the future. It abstracts the dirty details of how a model works similar to Huggingface and gives a clean API that you can orchestrate at a BFF level.

Features to be implemented

Neural coreference resolution (overview) is a natural language processing (NLP) task that involves identifying when two or more words or phrases in a text refer to the same entity or concept. This process is crucial for understanding the context and meaning of a text, as it helps in resolving ambiguities and connecting different parts of a text.

For example:
Non-coreferenced conversation:
User : 'Can you tell me where are the shops for paddy seeds?'
User : 'What is the price for them?'

Coreferenced conversation:
User : 'Can you tell me where are the shops for paddy seeds?'
User : 'What is the price for paddy seeds?'

How it works

Neural coreference resolution models usually involve various components, such as:

Feature Extraction:

Detecting the different parts of a sentence and their relationships with each other

Mention detection:

Identifying potential words or phrases (mentions) that may be involved in coreference relations (they all refer to the same entity)

Pairwise scoring:

Computing a score for each pair of mentions, representing the likelihood that they corefer.

Clustering:

Grouping mentions into clusters, where each cluster represents a single entity or concept.

Replacement:

For the last message in the conversation (or for all messages succeeding the first), replace the phrases in each cluster by the common word that gives the entire picture.

Deployment:

Deploy the above setup as a part of ai-tools package such that it can be dockerized and then accessed through an API setup.

Learning Path

Complexity

Hard

Skills Required

Python, NLP

Name of Mentors:

@GautamR-Samagra

Project size

8 Weeks

Product Set Up

See the setup here
The setup here is just an example or baseline solution of how it could be carried out. We are currently focusing this project on using the Spacy model to carry out coreference. Exploration of other techniques is encouraged only if the this one fails.

Acceptance Criteria

Unit Test Cases
OpenAPI Spec/Postman Collection
Dockerfile for this module

Milestone

Setting up spacy
Identifying type and format of data required to train
Creating synthetic data/collecting data for training
Training the model based on collected data
Carry out automated testing for models / integrate training based on our test cases
New trained model based on trained data

Reference

https://galhever.medium.com/a-review-to-co-reference-resolution-models-f44b4360a00

https://explosion.ai/blog/coref
explosion/spaCy#11585 (reply in thread)

C4GT

This issue is nominated for Code for GovTech (C4GT) 2023 edition.
C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/

The scope of this ticket has now expanded to make it the 'enabling conversation' part of 'FAQ bot'.
The FAQ bot allows a user to be able to provide content input in the form on csvs, free text, pdfs, audio, video and the bot is able to add it to a 'Content DB'. The user is then able to interact with the bot via text/speech on related content and the bot is able to identify relevant content using RAG techniques and be able to be able to respond to the user in a conversational manner.

This ticket covers the enabling conversation +small model deployment/finetuning pipeline. It includes the following tasks in its scope:

Implement circuit breaker to a different model if one fails

Sample implementation library

Alternatives

This when the models are deployed over Kong. Blog here.
Add Rate Limiter when using with Kong.

Update Dockerfile to pull csv file from some private source using token and copy it to required directory.

Distribute requests to multiple OpenAI Accounts

Rotated at an instance level injected by Infra. Load balancing - round robin through NginX.

Add CI that builds docker image and pushes to Github packages.

Setup Unit Testing Suite

Sanjay

Add security headers to Wrapper Quart App

error on src/text_translation/ai4bharat/deploy.sh file

I found some small issues in the src/text_translation/ai4bharat/deploy.sh file.

missing shebang line

#!/bin/bash

In line 12 give the error due to "../" this command.

ai-tools/src/text_translation/ai4bharat/deploy.sh

Line 12 in 83bec92

poetry add ../fairseq

Allow requests from '0.0.0.0' (any host), currently only localhost allowed

Refactor Remote Services

Remote service onboarding should not include code changes. They should be onboarded directly through repository.private.json which overrides the repository.json

Close use case - `Embeddings` with model `BERT` and `OpenAI`

Document how to create a new `concept`

Add Dockerfile for Wrapper

Add a header to the response when it comes from a cached function

Description

Add a header to the response when it comes from a cached function to indicate its origin.

Goals

Modify the function logic to include a header in the response when it is served from a cached function
Define a custom header, such as ai-tools-cached, to indicate that the response is from a cached function
Ensure the header is included in the response for appropriate requests

Expected Outcome

By adding a header to the response from a cached function, it becomes possible to differentiate between responses served from cache and those generated in real-time. This information can be useful for debugging and monitoring purposes, allowing developers to understand the performance and behavior of the cached functions.

Acceptance Criteria

The function logic is updated to include a custom header, such as ai-tools-cached, in the response when it comes from a cached function
The header is included in the response for appropriate requests
The presence of the header accurately indicates whether the response is from a cached function or not

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

Backend Development

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

Backend

Sub Category

API Development

Add Gitpod

Gitpod setup for faster hacking and contributions.

Supported Usecase <> Models list

Use Case	Model	Status
Use Case 1	Model 1	Add ticket reference here

@GautamR-Samagra

Separate out components

Need to separate out componets such that GPU required and no required ones can be hosted separately and each component can be deployed in individual docker containers

Create Embeddings API

@GautamR-Samagra

Step 1

Step 2

Step 3

Refactoring
#22

Auto Deployment to Staging when pushed to master

Webhook on the server => listen for push to master => docker container build => deploy through docker compose

Graceful Degradation in case an API fails

Add a retry mechanism to run the remote API again (text_translation/bhashini/remote model specifically)
Downgrade to running a local translation model

Sagemaker Setup

Description

Evaluate Sagemaker as a potential solution for deploying AI-tools and provide documentation to guide users in deploying all models to Sagemaker.

Goals

Research and evaluate the feasibility of using Sagemaker for deploying AI-tools
Identify the steps and configurations required to deploy all models to Sagemaker
Document the deployment process in a comprehensive sagemaker.md guide

Expected Outcome

By evaluating Sagemaker for deploying AI-tools and documenting the deployment process, users will have clear instructions on how to deploy all models to Sagemaker. This enables the easy adoption and usage of AI-tools in a Sagemaker environment.

Acceptance Criteria

Decision tree created on feasibility of using Sagemaker for AI-tools deployment, detailing when Sagemaker is advantageous
Detailed steps and configurations are identified for deploying all models to Sagemaker
A comprehensive sagemaker.md guide is created, providing instructions for deploying AI-tools on Sagemaker

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

AWS Sagemaker, AI-tools

Mentor(s)

@ChakshuGautam

Complexity

Hard

Category

Deployment

Sub Category

Infrastructure

Export Prometheus Metrics

Something like this but for quart.

Debug Server doesn't track src folder

The server doesn't restart when changes to src folder happens.

Add a health check API

Use this for any external dependent services.

A global /health endpoint that allows to check health of each model.
Update generate.sh to make health checks the part of generated docker-compose and nginx configuration file

Upload CSV API - Sanjay

Python decorator over an API Call

Input

Streaming CSV file
Fetch a file from a remote location

Implementation Details

Keep it in memory
Store it in folder /tmp on the server => Load it in memory through a CSV Reader

Output

CSV Reader

Error on deploy.sh file

I found some small issues in the deploy.sh file.

missing shebang line

#!/bin/bash

In line 3 given path is incorrect.

ai-tools/deploy.sh

Line 3 in e8c41e3

cd text_translation/ai4bharat & sh deploy.sh & cd .. & cd ..

Reducing the size of docker image

Current sizes

root@e25f48a1466e:/usr/local/lib/python3.8/site-packages# du -sh * | sort -hr
2.6G	nvidia
1.3G	torch
184M	triton
87M	torchvision
80M	plotly
76M	scipy
65M	cmake
49M	pandas
35M	sklearn
35M	scipy.libs
35M	numpy.libs
34M	sympy

Dependency Graph

aiohttp 3.8.4 Async http client/server framework (asyncio)
├── aiosignal >=1.1.2
│   └── frozenlist >=1.1.0 
├── async-timeout >=4.0.0a3,<5.0
├── attrs >=17.3.0
├── charset-normalizer >=2.0,<4.0
├── frozenlist >=1.1.1
├── multidict >=4.5,<7.0
└── yarl >=1.0,<2.0
    ├── idna >=2.0 
    └── multidict >=4.0 
async-cache 1.1.1 An asyncio Cache
asyncio 3.4.3 reference implementation of PEP 3156
en-coreference-web-trf 3.4.0a2 English transformer pipeline to demonstrate the experimental coreference components. Components: sentencizer, transfomer, coref, span_resolver, span_cleaner.
├── spacy >=3.3.0,<3.5.0
│   ├── catalogue >=2.0.6,<2.1.0 
│   ├── cymem >=2.0.2,<2.1.0 
│   ├── jinja2 * 
│   │   └── markupsafe >=2.0 
│   ├── langcodes >=3.2.0,<4.0.0 
│   ├── murmurhash >=0.28.0,<1.1.0 
│   ├── numpy >=1.15.0 
│   ├── packaging >=20.0 
│   ├── pathy >=0.3.5 
│   │   ├── smart-open >=5.2.1,<7.0.0 
│   │   └── typer >=0.3.0,<1.0.0 
│   │       └── click >=7.1.1,<9.0.0 
│   │           └── colorama * 
│   ├── preshed >=3.0.2,<3.1.0 
│   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│   │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
│   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
│   │   └── typing-extensions >=4.2.0 
│   ├── requests >=2.13.0,<3.0.0 
│   │   ├── certifi >=2017.4.17 
│   │   ├── charset-normalizer >=2,<4 
│   │   ├── idna >=2.5,<4 
│   │   └── urllib3 >=1.21.1,<1.27 
│   ├── setuptools * 
│   ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
│   ├── spacy-legacy >=3.0.10,<3.1.0 
│   ├── spacy-loggers >=1.0.0,<2.0.0 
│   ├── srsly >=2.4.3,<3.0.0 
│   │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
│   ├── thinc >=8.1.0,<8.2.0 
│   │   ├── blis >=0.7.8,<0.8.0 
│   │   │   └── numpy >=1.15.0 (circular dependency aborted here)
│   │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
│   │   ├── confection >=0.0.1,<1.0.0 
│   │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│   │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│   │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
│   │   ├── numpy >=1.15.0 (circular dependency aborted here)
│   │   ├── packaging >=20.0 (circular dependency aborted here)
│   │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
│   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│   │   ├── setuptools * (circular dependency aborted here)
│   │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│   │   └── wasabi >=0.8.1,<1.2.0 
│   ├── tqdm >=4.38.0,<5.0.0 
│   │   └── colorama * (circular dependency aborted here)
│   ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
│   └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
├── spacy-experimental >=0.6.1,<0.7.0
│   └── spacy >=3.3.0,<3.6.0 
│       ├── catalogue >=2.0.6,<2.1.0 
│       ├── cymem >=2.0.2,<2.1.0 
│       ├── jinja2 * 
│       │   └── markupsafe >=2.0 
│       ├── langcodes >=3.2.0,<4.0.0 
│       ├── murmurhash >=0.28.0,<1.1.0 
│       ├── numpy >=1.15.0 
│       ├── packaging >=20.0 
│       ├── pathy >=0.3.5 
│       │   ├── smart-open >=5.2.1,<7.0.0 
│       │   └── typer >=0.3.0,<1.0.0 
│       │       └── click >=7.1.1,<9.0.0 
│       │           └── colorama * 
│       ├── preshed >=3.0.2,<3.1.0 
│       │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│       │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
│       ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
│       │   └── typing-extensions >=4.2.0 
│       ├── requests >=2.13.0,<3.0.0 
│       │   ├── certifi >=2017.4.17 
│       │   ├── charset-normalizer >=2,<4 
│       │   ├── idna >=2.5,<4 
│       │   └── urllib3 >=1.21.1,<1.27 
│       ├── setuptools * 
│       ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
│       ├── spacy-legacy >=3.0.10,<3.1.0 
│       ├── spacy-loggers >=1.0.0,<2.0.0 
│       ├── srsly >=2.4.3,<3.0.0 
│       │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
│       ├── thinc >=8.1.0,<8.2.0 
│       │   ├── blis >=0.7.8,<0.8.0 
│       │   │   └── numpy >=1.15.0 (circular dependency aborted here)
│       │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
│       │   ├── confection >=0.0.1,<1.0.0 
│       │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│       │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│       │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│       │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
│       │   ├── numpy >=1.15.0 (circular dependency aborted here)
│       │   ├── packaging >=20.0 (circular dependency aborted here)
│       │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
│       │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│       │   ├── setuptools * (circular dependency aborted here)
│       │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│       │   └── wasabi >=0.8.1,<1.2.0 
│       ├── tqdm >=4.38.0,<5.0.0 
│       │   └── colorama * (circular dependency aborted here)
│       ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
│       └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
└── spacy-transformers >=1.1.2,<1.2.0
    ├── spacy >=3.4.0,<4.0.0 
    │   ├── catalogue >=2.0.6,<2.1.0 
    │   ├── cymem >=2.0.2,<2.1.0 
    │   ├── jinja2 * 
    │   │   └── markupsafe >=2.0 
    │   ├── langcodes >=3.2.0,<4.0.0 
    │   ├── murmurhash >=0.28.0,<1.1.0 
    │   ├── numpy >=1.15.0 
    │   ├── packaging >=20.0 
    │   ├── pathy >=0.3.5 
    │   │   ├── smart-open >=5.2.1,<7.0.0 
    │   │   └── typer >=0.3.0,<1.0.0 
    │   │       └── click >=7.1.1,<9.0.0 
    │   │           └── colorama * 
    │   ├── preshed >=3.0.2,<3.1.0 
    │   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
    │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
    │   │   └── typing-extensions >=4.2.0 
    │   ├── requests >=2.13.0,<3.0.0 
    │   │   ├── certifi >=2017.4.17 
    │   │   ├── charset-normalizer >=2,<4 
    │   │   ├── idna >=2.5,<4 
    │   │   └── urllib3 >=1.21.1,<1.27 
    │   ├── setuptools * 
    │   ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
    │   ├── spacy-legacy >=3.0.10,<3.1.0 
    │   ├── spacy-loggers >=1.0.0,<2.0.0 
    │   ├── srsly >=2.4.3,<3.0.0 
    │   │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
    │   ├── thinc >=8.1.0,<8.2.0 
    │   │   ├── blis >=0.7.8,<0.8.0 
    │   │   │   └── numpy >=1.15.0 (circular dependency aborted here)
    │   │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
    │   │   ├── confection >=0.0.1,<1.0.0 
    │   │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
    │   │   ├── numpy >=1.15.0 (circular dependency aborted here)
    │   │   ├── packaging >=20.0 (circular dependency aborted here)
    │   │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
    │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   │   └── wasabi >=0.8.1,<1.2.0 
    │   ├── tqdm >=4.38.0,<5.0.0 
    │   │   └── colorama * (circular dependency aborted here)
    │   ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
    │   └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
    ├── spacy-alignments >=0.7.2,<1.0.0 
    ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    ├── torch >=1.6.0 
    │   ├── filelock * 
    │   ├── jinja2 * (circular dependency aborted here)
    │   ├── networkx * 
    │   ├── nvidia-cublas-cu11 11.10.3.66 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * 
    │   ├── nvidia-cuda-cupti-cu11 11.7.101 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cuda-nvrtc-cu11 11.7.99 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cuda-runtime-cu11 11.7.99 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cudnn-cu11 8.5.0.96 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cufft-cu11 10.9.0.58 
    │   ├── nvidia-curand-cu11 10.2.10.91 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cusolver-cu11 11.4.0.1 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cusparse-cu11 11.7.4.91 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-nccl-cu11 2.14.3 
    │   ├── nvidia-nvtx-cu11 11.7.91 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── sympy * 
    │   │   └── mpmath >=0.19 
    │   ├── triton 2.0.0 
    │   │   ├── cmake * 
    │   │   ├── filelock * (circular dependency aborted here)
    │   │   ├── lit * 
    │   │   └── torch * (circular dependency aborted here)
    │   └── typing-extensions * (circular dependency aborted here)
    └── transformers >=3.4.0,<4.26.0 
        ├── filelock * (circular dependency aborted here)
        ├── huggingface-hub >=0.10.0,<1.0 
        │   ├── filelock * (circular dependency aborted here)
        │   ├── packaging >=20.9 (circular dependency aborted here)
        │   ├── pyyaml >=5.1 
        │   ├── requests * (circular dependency aborted here)
        │   ├── tqdm >=4.42.1 (circular dependency aborted here)
        │   └── typing-extensions >=3.7.4.3 (circular dependency aborted here)
        ├── numpy >=1.17 (circular dependency aborted here)
        ├── packaging >=20.0 (circular dependency aborted here)
        ├── pyyaml >=5.1 (circular dependency aborted here)
        ├── regex !=2019.12.17 
        ├── requests * (circular dependency aborted here)
        ├── tokenizers >=0.11.1,<0.11.3 || >0.11.3,<0.14 
        └── tqdm >=4.27 (circular dependency aborted here)
flake8 6.0.0 the modular source code checker: pep8 pyflakes and co
├── mccabe >=0.7.0,<0.8.0
├── pycodestyle >=2.10.0,<2.11.0
└── pyflakes >=3.0.0,<3.1.0
flask 2.2.3 A simple framework for building complex web applications.
├── click >=8.0
│   └── colorama * 
├── importlib-metadata >=3.6.0
│   └── zipp >=0.5 
├── itsdangerous >=2.0
├── jinja2 >=3.0
│   └── markupsafe >=2.0 
└── werkzeug >=2.2.2
    └── markupsafe >=2.1.1 
inference 0.1
matplotlib 3.7.1 Python plotting package
├── contourpy >=1.0.1
│   └── numpy >=1.16 
├── cycler >=0.10
├── fonttools >=4.22.0
├── importlib-resources >=3.2.0
│   └── zipp >=3.1.0 
├── kiwisolver >=1.0.1
├── numpy >=1.20
├── packaging >=20.0
├── pillow >=6.2.0
├── pyparsing >=2.3.1
└── python-dateutil >=2.7
    └── six >=1.5 
numpy 1.24.2 Fundamental package for array computing in Python
openai 0.27.4 Python client library for the OpenAI API
├── aiohttp *
│   ├── aiosignal >=1.1.2 
│   │   └── frozenlist >=1.1.0 
│   ├── async-timeout >=4.0.0a3,<5.0 
│   ├── attrs >=17.3.0 
│   ├── charset-normalizer >=2.0,<4.0 
│   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│   ├── multidict >=4.5,<7.0 
│   └── yarl >=1.0,<2.0 
│       ├── idna >=2.0 
│       └── multidict >=4.0 (circular dependency aborted here)
├── requests >=2.20
│   ├── certifi >=2017.4.17 
│   ├── charset-normalizer >=2,<4 
│   ├── idna >=2.5,<4 
│   └── urllib3 >=1.21.1,<1.27 
└── tqdm *
    └── colorama * 
openai-async 0.0.3 A light-weight, asynchronous client for OpenAI API - text completion, image generation and embeddings.
├── httpx *
│   ├── certifi * 
│   ├── httpcore >=0.15.0,<0.18.0 
│   │   ├── anyio >=3.0,<5.0 
│   │   │   ├── idna >=2.8 
│   │   │   └── sniffio >=1.1 
│   │   ├── certifi * (circular dependency aborted here)
│   │   ├── h11 >=0.13,<0.15 
│   │   └── sniffio >=1.0.0,<2.0.0 (circular dependency aborted here)
│   ├── idna * (circular dependency aborted here)
│   └── sniffio * (circular dependency aborted here)
└── pytest *
    ├── colorama * 
    ├── exceptiongroup >=1.0.0rc8 
    ├── iniconfig * 
    ├── packaging * 
    ├── pluggy >=0.12,<2.0 
    └── tomli >=1.0.0 
pandas 2.0.0 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.20.3
├── numpy >=1.21.0
├── numpy >=1.23.2
├── python-dateutil >=2.8.2
│   └── six >=1.5 
├── pytz >=2020.1
└── tzdata >=2022.1
pipdeptree 2.0.0 Command line utility to show dependency tree of packages
└── pip >=6.0.0
plotly 5.14.1 An open-source, interactive data visualization library for Python
├── packaging *
└── tenacity >=6.2.0
python-dotenv 1.0.0 Read key-value pairs from a .env file and set them as environment variables
quart 0.18.4 A Python ASGI web microframework with the same API as Flask
├── aiofiles *
├── blinker <1.6
├── click >=8.0.0
│   └── colorama * 
├── hypercorn >=0.11.2
│   ├── h11 * 
│   ├── h2 >=3.1.0 
│   │   ├── hpack >=4.0,<5 
│   │   └── hyperframe >=6.0,<7 
│   ├── priority * 
│   ├── toml * 
│   └── wsproto >=0.14.0 
│       └── h11 >=0.9.0,<1 (circular dependency aborted here)
├── importlib-metadata *
│   └── zipp >=0.5 
├── itsdangerous *
├── jinja2 *
│   └── markupsafe >=2.0 
├── markupsafe *
└── werkzeug >=2.2.0
    └── markupsafe >=2.1.1 
scikit-learn 1.2.2 A set of python modules for machine learning and data mining
├── joblib >=1.1.1
├── numpy >=1.17.3
├── scipy >=1.3.2
│   └── numpy >=1.18.5,<1.26.0 
└── threadpoolctl >=2.0.0
sklearn 0.0.post4 deprecated sklearn package, use scikit-learn instead
spacy-experimental 0.6.2 Cutting-edge experimental spaCy components and features
└── spacy >=3.3.0,<3.6.0
    ├── catalogue >=2.0.6,<2.1.0 
    ├── cymem >=2.0.2,<2.1.0 
    ├── jinja2 * 
    │   └── markupsafe >=2.0 
    ├── langcodes >=3.2.0,<4.0.0 
    ├── murmurhash >=0.28.0,<1.1.0 
    ├── numpy >=1.15.0 
    ├── packaging >=20.0 
    ├── pathy >=0.3.5 
    │   ├── smart-open >=5.2.1,<7.0.0 
    │   └── typer >=0.3.0,<1.0.0 
    │       └── click >=7.1.1,<9.0.0 
    │           └── colorama * 
    ├── preshed >=3.0.2,<3.1.0 
    │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
    ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
    │   └── typing-extensions >=4.2.0 
    ├── requests >=2.13.0,<3.0.0 
    │   ├── certifi >=2017.4.17 
    │   ├── charset-normalizer >=2,<4 
    │   ├── idna >=2.5,<4 
    │   └── urllib3 >=1.21.1,<1.27 
    ├── setuptools * 
    ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
    ├── spacy-legacy >=3.0.10,<3.1.0 
    ├── spacy-loggers >=1.0.0,<2.0.0 
    ├── srsly >=2.4.3,<3.0.0 
    │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
    ├── thinc >=8.1.0,<8.2.0 
    │   ├── blis >=0.7.8,<0.8.0 
    │   │   └── numpy >=1.15.0 (circular dependency aborted here)
    │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
    │   ├── confection >=0.0.1,<1.0.0 
    │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
    │   ├── numpy >=1.15.0 (circular dependency aborted here)
    │   ├── packaging >=20.0 (circular dependency aborted here)
    │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
    │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   ├── setuptools * (circular dependency aborted here)
    │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   └── wasabi >=0.8.1,<1.2.0 
    ├── tqdm >=4.38.0,<5.0.0 
    │   └── colorama * (circular dependency aborted here)
    ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
    └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
tiktoken 0.3.3 tiktoken is a fast BPE tokeniser for use with OpenAI's models
├── regex >=2022.1.18
└── requests >=2.26.0
    ├── certifi >=2017.4.17 
    ├── charset-normalizer >=2,<4 
    ├── idna >=2.5,<4 
    └── urllib3 >=1.21.1,<1.27

torch==2.0.0 is missing

When I run the ai-tools/src/text_translation/ai4bharat/deploy.sh file, facing this issue:

because of this line:

ai-tools/src/text_translation/ai4bharat/deploy.sh

Line 13 in 54f69fc

 poetry add gunicorn flask_cors sacremoses pandas mock sacrebleu tensorboardX pyarrow indic-nlp-library mosestokenizer subword-nmt xformers triton flask_cors webvtt-py 

Regarding this issue:
We should also need to add "torch==2.0.0" because xformers==latest depend on torch==2.0.0 and triton, which is already available but torch==2.0.0 is not.

Update workflow to make package public while pushing to github package and also automatically connect it to repository

Description

Update the workflow to make a package public and automatically connect it to the repository in GitHub Packages.

Goals

Modify the workflow to configure the package to be public upon pushing it to GitHub Packages
Automate the connection of the package to the repository
Ensure the workflow runs smoothly and the package is accessible to users

Expected Outcome

The updated workflow will automatically make the package public and establish a connection to the repository in GitHub Packages. This ensures that the package is easily accessible and discoverable by users who can benefit from its functionalities.

Acceptance Criteria

The workflow is updated to include the necessary configuration for making the package public
The workflow automatically establishes a connection between the package and the repository in GitHub Packages
The package is accessible and visible to users in GitHub Packages

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

GitHub Actions

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

DevOps

Sub Category

Maintenance