ethicalml / awesome-production-machine-learning Goto Github PK

Airflow-dvc provides a plugin to integrate DVC tool into Airflow systems. DVC is a version-control system that allows versioning of ML artifacts, data-science intermediate datasets and other large-volume files. It's the alternative for git lfs that has better support for pipelines and results reproduction.

Link to the PR

Thanks for curating a list of useful libraries that help people deploy machine learning projects!

Would it be okay to add the Haven-Ai library to the list? https://github.com/haven-ai/haven-ai

It's a library that we have been working on over the last 3 years that allows us to design codebase for quick reliable prototyping and also running, managing and visualizing large scale reproducible experiments.

I think it fits under Model and Data Versioning section in the README.md.

Thanks a lot!

Datmo joined One Concern. Is Datmo still maintained? https://medium.com/datmo/datmo-and-one-concern-join-forces-29d2627ca96d

The following tools are more appropriate for NLP
INCEpTION
Webanno
TeamTat

We have already updated the frameworks in each section to be ordered alphabetically. The sections themselves should be sorted alphabetically as well. The commercial section would still stay at the bottom given that the primary objective of this list is to showcase OSS tools. If anyone from the community would be keen to give it a shot we would be very grateful!

I've been using ploomber for model training, orchestration. It's helped keep my work reproducible. Would you accept a PR for this?

Katonic MLOps Platform is a collaborative platform with a Unified UI to manage all data science activities in one place and introduce MLOps practice into the production systems of customers and developers. It is a collection of cloud-native tools for all of these stages of MLOps:

-Data exploration
-Feature preparation
-Model training/tuning
-Model serving, testing and versioning

Katonic is for both data scientists and data engineers looking to build production-grade machine learning implementations and can be run either locally in your development environment or on a production cluster. Katonic provides a unified system—leveraging Kubernetes for containerization and scalability for the portability and repeatability of its pipelines.

It will be great if you can list it on your account

Website -
Katonic One Pager.pdf

https://katonic.ai/

Add Open Distro to the anomaly detection chapter. It is a really good tool with large documentation.

#184

Skafos is no ml platform anymore: https://skafos.ai/ where to move it?

HUAWEI recently open sourced MindSpore (github - mindspore-ai/mindspore). I'd consider it an open source end-to-end platform. Into which section should it be put?

For example, Apache PredictionIO and steppy in Model and Data Versioning? These projects keep unmaintained for several years.

FedML - The federated learning and analytics library enabling secure and collaborative machine learning on decentralized data anywhere at any scale.

https://github.com/FedML-AI/FedML

Hi,

Could we add a section to list the open-source tools available to design or run online experiments?

Some examples

• https://github.com/facebook/planout
• https://github.com/gojek/turing

Thank you!

There is trainset for lightweight UI-based visual labeling of time series data. Does someone know standalone tool alternative to trainset?

BTW: Grafana provides annotation API but I don't know an UI integration/tool which enable "drag & drop" style usage.

Hello,

I would like to present Substra: an open-source framework for privacy-preserving, traceable and collaborative Machine Learning.

Substra gathers data providers and algorithm designers into a network of nodes that can train models on demand but under advanced permission regimes. To guarantee data privacy, Substra implements distributed learning: the data never leave their nodes; only algorithms, predictive models and non-sensitive metadata are exchanged on the network. The computations are orchestrated by a Distributed Ledger Technology which guarantees traceability and authenticity of information without needing to trust a third party. Although originally developed for Healthcare applications, Substra is not data, algorithm or programming language specific. It supports many types of computation plans including parallel computation plan commonly used in Federated Learning. With appropriate guidelines, it can be deployed for numerous Machine Learning use-cases with data or algorithm providers where trust is limited.

• Whitepaper: https://arxiv.org/abs/1910.11567

Github:

• Substra Foundation: https://github.com/SubstraFoundation/welcome

• Framework: https://github.com/SubstraFoundation/substra

Since Speedster is already a part of Nebullvm, why not integrate Speedster into Nebullvm? @axsaucedo
The logic is straightforward. We only keep Ray rather than its specific component such as Rllib (a famous production-level reinforcement learning library).

• Nebullvm - Nebullvm is an ecosystem of plug-and-play modules to boost the performances of your AI systems. The optimization modules are stack-agnostic and work with any library. They are designed to be easily integrated into your system, providing a quick and seamless boost to its performance.

Would you mind reopening this pr for further discussion? #307

Not necessarily for big data projects, but for Adhoc machine learning ETL-ing and especially for its ease-of-use it should find its place in your list:

https://www.bonobo-project.org/
http://docs.bonobo-project.org/en/master/
https://github.com/python-bonobo/bonobo

In the overview section it states "Data Storage Layer" this makes sense looking at what is in there. But the section itself is called "Data Storage Optimisation". Shouldn't this also be "Data Storage Layer"?

Hi, @axsaucedo I found the discussion regarding #294 #293 is far from ended, I wonder if we can reopen them for further discussion? I appreciate it a lot!

Such as BindsNET, BrainCog, Norse, and snnTorch in #312 @axsaucedo

I was wondering if it could be useful to open a section on production (serving, ml specific monitoring of the model and the data distributions)?

In the Readme.md, the number of Github stars for Plotly.py is wrong. It lists 40 stars when it should be around 7k.

Creators of DVC released today https://github.com/iterative/cml

What is the recommended place to put this lib?

Thanks for creating an amazing list that help people setup a productive workflow.

Would it be possible to add Haven (https://github.com/ElementAI/haven) to the list?

The goal of Haven is to help in running, managing, visualizing, and sharing a large number of experiments in parallel.

Thanks!

Massive thanks for all the additions @fkromer ! Really great finds! Only thing before we can merge teh PRs, would be to add a short rescription to all the libraries in #83, #82, #81, #80, #79, #78, #77, #76, #75 and #74 - let me know if you can add those, if not I can also jump in and add descriptions.

Do we want to add a section for standards w.r.t. the development of ML models? One standard is e.g. DIN SPEC 92001-1.

The commercial platforms section may be miss-leading. Often it's not obvious if the platforms are end-to-end platforms with more less "complete" feature set (e.g. dataiku DSS) or commercial variants of single components of an overall e2e platform, like e.g. experiment management (e.g. neptune.ai) or model deployment (e.g. datatron). For new contributors it would be helpful to have a separate end-to-end ml platform section.

https://github.com/codalab/codalab-worksheets/, the famous experiment tracking tool powering HuggingFace, SQUAD, etc.

Vector databases are a relatively new phenomenon, but they might deserve their own section.

For the convenience of the maintainer of this repo, I've added two PRs, one without the new section one with the new section.

This reading material about Weaviate might be helpful in the argumentation why the vector database might deserve its own spot on this list :)

• https://db-engines.com/en/blog_post/87
• https://www.zdnet.com/article/weaviate-an-open-source-search-engine-powered-by-machine-learning-vectors-graphs-and-graphql/

A very good tool for EDA

https://github.com/pandas-profiling/pandas-profiling

I can issue a pull request if decided.

Apache Marvin is a platform for easy model deployment and versioning. Once the marvin server is up and running, it hides all complexity "under the hood": the data scientist just needs to write his code in an extended jupyter notebook (included), setup variables with Marvin's reserved names and correctly tag the relevant code cells using a dropdown.

Homepage: https://marvin.apache.org
Github: https://github.com/apache/incubator-marvin

See VertaAI/modeldb#290 (comment)

As discussed in #98 with @visenger , we'll explore adding a new section on outlier detectors, some outlier detection libraries identified include:

• https://github.com/SeldonIO/alibi-detect
• https://github.com/yzhao062/pyod
• https://github.com/yzhao062/SUOD
• https://github.com/hendrycks/outlier-exposure

I excluded libraries with less than 100 stars. If we can identify a couple more these should be enough for a new section.

TDengine is an open source time-series database and has more than 19K stars on Github.

In a training pipelines, we sometimes get tripped by how to best handle and pass around the 5gazillion hyperparameters of data preparation and training. There are some libraries that focus on good configuration for ML:

https://github.com/google/gin-config
https://github.com/PetrochukM/HParams

Is this topic too specific? If not where could it go?

@axsaucedo such as https://github.com/EthicalML/awesome-production-machine-learning/releases. The current one publishes almost two years ago.

NannyML allows estimating post-deployment model performance (without access to targets).
Has 1k+ stars, is maintained regularly, and more than 6m old :)

Tournesol is a collaborative content recommendation platform based on the latest research in AI security (some articles on the algorithms used have already been published). The goal of Tournesol is to identify top videos of public utility by eliciting contributors' judgements on content quality. All the code is open-source and available here https://github.com/tournesol-app/tournesol

If the suggestion is accepted, I may do a PR.

For example, Industry-strength Finance such as Qlib? Industry-strength Healthcare such as MONAI, torchxrayvision, hi-ml, InnerEye? @axsaucedo

The description for ggplot2 mentions that it is a Python library, but it is an R library.

Some tools in the FaaS module are designed to solve more general scalability issues rather than model deployment.
I suggest removing those available tools from our list since they seem misleading somehow. For example, we should not include Kubernetes in our list since it is a general tool that aims not for model deployment specifically.
check #333

Katonic MLOps Platform is a collaborative platform with a Unified UI to manage all data science activities in one place and introduce MLOps practice into the production systems of customers and developers. It is a collection of cloud-native tools for all of these stages of MLOps:

-Data exploration
-Feature preparation
-Model training/tuning
-Model serving, testing and versioning

Katonic is for both data scientists and data engineers looking to build production-grade machine learning implementations and can be run either locally in your development environment or on a production cluster. Katonic provides a unified system—leveraging Kubernetes for containerization and scalability for the portability and repeatability of its pipelines.

It will be great if you can list it on your account

Website -
Katonic One Pager.pdf

https://katonic.ai/

https://github.com/beniz/deepdetect is forked from https://github.com/jolibrain/deepdetect

It make wrong starts count

Superwise.ai - Machine learning monitoring solution for models in production

Monthly automated release with pdf conversion from main readme similar to first release.

I found out that a very useful package keras-tuner is missing from the list.

Keras tuner is a great package used for feature engineering and automated hyperparameter tuning in Keras and Tensorflow and can be really helpful in finding the best hyperparameters using various search methods.

1. https://voxel51.com/docs/fiftyone/
2. https://github.com/IndustryEssentials/ymir

As suggested and discussed in #94 with @rmminusrslash and @visenger we'll add a new section on ML serving, as there's been quite a lot of new projects in this space. A few ML Serving frameworks include:

• Seldon Core
• KFServing
• BentoML
• Cortex
• TFX
• TFSerivng
• ForestFlow

Any other suggestions? @rmminusrslash / @visenger would you be interested on explore adding a PR?