martimors / mlserver-inference Goto Github PK

The idea: Extensible and parametrized inference framework to use with mlserver (KServe compliant webservers).

The user configures a source and a destination using env-vars, akin to the way that for example Kafka Connect sources data from one place and inserts into another place.

Currently geared towards synchronous batch inference, but an asynchronous Kafka-based service can easily be implemented on top of this.

from pathlib import Path
from mlserver_inference_pipeline.predict import MlserverPredictor
from mlserver_inference_pipeline.extractors.random import RandomFeatureExtractor
from mlserver_inference_pipeline.destinations.csv import CsvPredictionDestination

MlserverPredictor(
    "http://my.model.example.com",
    "mlflow-model",
    RandomFeatureExtractor(n_features=10),
    CsvPredictionDestination(outpath=Path("file.csv")),
).predict()

Very much a WIP or PoC project.

Connectors that get features from a datastore are referred to as Extractors. An extractor must implement AbstractFeatureExtractor.

Connectors that insert predictions into a datastore are referred to as Destinations. A destination must implement AbstractPredictionDestination.

poetry install
poetry run pytest

• CLI
• Docker Image
• Airflow DAG