autonomi-ai / nos Goto Github PK

• nos serve -m stability-ai/stable-diffusion-v2: Serve optimized nos model (blocking)
• nos serve -d stability-ai/stable-diffusion-v2: Serve optimized `nos model (daemon/detached)
• nos serve -c deployment.yml: Serve collection of models (blocking)
• nos serve -d -c deployment.yml: Serve collection of models (daemon/detached)

Consider https://github.com/avast/pytest-docker and dockerized task decorator

Currently hitting dependency issues when trying to install nos in the base nos environment (cloudpickle, grpc, av etc.). Need to resolve these before v0.1.

Should provide any metadata Marcel needs for writing a UDF.

Docker can take path to env as an argument

Register models as part of the nos hub registry, with full build-time and runtime spec.

@hub.register(
    name="<org_name>/detection2d-detr-resnet-50", 
    build_spec=DevelopmentConfig(
        conda="autonomi-ai/nos-base-dev",
	resources=ResourceConfig(cpu=8, memory="8Gi", gpu=0.25, gpu_memory="4Gi"), # runtime resource
    ),
    runtime_spec=RuntimeConfig(
        conda="autonomi-ai/nos-base-runtime", # runtime conda env
	resources=ResourceConfig(cpu=2, memory="4Gi", gpu=0.25, gpu_memory="4Gi"), # runtime resources
    ),
)

• Platforms: Mac, Windows
• Modes: host, conda, docker (or docker-in-docker),

• Potentially consider W&B report integration with CI tests
• Related to #5

Currently we create a large docker image (11GB) for the base gpu image

Use gRPC gateway to serve REST API via reverse proxy.

• https://grpc-ecosystem.github.io/grpc-gateway/
• Add buf support with openapi v2 integration
• https://blog.logrocket.com/guide-to-grpc-gateway/

References:

• https://github.com/triton-inference-server/client
• https://github.com/triton-inference-server/server
• gRPC ingress for Ray serve (https://docs.ray.io/en/latest/serve/direct-ingress.html)
• https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#v2-protocol

We want to be able to query for model info from the cli, including:

• task
• model_id (e.g. huggingface/CLIP_ViT_base_32)
• model inference signature
• model URL?
• input resolution
• batch size?
• default model for task?

Documentation should be inline.

InferenceClient("localhost:50051") is taking significantly longer to establish a connection with WaitForServer() taking over 30+ seconds. The defaults InferenceClient("[::]:50051") however are instantaneous (< 1ms).

cc @mkornacker

Benchmark decorators with separate make test-benchmark target

#128 #132

Reduce bloat, move init, ready, id etc into the subclass. Right now we just have an inference runtime, but future releases might include runtimes for benchmarking, compilation etc.

#97

Similar to #17 but for conda environments

Avoid having to generate the pb2 files by the client and instead generate pb files on make dist and add it to the wheel file.

If we're able to build checksums for layer-wise weights, we should be able to only download the diffs and speed up model downloads significantly. This is particularly helpful if you're fine-tuning models (especially the last few layers, or changing only parts of an ensemble model).

• Github workflow CI: Support platforms: Windows, MacOS for basic models (SD v2, CLIP)

Create publicly accessible MkDocs for nos

We'd need this for users when reporting bugs and also for internal benchmarking purposes.