In conftest.py we currently can choose between a few models in the tiny_models directory:

From quick testing, it doesn't seem like any of these work with the streaming tests, so those were marked with xfail

We should find out whether it's related to caikit-np or if it is related to the way these models are created.

For reference:

• tiny models in caikit-nlp
• create tiny models with huggingface transformers

We currently are only asserting that the generated text is not empty. We should be able to consistently reproduce the same outputs and validate those e.g.

One way of consistently returning the same results could be by settings the rng seeds in the requests

Downloading a model during ci and setting the path to the model in the current pytest fixture will get CI running properly

Should be implemented similarly to #21

See for example https://github.com/vaibhavjainwiz/caikit-nlp-client/actions/runs/6749154072

Probably just a matter of preference, but I would prefer to use src-layout for this repo.

It should be enough to:

mkdir src
git mv caikit_nlp_client src
git commit

pre-commit session is not currently running:

https://github.com/vaibhavjainwiz/caikit-nlp-client/actions/runs/6736309915/job/18311386232#step:8:15

We should be able to set up a pytest fixture that sets up a caikit_nlp server instance and use that to test both grpc and http behaviour.

For example, to start a http and grpc server it is sufficient to run:

RUNTIME_LIBRARY=caikit_nlp python -m caikit.runtime

Installing caikit-nlp also requires torch which pulls down a lot of extra cuda dependencies which occupy a lot of space (and CI time). A full caikit-nlp install requires ~5GB of space. Installing the cpu version of torch, requires ~800MB

Specific versions can be installed by providing the index urls:

• cpu "https://download.pytorch.org/whl/cpu"
• cuda 11.8 "https://download.pytorch.org/whl/cu118"
• cuda 12.1 "https://download.pytorch.org/whl/cu121"

e.g.:

python -m venv .venv
source .venv/bin/activate
pip install --index-url=https://download.pytorch.org/whl/cpu torch
du -sh .venv # 813M

will install torch==2.1.0+cpu (not available on PyPi).

It could be worth it to explore whether we can avoid installing the full torch version in our nox sessions to speed up testing