Inferencing with `codellama-instruct` fails locally about spin HOT 7 CLOSED

I think the first level of directory is the model architecture and then the second layer is the actual model needed. I have not looked at the code but that is how it is parsed if I remember correctly.

from spin.

Makes sense to me. I'll close this issue and create a corresponding one, to update docs/tutorials.

from spin.

@ThorstenHans "Well known" inferencing models (which is currently llama2-chat and code_llama) can go directly in the ai-models directory. (Embedding models are different again.) It's only "not so well known" models (such as codellama-instruct) that need an architecture directory.

As you've noted, however, it's legal to put well-known models in the architecture directory anyway. It seems like your docs PR recommends always doing that, but I wonder if we should be draw the distinction of "well known" vs "not well known" so that we don't give the impression that "ai-models/llama is where you put files"?

from spin.

@ThorstenHans Could you try putting the model at ./ai_models/llama/codellama-instruct please? (note the extra llama directory) I know this is not what the docs say, just trying to figure out what is happening inside Spin here... Thanks!

from spin.

@itowlson that works 🥳

Did the desired location for models change at some point in time? All the docs and guides tell users to put models in ./spin/ai-models.

I tried also with llama2-chat. Which also works when I put the corresponding model file in ./spin/ai-models/llama

tree . -al
.
├── ai-models
│   └── llama
│       ├── codellama-instruct
│       └── llama2-chat
└── logs
    ├── dotnetpro-api_stderr.txt
    └── dotnetpro-api_stdout.txt

BTW @itowlson : Just to verify, I use the folder ai-models (dash) - as mentioned in the docs - , you suggested using ai_models (underscore)... is that folder checked as well?

from spin.

At this point I wonder if we wanna update docs/guides to match what you suggested or - if this is a bug and should be changed in Spin?

WDYT @itowlson

If we go with updating docs and guides, I'll close this one and open a corresponding issue to update the docs. Otherwise, we can use this to track the fix in Spin itself

from spin.

I believe the way this currently works is how it's always worked which would point to this being a bug in docs. If I remember currently, specifying the architecture (which is what this additional directory does) is necessary for us to know how to run the model correctly.

from spin.