Support for Phi-3 models about llmfarm HOT 7 CLOSED

development version

from llmfarm.

Make sure Metal=on, BOS=on, EOS=off. And try setting contextsize=1024. I got 8-9 Tok/sec.

Officially phi3 is only supported starting with llama.cpp release b2717. The latest LLMFarm commit uses b2692. The Testflight version uses b2135 which officially supports only phi2.

from llmfarm.

Hi. work normal with this template

<|user|>
{{prompt}}<|end|>
<|assistant|>

And BOS option enabled.

from llmfarm.

Hi. How can I make it generate until EOS? If I select the option, the app crashes.

from llmfarm.

Hi. work normal with this template

<|user|>
{{prompt}}<|end|>
<|assistant|>

And BOS option enabled.

BOS is enabled, I have set that prompt, but I am getting an error as reply for every message:
Load Model Error: [Error]
modelLoad Error
Load Model Error: [Done]

from llmfarm.

@guinmoon when you say "works normal" are you referring to the development version or the version in the App store?

The stable version from the app store isn't honoring the end token and the app crashes if you try enabling EOS.

from llmfarm.

Hi. work normal with this template

<|user|>
{{prompt}}<|end|>
<|assistant|>

And BOS option enabled.

BOS is enabled, I have set that prompt, but I am getting an error as reply for every message: Load Model Error: [Error] modelLoad Error Load Model Error: [Done]

In the TestFlight version I’m using ‘Phi-3-mini-4k-instruct-q4.gguf’

When setting up, I used the “Phi 2” setting template and then wrote the recommended prompt. On my iPhone 14 Pro I’m getting around 2-5 token per second.

Sometimes the <|end|> tag isn’t handle correctly, and it just skips over it and starts a new answer

from llmfarm.