Crashes on macOS with Llama2 model about llmfarm HOT 18 CLOSED

https://github.com/lake-of-fire/SwiftWhisperStream/blob/master/Sources/SwiftLlama/AI.swift

(which will be launched at https://chatonmac.com if you're curious)

just FYI some minor style feedback, you don't need parens for if-statements and the like in swift (can save you some typing)

from llmfarm.

I have updated the version of llama.cpp and will soon upload it to testflight, I hope the crashes will stop in the new version. Unfortunately, from your reports, I was unable to draw an unambiguous conclusion about the cause of the crashes.

from llmfarm.

This is because the context is shifted incorrectly when overflowing. In the new version of llama.cpp it seems to be possible to do it correctly. I will add it to the update as soon as I figure it out.

from llmfarm.

Thanks. I just realized Orca 3B works with Metal. So it was just llama2 3B that spews nonsense. I'll try your config.

from llmfarm.

Is it possible to reduce the UI hang on model initialization? I couldn't find any open src yet for doing multi-process control of it in swift

I know how to make swiftui not hang during model loading. I will do it in one of the next releases. Right now there are many more priority tasks. But thanks for the offer of help.

from llmfarm.

Let me know if you don't have access to the crash logs via TestFlight. Latest stable macOS release, your Llama2 7B model file, default settings. After chatting a few times in a row, it crashes.

Unfortunately, I don’t see any crash reports for the MacOS version. Tell me, does it crash when metal is on?

from llmfarm.

LLMFarm crashes.zip
I believe I was exclusively testing Metal because without it's unusably slow. I tried it without Metal once but didn't try to repro the crashes.

from llmfarm.

from llmfarm.

it would be nice to use the newly merged llama.cpp ios build as well. no examples for it exist yet

from llmfarm.

Tell me, did the update solve the problem?

from llmfarm.

from llmfarm.

Tell me, did the update solve the problem?

from llmfarm.

unfortunately it started reverting to german and then long rambling about a book I didn't mention after a couple messages (continuing the chat I had from above, without creating a new one - let me know if there's something else I should try)

from llmfarm.

Is there a config that is working for you on your device?

from llmfarm.

i use this config for test llama2
llama-2-7b-chat-q4_K_M_1693728799.json

from llmfarm.

Is it possible to reduce the UI hang on model initialization? I couldn't find any open src yet for doing multi-process control of it in swift

from llmfarm.

FYI I was able to easily do it in my fork by changing AI to an actor (with a couple associated changes). I won't contribute the changes back because the fork's diverged a lot but it's there if you need the reference.

from llmfarm.

FYI I was able to easily do it in my fork by changing AI to an actor (with a couple associated changes). I won't contribute the changes back because the fork's diverged a lot but it's there if you need the reference.

I seem to have solved the problem in the new version. I am new to swift so would love to see your solution. But I don't see your fork for some reason.

from llmfarm.