Comments (28)
Hey, I'd like to thank you for the amazing program and the continuous improvements.
I was curious to try this new feature with llama-cpp since I don't use ollama and it seems to be working:
What do you call a pineapple that lives under the sea? <ctrl+space>
What do you call a pineapple that lives under the sea? An under-pine in an a-peeling disguise!
With an API base URL: http://localhost:3030/v1/chat/completions
Is there anything specific you'd like me to test?
from qownnotes.
It seems that in Ollama, you also need to append the /v1/chat/completions
endpoint to the hostname, according to the OpenAI compatibility ยท Ollama Blog.
from qownnotes.
@eljamm, if you want to contribute a script for llama-cpp to https://github.com/qownnotes/scripts it would be great. ๐
Would be almost the same as https://github.com/qownnotes/scripts/tree/master/ollama-integration anyway ๐
from qownnotes.
Yes, that would be the next logical step. ๐ As long they are OpenAI API compatible...
from qownnotes.
Which self-hosted AI did you have in mind?
from qownnotes.
e.g. ollama https://github.com/ollama/ollama/blob/main/docs/api.md
https://ollama.com/blog/openai-compatibility
from qownnotes.
Ok, thank you. I'm currently struggling to get this into the script engine. I may need to rewrite some parts of the AI service handling...
from qownnotes.
I had to rewrite and rearrange a lot of stuff to make this happen. ๐
24.6.2
- A new scripting hook
openAiBackendsHook
was added, that is called when the OpenAI
service config is reloaded (for #3037)- For example, this is also done when the script engine is reloaded
- You can use it to provide config for custom OpenAI backends, like your own OpenAI API compatible LLMs
- Please take a look at the
openAiBackendsHook documentation
for more information
from qownnotes.
There now is a new release, could you please test it and report if it works for you?
from qownnotes.
I've already found some small issues (like you need to set an apiKey
, but there is none in ollama and you still need to reload the script engine manually after adding the script so the OpenAI UI reloads). But the main issue with ollama seems to be that the API result doesn't seem to be compatible with the API result from OpenAI. I'm currently looking into what's different. I didn't have this issue with Groq...
from qownnotes.
Hm, I only got an empty result back from ollama over the API. ๐ค What is your experience?
from qownnotes.
I added a script ollama-integration
to the script repository for you to test.
from qownnotes.
The requests are the same with the OpenAI API and Ollama, but what Ollama returns with https://github.com/ollama/ollama?tab=readme-ov-file#chat-with-a-model is not at all what is returned at https://platform.openai.com/docs/api-reference/chat/create!
from qownnotes.
Nice! Thank you very much for the hint. I'll try that then!
from qownnotes.
Hey, I'd like to thank you for the amazing program and the continuous improvements.
Thank you! ๐ This feature took longer than anticipated (and will take some more tweaking)!
from qownnotes.
It seems that in Ollama, you also need to append the /v1/chat/completions endpoint to the hostname
Works perfectly, thank you very much! http://127.0.0.1:11434/v1/chat/completions
works perfectly!
from qownnotes.
@pbek Glad I could help and I don't mind writing the script for llama-cpp, but in my opinion it would it be better if the script was written for any local AI server in general since they more than likely support the OpenAI endpoint and it would be redundant to re-write the script for each one every time.
What do you think?
from qownnotes.
Wrote a PR in qownnotes/scripts#237
from qownnotes.
Please see qownnotes/scripts#237 (comment). One script only supporting one endpoint will not do it if you have multiple backends...
from qownnotes.
Plus, you need to find out the correct endpoints yourself...
from qownnotes.
Well, as far as I know, all OpenAI-compatible backends support the /v1/chat/completions
endpoint, so the only thing that will change will be the server address.
In this regard, I thought a general script would be better as it's like a template which users can use to make their own scripts.
This means that we only have to maintain one script and the users won't think only the ollama or llama-cpp backends are supported.
from qownnotes.
Is something like qownnotes/scripts#238 what you had in mind?
from qownnotes.
it is already working with ollama. I could test it now. Thank you for your effort including this.
From the user perspective I have the following expectations:
- since you only need the endpoint to the ollama instance. I would expect I have to set my specific local URL on the same section (settings page -> AI) where the user sets the API keys for the other AI backends. Not a new script I have to include. That is also the case for other backends like llama-cpp. I would just expect a new input field in the AI section. If I set it then it will be available to choose from the "enable AI support" menu.
- I don't want to remember the models I have available on my ollama instance and then have to set them as a variable in qownnotes. I would expect that after setting my endpoint it automatically gets/reads all available models. So that I can choose from them, as I would with the Groq or the OpenAI backend.
But like I said, it is working for me and again thank you for adding this. Those expectations are just improvements for the usability that I wanted to share. It did confuse me for a second why I had to add another script to add my own AI endpoint, although there is a dedicated AI settings page.
from qownnotes.
since you only need the endpoint to the ollama instance. I would expect I have to set my specific local URL on the same section (settings page -> AI) where the user sets the API keys for the other AI backends. Not a new script I have to include. That is also the case for other backends like llama-cpp. I would just expect a new input field in the AI section. If I set it then it will be available to choose from the "enable AI support" menu.
I didn't want to go through the hassle of implementing a UI for adding multiple backends. Doing that in scripts turned out to be a hassle too, but at least it's more flexible now. There can be scripts preconfigured for certain backends, so you don't need to research the endpoint URL.
I don't want to remember the models I have available on my ollama instance and then have to set them as a variable in qownnotes. I would expect that after setting my endpoint it automatically gets/reads all available models. So that I can choose from them, as I would with the Groq or the OpenAI backend.
In the 3rd party OpenAI UIs I worked with, you were supposed to configure them too, because maybe you don't want to pay for an expensive one.
Is there even an OpenAI API to get the models?
But since the custom ones have script now those network requests to fetch the models (even to non-OpenAI APIs) could be done in those scripts! ๐ฅณ๐
from qownnotes.
24.6.3
- Improvements to the new scripting hook
openAiBackendsHook
were made
(for #3037)- The OpenAI UI elements will now be reloaded properly, when the script-engine
is reloaded after a script was installed - The
apiKey
attribute in the scripting hook is now optional, because local
APIs don't need it - There now is a script
ollama-integration
in the script repository that
allows integration with Ollama - There now is a script
llama-cpp-integration
in the script repository that
allows integration with llama.cpp (for #238, thank you, @eljamm) - In the AI settings there now also is the script-repository mentioned
- The OpenAI UI elements will now be reloaded properly, when the script-engine
There now is a new release, could you please test it and report if it works for you?
from qownnotes.
Tested again and everything is working as expected.
from qownnotes.
Thanks for testing!
from qownnotes.
@speedyconzales, you might like this: qownnotes/scripts@93735c6 (ollama-integration: fetch installed models automatically
)
from qownnotes.
Related Issues (20)
- [BUG] Line count font size does not resize with text HOT 9
- [BUG] Slow accessing settings after Nextcloud connection is defined HOT 7
- [FEATURE] Make text highlighting themeable too HOT 9
- [FEATURE] Option to show full file path in statusbar HOT 16
- [BUG] Freezes and "Not responding" when reloading script engine on Windows HOT 17
- [BUG] List Dot on the left side when in RTL mode in the preview window HOT 5
- [FEATURE] Add "AI" support HOT 20
- [BUG] Windows - Opening settings is slow -> Some registry keys are read 72 times HOT 12
- [I] "Move notes to subfolder..." does not stay in folder but jumps to root folder HOT 2
- [BUG] macdeoloyqt not found with jurplel/install-qt-action@v4 HOT 2
- [BUG] xml formatting in markdown HOT 4
- [FEATURE] Shortcut for switching from ltr to trl and dynamic one HOT 2
- [FEATURE][USABILITY] Switching between INS/OVR mode in QOwnNotes HOT 8
- [BUG] Cannot disable dark mode HOT 2
- [ Ctrl + insert ] does not copy selected text HOT 5
- [I] The notes were encrypted by me a few days ago with a password . Today i tried to decrypt them back and they are decrypting into some kind of other encryption HOT 1
- [FEATURE] the inserted images should be visible in the editor as well ( not only int the preview ) HOT 1
- [FEATURE] the preview should reload at changes HOT 3
- [SUPPORT] Newbie's questions HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qownnotes.