lmg-anon / mikupad Goto Github PK

Being able to see probability on the kobold API backend would be a nice addition

Was looking for something more story focused than chat based for a D&D storytelling AI and a gaming clan story making AI, I created in silly tavern.
Both have an extensive lore book.
I dont not see anyway to import those lore books into MIkupad though, so first question:

1. Do I have to redo the lore books?
   If its yes to the above then thats not too much of an issue, however it leads me to the next question, I dont see where the lorebooks are saved?
2. where are the lorebooks saved?
   We also have a bard AI that makes song lyrics, he will use either lorebook as the case my arise but in the webUI it looks like the lorebooks are tied to sessions, can they not be switched?
3. how do we switch lorebooks on the fly between existing ones?

As an aside, what originally made me look for alternatives to silly tavern (besides it just not being the right UI for our needs) is that the RAG support has been untouched and unloved for months, is there plans to add RAG support into Mikupad? we have over 30 years of D&D journals from sessions in txt format...14mb... would be good to have that searched and referenced by the story teller.

Undo/Redo currently works in a very simplistic way, it doesn't track for example changes made by the user. I like how NovelAI's undo/redo works so I think doing something like that would be nice.

what if u have npm start so u dont need both start.sh and start.bat and also it would just open it in ur browser

Edit: Sorry, I didn't realize this is Issue #12. Close?

Often we have to adjust the display width to realign the token highlights.

As a non-coder at first I imagined it has something to do with the highlight hitting the margin before the text does. When you shrink the display, you can see the highlight get pushed to the next line before the text. But it's not as simple as extending the highlighting's margin by 1 pixel. I remember that most tokens start with a space, and I notice a space at the end of a line may or may not be "rendered" depending on whether it fits.

No clue how to solve this without adding bloat. SillyTavern does Token Probabilities in a separate panel with uneditable words and only for the newest message. Mikupad has highlighting over editable text because it's cool like that.

It can be incredibly helpful for readability, especially in those circumstances where Mikupad is used to hand-craft data for training or in-context learning (which most often needs to be formatted in a specific or at least in a consistent way) to have some form of syntax highlighting, in a way similar to what SillyTavern does for dialogue and actions. However, given the free-form nature of text completion, it would be useful if customizable rules could be used instead of fixed ones.

To make UI implementation simpler, every Mikupad theme could come bundled a number of predefined "style classes" (designed to work properly with the theme and affecting things like back/foreground color, bold/italic/underline or even size, etc), and the custom regex matches could then be associated with them.

As a demonstration of the general idea, below is an example of a short chat-like formatted text with various portions styled for easy recognition. The styled example is from a text editor with syntax highlighting (+ additional image editing).

Since this is a novel-esque frontend I think it's only natural someone might want to export the main prompt box to plaintext format to upload it somewhere else for example. Though one can just copy and paste it to notepad I guess, so it's not really high priority.

I think the current way the promptOverlay works isn't stable enough. Probably the only way to solve this kind of problem is by throwing away the promptOverlay and doing something like this.

Like the title says. Sometimes when the text's in a bad position, and you hover over the edges of the text area, it shifts things by a line. I've attached a video to show what I'm talking about.
output.webm
Using Firefox 125.0.2

I get the following errors when attempting to access APIs on other servers (regardless of whether it's KoboldCpp, OpenRouter, or OpenAI proper). This happens both using the hosted demo and when running from the HTML file locally:

Firefox

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openrouter.ai/v1/internal/token-count. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openrouter.ai/v1/internal/token-count. (Reason: CORS request did not succeed). Status code: (null).

Uncaught TypeError: NetworkError when attempting to fetch resource.
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openrouter.ai/v1/models. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openrouter.ai/v1/models. (Reason: CORS request did not succeed). Status code: (null).

Uncaught TypeError: NetworkError when attempting to fetch resource.
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openrouter.ai/v1/token/encode. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openrouter.ai/v1/token/encode. (Reason: CORS request did not succeed). Status code: (null).

Uncaught TypeError: NetworkError when attempting to fetch resource.

Chrome

Access to fetch at 'https://openrouter.ai/v1/completions' from origin 'https://lmg-anon.github.io' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
openrouter.ai/v1/completions:1

   Failed to load resource: net::ERR_FAILED

mikupad.html:1165

   Uncaught TypeError: Failed to fetch
at openaiCompletion (mikupad.html:1165:20)
at openaiCompletion.next (<anonymous>)
at completion (mikupad.html:911:18)
at async predict (mikupad.html:2194:21)

mikupad.html:1 Access to fetch at 'https://openrouter.ai/v1/internal/token-count' from origin 'https://lmg-anon.github.io' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
openrouter.ai/v1/internal/token-count:1

   Failed to load resource: net::ERR_FAILED

mikupad.html:1072

   Uncaught TypeError: Failed to fetch
at openaiOobaTokenCount (mikupad.html:1072:21)
at getTokenCount (mikupad.html:885:23)
at mikupad.html:2417:30

mikupad.html:1 Access to fetch at 'https://openrouter.ai/v1/token/encode' from origin 'https://lmg-anon.github.io' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
openrouter.ai/v1/token/encode:1

   Failed to load resource: net::ERR_FAILED

mikupad.html:1094

   Uncaught TypeError: Failed to fetch
at openaiTabbyTokenCount (mikupad.html:1094:21)
at getTokenCount (mikupad.html:888:23)
at async mikupad.html:2417:24

Then I tried to use the Together AI text completion endpoint, it complained about logprobs being > 1 and it wouldn't let me generate tokens unless it was 1.

According to the API docs, logprobs is indeed "Number of top-k logprobs to return". But no matter how I passed the parameter, it complained whenever it was larger than 1, despite the docs saying its maximum value is a full int32. I ignore if there's a way around this, but I didn't find it.

The error looks like this:

{
    "message": "5 is greater than the maximum of 1 - 'logprobs'",
    "type": "invalid_request_error",
    "param": "logprobs",
    "code": null
}

On the other hand, models don't show up in the model dropdown because the list fetched from the API is not parsed correctly. The current code expects the list to be wrapped inside a data property, but the API's response contains the list straight.

I made pull request #54 as a way to fix this.

This is probably another one for when the live syntax highlighting thing is implemented, but I feel like it'd be nice to always have that visual feedback available.

It was recently added in 1.55.1 and I think it's a really cool feature :)

LostRuins/koboldcpp#600

It would be nice to incorporate a way to have this being accessible over your LAN, such as a flag like --listen "local_IP"

This would make it accessible on mobile devices as well as other devises where the database is not stored, ie. being hosted a server.

I really like the simplicity and simple writing pad of the software, and an addition like this would enhance it greatly.

Currently, mikupad will only cache its imports in the browser cache, so if that is cleared, and you do not have an internet connection, it won't load.
I'm not entirely sure if it's possible, but it'd be great if it could save a version of the dependencies locally, like it does with the sessions and settings, to fall back on when not connected to the internet later.

Not sure if this is intentional behaviour, but recent updates seem to have made the token probability display mutually exclusive with the highlight generate tokens option.

In older versions I could have the highlight disabled and still see tokens probabilities, but this is no longer the case; when you enable the highlight, the submenu for probabilities visibility disappears.

How it used to work in older versions:

How it works in most recent version:

Not a big deal, but it would be nice to see token probabilities without forcing you to turn highlighting on.

Hi! I've been trying to use mikupad with my custom OpenAI API server, but I don't have a /v1/token/encode endpoint. I was adding it, but unsure what the request body looks like from mikupad's side, and what sort of response it expects.

It would be nice to have a global storage for backends, allowing you to quickly switch between saved backend configurations.

A usability issue is that the position of the predict/cancel/regenerate buttons shifts around depending on the current app state. This makes them somewhat annoying to use with the mouse when there's a need to repeatedly use them.

Have you guys thought about creating a discord server? I see mikupad has a lot of potential, and making communities and gaining support are easier in discord, especially for those who are new to github.

One issue I have, but that seems to be on purpose, is that the cursor, after running the prediction, does not go to the end of the new generated text.

Personally I'd find it useful because if I like what the AI generated, I may wish to continue writing from there. Additionally, with very long generations, the text area will scroll down to the end, but the cursor will be up and out of view, which seems a bit unintuitive to me.

The code doing this has the following comment:

// Update the textarea in an uncontrolled way so the user doesn't lose their
// selection or cursor position during prediction

Since it is mentioned that it also preserves "selection", I assume there is a genuine case for wanting to preserve a selection. However, with no selected text, I find this hard to work with if I want to co-write with the AI.

For personal use I will modify it to move the cursor to the end unless there is a selection, but don't know if a PR for this would be in order.

Would it be possible to add Top_P, Top_K, to the ui?

It would be convenient for testing different instruct models in a single session, where if you select a different template, it would search and replace the prefixes/suffixes in the current session.

First of all thank you for sharing this work, it's great for conducting experiments :)
As of now, prompt templates are quite heavy on the eye when multiple messages are exchanged between the user and the LLM.
It would be great if there was an option to automatically collapse all text matching a set of strings to say <…>, so that it doesn't visually clutter the screen.
The text would expand and be editable when clicking.
The idea is a bit like https://github.com/KeitaNakamura/tex-conceal.vim...

Not too familiar with what options the various backends make available, but if at all possible, a field to ban or bias certain tokens and phrases would be great to have.

It would be great to have improved support when accessing from a mobile, i.e. a simple responsive design.
I have the following suggestions:

• collapsible top menu instead of fixed sidebar
• larger font size
• disabling text overlay, I noticed a significant computational effort (but possibly that could also be the fast streaming)

I just noticed if you set say minp=0.15 and turn off all the other samplers, that the token probability popup still shows all the remaining tokens with "0.00" probability. It would be nice if these could be hidden so as to get an idea at a glance what tokens have non-zero probability.

All the default themes show up very badly on e-ink tablet screens due to all of the themes having non-white backgrounds.

I've put in a PR with a new "eReader" theme that should improve this and will link it here now.

I love this project and think it could be really great one day with more improvements. Some directions I think would be interesting to consider:

Put the sessions box into its own panel and on the left side, mirroring the panel on the right. I've got so many sessions now it would be useful to have a larger list of them visible all the time.

The ability to give sessions tags and filtering the session list by tags, would also be useful.

A session branching tree system. Not entirely unlike in chat frontends, but something made for generic completion, and more powerful. For instance, you could have a single session, that then have branches, or children. They don't even need to actually share any of the same text. You could imagine it like the parent and child object system in 3D game engines (https://youtu.be/XtQMytORBmM?t=273). So for instance, there might be a button that says "clone to child" or something to that effect, and instead of creating an adjacent copy of the session, it creates a child. Perhaps the Regenerate button can be set, by the user, to automatically spawn a new child (or not), thereby creating the equivalent of a branch in LLM Chat frontends.

Generate from anywhere you place your cursor. This would be useful, say, when I just want the LLM to fill in something. Of course, most LLMs these days are not trained to be able to do fill in the middle tasks, but it would still be useful to have the ability to just start generating text somewhere back in context, instead of copying the bottom half of the text to the clipboard, deleting it, generating, and then pasting it back in.

The ability to set how many token probabilities you want to see listed. Sometimes I just want to see more possible logits. Letting the user control how many they want to see would be useful.

The ability to run commands on your desktop by scanning the context. For instance, maybe I'm working on something in Mikupad and want to let it do something on my PC, I could explicitly write out what commands I'd like to whitelist, and then it could scan the context and execute when it encounters those commands. I assume this kind of functionality is not possible through an HTML file and we'd need a different strategy to get it working. But, still, just an idea of something that I'd personally love to see happen somehow. Perhaps a script on your desktop could monitor the files from the browser, not sure.

A sampler settings save system, so we can change them quickly to certain specific values.

Highlight the instruct formatting strings or color them. Sometimes the text just visually feels like a huge goopy block when doing chat mode, depending on the model. Having the instruct formatting strings stick out in some way (say by making the text green?) would make it a lot easier to look at.

Overall I think some of these changes could make this a really powerful tool, more than it already is. I don't expect anyone to do this of course though. This project is already a gift.

The prompts should have </s> (or whatever their specific eos_token is) appended after the assistant's reply? eg:

USER: Write me the opening chapter of a 'Grimdark' stories trilogy in the style of Joe Abercrombie and Rob J Hayes. Use third-person omniscient POV. The boy king has just come to power, but the machinery of politics is moving fast... ASSISTANT: In the shadowed halls of the Blackspire ... his strings pulled by unseen hands in a dance that would end only in tragedy.</s>USER: continue ASSISTANT:...

I think Alpaca may be the only prompt template that doesn't use this (or at least it seems to screw the very few models that use multi-turn Alpaca like phind-codellama and some of the older wizard models).

Cool tool BTW - very useful for testing creative writing models with! 👍

With the implementation of token bias, the next step would now be phrase bias. The tokenize/detokenize functions which are used for the token bias implementation, of course, provide the tokenizations of the entire input string, but currently only the first token of multi-token strings is biased.

As I see it, the way to implement this would be the following (for negative bias):

1. Remove all multi-token strings from the normal bias function
2. Monitor the streamed tokens for the multi-token strings, stop generation when one of the strings is found, and remove the biased string from context
3. bias the first token of the string (possibly the remaining tokens of the string as well)
4. generate the number of tokens of the biased string, then stop again
5. remove biases from the string
6. resume normal generation

As for positive bias, I'm not quite sure how it should be implemented. Possibly biasing the first token, stopping when it's generated, then biasing the rest of the tokens?

I'm working on it, but the fact that most of my knowledge of React comes from reading the source of this project is coming back to bite me in the ass, as I can't get my state variables to update correctly. I'll probably have to read up on a lot of shit before I can get it anywhere near functional.
The code I've written so far's fair game if anyone wants to have a go at it. (Though you'd probably be better off just starting over.)

Just wanted to say your project is really great, I was wondering if you guys could add a characters tab (similar to lorebook) For writing, would be helpful! Also, if possible, some sort of extra RAG option maybe? if not, that's fine.

While text is being streamed during prediction, you can still press Ctrl+R to regenerate. This causes the text window to print tokens being streamed from two different requests and inadvertently interleaves the output resulting in jumbled text.

Started working on phrase biasing, and being able to monitor the stream of token ids instead of the token strings would make life much easier for me. I'd do it myself, but I couldn't figure it out.