serenadeai / serenade Goto Github PK

System Information

Serenade Version: v2.0.1

OS and Version: Windows 11 Version 10.0.22000 Build 22000

Application: VS Code

Issue Description

Bug 1 - Object Methods

When it comes to adding methods to an object, if we use the following shorthand, Serenade will stop working.

const example = {
  name: "Test",
  getName() {
    return this.name;
  }
};

If any object in the document has a method with the previous syntax, Serenade will throw an error with every subsequent command.

Bug 2 - Generator Functions

If there's any generator function present in your code, Serenade will throw an error with every subsequent command.

function* example() {
  yield "Hello World";
}

It doesn't matter where the * is, after the function keyword, before the function name or in the middle.

Bug 3 - Voice Commands to Traverse Arrays

Serenade groups all array elements on a global scale. As it currently stands, it's quite difficult to refactor arrays with Serenade. Let me explain. Example code:

function test() {
  const a = [1, 2];
}

function test2() {
  const b = [3, 4];
  const c = [5, 6, 7];
}

Saying element two takes me to the number 2 in the first array.

Saying element five takes me to the number 5 in the third array.

It doesn't matter where my cursor is, Serenade treats all elements as if they're in the same array. I think the expected behavior is that these commands would prioritize the arrays that are on the same line as the cursor. Or in case of nested arrays it would prioritize the array that the cursor is currently inside of.

Bug 4 - Private Methods & Attributes

In TypeScript, Serenade handles private methods and attributes well.
If I'm inside a class and say: add private method hello, I get what you would expect:

class Example {
  private hello() {
  }
}

However, I get the same result in JavaScript, which is wrong. In JavaScript, private methods and attributes are declared with a # sign. So, following the previous example, in JavaScript I should get:

class Example {
  #hello() {
  }
}

(Maybe related to "#9", but maybe not?)

System Information

Serenade Version: v2.0.2
OS and Version: Kubuntu 20.04.3 LTS (based on Ubuntu 20.04.3 Focal Fossa)
Application: VS Code

Issue Description

I downloaded "Serenade-2.0.2.AppImage", then ran it from my download folder. It opened the 2 windows + tray icon perfectly and I went through the tutorial, and I used it to install the VSCode plugin, and went into the settings and read tutorials, etc, all with a single execution of the AppImage. I also clicked on the setting to use verbose logging and viewed the log file, but eventually disabled verbose logging. Everything seemed to be working perfectly. Then after an hour I closed Serenade.

An hour later, I opened the same Serenade AppImage, but now it opens 2 empty windows (and the tray icon) and shows that it's busy with the "Loading..." animation, staying like that with the animation spinning even after 30 minutes of waiting.

I tried running it again multiple times, it keeps showing the same "Loading..." animation.
I tried deleting my whole "~/.serenade" folder and then running it, but that didn't change anything.

Note that the terminal does show this message whenever I run Serenade:
[2103072:0926/145012.999969:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.

How to Reproduce

Run "Serenade-2.0.2.AppImage" on Ubuntu 20.04, then close it, then run it again. (I'm not sure if the things I did in the settings & plugins caused a problem or if that was unrelated).

Screenshots

I'm also attaching the result of strace, in case it helps see what it's waiting for in an endless loop:
strace ./Serenade-2.0.2.AppImage 2> strace.txt

strace.txt

Note that I don't have a standalone GPU, I'm just using onboard graphics, and I'm using X not Wayland:

$ lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630]
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
00:14.2 Signal processing controller: Intel Corporation 200 Series PCH Thermal Subsystem
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.3 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Z370 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

System Information

Serenade Version: e.g., v2.0.2

OS and Version: e.g., Window 10

Application: e.g., VS Code

Issue Description

There is a javascript advanced tutorial in Serenade's settings but it opens a Python tutorial.

How to Reproduce

Open Serenade, go to settings -> docs -> scroll down to Advanced Javascript -> click open

Screenshots

System Information

Serenade Version: v2.0.1
OS and Version: Windows 10 Enterprise Build 19041 (20H2)
Application: Serenade

Issue Description

Serenade AI settings menu opens blank.

How to Reproduce

1. Open Serenade AI
2. Click the cog icon to the right of the "Paused" button

Screenshots

Context

Would it be acceptable for me to add Support for FSharp inside vscode and rider? I would look into how the csharp support was made and would try to replicate that with FSharp. But before I put any time into that, I first want your accepted Tag so that an eventual PR is accepted :) Also any help and pointers about where to start would be appreciated.

Goals

Speech to FSharp Code

Proposed Solution

Language Support

System Information

Serenade Version: 2.0.2, Pycharm Serenade plug-in 0.0.11

OS and Version: e.g., macOS Sonoma 14.4.1

Application: Chrome, PyCharm Community Edition, VS Code

Issue Description

Visual distraction, annoyance - Serenade should not be notifying users to install a plug-in they already have, and it already recognizes. Seems to know I'm in Jetbrains/PyCharm or Chrome as Serenade UI shows the logo of the given app (as opposed to "Other"). Occurs in both compact and standard UI. More minor issue with when closing a VS Code window as well.

How to Reproduce

Open Serenade, Open PyCharm Community Edition, install Serenade plug-in for PyCharm. Navigate away from PyCharm, navigate back - every time you navigate back to PyCharm you should get a new "Did you know?" pop-up notification in the Serenade UI. Same with if you open and close PyCharm.

Occasionally also happens in Chrome but not currently able to reproduce that. Very occasionally happens with VS Code plug-in as well. With VS Code when I close a given window it tells me this not great message to re-install, when it's simply because I closed a given window.

Screenshots

System Information

Serenade Version: v2.0.2

OS and Version: Ubuntu 22.04.2

Application: IntelliJ

Issue Description

When using Serenade on Ubuntu and having IntelliJ IDE open and in focus, an issue arises where Serenade incorrectly detects that Atom editor is selected instead of IntelliJ.

How to Reproduce

1. Ensure that Serenade is installed and running on Ubuntu.
2. Open the IntelliJ IDE.
3. Make sure that the IntelliJ IDE is in focus (active window).
4. Observe that Serenade incorrectly detects that Atom editor is selected instead of IntelliJ.

Screenshots

Additional info

Expected Behavior

Serenade should accurately detect the active application and recognize that IntelliJ IDE is in focus when it is open and being used.

Actual Behavior

Serenade incorrectly identifies the Atom editor as the active application when IntelliJ IDE is open and in focus. This leads to voice commands not working correctly.

Investigation:

During my investigation of this issue, I found next:

in function determineActiveApplication()(serenade/client/src/main/execute/system.ts)
I noticed that if function result includes atom (result.includes("atom")) serenade decides that the atom is in focus, and that this is being checked before checking if the application that is selected is IntelliJ

Further investigation how driver.getActiveApplication() works and what is the result of this function on my machine I discovered this by running a simple script from inside IntelliJ's terminal app (so IntelliJ would be in focuse 😄 )

const driver = require('serenade-driver')

driver.getActiveApplication().then(result => {
    console.log(result)
})

// /opt/idea/idea-iu-222.3345.118/jbr/bin/java-classpath/opt/idea/idea-iu-222.3345.118/lib/app.jar:/opt/idea/idea-iu-222.3345.118/lib/3rd-party-rt.jar:/opt/idea/idea-iu-222.3345.118/lib/util.jar:/opt/idea/idea-iu-222.3345.118/lib/util_rt.jar:/opt/idea/idea-iu-222.3345.118/lib/util-8.jar:/opt/idea/idea-iu-222.3345.118/lib/jps-model.jar:/opt/idea/idea-iu-222.3345.118/lib/stats.jar:/opt/idea/idea-iu-222.3345.118/lib/protobuf.jar:/opt/idea/idea-iu-222.3345.118/lib/external-system-rt.jar:/opt/idea/idea-iu-222.3345.118/lib/intellij-test-discovery.jar:/opt/idea/idea-iu-222.3345.118/lib/forms_rt.jar:/opt/idea/idea-iu-222.3345.118/lib/rd.jar:/opt/idea/idea-iu-222.3345.118/lib/externalprocess-rt.jar:/opt/idea/idea-iu-222.3345.118/lib/annotations-java5.jar:/opt/idea/idea-iu-222.3345.118/lib/annotations.jar:/opt/idea/idea-iu-222.3345.118/lib/async-profiler-windows.jar:/opt/idea/idea-iu-222.3345.118/lib/async-profiler.jar:/opt/idea/idea-iu-222.3345.118/lib/byte-buddy-agent.jar:/opt/idea/idea-iu-222.3345.118/lib/error-prone-annotations.jar:/opt/idea/idea-iu-222.3345.118/lib/groovy.jar:/opt/idea/idea-iu-222.3345.118/lib/idea_rt.jar:/opt/idea/idea-iu-222.3345.118/lib/intellij-coverage-agent-1.0.706.jar:/opt/idea/idea-iu-222.3345.118/lib/jsch-agent.jar:/opt/idea/idea-iu-222.3345.118/lib/junit.jar:/opt/idea/idea-iu-222.3345.118/lib/junit4.jar:/opt/idea/idea-iu-222.3345.118/lib/ant/lib/ant.jar-xx:errorfile=/home/milos/java_error_in_idea_%p.log-xx:heapdumppath=/home/milos/java_error_in_idea_.hprof-xms128m-xmx750m-xx:reservedcodecachesize=512m-xx:+useg1gc-xx:softreflrupolicymspermb=50-xx:cicompilercount=2-xx:+heapdumponoutofmemoryerror-xx:-omitstacktraceinfastthrow-xx:+ignoreunrecognizedvmoptions-xx:compilecommand=exclude,com/intellij/openapi/vfs/impl/filepartnoderoot,triedescend-ea-dsun.io.usecanoncaches=false-dsun.java2d.metal=true-djbr.catch.sigabrt=true-djdk.http.auth.tunneling.disabledschemes=""-djdk.attach.allowattachself=true-djdk.module.illegalaccess.silent=true-dkotlinx.coroutines.debug=off-dsun.tools.attach.tmp.only=true-xmx4096m-djb.vmoptionsfile=/home/milos/.config/jetbrains/intellijidea2023.1/idea64.vmoptions-djava.system.class.loader=com.intellij.util.lang.pathclassloader-didea.vendor.name=jetbrains-didea.paths.selector=intellijidea2023.1-djna.boot.library.path=/opt/idea/idea-iu-222.3345.118/lib/jna/amd64-dpty4j.preferred.native.folder=/opt/idea/idea-iu-222.3345.118/lib/pty4j-djna.nosys=true-djna.noclasspath=true-dsplash=true--add-opens=java.base/java.io=all-unnamed--add-opens=java.base/java.lang=all-unnamed--add-opens=java.base/java.lang.ref=all-unnamed--add-opens=java.base/java.lang.reflect=all-unnamed--add-opens=java.base/java.net=all-unnamed--add-opens=java.base/java.nio=all-unnamed--add-opens=java.base/java.nio.charset=all-unnamed--add-opens=java.base/java.text=all-unnamed--add-opens=java.base/java.time=all-unnamed--add-opens=java.base/java.util=all-unnamed--add-opens=java.base/java.util.concurrent=all-unnamed--add-opens=java.base/java.util.concurrent.atomic=all-unnamed--add-opens=java.base/jdk.internal.vm=all-unnamed--add-opens=java.base/sun.nio.ch=all-unnamed--add-opens=java.base/sun.nio.fs=all-unnamed--add-opens=java.base/sun.security.ssl=all-unnamed--add-opens=java.base/sun.security.util=all-unnamed--add-opens=java.base/sun.net.dns=all-unnamed--add-opens=java.desktop/com.sun.java.swing.plaf.gtk=all-unnamed--add-opens=java.desktop/java.awt=all-unnamed--add-opens=java.desktop/java.awt.dnd.peer=all-unnamed--add-opens=java.desktop/java.awt.event=all-unnamed--add-opens=java.desktop/java.awt.image=all-unnamed--add-opens=java.desktop/java.awt.peer=all-unnamed--add-opens=java.desktop/java.awt.font=all-unnamed--add-opens=java.desktop/javax.swing=all-unnamed--add-opens=java.desktop/javax.swing.plaf.basic=all-unnamed--add-opens=java.desktop/javax.swing.text.html=all-unnamed--add-opens=java.desktop/sun.awt.x11=all-unnamed--add-opens=java.desktop/sun.awt.datatransfer=all-unnamed--add-opens=java.desktop/sun.awt.image=all-unnamed--add-opens=java.desktop/sun.awt=all-unnamed--add-opens=java.desktop/sun.font=all-unnamed--add-opens=java.desktop/sun.java2d=all-unnamed--add-opens=java.desktop/sun.swing=all-unnamed--add-opens=jdk.attach/sun.tools.attach=all-unnamed--add-opens=jdk.compiler/com.sun.tools.javac.api=all-unnamed--add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=all-unnamed--add-opens=jdk.jdi/com.sun.tools.jdi=all-unnamedcom.intellij.idea.main

if you search the output of the console log you will find word atomic which contains atom 😄

.../java.util.concurrent=all-unnamed--add-opens=java.base/java.util.concurrent.atomic=all-unnamed--add-opens=java.base/jdk.internal.vm=all-u...

Solution Proposal

To address the issue, we propose modifying the logic of the application detection in Serenade. One possible solution is to move the check for the Atom editor to the end of the if-else statement. Since the term atom can be found in other applications, such as "atomic" in IntelliJ, this change will allow for more accurate identification. Additionally, it is advisable to reposition any other applications with generic names that may potentially be found in different languages.

Available for Further Investigation

If you require any further assistance or if additional information is needed from my Ubuntu system to aid in the investigation of this issue, I would be more than willing to help. Please feel free to reach out to me with any specific instructions or guidance on how I can assist further in resolving this problem.

Context

Please support VS Codium.

I do not want to be tracked by Microsoft so I use VS Codium. It is an OSS fork of VS Code that has all the same features, minus telemetry. It should work with minor code changes as it has all the same considerations as vs code. I installed the VS Code plugin to VS Codium but it does not seem to work. There are too many places that have hardcoded settings for VS Code for me to manage manually going through.

Goals

I want to use this with VS Codium instead of VS Code.

Proposed Solution

Make the VS Code editor configurable to support whatever version is installed

Alternatives

There is no downside to doing this.

System Information

Serenade Version: v2.0.2

OS and Version: macOS 14.3 (and previous Sonoma versions)

Application: focus command: I use ITerm, VSCode, and Chrome

Issue Description

After updating to macOS Sonoma, the focus command no longer works. It will sometimes bring the correct application to the front, but it does not actually focus it. Many times it doesn't work at all. It works the worst with focus VSCode. The launch command can replace focus for Chrome and iTerm, but this does not work for VSCode.

Matt says:
Here’s the underlying library that gets triggered, and likely where the fix needs to be:
https://github.com/serenadeai/driver

There’s a test for focus here that would confirm it’s in this library:
https://github.com/serenadeai/driver/blob/master/test/focus.js

How to Reproduce

Download serenade on macOS Sonoma and try focusing between applications.

System Information

Serenade Version: v2.0.2

OS and Version: Windows 11

Application: VS Code

Issue Description

Serenade crashes interacting with anything besides the tutorial buttons. This includes the pause toggle, and cloud button.

How to Reproduce

Fresh install of Serenade, working through the tutorial for VS Code. Tried to do the first 'next' voice command and the app crashed. This is repeatable every time I launch Serenade.

Screenshots

Context

I never have a reason to use the revision box in VS Code. However, serenade often mis-hears "indent" as "edit" and opens the annoying revision box. Tommy tells me that there is no way to suppress this behavior (since changing settings.json does not solve this version of suppressing the revision box.) and suggested that that I open an issue here.

I made a quick fix by adding this to custom.js:
serenade.app("code").command("edit", async (api) => {
await api.pressKey();
});

But this does not fix the case where serenade mis hears "line 88" as "line, edit".

Goals

Make an option to disable the 'edit' command in vs code.

Proposed Solution

Make an option to disable the 'edit' command in vs code.

Alternatives

Maybe there's a way to solve the "line 88" issue? This didn't work though:

serenade.app("code").command("line, edit", async (api) => {
await api.pressKey("g", ["ctrl"]);
await api.typeText("88");
});

Thanks!

Context

Much of the documentation for developing Serenade is not clear, incomplete, or hard to find. This is a meta ticket to outline the missing sections so we can fix them up. User facing documentation on some features could be part of this issue or a separate one

Goals

Ease of contributing or working on Serenade

List of missing areas

• Building and running Serenade
• Adding new programming language support or updating existing support
• Training language models

System Information

macOS 12.6

Serenade Version: e.g., v2.0.1
v2.0.1
OS and Version: e.g., macOS 12.4
12.6
Application: e.g., VS Code
vs code

Issue Description

chrome extension dialogue has no close option

How to Reproduce

it pops up and won't leave, periodically

Screenshots

System Information

Serenade Version: v2.0.2

OS and Version: Windows 7 Profesional

Issue Description

How to Reproduce

Install on Windows 7.

I suspect jsnode issue

Context

There is a modifier commandOrControl for key presses that switches depending on whether or not you are on mac between command (Mac) or control (Windows). Similarly there should be an optionOrAlt key. This is even more intuitive since mac doesn't have an alt, but it does have a seperate control and command key.

I should be able to do this, just noting it down here.

This looks promising
Only with over 1 year of no more commits on this github we can assume this project is death?
Used to build a commercial version available somewhere else?

v 2.0.0 works fine on my system. It constantly auto updates to v 2.0.2 that has a bug tracked on another issue pending triage.

Is there any way to BLOCK the constant auto update to the buggy version? I am quite tired of re-installing.

Context

I have caps lock and ctrl swapped on my machine via Xorg/X keyboard extension. When I tried a few voice commands in vs code, only a single letter would appear like "L" or "V". I'm assuming serenade is "pushing" ctrl+V or ctrl+L, but since I've swapped my keys, only the letter appears instead.

Perhaps the "ctrl" key can be user configurable somehow?

Is there a solution available to make the "show links/show inputs" command function with Chrome DevTools? I am suffering from RSI and this feature would be extremely useful for me. Please come up with a way to navigate through out DevTools.

Context

This is mainly something to make the interface a bit prettier, particularly as more people create plugins for Serenade.

Goals

I'd like to be able to pass a custom app icon from a plugin, so that as new plugins for new editors/terminals/browsers/etc. are created, the plugin author can provide an icon rather than it defaulting to a generic icon.

Proposed Solution

• The plugin can optionally pass an icon string in the initial active WebSocket message, which is a base64-encoded image (e.g. using a data:// url, so that it can be used directly with the src prop in img tags) to use as the icon when the associated app is focused

A few potentially helpful implementation details that I've figured out from exploring the source code:

• Add customAppIcon property (with some falsey value, maybe null or an empty string) to initialState in client/src/renderer/state/reducer.ts
• Update client/src/main/active.ts, specifically the Active.update() method, to include an extra value in the state (e.g. add customAppIcon: this.pluginManager.findApp(this.app)?.icon in the call to this.bridge.setState() on line 405)
• Update client/src/renderer/components/indicators/active-app-indicator.tsx to take the new customAppIcon state property, and use it as the icon src if its value is truthy, otherwise defaulting to the current logic.

I may have missed some other edge cases (e.g. what to do when the plugin disconnects, though that may already be handled by the current logic - I'm still reading the code), so please provide any additional guidance if necessary.

I'd be happy to implement this if the above sounds reasonable - it would give me the opportunity to set up my dev environment for Serenade and start contributing to the project.

Alternatives

The alternative is that you have to keep updating Serenade with an ever-growing list of icons ;)

Context

Right now you can specify applications where the revision box should be used. However I find that I often need to use it on specific web sites as well. However I don't want to use the the revision box on all websites, so I don't want to simply specify my browser in the revision box settings.

Goals

The ability to specify URLs in the revision box settings file.
Preferably with wildcards/regex to allow for only specifying specific parts of a website.

Proposed Solution

I do not know how to solve this, but I might attempt to figure it out.
At the very least I am available to exchange ideas with if necessary. :)

Alternatives

I don't know of any valid alternatives other than making Serenade work properly on all pages without the use of the revision box.

Context

I just tried serenade. I find it useful and I will use it in the feature as an extension to keyboard.

Goals

I want to do refactorings like renaming of a class over the whole project.

Proposed Solution

Support IDE Refactoring commands.

For example use refactor rename <selector> to call the IDE rename feature.

Alternatives

As an alternative I know about the possibility to write custom commands which may send the required keystrokes to IntelliJ.

System Information

Serenade Version: e.g., v2.0.1
2.02

OS and Version: e.g., macOS 12.4
Windows 11

Application: e.g., VS Code
VS Code

Issue Description

Without having Serenade server started, VS Code extension Spams errors on Logs (Window, Extension Host etc).
It creates an error every second, I guess it should be some other way to do it or suppress the error even if it will happens.
Logs are useless if they are full of garbage.
it looks like this:

2023-07-31 01:28:47.499 [error] connect ECONNREFUSED 127.0.0.1:17373: Error: connect ECONNREFUSED 127.0.0.1:17373
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
2023-07-31 01:28:48.509 [error] connect ECONNREFUSED 127.0.0.1:17373: Error: connect ECONNREFUSED 127.0.0.1:17373
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)

Thanks.

How to Reproduce

Installing Serenade extension and checking VS Code Window log.

Screenshots

System Information

Serenade Version: v2.0.2

OS and Version: macOS Monterey 12.6.2

Hardware: 2020 M1 Mac mini (i.e. I'm running Serenade using Rosetta)

Application: N/A (I happen to be using Serenade with VS Code, but this issue occurs even when VS Code is not running)

Issue Description

I'm working on an open-source Gregorian chant database, and Serenade keeps hearing "chance" whenever I say "chants". When I append serenade.global().pronounce("chance", "chants"); to custom.js and launch Serenade, the app doesn't respond to anything I say (i.e. "focus code" and similar commands have no effect, and Serenade doesn't give a numbered list of suggested commands to execute). With an empty custom.js (i.e. the default file, with a 24-line comment and nothing else), the app works just fine (other than mishearing "chants", of course)

How to Reproduce

• Add serenade.global().pronounce("chance", "chants"); to the bottom of custom.js

/* Serenade Custom Commands
In this file, you can define your own custom commands with the Serenade API.
For instance, here's a custom automation that opens your terminal and runs a command:
serenade.global().command("make", api => {

api.focusApplication("terminal");

api.typeText("make clean && make");

api.pressKey("return");

});
And, here's a Python snippet for creating a test method:
serenade.language("python").snippet(

"test method <%identifier%>",

"def test_<%identifier%>(self):<%newline%><%indent%>pass",

{ "identifier": ["underscores"] }

"method"

);
For more information, check out the Serenade API documentation: https://serenade.ai/docs/api
*/

serenade.global().pronounce("chance", "chants");

• launch Serenade app, unpause it, and speak a command. Sometimes, a small "Low volume" notification appears briefly. Serenade does not respond to the command

First of all: I really love Serenade! Is so easy to use and I can relax my aching hands. Thank you so much for that.

System Information

Serenade Version: v2.0.1

OS and Version: Windows Version 10.0.19043 Build 19043

Application: Chrome, VS Code

Issue Description

In German, words may contain umlauts (äöü).
Serenade does recognize them as separate words, such as other special characters (e.g. $, ., etc.).
Thus, commands such as 'call write 5 words' or 'delete 3 words' does actually delete less words than indicated.

How to Reproduce

(I tried with Chrome and VS Code)

1. Insert this text: 'Ich bin glücklich.'
2. Use the command 'go left 1 word' 3 times

Result: as characters are recognized as separate words, a word with a umlaut will be recognized as multiple words

System Information

Serenade Version: v2.0.2

OS and Version: e.g., Ubuntu 22.10

Application: Everything, crashes when accessing the microphone.

Issue Description

Audio input devices that don't support 16000hz of sample rate will crash the Speech Recorder:

Expression 'paInvalidSampleRate' failed in '/home/tmac/github/speech-recorder/tmp/portaudio/portaudio/src/hostapi/alsa/pa_linux_alsa.c', line: 2050
Expression 'PaAlsaStreamComponent_InitialConfigure( &self->capture, inParams, self->primeBuffers, hwParamsCapture, &realSr )' failed in '/home/tmac/github/speech-recorder/tmp/portaudio/portaudio/src/hostapi/alsa/pa_linux_alsa.c', line: 2721
Expression 'PaAlsaStream_Configure( stream, inputParameters, outputParameters, sampleRate, framesPerBuffer, &inputLatency, &outputLatency, &hostBufferSizeMode )' failed in '/home/tmac/github/speech-recorder/tmp/portaudio/portaudio/src/hostapi/alsa/pa_linux_alsa.c', line: 2845
PortAudio Error: Open Stream
Error number: -9997
Error message: Invalid sample rate

Dug into this very deep, issue comes from the Port Audio OpenStream function (microphone.cpp):

Pa_OpenStream(&stream_, &parameters, 0, sampleRate_, samplesPerFrame_, paClipOff, callback, &callbackData_);

Which is called from microphone.ts in the Serenade Client (client/src/main/stream/microphone.ts):

    this.recorder = new SpeechRecorder({
      device: this.settings.getMicrophone().id,
      sileroVadSilenceThreshold: this.settings.getChunkSilenceThreshold(),
      sileroVadSpeechThreshold: this.settings.getChunkSpeechThreshold(),
      ...
   });

The problem is that there isn't any sample rates being set here, so it defaults to the Speech Recorder library's default of 16000hz:

  ...
  options.sampleRate = options.sampleRate !== undefined ? options.sampleRate : 16000;
  ...

How to Reproduce

Have no audio input devices that support 16000hz. Not sure why none of mine do since the same microphone on Windows doesn't have any issues... Might be a new thing with Ubuntu 22.xx and the audio drivers? Here is the full error log, has some other ALSA errors in it.

Either way, I have a fix for this working locally:

...
type MicrophoneInput = {
  id: number;
  name: string;
  selected?: boolean;
  defaultSampleRate?: number;
};

type SpeechRecorderDevice = {
  id: number,
  name: string,
  apiName: string,
  maxInputChannels: number,
  maxOutputChannels: number,
  defaultSampleRate: number,
  isDefaultInput: boolean,
  isDefaultOutput: boolean
}
...
    const microphoneId = this.settings.getMicrophone().id;
    const selectedMicrophone = this.microphones().find(mic => mic.id === microphoneId);

    this.recorder = new SpeechRecorder({
      device: microphoneId,
      sampleRate: selectedMicrophone?.defaultSampleRate || 16000,
...

  microphones(): MicrophoneInput[] {
    const inputs: [SpeechRecorderDevice] = devices().filter((e: any) => e.maxInputChannels > 0);
    const defaultInputDevice = inputs.find(i => i.isDefaultInput);

    // very important to include the sample rate here
      // the speech processor does not handle default sample rates of devices
      // It defaults to 16000hz for each device, if it's not supported, the program will crash
    const microphones : [MicrophoneInput] = [{
      id: Microphone.systemDefaultMicrophone.id,
      name: Microphone.systemDefaultMicrophone.name,
      defaultSampleRate: defaultInputDevice?.defaultSampleRate,
      selected: Microphone.systemDefaultMicrophone.id == this.settings.getMicrophone().id,
    }];

    inputs.forEach(e => {
      microphones.push({
        id: e.id,
        name: e.name,
        defaultSampleRate: e.defaultSampleRate,
        selected: e.id == this.settings.getMicrophone().id,
      })
    });

    return microphones;
  }
...

Screenshots

Nothing interesting