c-loftus / sight-free-talon Goto Github PK

it would be useful to have a way to query the new openai text to speech model. I don't think this would be economical to use in large quantities but it may be helpful for certain specialized cases

I do not have a Mac so I have no way of testing or developing this at the moment. Would appreciate any help or the ability to test run code

We don't send the ipc command if the mode is sleep, so this means that on talon sleep the speech interrupt won't be restored

Microsoft has an immersive reader mode within Microsoft edge. this immersive reader mode is free and has a more natural text to speech voice than most and thus would be a good choice instead of the robot-like tts when the user needs to listen to a long article intended to be read with natural speech.

there are other programs that use this immersive reader mode like Microsoft word and so it would be useful to try and see if there is a common shortcut or voice command that could be used to query all of these modes

The Python program edge-tts that I experimented with seems to be using the same API endpoint in the background. However, it is much easier to just take advantage of the immersive reader functionality where it is an option.

Should be made easier for those with no or low vision. Many steps to install talon, knausj, nvda, remove character / word dictation. All these settings add significant user friction. Not entirely clear what the best way to do this though since knausj is a git repo and thus should really be cloned. Thus the user needs git and likely another step. Difficult to say what is necessary and what can be automated in the background when installing

I do not know to what degree this is possible. Let me preface my idea with that :)
I think, for me at least, the largest limitation Talon has is its reliance on Cursorless, mouse grids and essentially any variation of the concept of tagging UI elements with numbers, two-letter abbreviations etc.
Due to the sequential nature of screen readers, only one element at a time can be in focus, which means the rest of the screen is unperceivable while that happens. This means that without focusing the element in question first to find its identifier, this sorta kinda works with the Cursorless browser extension for example, screen reader users aren't going to know what to tell Talon in order to interact with that element unless they focus it through some other means, at which point they may as well just continue using the modality they were using, if able.
Mind you, this info is based on Talon as it was several months ago, i don't know if this has improved.
Dragon is, generally, able to receive a command like "Click OK" and find and interact with the right button. I believe there is a Talon OCR plugin that does something similar but in my tests, it wasn't always very reliable.
My thought was that NVDA has a rendition of the accessibility tree at any given point; that's what's used for object navigation, for eample. It also is able to use OCR using the windows 10/11 aPIs that far outstrip open-source offerings like Teseract at present. A mode might be developed that uses this information to provide Talon with this info and work with it in order to allow the user to access controls in view similar to Dragon or possibly even better?

It seems that emacspeak is a screen reader option used by some blind or visually impaired programmers. That being said, it is a bit convoluted to set up and may run differently in WSL2.

It is probably a very small portion of users who would use both Talon and emacspeak, but it might be interesting from a design perspective to see if there is potential for integrating the two.

I think I would need to discuss with someone who actually uses the software in order to get a better perspective

Echo this does not stop until I quit Talon. It would be good to have a command to stop it.

For users without eyesight who need to use Talon without looking at the computer, it would be useful to have a way to provide auditory feedback to ensure accurate command execution.

This auditory feedback should be quick in such a way that it does not interfere with dictation or cause an annoyance.

Use os.path.join(globalVars.appArgs.configPath, "talon_server_spec.json") instead of os.path.expanduser. This will also
handle portable versions of NVDA, or NVDA running with an alternate configuration path.

Waiting to finish this until I have the addon stabilized and working

I imagine that Talon has many ways to potentially interact with screen readers. For instance, instead of needing to hear every potential option on the screen, the user could say a command to jump directly to the option they want. This could be particularly powerful, when integrated with a large language model and talons located library to do on screen ocr.

I am not particularly familiar with screen readers, and if they directly support some sort of process, communication, and what the best way of sending information to them would be.

There would likely be times, in which text to speech from the screen reader would not need to be echoed, especially if we are echoing words from Talon, such as any log errors.

if a user wants to find out all potential Talon script commands that are able to be called, they have to either say one of the help commands which will trigger a graphical pop up, or look within the Talon file itself. neither of these solutions are particularly accessible

it would be helpful to have some sort of action that could echo out all the specific commands that could be said within a given file. it could potentially be cool to have some sort of tag that could be declared at the top of the file which would automatically allow access to a help command. this help command would take all the commands declared within that file and then echo them out to the user. that being said we would have to parse the Talon files somehow to get the spoken names of the commands so this might be rather difficult and we have to know exactly where the path the file is as well. however at the same time the syntax for Talon files is just everything before the: so it might be not too difficult to have a basic heuristic

This can be done by adding:

• New NVDA server command to get the state of the command before disabling the settings to speak / interrupt
• Using that state to determine the clientside ipc request for the post:phrase callback

We will probably need a better serialization JSON format and preferably this can be shared between both server and client in the repo

• Schema should include both the status code of the response and any content that gets returned i.e. the state of the variable that was requested

As of right now, some commands use the word "speak" while others use the word "echo." Sometimes, it makes more sense to use the word "echo," such as when Talon echoes back what the user said to confirm it was dictated correctly. However, it is probably preferred to have the same word used in all contexts. Additionally, it would be preferred to have some sort of consistent grammar, either verb subject or subject verb. I have to think about this more and potentially get some user feedback.

Terminals and other programs that operate upon a text log are usually not very accessible. it would be useful to try and find a way to watch the log file see if there is any errors and if so echo something to the user via text to speech.

This functionality could also be useful for users who do have full sight but still don't want to switch back and forth for the log on one monitor

Similar to cursorless or rango, it would probably make sense to have a more specialized name for this repository instead of a generic name.
Need to think of something that is clever, short, preferably easily, dictated with Talon and has decent seo. Should use inclusive language . Should imply the ability to use talon along with screen readers and low vision tools, but also not necessitate users to have a certain ability status. (Ie sightless talon is not a great name since it implies the program is only for non-sighted users)

I have the baseline Python code but no JAWS license. Could use help with testing this

If you are using a screen reader, there is currently a setting to allow the tts function to instead use the screen reader, instead of using the text to speech function with Talon

However, if you do that, you have to press down a key in order to read from the clipboard and if you press this key down, it could mangle text that is input via dictation

There might need to be some sort of lock, or other concurrency primitive in order to prevent this race condition

I'm not particularly familiar with COM functionality on Windows, but it seems like if COM gets in a messed up state, then the functions that rely on COM interfaces won't work correctly. This could include both the screen reader and the simple Windows TTS included in this repository.

This is fixed only by a reboot. it is not particularly clear what can cause this

This would be useful for both sighted and visually impaired users.
Talt has a series of useful contextual state variables. For instance, whether certain tags or modes are enabled or even more fundamental things like the title of the running application and the window name. It would be useful to be able to echo this out via text to speech. That could be useful for people of all backgrounds who want to debug their talents scripts but don't necessarily want to copy and paste things or open the debug window

With a vision impairment, it would be useful to quickly get a sense of what mode they are in or other important context

By default the speaker object within the python windows com library speaks out at rate that is relatively slow. It would be useful to see if there is a way to make it faster.

The default speaker within this library is particularly robotic. This is OK and some circumstances and I have been told that sometimes for accessibility reasons this allows for faster listening. I have personally not found that to be the case myself. Regardless, it is useful to have other options as well.

I have experimented with 11labs which provides a free API key, but there is pretty heavy rate limits.
I have experimented with the Python library, edge-tts, but it is awkward to call from a python subprocess. you can use it asynchronously with a streaming API, but Talon does not allow asynchronous python code so this would need to be called from another interpreter, which would likely be a hassle. Perhaps there is a better way.

Hopefully should integrate with both system tts and screen reader tts volume although the latter might be difficult.

For context, sometimes things like videos or video calls have a sound level that is much different and thus we need to normalize everything

Lots of potential for using cursorless to intelligently navigate around vscode when coding. Blocked until cursorless keyboard mode is done since the api doesn't expose everything at the moment.
Investigate using a pedal as well

We can essentially use the tree structure as a way to intelligently interact with the editor better than a typical screen reader.

Not sure why this is happening and is likely something on NVDA's side with the controller client and DLL injection. This happens regardless of my extension being installed. However, at the same time, Talon has undefined behavior if force quit from the terminal and the user is not even intended to launch this way, so perhaps it is not a significant issue.