clipboard interaction about turbo HOT 37 CLOSED

The issue you will find is that, since Turbo Vision can only dispatch one text event at a time, you need NcursesInput::getEvent to be invoked until all the clipboard text has been processed. However, it would be really unfortunate to add this logic in the NcursesInput class for several reasons:

• It would make NcursesInput more complex for a feature that does not concern Ncurses.
• It would be coupled to Ncurses, so the only way for other input strategies to benefit from it would be to duplicate code.

It may be possible to do something fancier with the current FdInputStrategy architecture. Let me write a skeleton of the idea I have in mind.

// An extension to FdInputStrategy

static void FdInputStrategy::putFront(FdInputStrategy *listener)
{
    for (size_t i = 0; i < listeners.size(); ++i)
        if (listener == listeners[i])
        {
            ready.push_front(i); // Replace std::queue with std::deque.
            break;
        }
}


class FAR2LClipboardInput : public FdInputStrategy {

    enum State { Waiting, Ready };
    
    State state;
    
public:
    
    FAR2LClipboardInput();
    
    bool getEvent(TEvent &ev) override;
    
private:
    
    bool peekStdinAndSeeIfTheExpectedSequenceIsThere();
    bool readClipboardText(TEvent &ev);

};

FAR2LClipboardInput::FAR2LClipboardInput() :
    state(waiting)
{
    addListener(this, StdioCtl::in());
}

bool FAR2LClipboardInput::getEvent(TEvent &ev)
{
    switch (state)
    {
        case Waiting:
            if (!peekStdinAndSeeIfTheExpectedSequenceIsThere())
                break;
            state = Ready;
            [[fallthrough]]
        case Ready:
            if (readClipboardText(ev))
            {
                FdInputStrategy::putFront(this); // We need to be invoked on next iteration of the event loop.
                return true;
            }
            else
            {
                state = Waiting;
                break;
            }
    }
    return false;
}

// Then, in THardwareInfo's constructor

#ifdef _TV_UNIX
    static TSignalHandler h;
    static FAR2LClipboardInput f; // add this line.
    // This will ensure FAR2LClipboardInput is always before NcursesInput in FdInputStrategy's queue,
    // and will also ensure that StdioCtl is initialized before FAR2LClipboardInput.
#endif

The above would allow you to have a peekStdinAndSeeIfTheExpectedSequenceIsThere function that gets invoked every time there's something in stdin so that you can check if it's a FAR2L clipboard sequence, and a readClipboardText that gets invoked on every iteration of the event loop until it returns false.

from turbo.

Yes, I forgot to add this to the TODO list. Another editor that already supports this is https://github.com/gphalkes/tilde.

But I currently don't have time to work on this. My suggestion is that either you get used to disabling automatic indentation before pasting, or you implement it yourself, depending on how much time you are willing to spend on this issue.

from turbo.

What's up unxed. Things have changed and the advice I gave you a month ago is no longer very useful.

If you look at the NcursesInput::getEvent function, you will notice it now contains the following:

int k = wgetch(stdscr);

switch (k)
{
    case KEY_ESC: {
        NcGetChBuf buf;
        switch (TermIO::parseEscapeSeq(buf, ev, mstate))
        {
            case Rejected: break;
            case Accepted: return true;
            case Ignored: return false;
        }
        break;
    }
    case KEY_MOUSE:
        return parseCursesMouse(ev);
    case KEY_RESIZE:
        return winchEvent(ev);
}

The parseEscapeSeq is implemented in source/platform/terminal.cpp and it looks like this:

ParseResult TermIO::parseEscapeSeq(GetChBuf &buf, TEvent &ev, MouseState &oldm)
{
    if (buf.get() == '[')
    {
        switch (buf.get())
        {
            case 'M':
                return parseX10Mouse(buf, ev, oldm) == Accepted ? Accepted : Ignored;
            case '<':
                return parseSGRMouse(buf, ev, oldm) == Accepted ? Accepted : Ignored;
        }
    }
    buf.reject();
    return Rejected;
}

As you can see, it's a place where you can check for input escape sequences, and it can be easily extended.

Additionally, there's a new method NcursesInput::hasPendingEvents that can cause NcursesInput::getEvent to be invoked even if no data is left in the file descriptor.

from turbo.

469e279 contains a preliminary fix for this.

from turbo.

Yes, that is an example of how I deal with escape sequences Ncurses cannot recognize.

from turbo.

OSC 52 copy support is now implemented, but this is not over yet. A definitive solution would be integrated into the public Turbo Vision API, but that will require some thinking.

from turbo.

Other possibilities also exist for using external clipboard in TUI apps.

1. OSC 52 escape sequences. xterm supports it, alacritty supports it, hterm does it also.

2. far2l's VTE has it's own terminal extensions for clipboard access. Ready to use here and now, but not documented at all (I am trying to write kind of documentation here). Supported in my fork of putty also.

Thinking of adding the second option to Turbo as such thing would be very useful for me personally. It also will work not only on local system (as xclip), but via SSH also.

from turbo.

Will look into the code, possibly try to implement option 2 (far2l's VTE extensions).

from turbo.

I wonder if you worked with Scintilla before. It took me over a week of frustration to understand what was going on.
But for the use case of copy-pasting, the situation is:

• No 'paste' event is emitted in the Turbo Vision event loop. Ctrl+C/Ctrl+V/Ctrl+X are simply forwarded to ScintillaBase.
• The magic happens in TScintillaEditor::Copy and TScintillaEditor::Paste (src/tscintilla.cc).
• There is currently one Scintilla::SelectionText object in the TurboApp class (src/app.h), which gets assigned to TScintillaEditor instances in EditorWindow's constructor (src/editwindow.cc).
• SelectionText is defined in scintilla/src/Editor.h.

from turbo.

I wonder if you worked with Scintilla before

Nope, I'm even not very experienced in c/cpp :) Just playing around

It took me over a week of frustration to understand what was going on

We're in no hurry, right? :)

PS: Thanks for tilde, looks like an acceptable solution for the first time.

from turbo.

Looked at the tilde. The good news is that your Turbo Vision approach is potentially far more promising.

Tilda uses the good old Unix approach "let's take the existing terminals and make TUI interface for them, as human friendly as they allow to make". Hence all these problems for the user right from the start: the need to set the behavior of Esc. Why do we even have to make user think about all this boring stuff?! He does not wants to dig into terminal standards, he wants to write a poem or love letter or some code in python!

Your path looks much more interesting: "Let's take a framework that is well suited for creating human-friendly text-mode applications and adapt it for use in Unix terminals". The only step that remains to be added is, "if the terminals fundamentally do not allow this, let's improve the terminals", as there is absolutely no way to make a comfortable text-mode application using 50-year-old teletype technology! Let's be honest: even a DOS program had more opportunities to be made user friendly than a Unix program, whose developers have to spend most of their (very limited!) time supporting every terminal that has ever existed in the world, whether it is in use now or not.

This is just the path that was chosen for implementing Linux (BSD, Mac) port of the most popular text mode application for Windows, Far Manager. The author of the port simply made his own implementation of the console window (of course it is also capable of running inside old style tty for compatibility/emergency reasons — if X could not be started, for example), and to support the necessary features through ssh he added into the port his own virtual terminal emulator with his own esc sequences — and now we have support for ALL possible key combinations, clipboard synchronization, desktop notifications and other delicious things in console — and all those stuff works even through ssh! Moreover, these extensions have even been added to the putty windows terminal client so that you can use all these wonderful things from Windows.

Let me explain with examples.

In the world of Unix terminals you cannot distinguish between key combinations "1", "LeftControl" + "1" and "RightControl" + "1" — Unix terminals simply do not send the required data. No ingenious hack can get around this, ever! You also can't support KeyUp events in any way — this data is not being sent either. Clipboard from a remote server? Ha, wait until KDE and GNOME add OSC 52 support (possibly never). Desktop notifications (for example, when a long-running operation ends)? The creators of Unix terminals did not even thought about this!

But all this becomes possible if you use the built-in Far Manager for Linux terminal and its terminal extensions. I'll try to find an opportunity to add them to Turbo, so that you can see how beautiful the world becomes when a console application for Linux, BSD or Mac gets all the power previously available only for Windows (and partially DOS) TUI apps. And, most importantly: if implemented, it would be possible to set the focus on improving the application itself, not on endless war with terminal limitations!

Cheers!

from turbo.

Hi unxed! Thanks for your comment.

Your path looks much more interesting: "Let's take a framework that is well suited for creating human-friendly text-mode applications and adapt it for use in Unix terminals".

Let's not forget that Borland is to be credited for the User Experience of Turbo Vision. All I did was keeping it away from the complexities of terminals.

The creators of Unix terminals did not even thought about this!

All "Unix terminals" do is inherit the standards created many years ago, to replace the vendor-specific sequences that used to be popular among machines a lot less featured than today's terminal emulators (https://en.wikipedia.org/wiki/ANSI_escape_code). I believe that it will be hard to get other developers to adopt non-standard features unless there is a broad consensus to develop a new standard.

Unix terminals simply do not send the required data. No ingenious hack can get around this, ever! You also can't support KeyUp events in any way — this data is not being sent either.

Borland chose not to expose KeyUp events in Turbo Vision API either, although it was possible to detect them in DOS and Windows. If this had not been the case, it would have not been possible to make a backward-compatible port: programs would require changes in their logic and design before porting. So this limitation in the API actually made my life easier. Sincerely, I think that having limited features forces developers to keep things simple. Is there a specific I/O capability (besides clipboard) that you miss in Turbo? A certain key combination?

If implemented, it would be possible to set the focus on improving the application itself.

The only thing about terminal capabilites that I currently miss in Turbo/Turbo Vision is true color support. Clipboard is another desirable feature. Besides that, my focus has already been on improving the application itself for a long time :).

In the world of Unix terminals you cannot distinguish between key combinations "1", "LeftControl" + "1" and "RightControl" + "1"

Almost only Windows Console and GUI users will miss these features. I don't think coupling the model of terminal applications with the concept of a keyboard is a very good idea: it would make terminal applications more complex. The reason why this is not a problem with GUI applications is because complexity is mostly absorbed by GUI toolkits. But on the terminal, TUI toolkits are not popular at all. Most terminal applications today are developed for ANSI terminals and developers seem to be glad to do as much as possible by themselves and reinvent the wheel every single time.

I'll try to find an opportunity to add them to Turbo.

Code related to terminal I/O belongs to Turbo Vision, so that all applications can benefit from it. New features will have to become part of the Turbo Vision API, and so it will take some effort to design API extensions. But experimental support can be first developed in Turbo and then moved to the library, as I did with Unicode support.

If you need to make changes to Turbo Vision, then you the following information will be useful to you:

• Terminal I/O support follows a Strategy Pattern. Implementations are hidden behind the PlatformStrategy, DisplayStrategy and InputStrategy interfaces defined in include/tvision/internal/platform.h. On Unix, where input happens through file descriptors, all input strategies inherit from FdInputStrategy.
• The adequate strategy is chosen at runtime from THardwareInfo::setUpConsole() in source/platform/hardware.cpp.
• Although the InputStrategy and DisplayStrategy interfaces are totally independent, in practice all implementations rely on Ncurses being initialized.

Cheers.

from turbo.

I believe that it will be hard to get other developers to adopt non-standard features unless there is a broad consensus to develop a new standard.

It's well known «chicken and egg problem» described in this Joel Spolsky's article. «Why in the world should I add new feature support code to my terminal if no app is using that feature?», — says terminal developer. «Why should I add the use of new terminal feature to my app if no terminal supports it?», — says app developer. As a result we stay forever with very limited terminals.

In far2l this circle was broken as it is both a terminal emulator and a console application. So if you are terminal developer, there is at least one useful app utilizing new standard, and if you are app developer, there is at least one (already two) terminal supporting it.

Some apps and terminals may add in future. As I can guess the main thing that prevents this approach to be used widely is lack of far2l packages in popular distros (will definitely be solved over time) and lack of clear documentation (I am trying to write one).

If you need to make changes to Turbo Vision, then you the following information will be useful to you:

Indeed useful, thanks!

from turbo.

Will look into the code, possibly try to implement option 2 (far2l's VTE extensions).

If you need to make changes to Turbo Vision, then you the following information will be useful to you:

During my yesterday's experiments with Turbo Vision, I managed to add partial support for the far2l terminal extensions mode there (so far only enabling it and keyboard input support in this mode, but this is a necessary base to work on clipboard support in the future). So far, the code is too dirty and messy to be released in public, so I just share what I tried and it works.

Thanks for the tip, it helped a lot! Turbo and cross-platform port of Turbo Vision are cool things, it's very interesting to digg into the source code!

from turbo.

In the meantime, could you please suggest me how to correctly access clipboard of the Turbo Vision editor from ncursinp.cpp, where the input is processed? The contents of clipboard will arrive in the form of ESC sequences, so it will be received exactly in this piece of code.

from turbo.

Hi unxed:

Note that NcursesInput::getEvent is invoked in two circumstances:

• When there is input ready to read() in standard input (because of addListener(this, StdioCtl::in());).
• When a SIGWINCH was received (because of addListener(this, winchFd());).

So, unless Ncurses does something strange with the standard input buffer, you should be able to read standard input ( StdioCtl::in()) manually. Or, if Ncurses does not mess with the relevant ESC sequences, it may enough to rely on the results of wgetch.

(Will add more details later.)

from turbo.

Yes, something like that, apparently, is needed. I will try, thanks!

from turbo.

469e279 contains a preliminary fix for this.

Does "preliminary" mean that some other work on this is planned?

from turbo.

Further work on this is not prioritary, but there's still room for improvement (e.g. OSC 52 support for SSH, although I'm pretty sure that ssh -X will do the trick for 90% of SSH users), and in the long term we can think of making clipboard support a standard Turbo Vision feature (like GUI toolkits do).

from turbo.

Started experimenting with adding far2l extensions support. Can you please suggest, how to "catch" raw escape sequences that ncurses can't parse? Not worked with ncurses previously, just do not know where to start.

For example, if terminal sends to application something like "\x1b_far2lok\x07", there in the code can I distinguish such string from keyboard/mouse related escape sequences and react appropriately?

UPD: Already found it, NcursesInput::getEvent and TermIO::parseEscapeSeq is that I need, yep?

from turbo.

Just another question. To support far2l extensions I need some base64 encode/decode lib. That is the preferred way to do it? Import code of any such lib to repo? Add dependency?

from turbo.

And one more question. In far2l terminal extensions keyboard events are encoded in the format of KEY_EVENT_RECORD structure as used in Windows. Parsing this structure is already implemented in win32con.cpp, so no need to write parsing code once again. But how to call Win32Input::getKeyEvent from terminal.cpp correctly?

from turbo.

Just another question. To support far2l extensions I need some base64 encode/decode lib. That is the preferred way to do it? Import code of any such lib to repo? Add dependency?

Maybe we can find an implementation that we can copy into our source code, since encoding/decoding base64 is not a very complex problem. I have already done this previously; for example, the COPYRIGHT file in the tvision project already contains copyright notices of third-party components that were copied into the project.

And one more question. In far2l terminal extensions keyboard events are encoded in the format of KEY_EVENT_RECORD structure as used in Windows. Parsing this structure is already implemented in win32con.cpp, so no need to write parsing code once again. But how to call Win32Input::getKeyEvent from terminal.cpp correctly?

We would have to extract the Win32Input::getKeyEvent (and Win32Input::getUnicodeEvent) method out of Win32Input so that it becomes a global function that can also be called from terminal.cpp. The Win32Input class can only be compiled on Windows.

The KEY_EVENT_RECORD struct is already defined in include/tvision/compat/windows/windows.h so that it can be used on Linux (#include <windows.h> should just work).

from turbo.

Managed to get far2l terminal extensions input model working in tvision & turbo! Now, for example, we can use turbo comfortable in putty (with all hot keys working, like Shift+Arrows, etc), as there is putty's fork with far2l extensions support. The code is somewhat dirty for now, need to clean it up before submitting a PR, but I will leave a very early patch here just FYI.

tv_far2l.tar.gz

@elfmz we are now going to have a pair of applications not related to far2l at all but communicating with each other using far2l terminal extensions, isn't is amazing?

from turbo.

Hi Unxed! Thank you very much for the patch. I have begun adopting your changes in the far2l branch of tvision.

I am facing a strange issue, though. Using elfmz/far2l@e37a3a4, when I run a Turbo Vision application with far2l input support, the application receives no input until I click anywhere on it with the mouse. Only then keyboard events begin working. Also, the application does not receive mouse events, regardless of whether it enables farl2 extended input or not. Is this normal?

from turbo.

when I run a Turbo Vision application with far2l input support, the application receives no input until I click anywhere on it with the mouse

#79 fixes it

Also, the application does not receive mouse events, regardless of whether it enables farl2 extended input or not.

In far2l extended terminal mode mouse events are also sent in special way, not as usual mouse escape sequences.

Currently only KeyDown far2l events are supported:
if (strcmp(out + 14, "K") == 0 )

Mouse events will have not "K", but "M" in the last byte. Parsing such events is not implemented yet.
Structure for base64-decoded data for mouse events is as folows:

  uint32_t (dwEventFlags)
  uint32_t (dwControlKeyState)
  uint32_t (dwButtonState)
  int16_t (pos.Y)
  int16_t (pos.X)
  "M" char — event type signature

Full description of far2l's extended input events is here:
https://github.com/elfmz/far2l/blob/master/WinPort/FarTTY.h

As for regular, standard mouse event escape sequences, seems that far2l do no support it at all for now:
elfmz/far2l#1427

UPD: fixed event type check to check last data byte, not always 15th one
magiblot/tvision@140abf5

from turbo.

Mouse support in default terminal mode (without extensions) added to far2l:
elfmz/far2l@ed2c198

from turbo.

Support for far2l's keyboard and mouse events is now finally merged into Turbo Vision. Thank you very much for the help, @unxed.

from turbo.

Nice to hear that! So great. Now users of putty4far2l, putty-nd and kitty can enjoy full keyboard support in turbo.

Next logical step should be adding support for far2l extensions that implement clipboard access. But using OSC 52 in pair with paste from terminal (maybe also implementing bracketed paste support) looks like viable and more standard-friendly approach that is much simpler to implement, keeping in mind that far2l supports OSC 52 and bracketed paste also.

from turbo.

But if we want clipboard sync to work in putty forks also, we should prefer far2l extensions as there is no osc52 support there. Actually we can support both clipboard interaction methods as far2l does.

from turbo.

Great, thanks! Added OSC52 support to my putty4far2l also.

from turbo.

Now copying from turbo to putty4far2l is working. Unfortunately, pasting from putty4far2l into turbo is not: I overlooked the fact that both putty4far2l and far2l do not process Shift+Ins combination in far2l extensions mode, but send it to the client application for processing on its side. So we still have to implement far2l extensions clipboard access model for smooth user experience.

from turbo.

To do so, we need some additional tricks. First of all, we should check if terminal actually supports far2l extensions or not. It can be done by waiting for "\x1b_far2lok\x07" reply to "\x1b_far2l1\x07". Also extensions state (enabled or not) should be exported somehow from TV so turbo can know if it should use extensions for clipboard or not. If we try to get clipboard using extensions in terminal that does not support them, we will send clipboard get request and wait for answer forever.

from turbo.

In putty4far2l, btw, paste is still possible via Shift+RightMouseButtonClick

from turbo.

Here is simple patch demonstrating how to get far2l's clipboard content using far2l tty extensions. Just POC, but works for me:
farclip_get.tar.gz

from turbo.

Definitely fixed as of magiblot/tvision#82 and 22b8827 😎.

Thanks for the help!

from turbo.

Fantastic! Thanks a lot!

from turbo.