Langoustine provides an easy way to build servers, but AFAIK there's no such thing for clients. If you were to implement a language client with Langoustine, what could that look like?

When you filter for, e.g. "completion", and the log is full of textDocument/completion calls from the client, you'll only see the client side of a request in the result of the filter:

It would be great if server "spans" were also tagged with that request name, so that they would also show up in that filter.

Related/alternative: #36

Sort of nitpicking at this point, but I think having a larger clickable area would be nice here:

Currently clicks are only registered when you click one of the colorful thingies - I'm suggesting to make it so that the entire selected div (as per screenshot) triggers the action.

e.g. in Metals it can get pretty big:

Probably having a single-line scrollable element would be good enough.

See scalameta/metals#4924 (comment)

No matter which direction it goes, it would be great to specify exactly the format of supported messages.

To move things along we can publish mini-libraries pre-configured with codecs for major JSON libraries.

The goal is for other LSPs written in other languages to be able to produce snapshots replayable with tracer.

Last time I tried to do it, something about http4s and lazy vals broke.

Platforms:

• Linux x86
• Mac x86
• (optional) Mac M1
• Windows x86

Name: langoustine-tracer-, see https://github.com/indoorvivants/sn-bindgen/releases/tag/v0.0.23 for platforms (matches Coursier)

It's not possible to scroll down on a large payload.

Browser: firefox on mac

Something feels sus about the height of the element:

I've finally realised why Metals has nice output in VS code and not in the logs - because VS Code just renders the windows/logMessage stuff as normal logs!

We can make it much nicer in the Logs panel.

This is also a good opportunity to set up machinery for reading the server's information as well - for example to capture server name

A video is worth a thousand screenshots, and a screenshot is worth a thousand words:

I suppose there's some sort of redundant re-rendering of the old interactions when new ones are added.

This is just too much:

import langoustine.lsp.requests.*
import langoustine.lsp.structures.*
import langoustine.lsp.json.*
import langoustine.lsp.enumerations.*

import langoustine.lsp.RuntimeBase.uinteger
import langoustine.lsp.RuntimeBase.DocumentUri

We should aim to somehow bring all of them into the same all object.

I thought of using export clause which seems designed for this exact purpose, but I worry about the IDE experience

There's quite a dangerous issue currently in textDocument.documenSymbol (my favourite, which makes it hurt even more):

  object documentSymbol extends LSPRequest("textDocument/documentSymbol"):
    type In = structures.DocumentSymbolParams
    type Out = Opt[(Vector[structures.SymbolInformation] | Vector[structures.DocumentSymbol])]
    
    given inputReader: Reader[In] = 
      structures.DocumentSymbolParams.reader
    
    given inputWriter: Writer[In] = 
      structures.DocumentSymbolParams.writer
    
    given outputWriter: Writer[Out] =
      upickle.default.writer[ujson.Value].comap[Out] { _v => 
        (_v: @unchecked) match 
          case v: Vector[?] => writeJs[Vector[structures.SymbolInformation]](v.asInstanceOf[Vector[structures.SymbolInformation]])
          case v: Vector[?] => writeJs[Vector[structures.DocumentSymbol]](v.asInstanceOf[Vector[structures.DocumentSymbol]])
          case a if a == Opt.empty => ujson.Null

The particular offender is

        (_v: @unchecked) match 
          case v: Vector[?] => writeJs[Vector[structures.SymbolInformation]](v.asInstanceOf[Vector[structures.SymbolInformation]])
          case v: Vector[?] => writeJs[Vector[structures.DocumentSymbol]](v.asInstanceOf[Vector[structures.DocumentSymbol]])
          case a if a == Opt.empty => ujson.Null

We cannot generically match on two vectors because they are erased.

A quick solution might be to identify this case and attempt to serialise the vector as one type, catch the casting exception and serialise as another.

A more involved solution is to generate special code which goes like this:

1. If the vector is empty, pick whatever type and serialise
2. If the vector is non empty, use the first element to distinguish the two vectors

Question is whether this solution is overkill (I don't think so, because trapping class cast exceptions seems like a bad idea in general)

Same principle as #20 but for requests/responses

This might be a me thing, but I'm not able to pass a flag to the underlying server (which is also a cs launch call):

cs launch tech.neander:langoustine-tracer_3:latest.release -- cs launch --ttl 0 com.kubukoz.playground:lsp_2.13:0.4.0

Is this something that could be fixed in the tracer, or could it be related to how cs launch passes parameters?

It's the right thing to do in most cases, and it's required under WSL

There exists perhaps a million reimplementation of the same basic data structures for LSP servers, used both for implementation and testing.

1. Going from offset to a (line, character)
2. Going from (line, character) to offset in the document
3. Getting contents of a line
4. Finding the (line, character) positions of a given offset range
5. Cutout text based on a position range
6. Apply a text edit to given range
7. etc.

It would be good to provide efficient implementations (may be even special cased for platforms!) of these operations, implemented in terms of the LSP structures like Position and Range, so that LSP authors don't reimplement the wheel unnecessarily.

You can see one such implementation here: https://github.com/artempyanykh/marksman/blob/main/Marksman/Text.fs

The companion for Position should inherit from a private trait PositionExtensions to define a few handy method:

def apply(line: Int, char: Int): Position
def documentBeginning: Position = Position(0, 0)

may be others?

the trait itself doesn't have to be part of generated code, we just reference it.

Motivation:

Range(
                  start = Position(line = uinteger(idx), uinteger(0)),
                  end = Position(line = uinteger(idx), uinteger(line.length))
                )

it's jsut too much typing

It would be cool to show the params in a decoded form, with URIs potentially being clickable. I'm thinking a "nested table" view would be a good start.

I know next to nothing about it, so I hope to get some info - would it be feasible to extend Langoustine with support for the Debug Adapter Protocol? Would this be mostly an addition of new codegen, or are there changes in more areas?

Can you do custom endpoints with Langoustine?

First of all, the LSPRequest type is sealed, so I don't think so. However, assuming if it wasn't:

The way I'd attempt it would be to basically copy-paste one of the existing request objects and the traits it implements, and try to adapt to my own model.

object diagnostic extends LSPRequest("textDocument/diagnostic") with codecs.requests_textDocument_diagnostic:
      type In = structures.DocumentDiagnosticParams
      type Out = aliases.DocumentDiagnosticReport

//...

private[lsp] trait requests_textDocument_diagnostic:
  import textDocument.diagnostic.{In, Out}
  given inputReader: Reader[In] = 
    structures.DocumentDiagnosticParams.reader
  
  given inputWriter: Writer[In] = 
    structures.DocumentDiagnosticParams.writer
  
  given outputWriter: Writer[Out] =
    aliases.DocumentDiagnosticReport.writer
  
  given outputReader: Reader[Out] =
    aliases.DocumentDiagnosticReport.reader

Can we make this possible in a convenient way?

Version Info

Scala 3.2.1
SBT 1.7.3
Scala.js 1.11.0
Langoustine 0.0.17

Description

Despite defining handleRequest, the server receives the request but does not return a response to it.
Sorry if there is something wrong with the way I write.

Reproduction frequency

always

Code

build.sbt

name := "IMPDocUtil"
ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "3.2.1"

enablePlugins(ScalaJSPlugin)

Compile / fastLinkJS / scalaJSLinkerOutputDirectory := baseDirectory.value / "dist"
Compile / fullLinkJS / scalaJSLinkerOutputDirectory := baseDirectory.value / "dist"

scalaJSLinkerConfig ~= { _.withModuleKind(ModuleKind.CommonJSModule) }

scalaJSUseMainModuleInitializer := true

javaOptions ++= Seq(
  "-Xmx2G",
  "-XX:+UseG1GC"
)

libraryDependencies ++= Seq(
  "org.typelevel" %% "cats-core" % "2.8.0",
  "org.typelevel" %% "cats-effect" % "3.3.14",
  "tech.neander" %%% "langoustine-app" % "0.0.17",
  "org.scalatest" %% "scalatest" % "3.2.14" % Test
)

main program

package com.github.chencmd.impdocutil

import scala.util.chaining.*

import jsonrpclib.fs2.catsMonadic

import langoustine.lsp.{requests as R, structures as S, enumerations as E, *}
import langoustine.lsp.app.LangoustineApp
import langoustine.lsp.runtime.*

import cats.effect.IO
import cats.implicits.given

import fs2.io.file.{Path, Files}
import fs2.text.*

object IMPDocUtilityLanguageServer extends LangoustineApp.Simple {

  def server: IO[LSPBuilder[IO]] = IO(create)

  def create: LSPBuilder[IO] = {
    LSPBuilder
      .create[IO]
      .handleRequest(R.initialize) { in =>
        sendMessage(
          in.toClient,
          E.MessageType.Info,
          "server activated"
        ) as S.InitializeResult(
          S.ServerCapabilities(
            // documentFormattingProvider = Opt(true)
          ),
          Opt(
            S.InitializeResult.ServerInfo(
              name = "IMPDoc Utility",
              version = Opt("0.0.1")
            )
          )
        )
      }
  }

  def sendMessage(
      back: Communicate[IO],
      messageType: E.MessageType,
      msg: String
  ): IO[Unit] = {
    back.notification(
      R.window.showMessage,
      S.ShowMessageParams(messageType, msg))
  }
}

The complete code, including other files, is in ChenCMD/IMPDocUtil.

🤷

To assist with testing LSPs, we can provide certain features.

Specifically, end-to-end testing of payloads.

Hi,

As a language server developer, I sometimes would like to look at language client and server interaction and be able to see for what kind of document a certain response was sent. For example, if I see an error response for a completion request, I would like to see for what state of the document this response was sent. I am thinking of a feature in the tracer that can go from textDocument/didOpen to the current request of interest, applying all content change notifications, to reproduce the document state.

Would such a feature be welcome? Do you also experience such need?

Another feature that could be built based on this one is ability to see the code surrounding a position mentioned in a request.

See https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens

This needs to be added to SemanticTokensEncoder companion object (as it doesn't depend on negotiated legend), and should operate directly on two Vector[uinteger]

LangoustineApp-created server processes appear not to when they get a SIGTERM.

Reproduction:

//> using scala "3.3.0-RC4"
//> using lib "tech.neander::langoustine-app::0.0.19"
import cats.effect.IO
import cats.implicits._
import jsonrpclib.fs2.given
import langoustine.lsp.LSPBuilder
import langoustine.lsp.app.LangoustineApp
import fs2.io.file.Files
import fs2.io.file.Path

object Main extends LangoustineApp.Simple {

  val writePid =
    Files[IO]
      .writeAll(Path("PID"))
      .apply(fs2.Stream(ProcessHandle.current().pid().toString).through(fs2.text.utf8.encode[IO]))
      .compile
      .drain

  override def server: IO[LSPBuilder[cats.effect.IO]] = writePid *> LSPBuilder.create[IO].pure[IO]

}

2. scala-cli --power package . -f -o server
3. ./server
4. in another tab, kill $(cat PID)
5. The server is still running, and needs a newline in stdin or a ctrl+C to shut down

if something wonky is happening in in/out, it'd be good to have a way to see that

Hi,

I think vscode has quite nice facilities that allow embedding the tracer into vscode. Some of the advantages, for example, include

• debugging your language server without switching from editor to browser
• platform for working with documents:
  • clickable url's that open documents with code highlighting (if we add #108, we can open locked documents with code highlighting and selected text)

Would you guys be interested in collaborating on something like this?

@since exists in both Scaladoc and the documentation for LSP spec, but it means different things - in Scaladoc it refers to the version of langoustine, in LSP spec it's the version of protocol.

We should not render @since from LSP spec in Scaladoc (replace it with since)

It might be a fun exercise because I have a modified macro to work around upickle 2 weirdness, and I have no idea what sort of wonders upickle 3 will bring..

I think server-sent requests (such as workspace/configuration) aren't showing up in the tracer.

For example, my formatter asks the client for some config, but it's not shown here:

Also, I'm seeing some weird mismatches (e.g. a request showing up with another one's body), especially around non-standard requests... I'll try to report that separately sometime.

Perhaps it will make IDE experience better if structures exposed builder-style pattern.

I.e. if you have a SymbolInformation structure:

case class SymbolInformation(
  deprecated: Opt[Boolean] = Opt.empty,
  location: structures.Location,
  name: String,
  kind: enumerations.SymbolKind,
  tags: Opt[Vector[enumerations.SymbolTag]] = Opt.empty,
  containerName: Opt[String] = Opt.empty
)

It has 3 required fields (location, name, kind) and several optional fields. All the fields become withCamelCase methods on the generated builder, but to create the builder you need to pass the required fields:

SymbolInformation
  .builder(name, location, kind)
  .withTags(v: Vector[SymbolTag])
  .withContainerName(s: String)
  .withName(s: String) // overwrites the value passed to the builder method
  .build // : SymbolInformation

... and move them to traits that those companions should extend.

Say you have a currently generated companion like this:

object WorkspaceSymbol:
  private given rd0: Reader[(structures.Location | WorkspaceSymbol.S0)] = 
    badMerge[(structures.Location | WorkspaceSymbol.S0)](structures.Location.reader, WorkspaceSymbol.S0.reader)
  private given wt0: Writer[(structures.Location | WorkspaceSymbol.S0)] = 
    upickle.default.writer[ujson.Value].comap[(structures.Location | WorkspaceSymbol.S0)] { _v => 
      (_v: @unchecked) match 
        case v: structures.Location => writeJs[structures.Location](v)
        case v: WorkspaceSymbol.S0 => writeJs[WorkspaceSymbol.S0](v)
    }
  given reader: Reader[structures.WorkspaceSymbol] = Pickle.macroR
  given writer: Writer[structures.WorkspaceSymbol] = upickle.default.macroW
  case class S0(
    uri: RuntimeBase.DocumentUri
  )
  object S0:
    given reader: Reader[structures.WorkspaceSymbol.S0] = Pickle.macroR
    given writer: Writer[structures.WorkspaceSymbol.S0] = upickle.default.macroW

First step is obvious - we can move the direct codecs somewhere else (into huge object codecs with lots of traits)

object WorkspaceSymbol extends codecs.WorkspaceSymbol:
  case class S0(
    uri: RuntimeBase.DocumentUri
  )
  object S0:
    given reader: Reader[structures.WorkspaceSymbol.S0] = Pickle.macroR
    given writer: Writer[structures.WorkspaceSymbol.S0] = upickle.default.macroW

where

object codecs:
  ...
  trait WorkspaceSymbol:
    private given rd0: Reader[(structures.Location | WorkspaceSymbol.S0)] = 
      badMerge[(structures.Location | WorkspaceSymbol.S0)](structures.Location.reader, WorkspaceSymbol.S0.reader)
    private given wt0: Writer[(structures.Location | WorkspaceSymbol.S0)] = 
      upickle.default.writer[ujson.Value].comap[(structures.Location | WorkspaceSymbol.S0)] { _v => 
        (_v: @unchecked) match 
          case v: structures.Location => writeJs[structures.Location](v)
          case v: WorkspaceSymbol.S0 => writeJs[WorkspaceSymbol.S0](v)
      }
    given reader: Reader[structures.WorkspaceSymbol] = Pickle.macroR
    given writer: Writer[structures.WorkspaceSymbol] = upickle.default.macroW

This should declutter the main codebase. For the anonymous structure, we can generate a S0Codecs inner trait in codec.WorkspaceSymbol and use it like this:

object WorkspaceSymbol extends codecs.WorkspaceSymbol:
  case class S0(
    uri: RuntimeBase.DocumentUri
  )
  object S0 extends S0Codecs

Isn't this nice.

On the one hand, LSP spec has several places where raw JSON values are used - as data parameter to errors, for example, or in various LSPAny situations.

For those, I really like the super concise, unsafe ujson.Value that upickle provides, and I think it should stay.

Now, how that value is produced is another story:

1. I struggle with upickle Scala 3 macros quite a bit
2. Its handling of optional values is not exactly fitting for langoustine needs and requires invention of new types (Opt)
3. A lot of manual construction of codecs takes place, which could be avoided

Additionally, jsonrpclib is already using jsoniter, so it would be good to align dependencies.

We need to see whether:

1. Jsoniter macros require less maintenance
2. We can output ujson's Value from jsoniter parsing
3. It will be better

I want to upgrade to upickle 3 eventually, but before that we need some guarantees about how JSON is encoded/decoded.

e.g. in InitializeParams, the rootUri is deprecated. However, because of scaladoc magic, the deprecation is recognized on InitializeParams itself. This results in silly Scaladoc rendering:

At least it happens in metals (vscode) on hover of InitializePArams:

Need two variants:

1. Using a custom extension
2. Using langoustine-vscode

Also make sure to mention #84 (comment)

As on some editors (looking at you, Neovim), it's sometimes impossible to click on the link that gets sent back

Also worth exploring augmenting the service info sent from the target LSP and adding to it

https://github.com/neandertech/langoustine/blob/main/modules/lsp/src/main/scala/generated/requests.scala#L1530-L1535

On top of mixings and extends, they seem to do the same thing, and are quite rare.
We should generate an anonymous class with the fields of all type constituents

neandertech/jsonrpclib@v0.0.5...main

Considerable amount of changes, which affect tracer

Tracer already keeps the request-response interactions in memory, but we can render them to gzipped bytes and keep the compressed representation of entire sequence of requests up to a certain point.

The user should be able to select a point in the timeline and request a json file with all the requests/responses. User can select the format in which to receive the file:

1. Requests only
2. Responses only
3. Requests and responses

For convenience, the tracer should also provide the exact curl command to run to get the same output.

Currently you can call clients like this:

.handleNotification(initialized)(
  _.toClient
    .notification(
      window.showMessage,
      ShowMessageParams(MessageType.Info, "hello from badlang server!"),
    )
)

it would be cool to do it this way:

.handleNotification(initialized)(
  _.toClient
    .notification(
      window.showMessage(ShowMessageParams(MessageType.Info, "hello from badlang server!")),
    )
)

feels more DX-friendly because of the error messages:

vs the status quo:

I know this may be something that should be resolved in the language server / in dotty, but perhaps the single-parameter way will seem better for other reasons.

Two things we want from a correctly configured Scribe logger:

1. It only writes to stderr (on all platforms), because otherwise it screws up the jsonrpc protocol
   • See https://github.com/neandertech/langoustine/blob/main/modules/tracer/backend/src/main/scala/logging.scala#L4-L8 on how it's done
2. On Node.js, it uses ANSI colours where appropriate
   • See keynmol/tree-sitter-grammar-lsp@a5b1179 on how it's done

Ideally we want to provide a def logger in LangoustineApp and all of its variants (so IO-based in LangoustineApp, unsafe in LangoustineApp.FromFuture) which is already pre-configured for the platform

It would be amazing if one could see both the request & the response for a given request while viewing the interaction's client or server. For example:

^ both of the JSON snippets could be shown whenever you open either the request or the response.

https://github.com/neandertech/langoustine/blob/main/modules/lsp/src/main/scala/tools/SemanticToken.scala#L24

Regarding tokens being multiline: microsoft/language-server-protocol#982

Safer to just remove this method altogether.

If you are investigating a server that is crashing (like I'm doing at this moment), it would be useful if Langoustine didn't shutdown with it.

Because we don't persist any data in memory on the client side, you immediately lose access to the logs/traces.

What I would like is

1. A --keep-alive CLI option for the trace command, that will not shutdown the tracer if the child process is shut down
2. A Keep alive toggle in the UI, which can enable the mode on an already running server (so you can enable it right before you perform an action that crashes your server)