usethesource / usethesource.github.io Goto Github PK

Salix is becoming stable and it is used by a lot of Rascal projects in research. Let's boost its support-level and horizon by moving Salix to usethesource.

• name: salix
• github: https://github.com/cwi-swat/salix
• project owner: @tvdstorm

vote

vote

• @PaulKlint
• @tvdstorm
• @jurgenvinju
• @DavyLandman

motivation

Salix is used:

• the IDEs for AlleAlle
• the IDE for Rebel
• the UI of DrAmbiguity

Salix fills a gap:

• we need to be able to code UI directly in Rascal
• it gives natural access to (interactive) data visualizations via its bindings of D3 and google charts, etc. which we also need
• it is and can be extended with bindings to all kinds of HTML5 UI features

Salix is efficient (enough):

• although the protocol is quite chatty and uses HTTP sockets now, it seems usable in most situations
• there is room for improving efficiency by moving to pure TCP or WebSockets by removing the latency of starting HTTP connections for every independent interaction

strategic considerations

• the salix programming model fuses pure Rascal with the HTML document model; if you know Rascal and if you know HTML then Salix is "easy" and does not require additional documentation
• HTML5 is here to stay and has a large audience
• Salix contributes to the "one-stop-shop" theme for Rascal, as opposed to the alternative model where people write javascript and html separately and make connections via util::WebServer

expected maintenance on the short term

• Salix needs to be included as a library with the Eclipse and VScode IDEs because currently as a deployed library it does not work (this is also an issue of the library story for Rascal in general)
• Salix has an experimental feature for "modules" which can be removed to arrive at a simpler model
• Salix expects an initial HTML file per application which should be changed to Rascal salix code that models that index file to close the abstraction
• Salix should work with an open TCP socket for every app instead of using the HTTP protocol.
• Salix could use a library for one or two of the CSS abstraction libraries (such as bootstrap.css)

this is to propose that the java support in the standard library, which is wholly based on the Eclipse JDT, be separated into an independent library project called rascal-jdt in the usethesource orginization. The idea would be that rascal-jdt be bundled with the Eclipse and VScode IDE's such that "batteries stay included".

• Project: rascal-jdt
• Lead: @jurgenvinju
• URL: github.com/usethesource/rascal/src/org/rascalmpl/library/lang/java/jdt

Motivation

• the rascal standard library is too large
• the rascal independent commandline jar contains the whole of Eclipse's JDT as a result of the JDT support being in the standard library
• we want to independently evolve the JDT support (independent version number)
• we want to independently track issues on this project

So..

• Let's not wait for the compiler project to finish to make this separation of concerns
• Let's package the new jar with all the IDEs we have, such that users of the library do not notice it has moved.

Votes

• @jurgenvinju
• @PaulKlint
• @tvdstorm
• @DavyLandman

Let's rename the default branch for our repositories from master to main.

Github has tried to make this process as smooth as possible: https://github.com/github/renaming

Roughly, you rename the branch inside github, and it will take care of most redirecting, and will send the user a message about changed branch name during a push.

The following commands will be have to executed locally for every clone of a repo that has been renamed.

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

this is an essential part of the usethesource infrastructure.

These are projects which will need to be upgraded still to java 11, which are not on usethesource (yet):

• flybytes
• clair
• php-air
• python-air
• js-air

Originally posted by @jurgenvinju in #16 (comment)

• Name: rascal-vscode, but should be renamed to rascal-language-servers
• Owner: @jurgenvinju
• github: https://github.com/cwi-swat/rascal-vscode

motivation

• we are moving to support the Rascal IDE and generated DSL IDEs via the Language Server Protocol
• this will be the prime way of interacting with Rascal programs and DSL code in different editor/IDE contexts
• currently VScode is only supported, but VScode is a very popular IDE infra-structure that is also the origin of the LSP
• there is no viable alternative
• this project is extensible with other code to plugin the generic LSP into specific IDE clients (emacs, Eclipse, etc.)

vote

• @PaulKlint
• @tvdstorm
• @jurgenvinju
• @DavyLandman

• the community is in need of a comprehensive archive/collection of Rascal libraries. Think grammars and front-end for specific programming languages, visualization libraries, etc.
• the goal is to make Rascal libraries "discoverable" and "usable", so as to enable more Rascal users to reach their own goals
• we need to think about this carefully, learn from others. For example for R there is CRAN.
• the functionality of entries into the library depends on Rascal's run-time system, i.e. how Rascal libraries are dynamically linked. With the compiler under-way, and this archive not being created over-night, let's consider this from a "compiled Rascal" viewpoint?
• example entries: rascal-csharp, rascal-dotnet, shapes, drambiguity, rascal-ecore
• see also the RAP 1 on rascal deployment, where we speak of a "rascal package server" named r2d2.usethesource.io

capsule by @msteindorfer is the essential library backing the implementation of Rascal's maps, sets and caches. Since it is essential to Rascal I propose we accept it as a usethesource project.

@PaulKlint @tvdstorm please agree.

@msteindorfer please fill in the CLA here: https://docs.google.com/a/vinju.org/forms/d/1uI6IsuKA-XMPt3jMbtQd9u0pQvDFQ99udbOoiB2La0A/viewform

Project

• https://github.com/cwi-swat/flybytes
• lead: @jurgenvinju

Motivation

This may be (is) quite an early proposal, for a project such as flybytes, but I see a not-so-distant future where the rascal compiler uses flybytes to directly generate (debuggable) JVM bytecode. For this to be possible flybytes needs to mature a bit more:

• support nested classes;
• support a flybytes extension library for generating code for lambdas and closures;
• finetune the code generator to generate even more idiomatic JVM code.
• write better documentation and create demonstration movies

Otherwise flybytes seems to be a perfect back-end for the Rascal compiler. It supports "block expressions", for example, which can make code generation considerably easier. Also other language features of Rascal (closures) could relatively easily be modeled as extensions to the flybytes compiler (i.e. flybytes to flybytes stages).

Flybytes also offers Mirrors which can be used to call any method or set/get any field in JVM bytecode directly from Rascal (using Java's reflection API and Rascal closures). It is used to test the flybytes compiler and decompiler, but it has many other applications in interfacing Rascal with arbitrary Java code. The Mirror bridge also marshalls basic types between JVM and Rascal.

Due to the (relatively complete) flybytes decompiler, it is easy to first write Java code to implement a Rascal language feature manually, then reverse engineer this into a flybytes AST pattern to extend the compiler. This feature can also be used when migrating from the Java code generator to a flybytes backend.

It would be cool if flybytes also supports an "intermediate code" debugger. Namely that one could step through the generated flybytes code instead of the original Rascal code that is associated with it. There is an idea on how to do this, by writing out indented AST values to a temporary folder and replacing the .src annotations with locations from these generated files, but it is not yet implemented either.

Another missing feature is an "API generator". Such a tool would read class files (using flybytes) and generate Rascal functions which will generate calls to the public API of such a class file. Currently users of flybytes must type in (copy/paste) the relatively complex method signatures into their code generators and use the "right method invocation primitive" from the JVM (invokeInterface, invokeStatic, invokeVirtual, invokeSpecial).

Question

I would like to invite you to have a look at flybytes and see what you think about the current quality and support. What would be required for you to accept and support this project at usethesource?

Vote

• @DavyLandman
• @PaulKlint
• @tvdstorm
• @jurgenvinju

This is to propose Clair as a usethesource project

• name: clair
• URL: https://github.com/cwi-swat/clair
• lead: @rodinaarssen

Motivation

• clair is an elaborate mapping of the CDT's abstract syntax trees to Rascal data-types
• this includes type and name resolution information
• it is noted that the trees are mapped after the application of the C pre-processor
• Clair is already used in several research and engineering projects (also industrial)
• A high quality and fully implemented front-end for C++ analysis is an enabler for all kinds of "killer-apps" in Rascal
• CDT is relatively stable

Questions

• @rodinaarssen do we have a 5 year horizon for Clair?
• can you lead this for the coming 5 years (with help)?

Votes

• @rodinaarssen
• @jurgenvinju
• @DavyLandman
• @tvdstorm
• @PaulKlint

@mahills; I'd like to discuss the promotion of php-air to usethesource here.

• link: https://github.com/cwi-swat/php-analysis
• What is the status of the project?
• Do you see a future for it in the coming 5 years?
• Should I have asked another question?

I have a very positive view on it. It seems an extensive and well-designed project with a lot of value for people who want to analyze PHP code using Rascal.

Votes

• @mahills
• @PaulKlint
• @jurgenvinju
• @DavyLandman
• @tvdstorm

it now brands use-the-source directly for Rascal; I think we should project a more general image of meta-programming. Perhaps only a grammar and not the header with Rascal and our names in it? Perhaps something even more generic?

see subject. looking for a volunteer.

This is to propose the iguana project to remain where it is at https://github.com/iguana-parser and remove it as a project mentioned on the usethesource.io website.

• we never adopted the iguana project for real into the usethesource.io organization, as per the community processes
• we have not been able to adopt iguana as the parser technology under Rascal (yet?)
• the iguana project has not been maintained in the last few years, as far as I can tell, and currently is not passing its own test suite

The usethesource community should look alive and the projects should be tightly connected. At the moment this is not the case for iguana. Perhaps in the future we can try again. For now let the usethesource website reflect the actual situation.

Vote:

• @jurgenvinju
• @PaulKlint
• @tvdstorm
• @DavyLandman

@joukestoel I'd like to discuss the promotion of AlleAlle to a UseTheSource project here.

• link: https://github.com/cwi-swat/allealle
• it's a very general tool for use in DSL construction; namely it can be used to model and check all kinds of semantic relations and explore them using model checking without having to think about the nitty gritty details of SMT solving.
• it's written in Rascal, but it is also for Rascal's "one stop shop" meme
• it's used to implement Rebel2

Questions:

• Do you also see a future for AlleAlle in the coming 5 years?
• What other application areas are there in the foreseeable future for AlleAlle?
• What are pressing issues which need resolution in short term?
• Did I miss an important question to ask?

This is to register our need to move all usethesource projects to Java 11 (and the associated JRE)

It requires us to synchronize the effort across all projects, hence this issue at usethesource.

Motivation

• Eclipse has moved to JRE 11 and JRE 11 is not backward compatible with Java 8
• Java 11 and subsequent versions have many improvements (speed and API)
• Currently Eclipse can not be tested by @jurgenvinju on his new machine due to this

Vote

This is to vote that we will move to Java 11, but not when. We have to plan the timing carefully so as not to interfere with other complex projects such as the compiler and the LSP.

• @DavyLandman
• @tvdstorm
• @jurgenvinju
• @PaulKlint
• @rodinaarssen // this will have to be done in clair as well
• @mahills // this also has implications for php-air I suspect

Timing

When to do this? The compiler project would be halted by this; i.e. new bugs caused by the move would mix in between the last bugs to solve. Otherwise we do not expect much interference with other projects. Maintenance on the rascal-eclipse project would become possible again after the move to Java 11. So we should not postpone this indefinetely..

Should we not aim higher?

Any reason why we do not move to a higher Java version than 11 immediately?

This is to register the intent to mark "rascal-eclipse" for deprecation and eventual removal. This includes the nested project for the update site, the feature, the actual plugin, and the developers-feature that would link Rascal JUnit tests to their source locations.

• Our (new) users gravitate (by about 90%) to using VScode
• The only feature the VScode extension is missing is the interactive Rascal debugger, otherwise it is feature complete, and even more featureful than rascal-eclipse
• Basic features that we depend on, such as terminal and HTML preview, in Eclipse are not maintained well. Otherwise, Eclipse has proven to be an extremely stable platform (also given the impulse abstraction that we maintain ourselves).
• Although IDE support for DSLs based on util::IDE can not literally be used to call util::LanguageServer from the rascal-language-servers project, the concepts are exactly the same and only some minor rewiring is requested.
• vis::Figure was not used by many students anymore, but still we have to bring the documentation of vis/* and salix-* up to the level of the Figure library
• by removing such code we improve the general sustainability of what is left
• we have to wait at least a year from this day (Feb 2nd, 2023) to give users time to migrate to VScode
• we can only drop Eclipse when we have the debugging feature in VScode, as it is essential for the bootstrapping of the compiler.

Please sign here for the consent of the idea:

• @jurgenvinju
• @PaulKlint
• @DavyLandman
• @tvdstorm

In the meantime we will make only small bugfix releases of rascal-eclipse, to help straggling users while they prepare to migrate.
I will add a documentation issue about documenting the migration from rascal-eclipse DSL implementations to VScode DSL implementations

impulse is the fork of Eclipse IMP for use with Rascal-eclipse and derived IDEs. For as long as we depend on it, it needs to be maintained and for this it should have presence on usethesource. I do suggest that impulse will only remain life on usethesource while we and the IDEs we generate also depend on it, and we make sure to communicate that this is the case on the project's webpage.

As it was Eclipse IMP before, the original authors from IBM have signed off on a CLA and vouched for the open-source licenses therein. An additional CLA is not necessary imho for these contributors from the past.

@tvdstorm @PaulKlint please agree.

rascal-value is the central API for data representation for Rascal. As such it is an essential component for usethesource and we should move it there asap.

@PaulKlint @tvdstorm please agree.

Since pdb.values was hosted on Eclipse.org before (and is now forked and split into rascal-value) the co-authors from IBM have signed a CLA with Eclipse before (Robert Fuhrer in particular), which I consider enough evidence of due diligence.

This is to start the work on an integral move to JDK 17 for all usethesource projects and the dependent libraries that are not on use the source.

The motivation is to keep up to date. In fact several of the libraries we depend on in library projects do not release versions of their class files anymore before JDK 17.

This means we go to JDK 17 as run-time JVM target and as source language. We can start using Java 17 features.

It has to go in one big bang because newer JDK 17 class files are not compatible with older Java 11 run-times.

Core:

• capsule
• vallang
• rascal
• rascal-maven-plugin
• rascal-eclipse
• rascal-language-servers
• rascal-core
• typepal

Libraries:

• clair
• flybytes
• php-analysis
• rascal-git

• naming the new project: typepal (all lowercase now)
• linking the github repository: https://github.com/cwi-swat/TypePal
• project owner: @PaulKlint
• Typepal is a generic type check/inference and name resolution framework for Rascal-implemented programming languages. It is used in the bootstrapping of the Rascal compiler itself, so it is a key component in the usethesource infra-structure.

This is to register the intent to mark the "impulse" project for deprecation and eventual removal.

• only rascal-eclipse depends on impulse and this is to be marked for deprecation and removal as well.
• it seems we have no outside customers for impulse who we could damage, also impulse runtime is hosted at the impulse generation still for archival purposes
• by removing such code we improve the general sustainability of what is left
• we have to wait at least a year from this day (Feb 2nd, 2023) to give users time to migrate to VScode

Please vote here for consent:

• @jurgenvinju
• @tvdstorm
• @PaulKlint
• @DavyLandman

The "project burial process" is described here. We can add more details as we go, as this is the first project we clean up after: https://usethesource.io/about/processes.html#:~:text=Project%20Burial%20Process

rascal is a central project on usethesource and it should be there; but let's hold to the process guidelines as in http://usethesource.io/about/processes.html and quickly move rascal through the incubation process to get things started.

The rascal project includes the following github repositories in the cwi-swat organisation:

• rascal
• rascal-eclipse
• rascal-shell
• rascal-feature
• rascal-update-site
• rascal-pom-master
• rascal-website
• rascal-pom-parent
• rascal-developers-feature
• rascal-eclipse-junit-runner

I propose we let the CLA signing go in parallel with moving the project, as an exception to the process since Rascal is so central and we have all contributed to it knowing it was open-source up-front.

I also propose to fix the problems with file LICENSE's as we go along.

@tvdstorm @PaulKlint please agree.

• In ParseTree.rsc in the standard library it says: anno loc Tree@\loc;
• The interpreter already simulates this by data Tree(loc src=|unknown:///|)
• Most of our Rascal code still uses the @\loc annotation syntax which is simulated by the interpreter and the compiler by .src
• This is scattered over:
  • rascal
  • rascal-core
  • typepal
  • flybytes
  • rascal-eclipse
  • rascal-language-servers
  • ... ?
• We have simple cripts to automatically rewrite locations to keyword parameters, including the renaming of "loc" to "src".
• We need to synchronize this in the same way as the Java11 move, starting with rascal we have to move one-by-one through the dependency graph.
• Its complex to do this while other changes are being done at the same time, since it can invite commit conflicts due to the many small changes in many files.