lukashornych / evitalab Goto Github PK

When using GQL or evitaQL console on touch screen, left and right swipes on query editor switches the side tabs. This not desirable because we use the left and right gestures for editor movement.

Original naming of data grid "data grid" is not very clear on what it is in context of evitaDB. May better name would be entity grid. This way it is clear that it somehow displays entities in some form of grid or table.

When several developers (usually WD with AD) are debugging a problem with data or query, they need to share query with variables. But sharing these information using messaging application is tedious and may lead to misunderstanding and not 1:1 copy of the original query/variables.

We could leverage the demo query opening feature with the future planned functionality of storing opened tabs in local storage for session restoration to allow providing a URL link that would contain actual URL of the evitaLab and base64-encoded object with opened tab (in which the link was created) with all the data in the tab (query, variables, ...). This way one developer could easily share entire debugging stack to other developer just via link.

One problem is that we didn't want to allow custom queries to be passed as URL parameter other than safe ones from documentation. We fear that someone may misuse the functionality for attacks, so this will need to be discussed further.

cc @novoj

We want to allow users to visualise the JSON result in GraphQL/evitaQL consoles.
Primarily we want to visualise the extra results (facet summary, hierarchy, histograms) to simulate parts of a e-shop. This should help newbies as well as developers already using the evitaDB to see what is possible with the evitaDB JSON result.

• GraphQL console visualiser
• evitaQL console visualiser
• common visualiser layer for both GraphQL and evitaQL consoles
• entities visualiser (similar to grid)
• extra results visualiser
  • facet summary
  • hierarchy
  • price histogram
  • attribute histograms
  • telemetry

TODO

Right now, there is no way to display prices of entities in data grid.

The idea is to display only a count of fetched entities in grid cell and in detail display entire pretty printed list of entities.

In evitaDB we have functionality to monitor client requests by assigning them ids. Specifically clientId and requestId as documented in the
Monitor section of evitaDB docs.

In evitaLab, the clientId would be consisted of static evitaLab prefix and value of evitalab_servername cookie (or standalone if omitted).
The requestId would be dynamically generated uuid for every call of evitaLab to evitaDB instance. Because in evitaLab the requests are not logically joined together like on website where the single page load is the joining element, we will not share request ids between requests right now (this can change in future).

cc @novoj

There are already features that are only applicable when they have configured needed data, e.g. GraphQL tools should be only usable when a certain connection have GraphQL API URL defined.

Currently, all such data are usually required, but this is not the desired behavior in future. Instead, we would like to make all of these required data optionally and disable the tools that needs it, but make it still visible so that the user has idea of all tools available if he/she add configures more data.

There must be an internal API for the toggling and disabling GUI part. TODO

To use actions in navigator, user needs to open context menu. It would be nice that the most used action would be automatically listed only on item hover overlaping the item title.

Especially in data grid, it would be helpful to prioritize certain constraints based on likelyhood of usage. Primarily the entityPrimaryKeyInSet constraint should be on the first place as it is probably the most used one in the grid.

Current semantic versioning doesn't seem to be appropriate/meaningful for users as the evitaLab is not library but end-user application. More appropriate versioning seems to be calendar versioning like uses e.g., IntelliJ IDEA

https://medium.com/javarevisited/application-versioning-strategies-de353a84faaa

Also, the commit messages (feat:, fix:, ...) should be, I think, chosen from end-user perspective, not from perspective of codebase.

This URL incorrectly displays the values in the columns.

I've selected Categories and added attributes status and visibility to the default listing.

We already have basic CSS styling in place for HTML rendered from Markdown texts. Now we also support rendering user HTML directly inside the grid property previewer.

Would it be possible to apply some basic styling for this as well? Unlike in the case of MD, there may be custom HTML tags and stylings but usually we render only partial HTML documents (like product descriptions) where the CSS is referencing stylesheet in the e-commerce system. That's why some basic styling like padding/margin and some positioning could help with readablity even more than we have now.

We could also explore rendering the user HTML inside iframe so that the user HTML couldn't interfere with evitaLab's HTML, but don't know if it's possible with local HTML. Also, the user HTML can be entire HTML document, not just part of body which could be problem with our current solution.

Subissue of #117

evitaDB server now provides server status endpoint (FgForrest/evitaDB#406). This and other tooling could be used to create new type of tab that displayes status of a server of a particular evitaDB connection like so:

Not sure which metrics/charts we will be able to render, but we can scrape the prometheus endpoint in time to render these charts in time from the point the tab is opened (without history).

The "Open logs" will be just shortcut to #142
The "Open runtime configuration" will open dialog window with printed YAML runtime configuration of evitaDB with actual values. A new API for converting and printing the io.evitadb.server.configuration.EvitaServerConfiguration config will be needed.

The tab will be available using context menu on connection or clicking the connection item in the explorer tree. This means that clicking the item in the explorer will not expand the item but will open new tab. Expanding will be possible only through the left-side arrow.

The default behaviour of tooltip in Vuetify is to render tooltip immediately when user hovers on specific component. This is not ideal in our situation. Usually we want to delay the render, so that it is displayed on if user hovers on specific component for some time, lets say 1-2s.

https://demo.evitadb.io/lab?sharedTab=N4IgLghgRgKgngBwKYgFwiQNwJaQI4A2AtAMYD2AdgM5kEoA040AChAE4QC2Vao5FFJCTDZKASQAmaEBKScyIRiQiQCZAOYA5LinRZcEEAF9GkKABEVh1KDwBXJGzjT7juAAoAOhQAEP8gR0wqIUXiDMbGQSdsKeIACU9N5+AGbYBGCOAEIeyX4+CGzYJEgAahAE2BJiFJpkAO7uiXl+hcVINRHtADLYVGBUYVAQVMVxzb75bSU1AMJ2bGxIFCS5IACiAKoASuNJk352VI4AYumZbF4HU0UlWUhg9UjL7gCMAAzv9D4ATJ-vE3y+RUYCKUDsmXuj2eoRaQJ8cWGYAucFIEAQEBIuDgcW+f0+3wAbP84X54nDyZNAT4lvZsEsrkDpkgABJ9MAaDicdwAZgB+yBILBENZ7M5XF5XwRICRKLRGKxYBxCTylMpihAmHY2GgdB46GAeW8RmMRiAA

• refactor types and classes to one class/type per file like in Java
• refactor packages based one functionality (data-grid, console, ...) instead of generic model, `services, etc at the top level
• simplify components names (remove package names from names of components
• rename services to manager and make their instances tab-specific (instead of app-wide) if possible to reduce passed data thru components
• split LabService to ConnectionManager and LabManager (or ExplorerManager)

I would take advantage of the fact that "EvitaLab" remembers the last queries used, e.g. the last 20. In the future, I could simply select and prefill the "Filter by" input with these queries.

In evitaDB if attribute is unique, it is automatically filterable but the explicit flag filterable is still empty in such case. We would like to state this fact in schema viewer in evitaLab as a side note maybe, something like:

(since the attribute is unique, it can be filtered by it)

We don't want to just simply toggle filterable flag without the side note, because it could indicate that developer explicitly set the attribute as filterable.

evitaDB now has new flag representative on entity/catalog attributes. We need to add support for it into the evitaLab.

• correctly display it in catalog/entity/attribute schema viewers
• load representative attributes automatically when opening data grid
• display representative attributes next referenced PK

This issue is just a reminder from our discussion on Discord. Currently it's not possible to select reference attributes as columns of a grid view.

Steps to reproduce:

1. define localized attribute in entity type
2. go to entity grid listing - no locale is selected when accessed for the first time
3. open column selection and enable all attributes to display
4. error is displayed:

I'd expect that no error will be displayed and the localized attribute column displays <NO LOCALE SELECTED>. This was the previous version behaviour.

Currently, long catalog names and entity types are being trancated. This is correct, but on mouse hover a tooltip with full name should be displayed.

Also it could be helpful if collections have their count next to them.

When an user attempts to sort entities by localized attributes, nothing is sorted because data locale is selected using dataInLocales rather than entityLocaleInSet which evitaDB ignores when ordering or filtering.

Because time pressure during initial evitaLab development, all UI text are hard-coded into actual components. Which is obviously bad idea and doesn't allow for multiple localizations.

That's why we need to implement something like https://www.npmjs.com/package/vue-i18n.

• choose i18n library
• get rid of all hard-coded texts
• prepare process for translations to other locales

Right now, there is no way to discover possible keyboard shortcuts that can be used.
It would be helpful to list all available shortcuts, even for codemirror.

• research possible library to display shortcuts
• create new tab component with all shortcuts to view and maybe search by name and actual shortcut
• create button in main navigation to access it
• test all shortcuts on mac
• fix CodeMirror shortcuts on macOS

In evitaLab we use markdown renderer (https://github.com/markdown-it/markdown-it) for rendering schema descriptions and entity property values. Currently, the renderer renders only base HTML without any CSS styling, therefore it's not pretty and not much usable when the markdown source in more complicated. Also, we would like the rendered markdown to match styling from evitaDB docs.

After some discussion with @MiroslavAlt, the simplest way to implement this right now, will be to copy needed CSS classes from evitaDB docs into evitaLab. In future, we may consider publishing some CSS lib with the common classes, but for now this copying should be enough. More so because we need only subset of what evitaDB supports, as in evitaLab there is only plain markdown without custom React components.

Note: before we can realize this, we need to finish the entity property value detail, because there is implementation of the new markdown renderer.

cc @MiroslavAlt

Our GD prepared some graphics changes (colors, icons, styling) that would match our evitaDB docs. portal and would look nicer.
We need to incorporate those changes into our codebase.

Ideally, we want most of the changes to done globally by defining custom theme in src/plugins/vuetify.ts (docs) and overriding default props for Vuetify components.
Other options are to override global CSS in src/styles/settings.scss or change styling directly in the custom components, but this should be used sparingly.

Currently used icons: https://pictogrammers.com/library/mdi/

To-Do

• create new Vuetify theme based on the prepared design
• modify icons based on the prepared design (consider changing icon font?)
• change CodeMirror theme to match the evitaDB docs if possible (some themes are at https://thememirror.net/)
• some custom components needs to be modified based on the prepared design
• change evitaLab icon to new one in README.md, .idea, favicon, and so on
• create new print screen from demo on our README.md with new design (@lukashornych)

The issue right now is that evitaQL parser return location of error with position of cursor in line, but the editors in evitaQL console don't display status bar with the current location of cursor. Therefore it is hard to navigate to the error.

• console
• entity grid

https://demo.evitadb.io/lab?sharedTab=N4IgLghgRgKgngBwKYgFwiQNwJaQI4A2AtAMYD2AdgM5kEoA040AChAE4QC2Vao5FFJCTDZKASQAmaEBKScyIRiQiQCZAOYA5LinRZcEEAF9GkKABEVh1KDwBXJGzjT7juAAoAOhQAEP8gR0wqIUXiDMbGQSdsKeIACU9N5+AGbYBGCOAEIeyX4+CGzYJEgAahAE2BJiFJpkAO7uiXl+hcVINRHtADLYVGBUYVAQVMVxzb75bSU1AMJ2bGxIFCS5IACiAKoASuNJk352VI4AYumZbF4HU0UlWUhg9UjL7gCMAAzv9D4ATJ-vE3y+RUYCKUDsmXuj2eoRaQJ8cWGYAucFIEAQEBIuDgcW+f0+3wAbP84X54nDyZNAT4lvZsEsrkDpkgABJ9MAaDicdwAZgB+yBILBENZ7M5XF5XwRICRKLRGKxYBxCTylMpihAmHY2GgdB46GAeW8RmMRiAA

Used locales are only visible in collection schema viewers. But in GraphQL we use catalog-wide locale/currency enums. It may be nice to display this aggregation across all collections in catalog schema viewer.
But it should probably have note, that it is a virtual list.

See query:

query {
  getProduct(primaryKey: 63049) {
    priceInnerRecordHandling
  }
}

With result:

Yet the grid displays:

Filter and order inputs in entity grid have functionality to ignore any new line characters (primarily when pressing Enter manually) to not break the UI. However, there is one side-effect to how the functionality is implemented. If there is a newline character in new registered text change, the entire CodeMirror transaction is discarded. This is fine when the query is manually typed, but when pasted as a whole, the entire query is a single change for the editor and thus it is discarded entirely.

Ideal solution would be to remove all new-line characters from the pasted text and than paste it into the editor.

Connections, catalogs and collection are not being sorted when rendered in the explorer which makes it harder to navigate through. All items in the explorer could be automatically sorted on client side to make navigation easier.

This issue was created to track ideas for lab usability improvements:

• for localized columns, instead of <null>, display <no locale selected> (and mark localized attributes in the column heading with some character - for example the same as selection widget: )
• distinguish column headers that are sortable from those that cannot be clicked on and sorted by (
  sign?!)
• allow selection of entire groups of column by their type (attribute, associated data, relation etc.)
• add page size 25 for bigger monitors
• add JSON / HTML renderer for preview of the column value
• allow following relations by clicking on the referenced primary key of the entity
  • allow reversed relations following by Ctrl+Click on the referenced primary key, this will either find all entities in currently viewed collection that has reference to that clicked entity, or ideally it would show small menu with all collection that has the clicked reference and find all entities in that target collection

It would practical to have application-wide symbol search feature. By this I mean the evitaLab would index all schema elements (catalogs, collections, attributes, references,...) for all initialized connections, and then with single keyboard shortcut user could search through all of the indexed data with possibility to "open" them (either open schema viewer, or in case of collection it could even open data grid).

Note: it would not index any data, just "symbols" defined in evitaDB schemas.

Vuetify's VTab component that is currently used for switching editor tabs is meant for such a dynamic use case and thus lacks some important UX features other editors have. For example:

• using a mouse scroll wheel to scroll a view when there is too many tabs to fit the screen
• change order of tabs using mouse

We could either try to modify the VTab component to support these features but i think it may require a completely custom component.

For some tabs, it make sense to allow only a single instance of particular tab for specific parameters. Those are tabs without changeable user data:

• keymap viewer
• schema viewer

Right now the data grid property selector is constructed rather manually because initially there were only two layers (sections and direct properties). Now we have already 3 layers (each property can have child properties). Although we don't expect more than 3 layers in future as even the third layer is rather difficult to render flattened, even the 3 layer layout is difficult to manage manually.

We could transform current data structures to a single tree of different nodes (section nodes, property nodes, ...), the entire rendering and user input handling could be largely automated. We could use the subtrees to toggle entire sections, filter nodes with references to parents (this way we could maybe hide entire sections if there are no filtered properties in them. We could also have property nodes directly in root next to the section nodes (this would be useful for prices as they don't have any children and a price section would be unnecessary for a single price property).

The tree structure could look something like this:

Root:
  - SectionNode
    filterableName: entity
    childrenNodes:
      - PropertyNode
        filterableName: primary key
        propertyKey: ...
        propertyDescriptor: ...
        selected: true
  - SectionNode
    filterableName: attributes
  - PropertyNode
    ...

This tree would be computed at the component initialization from input props and would be used to manager rendered selector.

The https://codemirror.net/docs/guide/ chapter "Affecting the view" there is example plugin that display document size. Our status bar should maybe be done similarly instead of custom component outside of codemirror.

It would be great if Ctrl+W would close the current tab, and Ctrl+Shift+W would close all open tabs. Soon I have dozens of tabs in my lab which is hard to navigate. The shortcuts would allow to quickly clear the context without a mouse.

We want to restore lastly opened tabs. We already have support for tab serialization thanks to the #96 so we can use that.

Handling storing/restoring + handling multiple tabs, shared tabs, demo requests

Tabs will be stored in local storage. The actual storing of current tabs will happen on onunload/onbeforeunload JS event, until that, the data will be only in memory of LabEditor component. The subsequent new evitaLab instance start, the editor will pick up lastly stored tabs (if any), remove them from storage and loads them to memory. And so on.

If a new evitaLab session will be opened in parallel to already existing one (in different browser tab), there will be no tabs to restore in storage (which is correct, we don't want to duplicate stale version of tabs that were already picked up), empty session will be created. Each session closing will store its tabs in storage. If multiple parallel sessions will be closed, the last one will be restored on next evitaLab session.

If evitaLab is created through shared tab or demo request, no session will be restored and the editor will not store the actual session on closing. It will be functioning in "playground" mode or something. This is to prevent unwanted overriding of previously crafted session when someone sends a link with shared tab to the user or the user will be reading documentation and opens example in evitaLab. We don't want to restore last session in this cases, because the user may accidently lose the previous session if closed parallel sessions in wrong order in hope that the shared tab or demo request is not actual session.

Subissue of #117

New version of evitaDB will be able to reason about different versions of the catalog, which are directly related to certain transactions (some of them could be grouped together - but still the granularity will be quite fine). evitaLab should introduce a new "menu item" called "Time Machine". It will allow to browse through all available versions of the catalog stored on the file system - for each record there will be information about

• catalog version
• offset date time of version persistence

For each of these versions, the catalog will be able to provide a list of changes made in a particular transaction (transaction, if several are collapsed within it). But this will be a more expensive operation because the WAL would have to be accessed, read, and aggregated. Also, the information will be missing if the WAL has rotated in the meantime and the information is no longer accessible.

So the UI should provide means to focus on a certain time period and list a few (10-20) versions of the catalog from that time period. For each of this period, it should try to access the WAL and create a digest of the changes made in that version.

Time Machine should be able to restore the snapshot of the catalog in one of these particular catalog versions into a new catalog and load it into a store so that the data becomes available for querying.

When connection to evitaDB instance is lost and user tries to open GraphQL console, evitaLab shows error message but the console keeps loading (even though its not loading anything) and the user has to close and reopen the tab. It will be nice if the console would stop loading when error occurs and display "retry" button.

To make evitaLab even more useful, we want to add several tools for database server/catalog/collection management.

For management of database server itself we would like to:

• status 🟢 -> solved in #103

  • URL, version, API URLs, ...
  • evitaDB runtime configuration
  • basic charts? (heap size, cpu usage, ...)

• logs + filtering 🟢 -> solved in #142

• JFR tracing (start, stop, download, configuration) 🟢 -> solved in #143

  • optionally embed JFR visualizer if possible

• create catalog -> solved in #144

Cluster management (in future):

• health, lags
• master/replica view

For catalog management we would like to:

• catalog preview -> solved in #161
• query recording -> solved in #150
• time machine 🟢 -> solved in #95
  • catalog version
    • mutation listing with filtering + mutation detail
    • create new catalog from version
• backup, restore 🟢 -> solved in #147
• drop 🟢
• unload?, load?
• replace, rename 🟢
• (un)make readonly
• create collection
• consistency
• API compatibility
  • REST
  • GraphQL

Entity collection management:

• index listing 🟢
  • size
  • consistency
  • recreate
• drop 🟢
• rename 🟢

Items marked with 🟢 have API support in evitaDB and can be implemented. Each section/item will be solved in separate issue.