Git Product home page Git Product logo

kyrix's Introduction

Kyrix - democratizing details-on-demand data visualizations

version build codestyle license contribution gitter

Kyrix facilitates the creation of data visualizations with details-on-demand interactions (e.g. pan and zoom, see the demo gallery1 above). In visualizations of such, the underlying dataset is often large. To deal with large data, Kyrix is focused on optimizing two goals: 1) usable declarative API library for visualization developers and 2) 500ms response time to user interactions, which is required to enable interactive browsing.

  1. This gallery GIF is itself a Kyrix application.

Installation instructions for the impatient

  1. Install Docker and Docker Compose (must be v1.17.0 or later). For Mac users, installing Docker Desktop will install Docker Compose automatically. Ubuntu 18.04 users can also simply install with sudo apt update; sudo apt install -y docker.io docker-compose;.
  2. Run sudo ./run-kyrix.sh --nba --build in the root directory. You might need to make run-kyrix.sh executable, i.e. sudo chmod +x run-kyrix.sh.
  3. Wait a couple minutes, then point your browser at <ip address>:8000 - remember that if you are using a cloud instance you may (probably) need to open your cloud provider's firewall for this port. If that sounds scary, you can create an SSH tunnel from your PC (e.g. Mac) using ssh -N <server ipaddr> -L 8000:<same ipaddr>:8000 to forward your laptop's port 8000 to the server via SSH tunneling.

note that you'll need to wait for a message saying Backend server started... like this:

Serving /project
New project definition coming...
There is diff that requires recomputing indexes. Shutting down server and recomputing...
Precomputing...
Done precomputing!
Completed recomputing indexes. Server restarting...
Backend server started...
*** done! Kyrix ready at: http://<host>:8000/

Getting started by reading a tutorial, and more documentations.

Setup-free big data visualizations

Kyrix is fully dockerized, with a live PostgreSQL database running right after docker startup. Front-end vis libraries often assume data fits in browser memory, and thus scale to only small data. By working with a containerized database, Kyrix scales to much larger data while being free of the hassle of installing and maintaining databases.

Docker works by creating fully isolated virtual machines ("containers") on your computer, making it much easier to achieve correct installations every time (learn more). See docker config details.

Declarative authoring

Kyrix offers two declarative grammars for authoring complex details-on-demand visualizations. Low-level Kyrix grammar is verbose but expressive. Kyrix-S grammar is a high-level and concise grammar designed for zoomable scatterplot visualizations. For example, Kyrix-S turns

{
    data: {  
        db: "nba",  
        query: “SELECT * FROM games"  
    },  
    layout: {  
        x: {  
            field: "home_score",  
            extent: [69, 149]  
        },  
        y: {  
            field: "away_score",  
            extent: [69, 148]  
        },  
        z: {  
            field: "agg_rank",  
            order: "asc"  
        }  
    },  
    marks: {  
        cluster: {  
            mode: "circle"
        },  
        hover: {  
            rankList: {  
                mode: "tabular",  
                fields: ["home_team", "away_team", "home_score", "away_score"],  
                topk: 3  
            },  
            boundary: "convexhull"  
         }  
    },  
    config: {  
        axis: true  
    }  
};

into

Embedding into your favourite web app

Kyrix offers APIs to integrate a kyrix application into a web application. There's no limit on what you can imagine: complex coordinated views, programmatic pan/zoom control, cross filtering...

Telling us about your Kyrix app

We'd like to learn more about your application requirements to make Kyrix better. If you have built a Kyrix app, send us a screenshot on Gitter!

Contributing to Kyrix

We welcome contributions! Please let us know about your change beforehand if it is big. Also, see contributing guidelines.

Kyrix is maintained by a group of researchers at MIT's data system group. The system is not possible without the contribution from these wonderful developers:


Wenbo Tao


Adam Sah


Xinli Hou


Xiaoyu Liu


Yedi Wang


Peter Griggs


Maxime Schoemans


Erica Zhou


Abhishek Bassan


Scarlett Zhang


Amy Zhang


Jim Peraino

Disclaimer

The current v1.0.0-beta release contains research-quality code, and is intended for demo purposes. The code should not be used for large-scale secure serving. Currently, v1.0.0-beta only supports data with up to 10~100 millions of data records. We plan to release a distributed version of Kyrix in the future.

Citing Kyrix

@inproceedings{tao2019kyrix,
  title={Kyrix: Interactive pan/zoom visualizations at scale},
  author={Tao, Wenbo and Liu, Xiaoyu and Wang, Yedi and Battle, Leilani and Demiralp, {\c{C}}a{\u{g}}atay and Chang, Remco and Stonebraker, Michael},
  booktitle={Computer Graphics Forum},
  volume={38},
  number={3},
  pages={529--540},
  year={2019},
  organization={Wiley Online Library}
}

kyrix's People

Contributors

abhishek-bassan avatar asah avatar brettgurman-real avatar dependabot[bot] avatar ericazhou7 avatar houxinli avatar mschoema avatar peterg17 avatar scarlettz98 avatar tracyhenry avatar xiaoyu4321 avatar yediwang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kyrix's Issues

zoom/panning "state" not stored in URL (e.g. #z=<level>&x=XXX&y=YYY)

Web users expect to be able to reload a page and go back to the current view.
In particular, this means that kyrix views are not shareable to other users and can't be bookmarked.

(to demonstrate this, pick any example, zoom/pan and then hit reload in the browser or try to copy/paste the URL into a new tab/window)

pushing predicate generation into backend (important security issue)

right now the predicate functions are run in the client. The resulting predicates are then passed to the backend. This will invite SQL injections...

Tentative solution is to put the generation back into the backend. Also, it's necessary to write a parser to ensure the predicates generated are in "good form"

nashorn-common-js breaks when requiring d3

This is because nashorn-common-js does not implement native node modules (e.g. http) while d3-request requires http.

The current workaround is to comment out the lines requiring d3-request in node_modules/d3/build/d3-node.js

installation/startup error if project table not created first

Ubuntu 16.04, Postgres 10.0, NodeJS v8.12.0, Maven (mvn) 3.3.9, Java 8,

The setup instructions read: "Go to tile-server/. Run mvn compile to build the server. Run mvn exec:java -Dexec.mainClass="main.Main" to start the server. After the server starts, it will prompt that it did not find the spec of the app and is waiting for it..." and then "cd compiler/examples/nba; node nba.js"

This results in "kyrix@kyrix ERROR: relation "project" does not exist at character 21" (seen in postgres logs). The bad SQL: select content from project where name = 'nba'

Perhaps the best workaround is to manually create the table using this idempotent SQL:
CREATE TABLE IF NOT EXISTS project (name VARCHAR(255), content TEXT, dirty int, CONSTRAINT PK_project PRIMARY KEY (name));

one-command script to load new datasource

so users can quickly move beyond the NBA example.
do this for both docker-compose and kubernetes
follow-on: create instructions on how to format the data (if not already present)
follow-on: create instructions on how to scale up for very large datasets that must be loaded in parallel

newScale function for zoom

Letting the zoom specify a function to calculate the scale of the destination canvas.

This is useful for creating ticketmaster-like geometric_semantic_zooms, where the zoomed-in canvas need to be scaled differently for different zooming entities.

And, an initial scale is needed for the initial canvas.

dbox 'fails' when panning is fast

right now front-end uses pendingBoxRequest to disable concurrent dbox requests.

However, this ignores all subsequent requests while a dbox request is still being processed, and will result in incomplete visualizations when the user already pans outside the returned new box.

make backend stateless

The backend is now stateful because it memorizes the user's history of positions.

This history should be instead maintained by the client, to make the backend stateless and parallelizable

canvas request can be removed

canvas request is used by the frontend to fetch canvas info and static layer data. However, we can probably assume these static info can fit in frontend's memory, and completely eliminate this kind of requests, which sometimes cause laggy animation.

Issue with using port 3306

Error messages:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project tile-server: An exception occured while executing the Java class. Address already in use -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project tile-server: An exception occured while executing the Java class. Address already in use

easy debug

  1. developer shouldn't need to delete project definition before modifying the spec

  2. after writing new spec, backend should automatically detect the changes and re-precompute accordingly

write an indexer that uses built-in postgres gist index

It is confirmed now that postgres has built-in geometry types and GiST indexes. So we should write a new indexer that uses it, test if it works fine, and then possibly get rid of PostGIS, which is a pain to install.

https://www.postgresql.org/docs/9.4/functions-geometry.html
https://www.citusdata.com/blog/2017/10/17/tour-of-postgres-index-types/
https://medium.com/@Alibaba_Cloud/principles-and-applications-of-the-index-types-supported-by-postgresql-481f59bab67d

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.