How do you feel about raising this issue to Zarr? It seems like something many people (myself included) would be interested in.

from ndarray.scala.

+1

from ndarray.scala.

@jakirkham sure, happy to discuss! Do you mean starting an issue there for brainstorming features, or implementing it over there, or something else?

I've obviously let things drag here, but as a small status update:

This branch has a web app that can display metadata about Zarr files stored in GCS, entirely client-side. It does an OAuth redirect to google, then hits the GCS HTTP API for file metadata and to retrieve the contents of e.g. .zarray files. Once that's fully working, all that will be left will be to, you know, build the whole page 😀

Of course, I am doing this so far here in Scala, and don't know what that will mean for collaborating on it with the larger Zarr community! Though we're going to have to explore that question one way or another.

Anyway, I'm interested in your thoughts, here or on a Zarr issue!

from ndarray.scala.

@ryan-williams I'd love to load a demo website in my browser that shows off your web app, even if it is a minimal proof of concept. I don't know how to run the code you linked to.

You might be interested to see related work by Maximilian Haeussler, who already made his own file format for quickly showing single-cell RNA-seq data on the web. His cellBrowser is a set of Python scripts to digest scRNA-seq data into a folder of compressed data files and a static HTML site. Javascript on the client-side makes range requests to download chunks of data from the files hosted by any HTTP server (apache, nginx, etc.), then Javascript draws data on a canvas, etc.

cellBrowser in action with 1.3M cells 🐎

https://cells.ucsc.edu/

Here are the files behind the 10xMegacell website:

https://cells.ucsc.edu/10xMegacell

from ndarray.scala.

@slowkow thanks for that pointer, I'd not seen cellBrowser, it looks wonderful.

What I'm proposing here is more like a "file explorer" for HDF5 / Zarr files (most likely hosted in GCS/AWS/ABS), and not specific to single-cell data.

I'd love to load a demo website in my browser that shows off your web app, even if it is a minimal proof of concept. I don't know how to run the code you linked to.

Yea, sorry! It's not worth trying to demo the current state of my "ndviewer", but I appreciate the nudge and am picking this work back up after the holidays. I will let you know when there's something you can play with. Soon 🤞🏻🤞🏻

For my records / if you're curious: I dusted off that branch last night; here's a rough state of things:

• Uri: portable interface for files (arrays of bytes that are block-{fetched,cached}, can be remote). Current implementations:
  • local files:
    • JVM
    • JS (using node fs filesystem module)
  • fetching http "files" in REST-ful fashion:
    • JVM (using http4s)
    • JS (with XHRs, for browser usage)
  • GCS:
    • wraps Http to read from Google Cloud
    • single implementation (useing platform-specific Http) works for JVM/JS!
• in progress: porting the zarr module to use Uri instead of hammerlab/path-utils (which is JVM only), as part of supporting Zarr in JS / in the browser
• web app
  • renders an empty page
  • performs a google oauth flow
  • fetches a test file from GCS

from ndarray.scala.