Problem

• Our server needs to pin a user's private file tree, but it can't crawl it because it's..... private
  • right now this is accomplished with loose pins which are cumbersome to deal with
• when a subtree is shared with a user, they should retain access to it even if the CID changes. So they need to be able to look up the current CID of the tree by using a given filepath

Solution

• Add a single-layer /pins node off of the FS root
• That node will include a list of all nodes in the private file system. The names of the nodes will be a hash of the name plus their DID which acts as a salt: sha(filename + did)
  An example snippet of this:

{ name: sha("private/index/file" + did), cid: "Qm123..." },
{ name: sha("private/index/file/index" + did), cid: "Qm456..." },
{ name: sha("private/index/file/isFile" + did), cid: "Qm789..." },
{ name: sha("private/index/file/mtime" + did), cid: "Qmabc..." },
...

Probably will live in its own library, but tracking for now:

Needs to be able to:

• Create EdDSA key
• Store via WebAuthn
• Retrieve key from _did.someusername.fission.name TXT record
• Validate self-signed tokens via PK

User feedback 💌

With the StrictNullCheck enabled in Typescript, users are required to do:

case Scenario.AuthSucceeded:
case Scenario.Continuation:
  if (state.fs != null) {
    if (state.fs.appPath != null) {
      const appPath = state.fs.appPath();
      ...
    }
  }

Ideally we would not need to do these "null" checks, but not sure how much we could eliminate these. Seeing that the presence of these things depends on the options given to initialise. state.fs can be undefined if given the loadFileSystem: false option (which one may do to load the file system themselves in a web worker), and fs.appPath can be undefined if the permissions.app option was not set.

Problem

Traversing the filesystem can take a long time especially for deeply nested & private files which require multiple back and forths to the network with decryption in between.

Solution

• Store a "cache" in the header of dirs that contains the subtree.
• Update the cache as the subtree is updated.

Note: implementation is similar to the current "pins" implementation that is used to track nested pins for the private tree

So that the IPFS instance is shared across tabs.
https://github.com/ipfs/js-ipfs/tree/master/examples/browser-sharing-node-across-tabs
https://github.com/ipfs/js-ipfs/tree/master/packages/ipfs-message-port-client
https://developer.mozilla.org/en-US/docs/Web/API/MessageChannel

We're deprecating https://github.com/fission-suite/typescript-client/ and moving most of that here.

Notes:

• I don't think we need to support the legacy username/password stuff here.
• Same thing for all the other deprecated things (correct me if I'm wrong)

Already implemented

• Create user account with DID & UCAN (sdk.lobby.createAccount, lobby/index.ts)
• Update user data root (sdk.dataRoot.update, data-root/index.ts)

Endpoints yet to implement

Apps

I would add this under sdk.apps

• GET /app ☞ sdk.apps.index
• POST /app ☞ sdk.apps.create
• PATCH /app/{App URL}/{New CID} ☞ sdk.apps.setContent
• DELETE /app/associated/{url} ☞ sdk.apps.deleteByUrl
• DELETE /app/{appId} ☞ sdk.apps.deleteById

User

• GET /user/whoami ☞ sdk.lobby.whoami or sdk.lobby.lookupUsername (not sure which)
• PATCH /user/did ☞ sdk.did.update
• PUT /user/exchange/keys/{did} ☞ sdk.exchange.addPublicKey (not a 100 on this)
• DELETE /user/exchange/keys/{did} ☞ sdk.exchange.removePublicKey (not a 100 on this)

The following occurs when I try to run:

var wnfs = await FileSystem.empty();
const cid = await wnfs.mkdir("private/gunclear/safe")

Error:

Unhandled Rejection (TypeError): o[Symbol.iterator] is not a function
./node_modules/fission-sdk/ipfs/basic.js/</__asyncValues<
http://localhost:3000/static/js/1.chunk.js:105016:96
./node_modules/fission-sdk/ipfs/basic.js/</add/</<
src/ipfs/basic.ts:11

   8 |     });
   9 | };
  10 | var __generator = (this && this.__generator) || function (thisArg, body) {
> 11 |     var _ = { label: 0, sent: function() { if (t[0] & 1) throw t[1]; return t[1]; }, trys: [], ops: [] }, f, y, t, g;
  12 |     return g = { next: verb(0), "throw": verb(1), "return": verb(2) }, typeof Symbol === "function" && (g[Symbol.iterator] = function() { return this; }), g;
  13 |     function verb(n) { return function (v) { return step([n, v]); }; }
  14 |     function step(op) {

step
http://localhost:3000/static/js/1.chunk.js:104996:17
./node_modules/fission-sdk/ipfs/basic.js/</__generator</verb/<
http://localhost:3000/static/js/1.chunk.js:104927:14
fulfilled

We should move generated docs to a separate branch. Open to other solutions :)

Problem

Getting a 422 status with the error message:

Time bounds are not a subset

Impact

Data-root update request fails.

• Spec
• Implementation

See docs/ for more detailed documentation based on the source code.

I get a 404. Is this just outdated? Should it perhaps only link to the gitbook?

Needs to check if the user is online before doing any DNS lookups.

Self-signed tokens with auth delegated via macaroons

• Spec
• Implementation

We need to walk trees for a few different features:

• mv between branches of the FS
• upgrade the structure of entire subtrees
• any sort of aggregation

API blocklist was recently updated, need to match it here.

Problem

Creating IPLD folders is cumbersome and not intuitive

Solution

• Add helpers for creating folders
• Add helpers for creating & sharing private folders

IPFS DNS resolver was offline today, we should have a fallback.

Problem

Currently metadata for a node is encrypted with that node's parent key (the key used to encrypt that node), not the node's own key (the key contained as a header value in that node). Thus, when a user is given access to a file, they don't get access to the metadata that goes with it

Encrypt metadata with the node's ownKey instead of the parentKey

Depends on fission-codes/fission#348 and the file system refactor.

Instead of using the Cloudflare API with query params, use https://dohjs.org/ and then send a DoH request using that library. This way we can easily switch between servers for looking up DNS TXT records and it's safer. Ideally we'd have a few fallback servers in case Cloudflare doesn't work.

https://github.com/curl/curl/wiki/DNS-over-HTTPS#publicly-available-servers
Try it out at https://dohjs.org/ to see what the results are of a DNS lookup.

So the filesystem doesn’t suddenly stop working. That said, window.ipfs is outside our control as well. Maybe switch entirely to js-ipfs (intead of get-ipfs), like we discussed before? 🤷‍♂️

Problem

In the auth redirect flow readKey, is confusing and could be interpreted as the symmKey

Solution

Change name to readPubkey or similar

The README should be short and have some dev stuff in it, the getting-started portion and the tutorial should be moved to Fission Guide. Besides that we also have the docs in this repo generated from the Typescript code, done by https://typedoc.org/ That cli tool can generate a JSON file as well, and we could make an app that takes such a JSON file and presents it nicely.

Problem

Right now the Filesystem is requires absolute paths to be passed to every function

Solution

Add a stateful Session class encapsulated by FileSystem so that a user can navigate the directory structure and use relative links

Some ideas here:

• #14 (comment)
• #14 (comment)

Summary

Problem

isUsernameAvailable checks if dataRoot.lookup returns null, but dataRoot.lookup also returns null now for the initial appointed CID.

Impact

When creating an account it shows the username as available when the user hasn't created a file system yet. Thus falsy advertising the username as available.

Problem

Every file system change I make adds a few MB to the indexed db.

200MB and I only have 30 quotes and a small list of quote ids on the file system.

Video

Here's a video (from a different session) that shows the storage-usage going up.
http://icidasset-public.s3.amazonaws.com/Kapture%202020-08-04%20at%2021.53.05.mp4

Relevant code

https://github.com/icidasset/quotes/blob/6cd2d1ee5058548eae744bfb0bcc3c8a9c2716ed/src/Main.js#L49

hardLink :: Name -> VNode
softLink :: Name -> Maybe VNode

This means that hard links are guaranteed to have some data at the pointer. Soft links may not, hence the Maybe.

Browser: Firefox 81
OS: MacOS 11

Looks like the websocket connection fails and IPFS fails to start, causing initialise to fail entirely.

Upgrade IPFS CIDs to v1

We need to do this for dag nodes & for files

Problem

Currently you can make a UCAN that expires later than its proof. This shouldn't be possible. Additionally, less important, but still should be validated as well, the nbf should be greater than or equal to the nbf of the proof. To be precise regarding the exp field, it should be less than or equal to the exp of the proof.

Solution

When making a UCAN, get the exp and nbf of the proof (if it is present), and then adjust the default values if needed.

https://github.com/fission-suite/webnative/blob/a4347b31cfec987ff718882e29cc88cff696091e/src/apps.ts#L24

This line, and the other similar ones for the other requests, should be something along the lines of:

proof: ucan_string

(see update-data-root function as an example: https://github.com/fission-suite/webnative/blob/a4347b31cfec987ff718882e29cc88cff696091e/src/data-root.ts)

Getting the proof

This is the "difficult" part of the changes need to be made.
We need to get the actual UCAN to use as the proof.
The easiest way to get this is by doing:

import * as ucanInternal from "./ucan/internal"
const proof = await ucanInternal.lookupFilesystemUcan("*")

We're revising various UCAN stuff soon, so this is just a temporary solution (hence the TODO's in the code).

PS.
In case you were wondering, this workaround works because the auth lobby returns a UCAN with the * resource. That UCAN is stored in indexeddb, which we then lookup by using lookupFilesystemUcan.

Summary

Problem

I am in the process of switching out the old Fission Typescript client with the new webnative sdk in https://github.com/fission-suite/ipfs-photo-gallery/ and have run into a Typescript error:

/ipfs-photo-gallery/node_modules/webnative/index.d.ts
TypeScript error in /Users/patdryburgh/Development/ipfs-photo-gallery/node_modules/webnative/index.d.ts(49,10):
'from' expected.  TS1005

    47 | export * from './filesystem';
    48 | export declare const fs: typeof FileSystem;
  > 49 | export * as dataRoot from './data-root';
       |          ^
    50 | export * as did from './did';
    51 | export * as lobby from './lobby';
    52 | export * as setup from './setup';

Impact

Not sure how to proceed until this error is fixed and not sure exactly what's causing it. I've created a PR in the IPFS Photo Gallery repo so you can see what I've done so far and suggest what I might be doing wrong. Happy to provide any other details you require :)

fission-codes/ipfs-photo-gallery#20

A user of the SDK shouldn't have to know about the "data root".

To do:

• The SDK should automatically update the data root after a file-system change
• The API endpoint should be configurable
• This data-root-updating call to the API should be debounced/throttled
• Should not be called when offline
• Should be called automatically when the user comes back online (after changes were made offline)

Problem

Filesystem supports directories & files with a bare minimum of metadata (currently just mtime)

Solution

Add metadata layer:

• dirs/files are 2-layers where a cid points to a folder containing metadata as well as an index link to the actual file/dir contents
• semver for metadata so that filesystems can interop with filesystems on a different version

Summary

Creating a new user on the Brave browser fails.

Problem

Throws this error Public key MIIB...QAB already exists
Which is when the function call mkdir fails out.

Impact

Brave users can not register to the service

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: macbook pro
• OS: macOS 10.15.6
• Brave Version 1.11.104 Chromium: 84.0.4147.105 (Official Build) (64-bit)

The API can return a 504 status when the IPFS gateway is still fetching the data from your in-browser js-ipfs node. We should retry this until it succeeds.

Additionally we should cancel long-running data-root requests, whenever a new request is made.

And other querying methods.

When setting the data root, we'll want to wait on the API to tell us the content was successfully fetched from our local js-ipfs node. In other words, the data from our file system lives on our local js-ipfs node and we need to wait until that is transferred to the Fission gateway.

More notes:

• The user cannot close the window before this action is finished, show a dialogue when trying to close the window.
• If this were not solved, you can have an infinite loader on your app and possibly lose your file system

https://www.npmjs.com/package/webnative

We have a DNSLink-lookup fallback that uses Cloudflare, but instead of returning the CID (like ipfs.io's api does), it returns an IPNS reference (eg. ipns=icidasset-test-6.fissionuser.net/public)

Problem

Encrypted DAG nodes are not dag-pb/dag-cbor formatted. They may be formatted as such under the hood but you won't be able to do file paths like Qm.../path/to/file.

Solution

We could define our own IPLD format for encrypted folders/files. This would also eliminate the need for a header subdir on every folder (thereby decreasing the # of objects needed to fetch by 2x)

Relevant discussion: ipld/ipld#64

Issue here is that connections can timeout and be closed.
This should be solved by libp2p soon-ish, see "keep alive" libp2p/js-libp2p#744

Seems to be a goal for the next libp2p release:
libp2p/js-libp2p#655

Problem

Adding a file to a folder needs to recurse up through the IPLD tree

Solution

• Add a helper for doing that^
• Handle private files by checking if the directory has a header folder w/ a key in it

Not sure if this is intentional or not

Object API is being (slowly) deprecated.
Need to prefix the path with /ipfs/ though.
See conversation ipfs/js-ipfs#3252 (comment)

Problem

FFS does not support file streaming. This is a limitation from WebCrypto. Large files have to currently be held in memory in their entirety.

Solution

Add file streaming
Contingent on: fission-codes/keystore-idb#16

Right now the structure of FLOOFS in IPFS mimics its structure locally (tree). We should serialize the private filesystem into a list before adding to IPFS

Details here: https://talk.fission.codes/t/floofs-restructuring/718

Problem

FFS should handle CIDv1. Right now this causes conflicts when accessing uploading files from the gateway or through ipfs UnixFS commands (such as ipfs.ls)
See #16 for more details

Solution

Investigate some solutions to get this working. It might be as simple as upgrading the server node to the latest version of IPFS (a good practice regardless)