filerjs / filer Goto Github PK

The same file system may be open from multiple tabs. File system instances should coordinate using same-domain postMessage for things like file watching.

The following things are done manually and should be automated:

• build idbfs.js and idbfs.min.js
• push idbfs.js, idbfs.min.js, tools/ and tests/ to gh-pages

I'm going to need fs.rename() for Brackets too: http://nodejs.org/api/fs.html#fs_fs_rename_oldpath_newpath_callback

Run the unit tests in both browsers and compare.

http://nodejs.org/api/fs.html#fs_fs_ftruncate_fd_len_callback
http://nodejs.org/api/fs.html#fs_fs_truncate_path_len_callback

NAME
getxattr, lgetxattr, fgetxattr - retrieve an extended attribute value

SYNOPSIS

   ssize_t getxattr(const char *path, const char *name,
                    void *value, size_t size);
   ssize_t lgetxattr(const char *path, const char *name,
                    void *value, size_t size);
   ssize_t fgetxattr(int fd, const char *name,
                    void *value, size_t size);

NAME

   setxattr, lsetxattr, fsetxattr - set an extended attribute value

SYNOPSIS

   int setxattr(const char *path, const char *name,
                 const void *value, size_t size, int flags);
   int lsetxattr(const char *path, const char *name,
                 const void *value, size_t size, int flags);
   int fsetxattr(int fd, const char *name,
                 const void *value, size_t size, int flags);

Safari and IOS support WebSQL instead of IndexedDB, so we should implement a WebSQL backend to get better browser coverage.

This relies on #20.

The algorithm in use is pretty dumb (take the next highest number), which can run out for long-lived applications that open lots of files.

First step here is to write a test case that exhibits the failure.

The rewrite for different backends landed, and I think we can take it a bit further. Where we currently have FileSystem, IndexedDBFileSystem, and WebSQLFileSystem, I think we only need/want FileSystem. To get there, I propose an API change like so:

// Create a FileSystem using whatever backed the current browser supports
// (one of IDBFS.StorageProviders.IndexedDB or IDBFS.StorageProviders.WebSQL)
var fs = new IDBFS.FileSystem({ name: "local" });

// Create an IndexedDB FileSystem explicitly
var idbfs = new IDBFS.FileSystem({
  name: "local2",
  db: IDBFS.StorageProviders.IndexedDB
});

// Create a WebSQL FileSystem explicitly
var websqlfs = new IDBFS.FileSystem({
  name: "local3",
  db: IDBFS.StorageProviders.WebSQL
});

// Create a "RAM Disk" FileSystem explicitly
var ramfs = new IDBFS.FileSystem({
  name: "local4",
  db: IDBFS.StorageProviders.Memory
});

In the case that no db option is provided to the FileSystem constructor, it will figure out what it supports (e.g., IndexedDB or WebSQL). However, if an explicit constructor function is provided for a Storage Provider (one we provide, or a custom one we don't--someone could add localStorage if they wanted without changing our code) the FileSystem constructor will attempt to use that instead. This will also make it possible to test different providers easily in tests, provide mocks if we want, etc.

The FileSystem creates and holds a reference to a db object using the appropriate Storage Provider constructor. All storage operations are delegated to this object, essentially what is happening now with IndexedDBContext. This also means that we can flatten the code paths for IndexedDBFileFystem, WebSQLFileSytem, and FileSystem into FileSystem. The advantage of this is smaller size, but also a much simpler model for extending with new types of storage providers, since you only need to send a new db into the FileSystem constructor vs. having to also implement a FooFilesystem, deal with scope issues on internal function calls in fs.js, etc.

The db object will provide ways to open the database, deal with transactions (real or simulated, depending on backend), do operations like get/put/delete, obtain a context to pass around to fs operations.etc. It won't be a large object, but will encapsulate all the backend specific bits of working with data. The FileSystem will do everything in terms of db operations when needing to work with a backend. There won't be any need for more specific types of file systems.

Our tests currently don't run in all environments because not every provider is supported. A better solution is for our tests to run once for every supported provider in the current environment.

It would also be nice to allow for explicitly setting the desired provider to run in the tests via the URL:

• ?provider=indexeddb
• ?provider=websql
• ?provider=memory

etc.

• Code files (IDBFS window object)
• README + docs
• Tests that refer to IDBFS

We can make use of the node.js test suite to provide good coverage and also ensure that this implementation has the expected behavior.

fs.should have a root directory sometimes fails

We have 3.0.2 in the tree, and 3.1.2 is out:

http://code.google.com/p/crypto-js/downloads/detail?name=CryptoJS%20v3.1.2.zip&can=2&q=

We don't want to have users accidentally delete their filesystems by clearing their browser data. We should make sure we understand what happens when each of the browsers we support, and each of the providers, goes through a "Clear Browser Data" operation. At the very least we should clearly document what will happen so people can make informed decisions.

We need an HTML5/in-browser way to visually interact with the file system, and specifically, to specify paths for Open and Save operations. Downstream consumers like Brackets will need this, but so will general users of the library:

If we consider how Brackets wants this done, we see that a file system needs some way to accomplish the following:

function showOpenDialog(allowMultipleSelection, chooseDirectories, title, initialPath, fileTypes, callback) {
  ...
  // Hand-back an array of strings: path or paths to open (e.g., a file, directory.
  callback(null, ['/a/path', '/a/second/path']);
}

function showSaveDialog(title, initialPath, proposedNewFilename, callback) {
  ...
  // Hand-back a single string: filename to use  
  callback(null, '/a/path/to/save');
}

It's possible that we only need 1 dialog, and we can re-use it for both the Open and Save cases by enabling/disabling bits of it. I only call out the two separate uses, since they both need to be supported, and each has some aspects that are unique.

The dialog doesn't need to do any work with the file system as such; it just let's the user specify a path or paths to use in some future operation, and calls the callback with the path(s). It can also call into the filesystem object to do various things it needs, for example: readdir to get a list of files, mkdir to create a "New Folder...", etc.

For our purposes, we probably need to design something generic looking that will work on all platforms in the browser. Consider a few examples of these dialogs in different platforms:

• http://www.pdxtc.com/wpblog/wp-content/uploads/save-as-new2.jpg
• http://www.macobserver.com/imgs/tmo_articles/20130329saveas4.jpg

Here's a rough list of things I can imagine wanting to to be able to do:

• The showOpenDialog and showSaveDialog dynamically create a centred div in the page, which will host the rest of the dialog's UI. We'll bundle the UI into the library so you get it for free when you use a file system. Because we're using RequireJS, we can use the text plugin to load and create a dom fragment from bundled HTML/CSS resources.
• A dialog's view of the world is confined to the file system on which it was invoked. That is, if I have a file system named _fs, and I do _fs.showOpenDialog(...), the things I'll see will all be in _fs. I'm not sure if we want to have the concept of multiple volumes, which would be other file systems (e.g., fs1, fs2). I think that's out of scope for this bug.
• A save or open dialog has a list view of some kind, showing the files and directories in initialPath or the root directory (i.e., "/"). There is a way to descend into child directories (e.g., double-click and the list view re-loads with that child-directory as the current path).
• It would be nice if you could create a new folder by clicking a "New Folder" button or something, so you can manage your files a bit within the dialog. Later we might want other features like being able to delete, rename, etc.

Currently, none of the operations are updating atime or mtime (if the file is changed).

As discussed in #55, it would be nice if you could use Filer in place of node.js' fs module. This issue is about fixing any issues we have using Filer in node.js, updating docs, etc.

The minified lib is currently 600K which is way too big. Some things we could try to reduce it:

• remove lodash.js -- it's hardly used, and we can cherry-pick the bits we do use/want and dump the rest
• remove TripleDES, Rabbit encryption types, and just use AES
• remove the TextEncoder/TextDecoder code, and try using the CryptoJS UTF8 encoder, or some other smaller one, preferring native TextEncoder/TextDecoder if they exist in the browser
• See if we can tune uglify/require to do better compression (e.g., dead code removal)

The read() method on node's fs can be called one of two ways, and we don't currently support the old way (from https://github.com/joyent/node/blob/master/lib/fs.js#L413-L438):

fs.read = function(fd, buffer, offset, length, position, callback) {
  if (!util.isBuffer(buffer)) {
    // legacy string interface (fd, length, position, encoding, callback)
    var cb = arguments[4],
        encoding = arguments[3];

    assertEncoding(encoding);

    position = arguments[2];
    length = arguments[1];
    buffer = new Buffer(length);
    offset = 0;

    callback = function(err, bytesRead) {
      if (!cb) return;

      var str = (bytesRead > 0) ? buffer.toString(encoding, 0, bytesRead) : '';

      (cb)(err, str, bytesRead);
    };
  }

  function wrapper(err, bytesRead) {
    // Retain a reference to buffer so that it can't be GC'ed too soon.
    callback && callback(err, bytesRead || 0, buffer);
  }

  binding.read(fd, buffer, offset, length, position, wrapper);
};

We don't (yet) support a stream-based file system, however, it might be nice to reserve file descriptors 0, 1, and 2 in case we do want to do something later with them. At the very least, our FD code could start handing out numbers at 3 and going up from there.

Currently, you must create a FileSystem object for each file system you want to use. It's more unixy to have a virtual file system and mount real file systems of different types inside it.

This also allows for the in-memory file system to cache data from mounted real file systems to improve performance.

http://nodejs.org/api/fs.html#fs_fs_symlink_srcpath_dstpath_type_callback
http://nodejs.org/api/fs.html#fs_fs_readlink_path_callback

1. fs.rename is missing from the API doc - should add something like...

   fs.rename(oldPath, newPath, callback)

   Asynchronous rename(2). Callback gets no additional arguments.

2. Correct "agruments" typo that is repeated -- should be arguments

3. For fs.link oldPath/newPath is confusing as param names -- use srcPath/dstPath instead

I'm working on porting Brackets on to idbfs, and it looks like I'm going to require the following things, at least to start. Some of them are already done, some might need to be added (for sure readdir):

• stat/exists
• readdir
• mkdir
• rename/mv
• readFile
• writeFile
• unlink/moveToTrash

As I go deeper I may find others, but this is the first cut at what I need for sure.

IndexedDB is great, but it's not well supported cross-browser:

http://caniuse.com/#search=indexeddb

With the WebSQL polyfill (see http://nparashuram.com/IndexedDBShim/) you would probably add some more webkit-based browsers (good to test that with you stuff, and make sure it works).

To support even more browsers, it would be great to use HTML5 Storage as an alternative backend. This has obvious down sides, in particular the amount of storage available (see http://dev-test.nemikor.com/web-storage/support-test/). In general we're talking 2.5-5 megs depending on browser; not a ton of space, but enough to do certain types of things (consider this the "Floppy Disk" to your "Hard Drive"). However, the support matrix is really interesting here:

http://caniuse.com/#search=storage

Tests occasionally fail but then pass on subsequent runs.

We want to support WebSQL in order to provide support for older browsers that don't (or won't) support IndexedDB. However, if a user is on a browser that doesn't support IndexedDB one day, but does the next (e.g., they get an update that adds it), we need to provide some way to migrate an existing WebSQL-backended File System to IndexedDB. The opposite direction likely isn't necessary, since WebSQL is a dead spec, and replaced by IndexedDB.

They're not adding much to the code and when.js is quite large. Pass callbacks instead.

http://nodejs.org/api/fs.html#fs_fs_utimes_path_atime_mtime_callback

The node.js implementation of write() supports two arg layouts, and we only support one of them (see https://github.com/joyent/node/blob/master/lib/fs.js#L468-L504)

// usage:
// fs.write(fd, buffer, offset, length[, position], callback);
// OR
// fs.write(fd, string[, position[, encoding]], callback);
fs.write = function(fd, buffer, offset, length, position, callback) {
  if (util.isBuffer(buffer)) {
    // if no position is passed then assume null
    if (util.isFunction(position)) {
      callback = position;
      position = null;
    }
    callback = maybeCallback(callback);
    var wrapper = function(err, written) {
      // Retain a reference to buffer so that it can't be GC'ed too soon.
      callback(err, written || 0, buffer);
    };
    return binding.writeBuffer(fd, buffer, offset, length, position, wrapper);
  }

  if (util.isString(buffer))
    buffer += '';
  if (!util.isFunction(position)) {
    if (util.isFunction(offset)) {
      position = offset;
      offset = null;
    } else {
      position = length;
    }
    length = 'utf8';
  }
  callback = maybeCallback(position);
  position = function(err, written) {
    // retain reference to string in case it's external
    callback(err, written || 0, buffer);
  };
  return binding.writeString(fd, buffer, offset, length, position);
};

I recognize that this project is trying to stick close to the node APIs however ES6 is introducing Promises as a natively implemented feature so it might be worth considering.

See - https://github.com/domenic/promises-unwrapping

In issue #36 we refactored the way the constructor works, and introduced the idea of pluggable storage providers. This needs to be reflected in the README.

I am a bit confused by the similarity of this project, filer.js and idb.filesystem.js by Eric Bidelman of Google. How does filer differ from using both latter projects? They provide similar methods and support both FileSystem API and IndexedDB, so why should I use your project?

https://bugzilla.mozilla.org/show_bug.cgi?id=912202

Since we're adding WebSQL support along with possibly other backends, it makes sense to rename IDBFS since it's no longer just IndexedDB. @humphd has suggested WebFS.

Would be nice to have a few larger tests that create and manipulate a file system and compare the final state to some known good state. Could probably use the memory provider to do this.

Similar to the encryption providers, we need a provider wrapper that inflates/deflates data going into a provider's put()/get() methods.

Doing this is pretty trivial, but requires that we first pick a zlib JavaScript implementation. There are a bunch of them kicking around.

I'm happy to do this if I can get advice on which lib to use.

Thoughts?

This involves using extended attributes to store crypto keys for each directory node.

Chrome and Firefox are currently supported, and we're working to support Safari. Since I don't have access to Safari, being able to run automated tests somewhere would be great.

https://github.com/rvagg/node-levelup

API methods that accept buffer parameters need to be passed an ArrayBufferView. The documentation should state this explicitly because those methods will not accept ArrayBuffer parameters.

Access/modified times are not currently updated. Might be good to add a mount option to control these, since there's are performance implications.

This means creating wrapped Transaction and ObjectStore prototypes, and rewriting read_object, write_object, and delete_object to make storage calls that are not backend-specific.

In cases where we aren't changing data or metadata, we can use IDB_RO instead of IDB_RW to allow better concurrency.

ArrayBuffer objects have the disadantage that they always have to live in memory. So if your file entries contain ArrayBuffer objects which describe the file contents, that means that the whole file contents is read into memory whenever the IDB entry is read.

However if you use Blob objects to hold the data, that means that that only the metadata about the Blob needs to be read at the time when the IDB entry is read. The actual Blob contents doesn't need to be read into memory until the Blob is read using FileReader.

Another advantage that Blob gives is that it enables the IDB implementation to store the Blob data outside of the database. This improves performance of the database since it can be kept more compact. Firefox currently does this, though I think IE does not. Not sure what chrome does since they only recently added Blob support (not sure if it's even landed yet).

The lack of Blob support in Chrome might be the main sticking issue here though...

Currently each file is stored in a single object. In some cases this can cause lots of unnecessary reads/writes. This can be fixed by assigning a per-file block size and converting the data pointer to a list of pointers to fixed-size blocks.

(I marked this as good first bug, but this work is somewhat more involved than other bugs with that label.)

Should be possible to export and save (externally) a file system image, or import one (externally, or?)

Needs to be wrapped up and published.