josephg / sharejs Goto Github PK

Need documentation on the OT system.

This should cover:

• How it works
• How to write your own OT types

If I call client.open() twice from the same process, and in each callback I register a .on('remoteop') handler, the two connections don't receive each other's ops.

My use case for this: two vims locally connecting to the same sharevim server.

ShareJS installed me the version 0.7.3 of socket.io, but when I ran the exampleserver the demos weren't working.
I kept receiving the message socket.io - 'has no method send'.
I installed socket.io 0.6.0 and now everything seems to be good.

PS: It's very plausible that I made something wrong, I'm new to node.js.

We need a decent API for editing JSON objects.

I'm thinking of something like this:

Document paths are more easily written using strings, separated by dots (.). We'll have an escape character for literal dots. All functions which take path strings can take arrays instead.

Unfortunately, if we use \ as the escape character you'll have to write literal dots as \\.. I'm not sure what would be better.

Document API methods:

• doc.get('a.b.c'): Get the object at path ['a','b','c']. Equivalent to doc.snapshot['a']['b']['c']
• doc.set('a.b.c', 55): Set ['a','b'] to 55. This will emit an op which replaces whatever is there now with the new value. Cannot be used on strings or numbers.
• doc.insert('a.b.4', 'hi'): This requires that there's a list, object or string at 'a.b'. If its a:
  • string: Insert 'hi' into the string at character position 4. (Throws exception if the inserted element isn't a string)
  • array: Insert 'hi' as a new element into the list
  • object: Set object[4] = 'hi'. Same as doc.set('a.b.4', 'hi').
• doc.wrap('a.b.c'): Get a fake document object which wraps the element at ['a','b','c']. If the element at the named path is moved, the result should keep working. If this element is a:
  • string: The object has the API methods from text documents (getText(), insert(), del(), on('insert', fn), on('delete', fn)). Operations effect the named path. (This allows you to bind JSON document string elements to a text editor)
  • object or array: The object inherits the JSON API methods.
  • number: The object has add(val), subtract(val) and getValue() methods, as well as an .on('added', function(byValue) {...}) event handler.

We'll need some events as well... I'm not really sure what these should look like.

We could rename doc.get() to doc.getRaw() and call doc.wrap() doc.get() instead. I have a feeling that editing objects via wrappers will be the most common way to edit them.. but I'm not sure if this API is as clear.

The authorization code should be able to specify that a client's view of a document is readonly.

This message should be passed to the client when the document is opened, and appear as a property on document (doc.isReadOnly)

The ace & textarea views should display the text area accordingly (ie, disabled).

The current text OT type has quadratic transform time for transforming lists of operations.

It should be possible to speed this up immensely by using clever algorithms.

Can you add a method to set the text in a specified document? I want to overwrite the entire content of a sharejs document but this is not so straightforward with the current API.

The client's test coverage is unusually low. It should have tests as thorough as the model tests.

These tests are conspicuously missing:

• Disconnect then reconnect works
• Documents are reopened when sharejs reconnects
• A closed document is not automatically reopened when sharejs reconnects
• A closed document can be reopened
• When a document is reopened, it specifies the type, the version, and doesn't request a new snapshot.
• open() still (eventually) works when the connection is closed but reconnecting
• Ops applied while a document is closed get sent to the server when it reconnects
• Remote ops applied while a document is closed get (correctly) applied locally
• Disconnect while an op is in-flight. The op should be resent when reconnection happens, with dupIfSource: set correctly
• connecting, connected, handshake, ok events are all fired from the connection
• Connection status events are fired from the document
• Bare sharejs.open returns the connection object
• The bare sharejs.open connection is closed when all documents have been manually closed

I bet at least one of these things doesn't currently work correctly.

getOps should be exposed in the REST and socket.io interfaces, so the client can request historical operations for timeline views and things like that.

To make sharejs work with closure, we've sacrificed a fair bit of readability. I've moved to uglify now, and all that nasty syntax should go away.

I don't see much benefit in cleaning up the OT code (Text & JSON) but client.coffee is one of sharejs's main entrypoints and its pretty heavily edited. That code should be cleaned up to remove all the @['awful']() method calls.

The method commonPath returns 0 when called with the following arguments.

commonPath(["name"], ["components"])

Which results in the error: "paths match length and have commonality, but aren't equal?"

Looks like this is happening here:

https://github.com/josephg/ShareJS/blob/master/src/types/json-api.coffee#L169-172

I'm not familiar with what is going on here, but it seems like this could be a bug.

Maybe this is the wrong way to think about ShareJS, but is there a way to get a list of currently active sessions (potentially with metadata)?

I'd love to implement something like this:

Shouldn't webclient/* be in the .gitignore - it's programmatically generated, no?

ShareJS is taking ages to connect initially when the server is awhile away.

Reduce the number of roundtrips needed.

If you do the following things:

sharejs.open 'doc1', (doc) ->
  # ... later
  doc.close()
  doc = null

  sharejs.open 'doc2', (doc) ->

... then the web client breaks. This problem shows up in the ace editor example.

Currently, the nodeunit tests aren't run on the closure compiled client code. There's a bunch of ways closure can break the client - and without tests, there's no confidence that it works.

The compiled share.js file should be wrapped in a nodejs module or something and the client tests should be run on it.

Sharejs needs heaps of changes for socket.io 0.7:
https://github.com/LearnBoost/Socket.IO/wiki/Migrating-0.6-to-0.7

Though these changes will have to wait for a nodejs-based client library to be available:
https://groups.google.com/forum/#!topic/socket_io/HQsu8Xp7axQ

The README says:

Note: Until I update sharejs in npm, you'll need this too: cd node_modules/share cake build

Any chance this can happen soon? I'd like to avoid the building step when deploying.

ShareJS should serve minified code only when NODE_ENV=production is set.

Or if there is a way to disable it, it should be documented better :)

I had today hard time debugging it.

The server API should be documented.

Primarily, the following APIs need docs:

• Creating a new server programatically
• Model APIs

Hi,
I installed ShareS this morning, and when I typed the commande bin/sharejs I've gotten this error in the server terminal :(

root@mcmyt-server:~/node/ShareJS# bin/sharejs
Server running at http://127.0.0.1:8000/

_linklist.js:65
item._idleNext = list._idleNext;
^
TypeError: Cannot read property '_idleNext' of undefined
at Object.append (_linklist.js:65:24)
at Object.active (timers.js:136:9)
at Socket._onReadable (net_legacy.js:663:12)
at IOWatcher.onReadable as callback

Best regards,
Moncef

It should be possible to lock down a sharejs server, and restrict read / edit access to documents.

This can be done by adding some access control methods to the options file. These methods could be called by the model to check whether or not a user is allowed to perform the requested operation on the requested document.

Currently, JSON OT uses the "text" type for strings it contains.
I would like to be able to change this (per node in the json) to text-composable or other types that may be added in the future.

ShareJS needs a set of OT functions for editing rich text. They should support annotations (bold, italics, etc) of some sort and there should be a plan on how to embed objects like images in documents.

Summary:
When using the attach_textarea functionality, and inserting text, Opera and IE8 will insert the text at an off-by-one location versus where it should be.

Steps to reproduce:

1. Go to http://sharejs.org with Chrome and Opera
2. Using Chrome, clear the "etherpad in 4 lines" and type 123

Expected results:
3. Opera shows 1, 2 and 3 on consecutive lines

Actual results:
3. Opera shows 1, empty line, empty line, 3, 2

Notes:

• Reloading Opera makes it show properly.
• IE8 has the same issue, IE9 is fine

It should be possible to edit JSON objects using sharejs. A spec for editing JSON is in a block comment in src/types/json.coffee. This should be fleshed out and implemented.

Example authentication code for both server and the client would be really nice.

I have a project management application that I'm working on and I'm finding that the ability to nest arbitrary sharejs documents within a sharejs json document would be extremely helpful. Take these objects for example:

{
  name : 'some project',
  description : 'some description',
  tasks : [
    <<[task] sharejs json document>>,
    <<[task] sharejs json document>>,
    <<[task] sharejs json document>>,
  ] 
}  // project

{
  name : 'task 1',
  description : <<[description] sharejs text document>>
} // task

In this case, I'm imagining a project document with a set of task documents inside. The projects path 'tasks' can be operated on just like a regular sharejs list. Each task is itself a sharejs json document, and each task contains a sharejs text document at path 'description'. This text document behaves just like a regular sharejs text document.

I can get almost all the behavior I want just by flattening these objects, but the text document part is problematic. I'd love to be able to take advantage of all the fancy stuff built-in to the text document type while being connected to a json document.

Disclaimer: I'm just beginning to play with web sockets, so I may be "doing it wrong". If that's the case maybe you can provide some pointers.

Any thoughts?

Might want to mention these in readme:

nodeunit
redis

The redis database layer is pretty slow at the moment because the code insists on storing / retrieving everything from the server on every operation and query.

There are two easy ways this could be made faster:

• Introduce a document cache between the database and the model code
• Make the database layer only commit new document snapshots infrequently.

This change should make the redis database perform about as well as the memory database, which is about a 2x performance boost to the whole system.

The clone function in the JSON OT code isn't ok:

• Its slow
• It requires browser support for JSON
• It shouldn't be necessary, damnit

It should be removed.

To do so, we'll probably need a way to properly roll back operation application. If an op gets half-applied, then an exception is raised during processing, we should instead invert the applied op components and apply them to restore the state.

I just ran the benchmark tests on my laptop and on my ubuntu box (11.10). From my mac laptop <-> my mac laptop, I get 1750 ops per second. If either the client or the server (or both) are running on my ubuntu box, I get exactly 40ms of latency with each request. So the number of ops processed per second drops from 1750 down to 25.

I don't know where that latency is coming from. Its using socket.io 0.8.6. I'm not sure if sharejs.org has the same problem.

ShareJS seems to work fine on my local machine. But when I put it on a public server, I've been getting the following error where it appears that a socket.io connection fails:

d is undefined
((function(){var a,b,c,d,e,f,g,h,i,j,k...on("connect failed")}}()})).call(this)

share.js - Line 1
I don't know what the d variable is, so its quite difficult to attempt to debug.

Other features of my scenario:
I'm using redis as the default data store.
Redis seems to get a connection from sharejs server.
The ace editor attaches fine on the client side, though sharing is not enabled due to the aforementioned error.
I have replicated this error in my own project as well as the examples provided.

I was to give a demo of this project to a group in a couple of days, so hopefully I can resolve this soon!

If you need additional details, let me know.
Thanks,
Dev

The new version of socket.io lets the client / server respond directly to a message. The socket.io interface should be rewritten to support that, since its way nicer.

I'm getting the following warning:

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.

Trace:
at Socket. (events.js:139:15)
at Socket.once (events.js:160:8)
at EventEmitter._setBackChannel (share/node_modules/browserchannel/lib/server.coffee:250:24)
at Object.handle (share/node_modules/browserchannel/lib/server.coffee:516:28)
....

Is there a way to start a share session with code based on the contents of Ace?

Right now I'm doing this:

sharejs.open(docID, 'text', options, function(doc, err) {
  if (doc.created)
    doc.submitOp({i:editor.code(), p:0});

  doc.attach_ace(editor.ace)
})

Uglify is faster, and has better node.js support than closure.

Figure out if we end up with a significantly bigger filesize. If we don't, switch to it.

Building the webclient caused the following error:

jeunet:ShareJS soph$ cake webclient

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Command failed: cat: webclient/concatenation.js: No such file or directory

    at ChildProcess.exithandler (child_process.js:102:15)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:829:12)
    at EventEmitter._tickCallback (node.js:126:26)

Running CoffeeScript version 1.1.1

Since objects cannot have integer keys in JS, it's sufficient to check the type of the last path component to disambiguate object operations from operations on lists and strings.

object[0] === object['0'] always.

To go deeper into pairing down the number of OTs, taking a bracketed list as an argument to list operations allows for changing multiple items and disambiguating the operation from a string operation, further removing the need for different OTs based on type.

At the moment, documents are implicitly created when their type is set. This must be the first operation applied to a document.

This is needlessly confusing. Creating documents should be a first-class thing.

• There should be a special 'create document' message in the socketio wire protocol.
• The model should have a create(docName) method or something like it

When I try the "wiki demo" in IE9 I get this error:
"Object doesn't support property or method 'trimRight'"

It only seems to happen when the following ace option is used:
editor.session.setUseWrapMode(true);

I haven't tried it yet, but probably this issue gets solved by just updating the ace.js :)

For example, if the textbox is empty and both people start typing at the same time, what they type gets interleaved, resulting in mangled text, when the desired behavior is that one complete sentence comes after the other.

This might not be quite a bug, the OT algorithm might be working correctly, but this is still an issue. EtherPad, notably, behaves correctly in this situation, and I don't think it's just because it has higher latency due to lack of WebSockets.

The fix is probably to be smarter about where to put the cursor after applying a remote op. What is currently happening, is that if the cursor is at the end of a line, and a remote op inserts something at the end of the line, the cursor comes after what was inserted. I think that if instead, what the remote op inserts comes after the cursor, then rather than interleaving, text inserted remotely should build up further and further away from text being inserted locally.

Is it possible to have different sessions, served from the same server instance?

E.g: If I run a server at http://foobar.com:8000/ , every browser pointed at http://foobar.com:8000/, regardless of parameters, such as http://foobar.com:8000/?session=1, sees the same edits.

I want to be able to serve different sessions of a collaborative app from the same base url. Is there any such notion included with ShareJS?

Currently there's lots of awful looking code like this:

        @socket['on'] 'connect', @connected
        @socket['on'] 'disconnect', @disconnected
        @socket['on'] 'message', @onMessage

and this

    @['close'] = @close = (callback) ->

and this

        if last['i']? && c['i']? and last['p'] <= c['p'] <= (last['p'] + last['i'].length)

... all in order to avoid names being munged by the closure compiler. There are two ways to fix this:

1. Create a closure externs file to define things closure isn't allowed to rename
2. Use uglify instead of closure to minify sharejs