orbitjs / orbit Goto Github PK

Consider logging on startup.

I'm using orbit.js with ember-orbit and setup a store using JSONAPISource my application route has a model hook that finds posts return this.store.find('post'). This results in an api call /posts and the server returns a doc like { posts: [ { id: "1", title: "x", body: "y" }, /*...*/ ], meta: {/*...*/} } See: http://pixelhandler.com/api/posts

The test for finding an array of records uses only an array as the mock payload. Would this example, https://github.com/orbitjs/orbit.js/blob/master/test/tests/orbit_common/unit/jsonapi_source_test.js#L312 need to use a named root in the payload? (like the example at http://jsonapi.org) I noticed the tests use the content of the named root as described on jsonapi.org e.g. https://github.com/orbitjs/orbit.js/blob/master/test/tests/orbit_common/unit/jsonapi_source_test.js#L205 is the object only.

In the success function, after calling findQuery, https://github.com/orbitjs/orbit.js/blob/master/lib/orbit_common/jsonapi_source.js#L232 the raw is the entire payload not payload.posts so raw.forEach errors as forEach is not defined on an object.

I have a blog repo with a branch for integrating ember-orbit with the JSONAPISource from orbit.js

If you want to try it out this should work...

git clone [email protected]:pixelhandler/blog.git
cd blog && git fetch origin && git checkout try-ember-orbit-with-ember-cli
cd client 

edit client/bower.json with full path to dist (builds) of ember-orbit and orbit.js e.g. see https://github.com/pixelhandler/blog/blob/try-ember-orbit-with-ember-cli/client/bower.json#L18-L19

make install
ember server --proxy http://pixelhandler.com/api

Error after loading localhost:4200 is Error while processing route: index undefined is not a function (evaluating 'raw.forEach')

From what I can tell ember-orbit/orbitjs doesn't handle links that contain multiple words in their name e.g. task-list. I think there's a couple of options for handling it:

1. ember-orbit's store.js camelizes any links that are included in paths before sending the operation
2. include dasherized links in the operation's path and do the camelization in the Sources.

@dgeb do you have a preference or any other ideas?

When I use JSONAPI Source, I get a lot of Cache._transformRef() exception, I believe it is because my JSONAPI server which I build on top of Sails.js does not return linked models. Anyway it doesn't seem to prevent the proper working of Orbit, but I am getting a lot of warning filling up my console.
Is there a way to turn off these warnings?

Here is a full view of one of these warnings :

Thanks in advance.

With following configuration (connecting memory to localStorage, and then localStorage to JSON API):

    new Orbit.TransformConnector(memorySource, localStorageSource, {
      blocking: false
    });
    new Orbit.TransformConnector(localStorageSource, memorySource, {
      blocking: false
    });

    // Connect localStorage <-> JSONAPISource
    new Orbit.TransformConnector(localStorageSource, jsonApiSource, {
      blocking: false
    });
    new Orbit.TransformConnector(jsonApiSource, localStorageSource, {
      blocking: false
    });

I get the following log:

[ORBIT] [memory] didTransform Object {op: "remove", path: Array[2]}
[ORBIT] [localStorage] didTransform Object {op: "remove", path: Array[2]}

Note that the change doesn't propagate to the JSON API.

With following configuration (connecting memory to localStorage, and also to JSON API):

    // Connect MemorySource <-> LocalStorageSource
    new Orbit.TransformConnector(memorySource, localStorageSource, {
      blocking: false
    });
    new Orbit.TransformConnector(localStorageSource, memorySource, {
      blocking: false
    });

    // Connect MemorySource <-> JSONAPISource
    new Orbit.TransformConnector(memorySource, jsonApiSource, {
      blocking: false
    });
    new Orbit.TransformConnector(jsonApiSource, memorySource, {
      blocking: false
    });

I get what I expect:

[ORBIT] [memory] didTransform Object {op: "remove", path: Array[2]}
[ORBIT] [localStorage] didTransform Object {op: "remove", path: Array[2]}
XHR finished loading: DELETE "http://localhost/api/ember/models/24d79ed6-935b-3ec4-4d5a-569a9beb5930". vendor.js:8718
[ORBIT] [JSON API] didTransform Object {op: "remove", path: Array[2]}

Is this expected behavior, or should the first case change the changes all together?

If you are using a LocalStorage source for active development, and you add a new key to the EO.Model the records that are loaded via reset() on page load from local storage end up not being able to change that value (since they don't have that key???)

I've added this in my EO.Store in my initalizer to work around it for now:

  init: function() {
    this._super.apply(this, arguments);
    var source = this.orbitSource;
    var data = source.retrieve([]);
    // loop over all stored data on initalization
    Object.keys(data).forEach(function(type) {
      // find the current model description for this type
      this.schema.modelFor(type);

      // for each entry in the database, upgrade the defaults
      Object.keys(data[type]).forEach(function(key) {
        this.schema._schema.initDefaults(type, data[type][key]);
      }, this);
    }, this);
  }

This should modernize the module output and improve compatibility for orbit and ember-orbit with Ember-CLI.

I'm working on this now locally.

How about including both the 'test' and 'replace' operations in payload for JSONAPISource Patch requests (updates)?

By sending the 'test' operation with the model's original value and sending the 'replace' operation with the model's changed value together... the server can verify the operation if needed. The use case would be to prevent a PATCH (update) from a client that has a stale (cached) model property. Since the HTTP PATCH is intended to be atomic this would allow enforcement of changes to a system without any push support. So when a client is using connected sources for memory and JSONAPI the server may choose to throw an error if the client sent a test that is not valid (most likely case would be a more recent change made by the server or another client of the same API).

Example payload for JSONAPISource sending PATCH request:

[
   { "op": "replace", "path": "/a/b/c", "value": 42 },
   { "op": "test", "path": "/a/b/c", "value": "C" }
]

Relevant links in the JSON Patch spec:

• test: http://tools.ietf.org/html/rfc6902#section-4.6
• Error Handling: http://tools.ietf.org/html/rfc6902#section-5

Perhaps this should be an issue for ember-orbit as well (when model changes make operations for both replace and test, here: https://github.com/orbitjs/ember-orbit/blob/master/lib/ember-orbit/attr.js#L19 ).

See this branch for suggested test operation support: https://github.com/pixelhandler/orbit.js/compare/test-operation

The latest rev of JSON API no longer requires PATCH for any operations, although it allows a fully redundant set of operations with PATCH.

The JSONAPISource needs to support POST/PUT/DELETE for all operations, including operations on relationships. PATCH support should probably be an option for JSONAPISource rather than removing it entirely. It's quite useful for supporting bulk operations and could be favored by some APIs.

The following specific actions are needed to make JSONAPISource compliant with JSON API:

• Use PUT for updating attributes
• Use POST/PUT/DELETE for updating relationships
• Define buildRelationshipURL to allow for different endpoints for relationships

I can't see why sometimes (hate to describe with that term) promises are "waiting for parent" / "rejected" when everything seems clean.

I'm using Orbit, connecting memorySource to localStorage and JSONApi asynchronously. These promises are rejected even for memory, so when adding or deleting any one object the following happens:

The save action is basically:

var hash = this.getProperties('name', 'other_attrs');
this.store.add('model', hash);

What do those promises mean? Even if any one source fails memory should go through, because it's async, right?

@dgeb the relationships in ember-orbit appear to be set up to always load synchronously from the source. How would you suggest allowing models to load relationships asychronously? An option that I'm considering is to override source.retrieve to return immediately but trigger the fetch for the relationship in the background which then notifies ember-orbit's store once it's loaded. Does this seem like a reasonable approach?

According to the spec updates to hasMany links should have an array as their payload. Currently orbit.js is sending a single value, see the test at https://github.com/orbitjs/orbit.js/blob/master/test/tests/orbit_common/unit/jsonapi-source-test.js#L260.

I think it the update should be a change at https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/jsonapi-source.js#L299

from

json[relResourceType] = relResourceId;

to

json[relResourceType] = linkDef.type === 'hasMany' ? [relResourceId] : relResourceId;

@dgeb if that seems correct I'll go ahead and put the PR together.

Hey guys, I've been going through the README trying to play around with Orbit, and I noticed at least one case where documentation does not line up with what is actually available. For example:

OC.LocalStorageSource;

Is referenced in the documentation, but the class that Orbit makes available appears to be called:

OC.LocalStorage;

I would be happy to submit a cleanup pull after playing around, if you are able to provide some direction in terms of which form is preferred.

A call to find('post', {secondaryKey: '123'}) should return an object instead of an array, similar to a call to find('post', '456').

Right now, finding my way through the API with Github and the browser's debugger. Also googling how to generate an API for a JS package.

In my current development scenario i have two rails projects: An API project and one project where the main web app is running in. In a productive environment the API project would run on the same host, but in the development scenario i am running two independent rails servers. I know that usually the same-origin-policy would forbid ajax requests to the API server, but for my development purposes I have just deactivated the security features in chrome. However, orbit does not currently allow to change the host in the jsonapi source settings (or i haven't figured out how). As I said, usually that would make sense, but for developing purposes it would be nice to have an option to adjust the host (currently I have just hardwired the host inside the library).

There is also a host variable inside the code, but as it seems you can't pass a value with the options hash (only namespace and headers).

The change would be a one-liner, but I wanted to double check with you first.

Will the following use case be supported?

We have 3 data sources,

1. Memory (Javascript objects)
2. Local (idb/websql/localForage)
3. Server (jdbc?)

Two connectors in main thread to have duplex connection between memory and local.

1. Memory to Local Connector
2. Local to Memory Connector

Two connectors in a web worker dedicated to synchronizing via REST

1. Local to Rest Connector
2. Rest to Local Connector

Web worker and main thread will only share the "Local" data source. Will have to probably keep a store for tracking updates instead of in memory variables, not sure how orbit is managing sync.

The plan is to persist to Local as app runs, and let the web worker synchronize to and from server. I wonder if orbit will be of help in this scenario?

PS: To be frank I haven't looked much deep into orbit as I am already too much tired doing plenty of tests on different wrappers, poly fills and adapters.

It should be possible to squash a set of operations such that related operations are merged. This functionality will be useful to condense any set of operations to a minimal set, which in turn will minimize network usage.

I'd like to start this work with a generic function that takes an array of operations and returns a "squashed" array of operations.

Records on the server side which are pulled with find and yet do not exist in local databases do not fire the didTransform event. Therefore, they do not alter e.g. localStorage, even with an activated TransformConnector.

Not sure if this a design decision, but I think freshly pulled records from the server should be inserted into the local databases (with activated connectors).

The target source's transform queue should be clear before this.target.retrieve is called from TransformConnector#transform.

Currently, TransformConnector pretty much breaks down when a primary key contains a /

I've been implementing a Source for Firebase and one of the things that surprised me was that each source maintains it's own cache. As far as I can tell this is to facilitate synchronous access through the retrieve method. Rather than maintaining caches for each individual Source perhaps it would be simpler/more efficient to keep one inside the ember-orbit Store instead? This way the Source implementations don't need to consider the cache at all (less to understand when implementing a Source) and the information isn't duplicated in memory for every Store.

This will allow for sources to ignore didTransform events raised by the other while transforms are being applied. This could eliminate the need for many sources to maintain caches.

Possible approaches:

• make two uni-directional connectors aware of each other
• create a separate bi-directional connector that encapsulates two uni-directional connectors
• create a completely separate bi-directional connector

I need to store the order that objects appear in a hasMany. I optimistically tried to just work with the natural order but it looks like this isn't possible (I'm using localstorage and the storage format appears to use a map rather than an array). It looks like my best option is to maintain the order in a separate field (have a position field on each object is another option but seems like it would be a lot more chatty in terms of transforms). @dgeb have you given any thought to this yet?

Hi

For a project I need to apply diff on long texts (news articles). I would like to know if it is possible to apply the Myer's diff algorithm for this type of data. I am still a bit confused where I could add this feature... add method or override existing one in Document? In TransformConnector?

Some javascript implementation of the algo: jsdiff, diff_match_patch, jsondiffpatch

Thank you.

I note in the documentation that "arguments for actions can be customized for your application" - but don't see how.

The example uses find with type and id arguments. How would I support a find action with different arguments, for example, find(type, account_id, user_id, name)?

It seems each source is responsible for updating the inverses, they're then propagated via didTransform to other connected sources(except in the case of JsonApiSource which appears to rely on the server to determine the inverses). Is this correct and if so does this mean that when an update propagates through the sources you get multiple inverse operations propagating in response?

See #70

I have not been able to build this library for use in a browser. The default settings result in an AMD with some dependency path issues ("orbit/main" vs "./orbit/main").

I have tried modifying the transpiler task to global but it doesn't output valid JS.

Could you add a dist folder to your github repo so that users can just use that for the source instead of having to go through the build process.

Thanks!

Perhaps the https://github.com/stefanpenner/ember-inflector project can be used to provide better support for inflection (pluralize/singularize) used by the Schema prototype (class).

I've noticed that schema only allows one to specify the id handling globally for all models.

I have models that use UUID's however there are others that have natural keys and I can't say that globally for all instances of all models that the id attribute can be handled uniformly.

If I have API's from a number of sources, there isn't any particular global rule that can be used, for some models a UUID may be fine and in other cases the api might use type/small-integer-id or slug, in some models the remote id attribute name may vary.

I can see that the IdMap segregates model types, to keep their key spaces distinct, and handles the attribute type names too, however the schema model definition limits what can be expressed.

I believe the schema definition needs to be enhanced to:

• allow configuring the remote id attribute(s) individually per model
• allow configuration of a model specific id generator (since different API's might use different schemes for client assigned ids)
• also indicate if the generated id can be used both locally and remotely.

In relation to my other issue #40 regarding embedded models/links, I am exploring if schema would be better described treating everything (ids, attrs, links) as just model property definitions.

Example:

 planet: {
       properties: {
         id: {type: 'id', generator: V4UUIDGenerator },
         name: {type: 'string'},
         classification: {type: 'string'},
         moons: {type: 'hasMany', model: 'moon', inverse: 'planet'},
         features: {type: 'hasMany', model: 'feature', embedded: true}
       }
     },
     moon: {
       properties: {
         id: {type: 'id', local: '_id', generator: OrbitDefaultGenerator },
         name: {type: 'string'},
         planet: {type: 'hasOne', model: 'planet', inverse: 'moons'},
         features: {type: 'hasMany', model: 'feature', embedded: true}
       }
     }
     feature: {
       properties: {
         // note absence of id field
         kind: {type: 'string'},
         description: {type: 'string'}
       }
     }

I think its important to keep things under a 'properties' key in-case there are future model/schema concerns that are not property specific.

Let me know your thoughts on your preferred way forward. I am trying things out with Orbit so I am willing to work up a PR with my changes if you're open to taking this approach or an alternative that provides similar capability. I am also aware that such a schema definition change also requires changes to Ember-Orbit.

instead of local storage you should used indexedDB (and WebSQL for older browsers) better size limits and faster

I'm looking for an event similar to didFind but which would fire every time source received an updated object data. In my case, this happens through linked property of jsonapi response. I've tried to use didTransform event, but it does not suit me well since it contains raw data, but otherwise works with some filtering applied, which is quite dirty.

I'm not sure this is good idea at all but if you think that it's worth considering, I'd love to contribute (with a little guidance).

Thanks for a nice project though! ✨

After playing around with the synchronization between REST-API and localStorage, I realized that data entries in localStorage doubled after calling find on the REST-source (due to a reload of the page). It seems like the find call on the REST source just triggers a copy (instead of a sync) to the synced storage. Therefore, I tried to clean localStorage every time shortly before the new find was called. Unfortunately, the REST source suddenly returned more entries than in reality on the REST source available. I also checked the cache size, but the cache was empty. Any idea where these 'phantom copies' are coming from?

Very much enjoying this library, nice work!

I've noticed with the LocalStorageSource that data can be initially loaded from LocalStorage when constructing the object, or manually at a later point. However, this doesn't seem to send any transforms across to other connected sources (I'm not sure if I missed something), is there a way to sync sources after a localStorageSource.load()? Particularly want to make sure I don't do something hacky.

This weekend I want to start looking at replicating between orbitjs and a rails backend. I'm probably going to start simple and just send some json patches over websockets. This will only solve part of the replication problem though as using json patches alone isn't enough to know if the server or client side have diverged. This is more likely to happen on the server side e.g. a user's privileges might have increased so they need to do a full sync on any resources that have been added.

Having spent some time with pouchdb I've had a look at the replication protocol used by couchdb and while it solves the replication problem it's not particularly well suited to a rails backend (it uses global revisions which don't map well to a sql store).

At this stage I'm just going to be getting more familiar with websockets and how using json patches alone would work. In the longer term though I want to find a robust way to handle replication. Does anyone have any experience in this area or have any ideas about what a good approach might be?

Hey,

I am currently testing orbit.js, but ran into some trouble: After including the necessary js files and launching the html site, I get the following error:
Uncaught Error: Assertion failed: MemorySource requires Orbit.Promise to be defined. I guess its related to your note in the readme regarding the dependency of promises. However, I am not really sure how to include these dependencies properly... Could give me a hint (and maybe add that part to the readme)?

Edit: At least it seems to work by adding
Orbit.Promise = RSVP.Promise; at the top of my js-file, is this the way to go?

The local storage source should respond to the storage event in order to know when its contents have changed. It should then perform a diff and trigger didTransform for each change.

Since diffs might be extensive, it would also make sense to provide the option to store data by type in different storage areas instead of all together.

When trying to load existing data from LocalStorageSource, I'm getting an undefined is not a function in this line: https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/schema.js#L418

The problem is that Schema.prototype.registerAllKeys tries to call forEach on an a plain object. Apparently there is a discrepancy between what the function expects as data and what LocalStorageSource actually provides. Specifically, the function seems to expect an array for each model while LocalStorageSource provides an object mapping ids to model objects.

Currently, remote ids used in links are not resolved into local ids as records are loaded into the cache. Therefore, relationships aren't properly resolved for remote data.

Firstly, what a promising library that looks like it will solve the client synchronisation problem with a good design approach.

@dgeb I have covered the embedded links issue on the json-api spec and I have been reviewing orbit for use in one of my apps. I have to say it looks very good, except for a couple of things I'll cover below.

There are two areas that with some minor changes could potentially allow model structures that have embedded attributes with links, the first being orbit-common/schema.js and the second being orbit-common/source.js.

If the schema.js instead just relied on the type being 'has-many' or 'has-one' to build up __rels[] there would be no need to segregate attributes and links.

The second change is to add support for embedded has-one and has-many models which don't (necessarily) have ids (or at least dont have server ids).

With schema avoiding segregation of links and allowing nested/embedded models (ie any value type that requires a breakdown into sub properties) , Orbit would move a step closer to supporting data sources and resources that have embedded structure.

The second area is orbit-common/source.js, which defines findLink(), addLink() and removeLink(). It is not clear that these are needed in the public api if the existing methods simply check if the property name exists in__rels[] and do an addLink/removeLink vs normal property handling. It would be nice to be agnostic to whether a property is an attribute or a link.

I would be keen to get your feedback on the above, as I'm seriously wanting to use Orbit if I can a path forward to supporting resources with structure/embedded models/links.

I'm also prepared to work up a PR to get there if you're interested and with your insight into the code if the proposal is viable and there aren't any obvious blockers.

Hi,

I'm wondering what the best strategy is to handle offline mode with Orbit.

Our use case is:

1. Users enter all form data
2. Syncs to local storage
3. Background syncs to server anytime there is network access

Questions:

1. How would I go about something like this with Orbit and Ember.
2. Are there hooks to watch for failed attempts, how can we periodically retry to sync.
3. What's the best strategy to test for network connectivity before attempting to save, and how could I make the transform connector only work when connection is available?

Thank you!

Some has-many relationships, such as tags on a post, should be updated only as a set. In other words, it shouldn't be possible to add / remove individual members, but only replace the entire array of members.

This should be indicated with a flag on the relationship (actsAsSet: true) and individual sources should respect this flag.

Right now, the JSONAPISerializer looks at each field in the "links" field and tries to assign the link to the record based on the schema https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/jsonapi-serializer.js#L236

The problem is that the server may be returning additional links that we don't have implemented in our orbit schema on the client (the client may only be implementing part of the server schema and ignoring fields that are not relevant to the particular client application). So it is possible that linkSchema = schema.models[model].links[link]; evaluates to null, which will cause the next line to throw an error.

I propose either adding the line if (!linkSchema) return; after https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/jsonapi-serializer.js#L236,

or changing https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/jsonapi-serializer.js#L238 to be if (linkSchema && linkSchema.type === 'hasMany' && isArray(linkValue)) {
and https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/jsonapi-serializer.js#L247 to be } else if (linkSchema && linkSchema.type === 'hasOne' && (typeof linkValue === 'string' || typeof linkValue === 'number')) {
depending on whether we want https://github.com/orbitjs/orbit.js/blob/master/lib/orbit-common/jsonapi-serializer.js#L252 to be hit in the event that there is no schema for the linked resource.

In at least one instance, the library passes null as the onFulfilled handler for a Promise. If Orbit.Promise is set to Chrome's native implementation of ES Promises, this results in the following exception being thrown:

Uncaught TypeError: onFulfilled must be a function or undefined 

It's possible that this is just an implementation detail that will go away as ES6 Promises are implemented more widely (the A+ Promise specs do not complain if onFulfilled is null, for instance), but for now it can be fixed by swapping the null with undefined.

How can we create a new record in Memory, bind it to a form, and then when all validations pass synchronize it with other sources? Something to do with blocking a transformation until a validation passes?

I understand that the JSONAPISource was designed to work with a JSON API compliant data source. However, it would be nice if it was easily customizable to support variations of the spec. Specifically:

• Allow providing specific resources paths for certain models in the schema
• Provide a simple hook for making adjustments to serialization/deserialization. The current deserialize method is not very suited to be overwritten since one would need to copy/paste a lot of code. For example, what if instead of {users: [..], ...} I want to parse something like {data: [...], ...}? This should be possible through a simple hook like parseData: response -> response.data.

Perhaps it would make sense to create an intermediate model like AjaxSource or something that covers some of the common ground of remote sources in general but provides more flexibility regarding encoding, serialization etc.

jsonapi-source.js#L695
/links/ fragment is optional in JSON API spec (http://jsonapi.org/format/#urls-relationships).
For the majority of REST frameworks the usual behaviour is resource/:id/subresource, e.g. /photos/1/comments. It may be hard to maintain a different behaviour.

The REST clients shouldn't rely on links. But it's hard to achieve. Usually you have to do redundant requests just to know links. But the great thing is that Orbit.js can store resource copies locally. So it can reuse links. There is no reasons to hardcode their structure.
If it wouldn't rely on links it will be the most RESTful client I know :)

@dgeb I've so far managed to avoid __rev but now that I'm connecting the Firebase source with other sources I can see transforms for __rev coming through and don't know what to do with them :) Could you give me a quick summary of what they're used for?

Currently many parts of the code use conditionals like if(path.length === 4) which makes it hard to read what's happening. Would be great to add the following helpers

• operation.model() - record type
• operation.modelId() - record id
• operation.link() - name of link
• operation.linkValue() - value of the link
• operation.path() - always a string i.e. 'planet/1/__rel/moons'
• operation.splitPath() - always segments i.e. ['planet', '1', '__rel', 'moons']
• operation.valueType() - link/record/meta (requires schema)
• operation.linkType() - hasOne/hasMany (requires schema)

The last two should probably live somewhere else as I don't think operations should be carrying the schema around with them. Perhaps schema.valueType(operation) and schema.linkType(operation).

In orbit-firebase I've also introduced an OperationMatcher which is used like:

var matcher = new OperationMatcher(operation);
if(matcher.matches("add", "record")) ...

This might be a better approach as the most common usage for that information is conditionals.

edit: added operation.path() and operation.splitPath()