mozilla / toodle Goto Github PK

It's currently hard to tell toodle subcrates apart from external dependencies. The subcrates should be renamed so that they are all prefixed with toodle_, so toodle_ffi_utils and toodle_store etc.

Create a JS interface to Mentat allowing all of mentat's FFI exposed functionality to be callable from JS.

Update transaction parser to accept vocabulary for creating a view from a complex query.

• Should use query parser to parse query
• Should use transaction parser for everything else.

Also create programmatic API for creating materialized views without having to do query parsing

We want to be able to use the write and transaction API from inside native code, not just from inside Rust so we need to create an FFI interface for it.

One of the things we want to demo is the ability to append new data types to an app, and have that sync without it breaking all the other clients. In order to do that we should take a field we currently have,create a version of the apps without that field and then we can "add" the field back in for the demo.

We are using a field that already exists as it means we can reuse already written code in the demo rather than having to write new code which will take longer.

Create an Android framework wrapping Mentat that can be imported to Android Studio with Ant, Maven or Gradle.

Allow the caching of attribute data.

Create a mechanism by which a client can pass a keyword into mentat and allow that values against that keyword to be cached in memory.

Things to think about:

• Do we want 2 levels of caching, eager and lazy? Or just one, in which case do we want either eager or lazy caching?
• Do we want a cache per connection? If so, where should this go and how do we ensure that we don't have multiple copies of the same cache? If not, what is the best way of sharing that cache between threads?

Beef up API documentation such that there are lots of useful, easy to understand resources for new developers to go to for reference.

Applications should be able to trigger a sync. The most basic version of this looks like pull-to-refresh in Toodle - or a "sync now" menu button.

A more complicated version on Android, which we might want to explore at some point, looks like SyncAdaptor integration - letting OS kick-off and schedule our syncs whenever appropriate for the device.

When creating a view we need to be able to specify:

• a particular schema field should be kept in cache
• a complex query should be used to create a snapshot of data for querying.
• schema cache versions to be tied to schema snippit version
• complex query versions should be linked to schema snippit versions and have versions themselves
• complex query versions should be able to be defined as JSON, Cached or SQL tables.

When a cached attribute is marked as unique identity, we should create a reverse map to go from value to entity. That’s useful for applications, but will also be useful for the transactor itself.

Currently, building on Xcode with bitcode enabled fails as the Rust library is not built Bitcode compatible. This issue tracks enabling the library to be build with Bitcode enabled so we don't force iOS developers to disable Bitcode on their apps.

We want to be able to management vocabulary directly from native code, not just from Rust so we need to create an FFI interface for it.

We need a bunch of tools to make cross compilation easy. Ideally this would include a script that can be run passing in only the platforms we want to set ourselves up on that will involve setting up the cross platform environments, and another for performing cross platform compilation for requested environments.

The caches held in Schema should be migrated to use the new caches.
EntidMap
IdentMap
AttributeMap

This involves creating a Swift framework wrapping Mentat's FFI. The framework should be compatible with SwiftPackageManager, Carthage and Cocoapods.

There are plenty of platform specific logging crates out there, but we want to log once and have it use the correct logging for all of our target platforms.

This issue tracks investigations into discovering existing crates that allow us to do this, or wrapping existing logging crates for Android, iOS, C++ and JS into a single crate that does platform detection.

Here's how that looks (docs + test):

https://github.com/mozilla/mentat/blob/master/src/vocabulary.rs#L12

See #26.

Same as Android/Swift/JS but this time C++/XPCOM compatible.

Before we write we should update any cached values.

Caching needs to be usable from native code, not just Rust code. An FFI interface needs to be created.

When a complex query view transaction statement is executed, a snapshot of the defined query results should be created and stored in an SQL table.

• Query used to create view needs to be stored somewhere for updating later

On iOS and Android

This issue tracks investigating how to make debugging a Rust library inside Android Studio or Xcode possible.

This issue is less important for Mentat as we don't expect developers using the Mentat crate to need to debug it!

We want to call log::debug, log::info & log::error from inside our Rust code and have it automatically use the correct logging tool for the architecture we are built for.

When materialized views are created they need to register for change notifications with the transaction log observer.

This issue involves the creation of a standard materialized view transaction listener that registers for changes on the attributes that are included in the view.
On notification, the view should be destroyed and recreated.
Deregister listener on removal of view.

How to build and deploy for Android, iOS and WebExtension.

At present the Toodle store layer builds transaction strings and calls conn.transact multiple times. This is inefficient in several ways: string parsing, multiple SQL interactions, multiple commits, multiple fsyncs.

It should instead use EntityBuilder to construct terms, and use begin_transaction to couple a SQLite connection to the Conn to get an InProgress, which we later commit.

Here's how that looks:

https://github.com/mozilla/mentat/blob/master/src/entity_builder.rs#L413-L446

Currently everything is stored in a Vec regardless of whether the values are unique or non-unique, causing a slow down in fetching as we are constantly referencing a vector. This issue covers creating 2 caches inside EagerCache, one for unique and one for non-unique attributes such that we can always use the most efficient caching for our attributes.

Currently Cheader and it's ilk do not generate Swift compatible C headers. This issue tracks contributing to an existing C header generation tool in order to autogenerate C headers from FFI for Swift projects.

We need to ensure that:

1. We version check against schema fragment versions and error if versions are not correct
2. We allow multiple versions of the same view to be present as long as their schema fragment versions match

Currently TxObservers only handle a set of matching attributes to filter whether it wants to be notified about a set of transactions. We need to be able to specify a set of filter criteria to further exclude notifications from an object.

For clients this might be - Providing an entid so a notification will only get fired for a specific record and not for all records that touch an attribute.

For materialized views this might be aggregation, negation or comparison logic, i.e. only notify me about transactions that have a value less than a constant, or that change number of times a url has been viewed.

This involves exposing both the datalog query and transact mechanisms, InProgress and the vocabulary management and entity building APIs.

All associated structs, such as NamespacedKeyword and Variable will need to be dealt with in some way.

• Allow clients to fetched cached data
• How do we include an in memory value in a filter?

Currently when running install-native-manifest.js the path that is used is hardcoded to the debug build path. Add ability to pass in --release or --debug to the script to set the path according to which library you want to run.

Observe transaction log.
Allow Listeners to to register for notification on attributes.
Call correct listeners when changes to the values associated with registered attributes occur & provide changeset.
No not call listeners that are not affected by changes.

Currently (13/3/2018) the Android app footprint size is huge (70mb). We need to ensure that we can build Android apps with a sensible footprint. This ticket tracks discovering build options and streamlining our imports to reduce our release Android library size.